How to web scrape and crawl through a website

[TinyNinja]

Hey everyone,

I want to help out a friend who checks prices on a website every week manually. He is a contractor and looks up prices from his vendor weekly to give his customers accurate quotes for parts.

Is it possible to build a web crawler + scraper to go onto the website. Log in with his credentials --> go through items he uses --> Save the prices --> export data to an Excel sheet ????

Can this be done properly with VFP9 or do I need to use another software or is this even ok to do?

Thank you for the help!

 

[Mike Lewis]

In short: Yes, all those things can be done. But there are lots of separate pieces - and lots of detail.

I'll give you a summary of the steps, but I think it is asking too much to expect anyone to give you all the details in one go:

1. Use Automation (or the Internet Transfer Control) to access the website, follow the links, and download the pages.

2. Use VFP's string-handling functions to parse the downloaded pages and extract the items of interest.

3. Build a DBF with that information.

4. Either use VFP's export features to create an Excel file (or a CSV) that Excel can open directly, or use Excel Automation to have Excel import the data.

5. Finally, use ShellExecute() to open the Excel file for the user to interact with (if necessary).

I suggest you tackle each of those steps in turn, and come back with detailed questions if you get stuck.

One other thing to consider: Can you ask the supplier to provide the prices in some other form? If you could obtain them direct from the supplier as a text file, or an XML file, or something similar, it would eliminate most or all of the above.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

 

[TinyNinja]

Hey Mike,

Thank you for a step by step overview. That is very helpful. I will check back later with progress questions. I have also asked my friend if he can get a file from the vendor to make it easier.

 

[strongm]

> export data to an Excel sheet ????

You can pretty much do all of this in Excel 2016, under teh ...

Data > Get and Transform> New Query > From other Sources > From Web

In older versions, as I recall, it was

Data > From Web

but that wasn't as effective. But you can download Power Query for Excel 210 upwards, which provides the Get and Transform functionality.

 

[TinyNinja]

Hey StrongM,

Thank you for this insight. I was unaware Excel could do that. I will check this out too.