Automate web browsing

[RottPaws]

I need to write a script to mine data from a web site. I could do it in Perl, but now I'm stuck on an environment that only has PHP and I'm a newbie in PHP.

If anyone could point me to a tutorial or an example script for what I need, that would be great!

Here's what I need the script to do:
* open a web page and get a list of <option> values from a <select> control.
* loop through the options and submit the form for each
* find the list of links on the results page and follow each
* parse the target page and retrieve table structured information from a to be added to a database or spreadsheet.

Thanks,

_________
Rott Paws

...It's not a bug. It's an undocumented feature!!!

 

[jpadie]

that sounds like screen scraping a product site. i hope that you have the site-owner's specific and explicit permission to do this.

i would:

(i) use a combination of file_get_contents and regex to get the options and the form structure.
(ii) use curl to post the form and capture the results
(iii) use regex to parse the results

plenty of examples on all of these in the php manual

 

[RottPaws]

Thanks for the info. I'll give it a try.

_________
Rott Paws

...It's not a bug. It's an undocumented feature!!!