php curl to go through pages

[richardko]

hi,
i am not sure if this is related with php but since the code i am using is all php i thought i will ask it here.

i am trying to get contents of a certain website. the contents are such that there are multiple pages and the first page has links for the rest of the pages. I can easily download the first page however when i use recursive function to download the second page I seem to get redirected to the first page.

Since it was weird for it to do so I copied to link to the second page (hard-copied) and tried to use curl to obtain it but the result from curl was still the very first page.

Now that I am writing about this - I think its some session variable that is set when I access the first page.
So the question is if there was a way to spider all the pages with/without using curl functions?

thanks

 

[sleipnir214]

If you need cookie support (which the most common way session variable IDs are transmitted between a web server and a web client), then the easiest way to fetch the site is probably cURL.

You could, I suppose, use socket functions to fetch the pages in question, so you could parse all HTTP headers and have your code maintain all the cookie information, but cURL can do that for you fairly painlessly.



Want the best answers? Ask the best questions! TANSTAAFL!