get base or full url... for relative urls via php curl

[benedictbutlin]

I'm using curl to get links from other site pages

absolute URLs are all good, but when i come across relative urls... i'm trying to figure out the best way to convert the relative url into an absolute url

if the site were my own server there's different way's to get the base or full url, but on a server that is not mine how can this be done?

cheers!

 

[jpadie]

Are you scraping the other site legitimately? (you have the owner's express consent?)

 

[benedictbutlin]

it's all legit. the sites are on different servers

 

[jpadie]

then the easiest solution is to use something like simplebrowser. let it simulate the clicks for you.

you can do it using cUrl yourself of course. for each page you would need to compare each href against the url of the calling page.

use a combination of parse_url and pathinfo to get the domain part, the path, the script name, the query string etc and then armed with that information you can build relative urls from the path.

also make sure that you are honouring any base meta directives (in the same way).

but personally I'd use simplebrowser (or similar) ...