Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations strongm on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

get base or full url... for relative urls via php curl

Status
Not open for further replies.

benedictbutlin

Technical User
Oct 12, 2012
16
GB
I'm using curl to get links from other site pages

absolute URLs are all good, but when i come across relative urls... i'm trying to figure out the best way to convert the relative url into an absolute url

if the site were my own server there's different way's to get the base or full url, but on a server that is not mine how can this be done?

cheers!


 
Are you scraping the other site legitimately? (you have the owner's express consent?)
 
then the easiest solution is to use something like simplebrowser. let it simulate the clicks for you.

you can do it using cUrl yourself of course. for each page you would need to compare each href against the url of the calling page.

use a combination of parse_url and pathinfo to get the domain part, the path, the script name, the query string etc and then armed with that information you can build relative urls from the path.

also make sure that you are honouring any base meta directives (in the same way).

but personally I'd use simplebrowser (or similar) ...

 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top