Using cURL to obtain information from a site 1

[Chucklez]

Originally, I asked this question in the html forums, and they directed me to use cURL. Now, after spending the last day and a half looking it over, im stuck.

Here is my current code:

<?php

$HTTP_method = 'http';
$hostname = 'app.the-reincarnation.com';
$cgi = '/cgi-bin/rank.cgi?guild=15';

$curl_handle = curl_init ();

curl_setopt ($curl_handle, CURLOPT_URL, $HTTP_method . '://' . $hostname . $cgi);
curl_setopt ($curl_handle, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($curl_handle, CURLOPT_POST, 1);
curl_setopt($curl_handle, CURLOPT_HEADER, 1);
curl_setopt($curl_handle, CURLOPT_REFERER,"[URL unfurl="true"]http://app.the-reincarnation.com/cgi-bin/guilddetails.cgi?guild=12");[/URL]

//curl_setopt($curl_handle, CURLOPT_COOKIEFILE,"cookiefile");
//curl_setopt($curl_handle, CURLOPT_COOKIEJAR,"cookiefile");  

$result = curl_exec ($curl_handle) or die ('There has been an error');
curl_close ($curl_handle);

print $result;
?>

Whenever I run this, I do not get any of the information I am expecting. I am directed to a page (click here to see) that tells me my entry is expired. Here is what the header returns to me:

HTTP/1.1 200 OK Date: Thu, 19 Jan 2006 10:07:42 GMT Server: Apache/1.3.34 (Unix) mod_perl/1.29 Pragma: no-cache Cache-Control: private Set-Cookie: uid=; path=/; expires=Tue, 20-Dec-2005 10:07:42 GMT Set-Cookie: mage=; path=/; expires=Tue, 20-Dec-2005 10:07:42 GMT Set-Cookie: mageid=; path=/; expires=Tue, 20-Dec-2005 10:07:42 GMT Set-Cookie: sessionid=; path=/; expires=Tue, 20-Dec-2005 10:07:42 GMT Connection: close Transfer-Encoding: chunked Content-Type: text/html

In looking at that, it appears the cookies (mageid,path,sessionid, etc.) contain NO data, correct?

So basically, I have concluded at this point the reason my code isnt working is because I have not sent over the cookie information to tell the server it is me trying to retreive this data. Correct me if I am wrong.

So, the purpose of this post is this:

How do I first obtain, then send this data to the server in order to acquire the info I am so desperately trying to receive?

 

[sleipnir214]

You're right -- the script with with your script is trying to interact is expecting some cookies to be already set. The script, not getting the cookies it expects, correctly clears all the offending cookies.

The app with which you are trying to interact should have a login script. It appears to me that you are going to have to interact with that script to get set all the cookies you need, then use those cookies in further communications.

Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[Chucklez]

So I was right in my previous beleifs. Now, let me throw this in.

Currently, I am logged onto that site via the login script that you mentioned I will need to interact with. Heck, in another browser window, I am open to that exact page, so therefore all cookies have been set with mageid, sessionid, etc. etc.

Now, whenever I attempt to run the url link I gace above, I am returned the exact same thing. I know the cookies are set as I can view that page from another browser. I have told the script that I am coming from somewhere inside its own domain. But whats the problem?

 

[sleipnir214]

Your script is not handing the other script the right cookies.

Even if your cookie jar had the right data in it, the lines:

//curl_setopt($curl_handle, CURLOPT_COOKIEFILE,"cookiefile");
//curl_setopt($curl_handle, CURLOPT_COOKIEJAR,"cookiefile");

are commented out.

Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[Chucklez]

Sorry, when I posted it here I forgot to take that comment tags out. Since then I have taken those out,and still nothing.

Here is how I am populating cookiefile with the contents:

<?
       $cookie_file_path = "cookiefile";
        $url = '[URL unfurl="true"]http://app.the-reincarnation.com/cgi-bin/guilddetails.cgi?guild=15';[/URL]
        $agent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461)";
        
        $ch = curl_init(); 
        curl_setopt($ch, CURLOPT_URL,$url);
        curl_setopt($ch, CURLOPT_USERAGENT, $agent);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); 
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
        curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
        curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);

        $result = curl_exec ($ch);
        curl_close ($ch); 
		  echo $result;
		  ?>

Once cookiefile is populate, I open it with a text browser. The cookie names are in there, but the values of those cookies do not match AT ALL the values of the actual cookies.

Value in cookiefile:

# Netscape HTTP Cookie File
# [URL unfurl="true"]http://www.netscape.com/newsref/std/cookie_spec.html[/URL]
# This file was generated by libcurl! Edit at your own risk.

app.the-reincarnation.com	FALSE	/	FALSE	1135090217	uid	
app.the-reincarnation.com	FALSE	/	FALSE	1135090217	mage	
app.the-reincarnation.com	FALSE	/	FALSE	1135090217	mageid	
app.the-reincarnation.com	FALSE	/	FALSE	1135090217	sessionid

So, am I not retreiving the proper values into cookiefile?

 

[sleipnir214]

You are not getting the correct cookie values because your script is not interacting with the correct script on re-incarnation's server.

As I have said before:

The app with which you are trying to interact should have a login script. It appears to me that you are going to have to interact with that script to get set all the cookies you need, then use those cookies in further communications.

Your script has to perform the same operations that a user would at the login script, [ignore]

http://app.the-reincarnation.com/perl/login.pl

[/ignore]. If you take a look at the HTML that script produces, you will see that the form on that page submits three fields to [ignore]

http://app.the-reincarnation.com/perl/login2.pl

[/ignore], "login", "pwd" and "validation".

If your script submits the correct values in those fields, [ignore]

http://app.the-reincarnation.com/perl/login2.pl

[/ignore] will set the cookies you will probably need.

Your script can then use those cookies to further interact with the site.

See faq434-2502 for information on sending POST-method data to a script using cURL.

Want the best answers? Ask the best questions!

TANSTAAFL!!

 

[Chucklez]

Thanks. I went back a re-read your old posts several times, and found the point you made earlier. Tried it out, and with a little finagling, got it to work.

I had to send it in to login2.pl first to create the cookies, THEN send it to guilddetail.cgi. It works. not quite the way I envisioned, but it works.

Now I just need to parse the info out and obtain the table i need.

Take a star for your help.