LWP, HTTPS and cookies

[plotzer]

Hello,

I am trying to pull some info from a HTTPS website. I am using LWP with the Crypt-SSLeay module for https support. I am able to connect to the site, but I am returned a page saying my browser is not cookie enabled and the page I want is not accessible. Is it possible to access such data outside of a browser via perl and fake the server into thinking you have cookies enabled. I know there are some modules within in LWP for cookie handling, but before I spend alot of time attempting this I want to make sure it is possible to do what I want to do. My code is as follows:

#use strict;
#use warnings;

use LWP::UserAgent;

my $uri = '

https://www.comcast.com/Localizatio...tate=&IsPermanent=on&Submit.x=24&Submit.y=4';

my $ua = LWP::UserAgent->new;
my $req = HTTP::Request->new(GET => $uri);
my $res = $ua->request($req);

if ($res->is_success) {

print $res->as_string;

} else {

print "Failed to GET '$uri': ", $res->status_line, "\n";
}


If I enter the url in my browser, the correct page is returned.

Thanks in advance.

 

[arcnon]

you have to send the cookie data also otherwise you'll get cookies are not enabled.

heres some stuff about this:

http://secu.zzu.edu.cn/book/Perl/Perl Bookshelf [3rd Ed]/lwp/ch11_01.htm

 

[plotzer]

I've still been unable to solve the cookie thing using perl. I've been attempting to use the HTTP::Cookies::Netscape module to accomplish this but the site I am trying to access still does not recognize cookies as being enabled. I've included my code below to see if anyone has any clues what I might be doing wrong.

Thanks



use LWP::UserAgent;
use HTTP::Cookies;
use HTTP::Cookies::Netscape;

my $url = "

https://www.comcast.com/Localizatio...tate=&IsPermanent=on&Submit.x=56&Submit.y=7";

my $browser = LWP::UserAgent->new;
open (FILEHANDLE, ">comcast_test.htm") or die "no such file";

$browser->cookie_jar( HTTP::Cookies::Netscape->new(
'file' => 'C:\Documents and Settings\administrator\cookies.txt',
'autosave' => 1
));


my $response = $browser->get($url,'User-Agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6) Gecko/20040206 Firefox/0.8');

print FILEHANDLE $response->content;