Extract the header of a Web site 1

rubis · Sep 17, 2002

Hi,

I would like to extract the information of the header from a Web page. I've found an example as the following.

use LWP::Simple qw($ua get);
use HTTP::Headers;
use HTML::HeadParser;

$ua->proxy(http => "
http://www.proxy.com/");
$h = HTTP::Headers->new;
$p = HTML::HeadParser->new($h);
$url = '
http://www.yahoo.com';

$content = get($url);

$p->parse($content);

print $h->header('Title')."\n";
print $h->header('Content-Base')."\n";
print $h->header('Last-Modified')."\n";
print $h->header('Content-Length')."\n";
print $h->header('Meta')."\n";

Everything seems fine EXCEPT no info about "last-modified", "content-length" and "meta". I checked on Yahoo Page. In the <head>..</head>, they don't put "last-modified" and "content-length". Does it mean that they have to put those info on the Web so I can get it? Also, there is a meta tag on the page and I'm wondering how can I print out the meta tag.

Thanks,

stillflame · Sep 17, 2002

$h->header('Foo') will access <meta http-equiv="Foo" content="...">

and no,
you cannot get at data unless it is actually there.

hth "If you think you're too small to make a difference, try spending a night in a closed tent with a mosquito."

rubis · Sep 17, 2002

to use 'Foo', the meta tag has to be <meta http-equiv="Foo" content="..."> ???

if the meta tag is <meta name="description" content="...">, is the syntax should be "$h->header('Description')" ?? I've tried it but it doesn't work. Any suggestion?

Thanks,

badco · Sep 17, 2002

Here is another way you can try. I am just giving you an example here without proxy address part. As you can see I chose to print out all data and then extract specifically "last modification data". For content-type you use the "0" index and for content-length you use "1" index.

[tt]
use strict;

use LWP::Simple;
my $data = head('

http://website');

print $data->{'_headers'}->{'last_modified'};

use Data:

umper;
print "\n\n", Dumper $data;

my $moddate = (head('

http://website'))

[2];

my ($timer) = scalar localtime($moddate);
print "\n\nLast modified date: $timer\n";[/tt] =================
Bad Company Music
=================

rubis · Sep 18, 2002

Thank you. It does work fine!!!

Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Extract the header of a Web site 1

rubis

Programmer

stillflame

Programmer

rubis

Programmer

badco

Programmer

rubis

Programmer

Similar threads

Part and Inventory Search

Sponsor