Retrieving the source of a webpage 1

[Supra]

I've tried using LWP to get the source of a webpage, but I keep getting an error. This makes me think my server doesn't support LWP (Netfirms.com). Is there any other way to retrieve the source from a webpage other than LWP? All I need is something simple like:

$URL = "

http://www.google.com";

$HTM = GET blah blah blah

I'm new to Perl so any advice one can give would be great.

 

[Supra]

Nope, no dice.

#!/usr/bin/perl

use CGI::Carp 'fatalsToBrowser';

use warnings;
use strict;

use lib '/mnt/web_i/d26/s15/b01eb8dc/[URL unfurl="true"]www/modules/';[/URL]
use CGI qw/:standard/;
use LWP::Simple;

print header, start_html();

my $to_get = "[URL unfurl="true"]http://www.yahoo.com";[/URL]
my @caught = get($to_get);

print "Printing source..\n";

foreach(@caught) {
	print;
}

Just says "Printing source.." still. Is it me or am I destined never to be able to do this?

 

[PaulTEG]

[cgi-bin]$ cat test.pl
#!/usr/bin/perl

use warnings;
use strict;

use CGI qw/:standard/;
use LWP::Simple;

print header, start_html();

my $to_get = "[URL unfurl="true"]http://www.yahoo.com";[/URL]
my @caught = get("$to_get");
print @caught;

This is the code that worked for me

--Paul

 

[Supra]

This line:

[cgi-bin]$ cat test.pl

Threw a 500 error at me. Once I took it out, it's the same code I used before. Has anyone else had this problem?

 

[Supra]

I contacted Netfirms Support to inquire about LWP and they told me "LWP is not supported at this time.". So I responded and asked about IO::Socket and gave them my error and the exact code I used and they replied "Please check our Supported Perl Modules list". IO::Socket is in the list. Now I'm installing the Perl modules from

http://cvs.sourceforge.net/viewcvs.py/libwww-perl/lwp5/lib/

and hoping this will work. Maybe it's time for a different server :\

 

[Supra]

Ok still nothing shows up on the page. No errors or anything, just no content. Could it be a problem with the Simple.pm file I got from SourceForge? Or, should I use a method other than Simple? This is driving me nuts - I'm so close yet so far. Thanks again to everyone who's helped me. Without your help I would've thrown my machine out by now

 

[Supra]

After fiddling with the script a bit, I got something new to pop up. Please take a look at it - maybe you can figure out what's going on.

http://juiced.netfirms.com/cgi-bin/test.pl

 

[duncdude]

Supra

Your script seems to work fine!?

What happens if you run this:-

#!/usr/bin/perl

use LWP::Simple;

print "Content-type: text/html\n\n";

$document = get("[URL unfurl="true"]http://www.google.com");[/URL]

print $document;

Kind Regards
Duncan

 

[Supra]

Absolutely nothing. It prints the Netfirms header at the top of the page, but nothing else at all. I'm telling you, it's because I had to install LWP locally. Here's the code I'm FORCED to use since Netfirms doesn't support LWP:

#!/usr/bin/perl

use CGI::Carp 'fatalsToBrowser';
use lib '/mnt/web_i/d26/s15/b01eb8dc/[URL unfurl="true"]www/modules/';[/URL]
use LWP::Simple;

print "Content-type: text/html\n\n";

$document = get("[URL unfurl="true"]http://www.google.com");[/URL]

print $document;

I tried using an IO::Socket::INET example last night, but I kept getting some kind of timeout.

 

[Supra]

I moved the script to a different server and tried everything listed in this forum. Can you guess the result? IT STILL DOESN'T WORK! This is crazy. I did see that one of the methods titles the page as "Untitled Document" and the other leaves the title as "

http://snow.prohosting.com/eqsource/cgi-bin/test.pl",

which I'm sure you can figure out is the path to the new server and script. The only logical conclusion I can come to is that free servers with Perl support have some sort of restriction against LWP. ProHosting DOES show LWP in the supported module list. I'll do some more research on it and let you know if I find anything out. Thanks again for your repeated help!!!!

 

[duncdude]

Supra

The ProHosting URL doesn't work for me...

just gives:-

Content-Type: text/html; charset=ISO-8859-1 
Free Web Hosting | Web Hosting - ProHosting.com - Sign Up!
 
Game Server Hosting | Managed Server Hosting 


Printing source..

Kind Regards
Duncan

 

[Supra]

That's pretty much the point :|

I moved the script to a different server and tried everything listed in this forum. Can you guess the result? IT STILL DOESN'T WORK!

I'm just in awe that it doesn't work on 2 commonly used free servers that support Perl. Especially ProHosting, since it supports LWP. I may just can the whole idea.

 

[duncdude]

Hi Supra

Can you save the

http://snow.prohosting.com/eqsource/cgi-bin/test.pl

as a text file and save it in the same location

i.e.

http://snow.prohosting.com/eqsource/cgi-bin/test.txt

or, alternatively, can you supply login/password for prohosting site?


Kind Regards
Duncan

 

[Supra]

http://snow.prohosting.com/eqsource/cgi-bin/test.txt

is now available.

 

[Supra]

PS.. the inserted advertisement code does NOT appear when downloading the file via FTP, which means it is only inserted when the file is run. For a second I thought that was why my Perl scripts were not working, but if this was the case, they would definitely error out instead of showing "Printing source.." and then nothing.

 

[duncdude]

thanks - got that!

now try this link:-

http://www.my-mot.co.uk/cgi-bin/supra.cgi

Kind Regards
Duncan

 

[Supra]

Works perfectly. NetFirms and ProHosting suck. What code did you use to do that? Probably the same I used

 

[duncdude]

weird?

I used this EXACT code - I had to take off the bottom of your script as it caused an error

#!/usr/bin/perl

print "Content-type: text/html\n\n";

use CGI::Carp 'fatalsToBrowser';

use warnings;

use CGI qw/:standard/;
use LWP::Simple;

print header, start_html();

my $to_get = "[URL unfurl="true"]http://www.yahoo.com";[/URL]
my @caught = get($to_get);

print @caught;

Kind Regards
Duncan

 

[duncdude]

I'm off out now for a few hours - good luck!


Kind Regards
Duncan

 

[duncdude]

any joy?


Kind Regards
Duncan

 

[Supra]

Nope. Looks like yours only because the "Content-Type: text/html; charset=ISO-8859-1" shows up top, and the page is titled "Untitled Document", but no source. Must be because I uploaded the Perl modules locally. Maybe I did it wrong. I created a folder named modules in the

http://www folder.

I wasn't allowed to create a folder where

http://www and

cgi-bin folders reside. I uploaded all the Perl modules for

http://wwwperl

to the modules folder (including folder names such as HTTP and LWP). Then I added the line

use lib '/mnt/web_i/d26/s15/b01eb8dc/[URL unfurl="true"]www/modules/';[/URL]

to use the modules in the folder. So my EXACT code is:
[code}
#!/usr/bin/perl

print "Content-type: text/html\n\n";

use lib '/mnt/web_i/d26/s15/b01eb8dc/

http://www/modules/';

use CGI::Carp 'fatalsToBrowser';

use warnings;

use CGI qw/:standard/;
use LWP::Simple;

print header, start_html();

my $to_get = "

http://www.yahoo.com";

my @caught = get($to_get);

print @caught;
[/code]
But no dice. I'm starting to think about paying for web hosting. I can't really afford it if it's $10/month since I have a crapload of other bills plus rent, but it might be the only thing I can do to get this working.