LWP::Useragent causing HTTP 501 error 1

[waiterm]

Hi,

The following block of code is causing a "501 Protocol scheme 'http' is not supported" error. Any suggestions as to why this might be happening, the version of perl is pretty old (v5.003 i think) but it's being hosted on a remote server.

my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/5.0');
my $response = HTTP::Request->new('GET', '[URL unfurl="true"]http://uk.shopping.com/');[/URL]
my $response = $ua->get('[URL unfurl="true"]http://uk.shopping.com/');[/URL]
print $response;
if ($response->is_success) {
	my $text = $response->content;  # or whatever
}  else  {
	die $response->status_line;
}

I tried using Mechanize but couldn't get it working using a library from my own version of perl (not compatible with v5.003) so have gone for the Useragent approach, which again works fine on my own server just not on the remotely hosted site.



Rob Waite

http://www.hostpipe.co.uk?src=TT

 

[audiopro]

Not really sure what you are doing here, beyond me I'm afraid but you are declaring $response twice, is that meant to be?

Keith

http://www.studiosoft.co.uk

 

[waiterm]

In it's simplest fashion, it is literally just getting a page and assigning it to the $text variable.

One of the $response declarations should not have been there, although it still refuses to work no matter which one I use!


Rob Waite

http://www.hostpipe.co.uk?src=TT

 

[PaulTEG]

I'd check the dependencies of LWP::UserAgent, though you're getting a 501, might be a firewall issue, in that the server can accept connections, but not create them

Check with your host

--Paul

cigless ...

 

[duncdude]

can you use this?

[b]#!/usr/bin/perl[/b]

print "Content-type: text/html\n\n";

use LWP::Simple;

$url = get("[URL unfurl="true"]http://uk.shopping.com");[/URL]

print "$url";

Kind Regards
Duncan

 

[waiterm]

Hi Duncan/Paul,

Tried the script you suggested and recieved:

'd:\h\o\hostpipe\

http://www\ecaps\temp\test.pl'

script produced no output

So, I think you may well be onto something with regards to the firewall issues. Do you know of any hosting companies who allow the creation of connections?



Rob Waite

http://www.hostpipe.co.uk?src=TT

 

[PaulTEG]

I'd ask the hosting company, could they check for outbound dropped packets coming from the machine, and does it pose an actual threat or is the firewall just over protective.

Many admins lock stuff down until a legitimate request is made to free up the resource.

Are you on a shared server?
Your use of LWP, is it totally, or quasi legit?
Is there only one IP for your server?

If you answered No, Totally, Yes then they should be able to create a rule for you to create outbound connections from that specific IP

HTH
--Paul

 

[waiterm]

Paul,

Many thanks, I have posed the question to our hosting company, and will leave a post here with their answer. The use of LWP is totally legit, we are hoping eventually move it onto a dedicated server where we can set up the system as we wish, but for the time being it's purely a proof of concept project.

Anyhoo, thanks again, will keep you informed.

Best regards

Rob


Rob Waite

http://www.hostpipe.co.uk?src=TT

 

[duncdude]

I can get the page with no probs with the tiny script i posted so i am sure Paul is correct


Kind Regards
Duncan

 

[waiterm]

Hi,

I now don't think it's a firewall issue, I've tried the following script:

#!/usr/bin/perl
print "Content-type: text/html\n\n";
use LWP::Simple;
$url = get("[URL unfurl="true"]http://uk.shopping.com");[/URL]
print "$url";

and it appears to work fine. So I now think there is an issue with my creation of the useragent using LWP::UserAgent.

Are there any major differences between the latest version of activeperl and the old version running on the remote server (v5_003)??

Rob Waite

http://www.hostpipe.co.uk?src=TT

 

[PaulTEG]

To rule out the firewall issue, request a page from the localhost on the server, using "

http://localhost"

Let us know how you get on

--Paul

cigless ...

 

[waiterm]

Hiya Paul,

Many thanks, I've managed to get around the problem. I was using a library from my newer version of perl and linking to it for Mechanize which caused an initial problem, then my UserAgent script wasn't quite right. It now appears to be successfully requesting and processing the page now using the following script:

#!/usr/bin/perl
print "Content-Type: text/html\n\n";

use CGI::Carp qw(fatalsToBrowser);
use LWP::UserAgent;

$ua = LWP::UserAgent->new;
$ua->agent("$0/0.1 " . $ua->agent);
$ua->agent("Mozilla/8.0");
$url = 'webaddressgoesinhere';
$req = HTTP::Request->new(GET => $url );
$req->header('Accept' => 'text/html');
$res = $ua->request($req);
if ($res->is_success) {
	print $url."<br />";
	$text = $res->content;
} else {
	print "8::" . $res->status_line . "\n";
}

It seems a little bit silly that it should default to the 'lib' directory I had created as opposed to the fully compiled version of Perl on my host's server, but hey, at least it works now!



Rob Waite

http://www.hostpipe.co.uk?src=TT