Retrieving information from another PAGE.

[jonthequik]

I'm trying to get information out of a page for use in a database. I can't get access to the information from the server because it's loaded through CGI. I CAN get the information by using View Source and pasting the code into a program that searches for the information. What I'm trying to do is to bypass the view source part and just have the program look into the page for the information itself. If I have a window or frame name, is this possible and if so, how? Jonathan Hannan
Computer Repair, Webdesign
HTML, CGI, PERL, JavaScript, XML

 

[tanderso]

Are you trying to steal information off of another website? If so, then I cannot help you. If not, then why not simply get the information in a more friendly format?
Sincerely,

Tom Anderson
CEO, Order amid Chaos, Inc.

http://www.oac-design.com

 

[jonthequik]

The information is from a game called Archmage. You have to login to play the game and it stores a cookie so that you cannot play multiple accounts or use multiple browsers with the game. It's webbased and our "Status Report" is produced by a file called report.cgi. I'm trying to get that report to save to a variable so I can enter in some of the information in calculators and things. I already have programs in place to do the same work when I MANUALLY enter the information. It's not stealing, just being lazy.

I have figured out how to retieve pages with LWP, but the cookie prevents me from getting it. I am thinking the only way I can do it with this page is to either view the source and copy that to my CGI, or find a way to have the CGI get the source itself. Any ideas? Jonathan Hannan
Computer Repair, Webdesign
HTML, CGI, PERL, JavaScript, XML

 

[tanderso]

Well, you could use http to directly query the webserver from your CGI script, in effect acting like a browser. I believe there is a Perl module which makes such actions very simple.

Or, you could use a JavaScript/CGI combination. First, access the DOM using JavaScript and then pass the info to your CGI script.

window.location=report.cgi?source=document.body.innerHTML

to get the whole source, or

window.location=report.cgi?source=document.body.innerText

to get the text portion. I don't know whether or not those objects exist in NS or not, but they do exist in IE.
Sincerely,

Tom Anderson
CEO, Order amid Chaos, Inc.

http://www.oac-design.com

 

[jonthequik]

I looked into this method and there has to be an ID= somewhere in the code to reference. I've figured out how to capture the source, but now there's a problem. I have to login to play the game. The login downloads a cookie (which I cannot find on my system) and using a CGI script to try and capture the code is considered to be from another browser which logs me out. Any other ideas? Jonathan Hannan
Computer Repair, Webdesign
HTML, CGI, PERL, JavaScript, XML

 

[luciddream]

you can accept cookies using LWP.

use LWP;
use HTTP::Cookies;
use HTTP::Request;
$| = 1;

$ua = new LWP::UserAgent;
$ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt", autosave => 1));
$ua->agent('Mozilla/4.0');
$request = HTTP::Request->new(GET => '

http://www.page.com/page.htm');

that will allow you to accept cookies. adam@aauser.com