How can I "view" a website programmattically? 2

[colttaylor]

I would like my program to simulate a browser and view a particular website. I would then like it to send some keystrokes into a pair of edit boxes on that website, effectively logging itself into the website. After that, I would like it to receive the contents of the resulting (successful login) page, parsing out a few significant bits of information (the website is extremely structured so the parsing is the easy part). Finally, I want the program to log out of the website by "clicking" the logout button on the website.

Anyone know how to do this kind of thing?

Thanks in advance,
Colt.

If it's stupid but it works, it isn't stupid

 

[Griffyn]

Hi Colt,

Easy enough to do, although there's one step that's not necessary. Remember that when you press a webpage 'Submit' button (or whatever), what's actually happening at that point is the .cgi or .pl or whatever script page that checks the creditials is being called, and having the entries in the text boxes passed to it as arguments.

eg.

http://www.site.com/login.cgi?login=Colt&pass=hardtoguess

You can see this by looking at the web address of this page and seeing how the viewthread.cfm file is a script that takes arguments so it knows what thread to display.

So, you can simply skip the first step of populating edit boxes on the webpage, and simply use an HTTP component to ask for the login.cgi (or whatever) page and supply it with your credentials.

Logging out is done in exactly the same way.

Now - knowing the names of the arguments you need to pass to the login and logout pages is what you need, and for that you need to view the HTML source of the page containing the Login button, and the one containing the logout button. This is because secure pages usually (hopefully!) don't display the passed arguments in the webaddress bar like this page.

That should get you started - post back if you need more help.

 

[Stretchwickster]

You will probably find Delphi's TWebBrowser component helpful. You can find it on the Internet page of the component toolbox.

Here is an example of some code I have used to do a similar thing to what you are asking. It uses an Indy component called TIdHTTP. It logs in to a website I use and then retrieves the HTML of the page and assigns it to a rich edit box. After this process, I parse through the HTML code in the rich edit box and strip out the information I require:

var
  PostDataStream : TStringStream;
  ParamData : TStringStream;
begin
  PostDataStream := TStringStream.Create('');
  ParamData := TStringStream.Create('');
  ParamData.WriteString('username=' + Username);
  ParamData.WriteString('&password=' + Password);
  ParamData.WriteString('&x=43');
  ParamData.WriteString('&y=8');
  try
    //Connect
    HTTP1.Post('[URL unfurl="true"]http://www.blah.com/action.cgi',[/URL] ParamData, PostDataStream);
    richEditHTML.Text := PostDataStream.DataString;
  finally
    ParamData.Free;
    PostDataStream.Free;
  end;
  // Parse the rich edit text here
end;

HTTP1 is my TIdHTTP component which can be found in the Indy Clients page of the component toolbox.

By the way, the website given is a ficticious URL.

Clive

http://www.kucu.co.uk

Ex nihilo, nihil fit (Out of nothing, nothing comes)

 

[Stretchwickster]

I forgot to mention - in order to view a website programmatically using a TWebBrowser component you can use the following code:

var
  data: OleVariant;
begin
  WebBrowser1.Navigate(URLEdit.Text, EmptyParam, EmptyParam, data, EmptyParam);
end;

Hope this helps!

Clive

http://www.kucu.co.uk

Ex nihilo, nihil fit (Out of nothing, nothing comes)