Site responses

[bogboy1978]

Does anyone know of a way of testing responses from websites. I want my app to go through all sites in a txt file and see what response it gets from each. I don't want to actually load any of the sites, i want the app itself to mimic going to the site and then return the response it got. None of the sites I want to visit are password protected so I don't need to check for that, but I do need the program to tell me if a site responds with 404 page missing (dead link) and when the site works fine (live link). If possible it would also be useful to detect if any of the pages are redirecting to another site or page.

I really don't know how to word this, I have done my best and I hope someone out there makes sense of this.

Thanks in advance.

 

[Griffyn]

I've got only a basic understanding of web servers, so let anyone contradict me - I don't think you will be able to reliably test for 404 errors. This is because to a web browser, or web client program, there is no difference to a 404 page, or any other page. The server exists, and decides what to return to the client. You could parse the webpage's title, or body, but there won't be any guarantee of 100% detection of 404 errors.

If the server was down, then that's different, because your client program can detect that it's not getting any response from the server.

In terms of getting started - your question is quite broad. What part of your program are you needing help with?

 

[Griffyn]

Ok, I've just thought of a reliable way to detect 404 errors

While parsing each entry in your text list, let's take one:

http://www.tek-tips.com/pagetotest.html

Your program should retrieve the page, and then also try to download

http://www.tek-tips.com/ThisPageDdoesNotExist

or something suitably unlikely. Then compare the two together. If they're the same - you've got a 404. Comparing the header is probably enough.

 

[mattlacey]

Obviously there are more situations that you will have to deal with just OK and URI not found errors.

Just check the HHTP Responce code for the request. This will be able to tell you exactly what happened with your request.

How you do this will depend on the HTTP component you are using.

If you are not using a component, a call to HttpQueryInfo with the HTTP_QUERY_STATUS_CODE flag should be all that's needed.

 

[Stretchwickster]

I've used an indy component called TIdHTTP in the past to connect to a website. The following is a snippet of the code I use. I actually log in to a website so you can ignore that bit.

var
  PostDataStream : TStringStream;
  ParamData : TStringStream;
  ...
begin
  PostDataStream := TStringStream.Create('');
  ParamData := TStringStream.Create('');
  ParamData.WriteString('username=' + Username);
  ParamData.WriteString('&password=' + Password);
  ParamData.WriteString('&x=43');
  ParamData.WriteString('&y=8');
  try
    //Connect
    HTTP1.Post(URLOfPage, ParamData, PostDataStream);
    richEditHTML.Text := PostDataStream.DataString;
  finally
    ParamData.Free;
    PostDataStream.Free;
  end;

HTTP1 is my TIdHTTP component. Simply use HTTP1.ResponseCode (after connecting to the page) to return the integer response code or use HTTP1.ResponseText which specifies the text for a response received from a HTTP server. See the Delphi help (TIdHTTP) for details of all the different response codes.

Hope this helps!

Clive
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"To err is human, but to really foul things up you need a computer."
Paul Ehrlich
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To get the best answers from this forum see: faq102-5096

 

[Stretchwickster]

To be more specific, the following code will detect a 404 error after connecting:

if HTTP1.ResponseCode = 404 then
  ShowMessage('404 detected');

Clive
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"To err is human, but to really foul things up you need a computer."
Paul Ehrlich
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To get the best answers from this forum see: faq102-5096

 

[buho]

Be warned some sites will not tell you you are getting a 404 Not Found but will redirect you to a page saying you have an error while returning a 2xx (ok) or may be 3xx (redirection) code.

Bad programmers are everywhere, and in the 'net most than in any other place.

buho (A).

 

[bogboy1978]

As a bit of a newbie to delphi your advice took a little looking up and working out, but it worked a treat.

Just wanted to say thanks to you all.

 

[Stretchwickster]

Glad to be of service!

Clive
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"To err is human, but to really foul things up you need a computer."
Paul Ehrlich
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To get the best answers from this forum see: faq102-5096