Could need some help with HTTP requests 1

[frag]

Hi!

I am trying to write a program that connects to an URL and writes the response to file... actually I only want to save the BODY of an HTML-file to a text file.

There was a post in this forum with a link to some java-examples a few days ago and I tried to use one of the classes from these examples but I keep getting 400 and 403 errors when trying to connect.

Here is the link:

http://www.nsftools.com/tips/JavaTips.htm#getinternetfile

Could perhapse someone give me a working peace of code that will make a connection (via proxy, none auth.) and just prints the responde on the screen?

Thank you very much...

cheers

frag



patrick.metz@epost.de

 

[frag]

Ok, I threw this class mentioned above away and coded this:

import java.io.*;
import java.net.*;

public class HttpResponse
{
	public static void main (String args[])
	{
		if (args.length == 0)
		{
			System.out.println(&quot;USAGE: java HttpResponse <URL> &quot;);
			return;
		}
		System.out.println(args[0]);
		String strURL = args[0];
		String strProxy = &quot;proxy.de.domain.com&quot;;
		int port = 8080;
		try { 
  			URL u = new URL(&quot;http&quot;, strProxy, port ,strURL); 
  			HttpURLConnection huc = (HttpURLConnection) u.openConnection(); 
  			huc.setRequestMethod(&quot;GET&quot;); 
  			huc.connect(); 
   			int code = huc.getResponseCode(); 
  			System.out.println(&quot;Response code was: &quot; + code);
  			huc.disconnect(); 
		} 
		catch (IOException e){
			e.printStackTrace();
		}
	}
	
}

The program can connect to a given URL and will return the responde code. But there is something very strange going on...

java HttpResponse [URL unfurl="true"]http://www.spiegel.de[/URL]

will return error code 403 (but the site can be viewed in my browser)

java HttpResponse [URL unfurl="true"]http://www.heise.de[/URL]

will return 200 -> OK

Why does the first site return an error code? And how can I save the BODY information?

thx

frag



patrick.metz@epost.de

 

[sedj]

Hmmm, not surewhy you are getting the 403, but it may have something to do with the proxy - 403 is "forbidden" - ie the server recieved your request, and understood it, but refused to fulfill the request.

I took out the proxy stuff and both URLs work fine.

I also added in some code to read the response (ie download the URL page)

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

import java.io.*;
import java.net.*;

public class HttpResponse
{
    public static void main (String args[])
    {
        if (args.length == 0)
        {
            System.out.println(&quot;USAGE: java httpResponse <URL> &quot;);
            return;
        }
        System.out.println(args[0]);
        String strURL = args[0];
        //String strProxy = &quot;proxy.de.domain.com&quot;;
        //int port = 8080;
        try {
              //URL u = new URL(&quot;http&quot;, strProxy, port ,strURL);
              URL u = new URL(strURL);
              HttpURLConnection huc = (HttpURLConnection) u.openConnection();
              huc.setRequestMethod(&quot;GET&quot;);
              huc.connect();
               int code = huc.getResponseCode();
              System.out.println(&quot;Response code was: &quot; + code);

				BufferedReader br = new BufferedReader(new InputStreamReader(huc.getInputStream()));

				int b = 0;
				while((b = br.read()) != -1) {
					System.err.print((char)b);
				}

				br.close();
              huc.disconnect();
        }
        catch (IOException e){
            e.printStackTrace();
        }
    }

}

 

[frag]

Hey, thanks... I was abel to make your program writing the response to a file instead of writing it on the screen.

But I still have some problems...

If I use

http://onvista.de

I get 200 as return code and the file is saved correctly.

But I want to be able to connect to this URL:

http://zertifikate.onvista.de/compa...ORT=FIGURE1_DESC&HIDE_KNOCK_IN_OUT=0&OFFSET=0

I keep getting 404 (not found) errors if I try to connect to the above URL via the program... my browser will display the page correctly.

Can someone test this without proxy-settings please?

cheers

frag

patrick.metz@epost.de

 

[sedj]

Sending parameters (ie everything after the '?' on the URL) must be done slightly differently to how you micht think.

Basically, you must write out the parameters to the URLConnection's output stream. See the code below, I've updated it to do this ...

import java.io.*;
import java.net.*;

public class HttpResponse
{
    public static void main (String args[])
    {
        if (args.length == 0)
        {
            System.out.println(&quot;USAGE: java httpResponse <URL> &quot;);
            return;
        }
        System.out.println(args[0]);
        String strURL = args[0];
        //String strProxy = &quot;proxy.de.domain.com&quot;;
        //int port = 8080;
        try {
              //URL u = new URL(&quot;http&quot;, strProxy, port ,strURL);
              URL u = new URL(&quot;[URL unfurl="true"]http://zertifikate.onvista.de/comparison.html&quot;);[/URL]
              HttpURLConnection huc = (HttpURLConnection) u.openConnection();
              huc.setRequestMethod(&quot;GET&quot;);
			  huc.setDoInput(true);
			  huc.setDoOutput(true);
              huc.connect();

			  // new code here ....
              String params = &quot;FORM_SEND=1&STEP=3&KNOCK=0&SEARCH_VALUE=metro&NUR_OPEN_END=0&TYPE_LONG_SHORT=&REALPUSH_KURS=ALL&ID_CERTIFICATE_TYPE=140&ID_NOTATION=20735&OPEN_END=1&NAME_FIGURE1=CN_LEVERAGE_ASK&NAME_FIGURE2=STRIKE_PRICE&NAME_FIGURE3=COVER_RATIO&NAME_FIGURE4=KNOCK_OUT&SORT=FIGURE1_DESC&HIDE_KNOCK_IN_OUT=0&OFFSET=0&quot;;
			  byte[] data = params.getBytes();
			  DataOutputStream dos = new DataOutputStream(huc.getOutputStream() );
			  dos.write(data);
			  dos.close();
			  // ... to here


               int code = huc.getResponseCode();
               System.out.println(&quot;Response code was: &quot; + code);

				BufferedReader br = new BufferedReader(new InputStreamReader(huc.getInputStream()));

				int b = 0;
				while((b = br.read()) != -1) {
					System.err.print((char)b);
				}

				br.close();
              huc.disconnect();
        }
        catch (IOException e){
            e.printStackTrace();
        }
    }

}

 

[frag]

problem is:

the prog doesn't want to conenct to

http://zertifikate.onvista.de/comparison.html

at all

keep getting 404 errors...

patrick.metz@epost.de

 

[sedj]

Are you still going through the proxy ? The code I posted (second version) works fine for me - I see the page content ....

 

[frag]

Yes still going through the proxy.

It doesn't like

http://zertifikate.onvista.de/comparison.html

but it likes

http://zertifikate.onvista.de

patrick.metz@epost.de

 

[sedj]

Sounds defintely like a proxy config error ...

 

[frag]

Ok, I give up.. tried both of my company's proxies and both wii come back with a 404 code when trying to connect to

http://zertifikate.onvista.de/comparison.html

(tried it without the parameter string).

patrick.metz@epost.de