Saving web pages to text files with Java????

[Albuckj]

Hi,

I need a Java program to go to a URL and then save the web page in a text file. I have successfully done this by finding the GetURL method (see bottom of page). But I need to do it for a URL where a standard username and password prompt appears. I know I need to use the following piece of code somewhere in the GetURL program to bypass this authorisation:

HTTPConnection con = new HTTPConnection(this);
con.addBasicAuthorization("protected-space", "goofy", "woof"

The problem is that I am a complete Java novice and am desperately trying to finish a project, therefore I just don’t have the time to learn the language.
Can someone PLEASEEEEEEE tell me how I can incorporate the above code into the following GetURL program (which I know works for URL’s without the authorisation needed):

[by the way, the URL is in the program below and the username = Bazza99 and the password = upge5w7u]

import java.io.*;
import java.net.*;

/**
* This simple program uses the URL class and its openStream() method to
* download the contents of a URL and copy them to a file or to the console.
**/
public class GetURL {
public static void main(String[] args) {
InputStream in = null;
OutputStream out = null;
try {
// Check the arguments
if ((args.length != 1) && (args.length != 2))
throw new IllegalArgumentException("Wrong number of arguments"

// Set up the streams
URL url = new URL("

http://www.advfn.com/cmn/fun/advfnFun.php?epic=BAA";;;);

// Create the URL
in = url.openStream(); // Open a stream to it
if (args.length == 2) // Get an appropriate output stream
out = new FileOutputStream("badata.txt"
else out = System.out;

// Now copy bytes from the URL to the output stream
byte[] buffer = new byte[4096];
int bytes_read;
while((bytes_read = in.read(buffer)) != -1)
out.write(buffer, 0, bytes_read);
}
// On exceptions, print error message and usage message.
catch (Exception e) {
System.err.println(e);
System.err.println(&quot;Usage: java GetURL <URL> [<filename>]&quot;
}
finally { // Always close the streams, no matter what.
try { in.close(); out.close(); } catch (Exception e) {}
}
}
}

 

[rsfoley]

Below is a simplified version of your code, using hardcoded values. This will show you how to access a password protected URL. It works with your provided link and the user name and password you provided. I hope you helps you.

Rodney

http://www.javathis.com

P.S.
In case you were wonder why I do no use something like the example you posted:

HTTPConnection con = new HTTPConnection(this);
con.addBasicAuthorization("protected-space", "goofy", "woof");

The class HTTPConnection does not exist in sun's JDK, I am assuming you may have ment HttpURLConnection, which can not be constructed directly since it is an abstract object. You can only obtain one from a URL object. Plus the method shown above does not exist on any Sun URLConnection object. Instead you need to call setRequestProperty using the key &quot;Authorization&quot;. Details in the code below.

import java.io.*;
import java.net.*;
import sun.misc.*;

public class GetURL
{
  public static void main(String[] args)
  {
    InputStream  in  = null;
    OutputStream out = null;

    String login    = "Bazza99:upge5w7u";
    String website  = "[URL unfurl="true"]http://www.advfn.com/cmn/fun/advfnFun.php?epic=BAA";[/URL]
    String fileName = "badata.txt";

    try
    {
      String ecodedLogin = new BASE64Encoder().encode(login.getBytes());

      URL url = new URL(website);

      // Only need a URLConnection object.
      URLConnection con = url.openConnection();

      // Set the Basic Authorization with the
      // encoded User name and password
      con.setRequestProperty("Authorization", "Basic " + ecodedLogin);

      in  = con.getInputStream();
      out = new FileOutputStream(fileName);

      byte[] buffer = new byte[4096];
      int    bytes_read;

      while((bytes_read = in.read(buffer)) != -1)
        out.write(buffer, 0, bytes_read);
    }
    catch (Exception e)
    {
      System.err.println(e);
    }
    finally
    {
      try
      {
        in.close();
        out.close();
      }
      catch (Exception e) {}
    }
  }
}

 

[Albuckj]

Cheers for the information rsfoley, it's much appreciated. I will try this later tonight at home, and I'll let you know how it goes.

All the best,

Alex.