Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Register Log in
  • Congratulations Mike Lewis on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

http versus https 1

Status
Not open for further replies.
plotzer

plotzer

Programmer
Aug 28, 2001
35
0
0
US
I am trying to scrape some info from a website by submitting a URL. I've had success with this code before, but I am now attempting to run the same code against url that uses the https protocal rather than http. I understand that this is some sort of secure protocal. So, I am getting the following error when trying to access this url:

Exception in thread "main" java.net.MalformedURLException: unknown protocol: https


my code is as follows:

import java.net.*;
import java.io.*;
import java.util.Date;

class getZips
{
public static void main(String args[]) throws Exception {
Thread t = Thread.currentThread();

try
{
Thread.sleep(30);
}
catch (InterruptedException e)
{
System.out.println("Main thread interrupted");
}
String mapUrl = "
URL hp = new URL(mapUrl);
URLConnection hpCon = hp.openConnection();


int len = hpCon.getContentLength();
if (len > 0)
{
InputStreamReader isr = new InputStreamReader(hpCon.getInputStream());
BufferedReader inData = new BufferedReader(isr);

String str;
while ( ( str = inData.readLine() ) != null )
{
System.out.println(str);
}//while

}//if

}//main
}//class





Does anyone know if it is still possible to do what I want to do via the https protocal or I am out of luck?

Thanks in advance
 
Sort by date Sort by votes
You need to use an HTTPS prtocol object :

Code: 
URL url = new URL("[URL unfurl="true"]https://www.server.co.uk/resource");[/URL]
com.sun.net.ssl.internal.[URL unfurl="true"]www.protocol.https.HttpsURLConnection[/URL] c = (com.sun.net.ssl.internal.[URL unfurl="true"]www.protocol.https.HttpsURLConnection)url.openConnection();[/URL]
 
Upvote 0 Downvote
sedj,

Thank you.. I was able to make the connection, though I now have another problem and was wondering if you could give a little guidance. Like I said, I was able to connect, but I was not able to retrieve the content of the page. I dont know alot about secure http or ssl, so I was wondering if you know why I am not returning any content. Could the site just be blocking me from this. My code as of know looks like the following:

import java.io.*;
import java.util.Date;
import javax.net.ssl.HttpsURLConnection;
import java.net.*;
import java.net.HttpURLConnection;



class getZips
{
public static void main(String args[]) throws Exception, IOException {
Thread t = Thread.currentThread();
HttpsURLConnection hpCon = null;
try
{
Thread.sleep(30);
}
catch (InterruptedException e)
{
System.out.println("Main thread interrupted");
}
//String mapUrl = " String mapUrl = " try
{
URL hp = new URL(mapUrl);
hpCon = (HttpsURLConnection)hp.openConnection();
//int code = hpCon.getResponseCode();
}
catch (IOException ioe)
{
System.out.println(ioe);
}

int len = hpCon.getContentLength();
System.out.println("Length: " + len + "\n");
if (len > 0)
{
InputStreamReader isr = new InputStreamReader(hpCon.getInputStream());
BufferedReader inData = new BufferedReader(isr);

String str;
while ( ( str = inData.readLine() ) != null )
{
System.out.println(str);
}//while

}//if

}//main
}//class




Thanks in advance
 
Upvote 0 Downvote
plotzer :

You're code runs fine ... remember that https is a slower protocol that straight http.

I have however rewritten your code a bit, you may find it useful (it will save binary data from the stream into a file).

Code: 
import java.io.*;
import java.net.*;
import javax.net.ssl.HttpsURLConnection;

public class Download {

	public void download(String u, String dest) throws IOException {
			URL url = new URL(u);
			HttpsURLConnection huc = (HttpsURLConnection)url.openConnection();
			huc.setRequestMethod("GET");
			huc.setDoInput(true);
			huc.setUseCaches(false);
			huc.setFollowRedirects(true);
			System.err.println("[Download] from : " +huc.getURL() +", to : " +dest);

			int cl = huc.getContentLength();
			DataInputStream dis = new DataInputStream(huc.getInputStream());
			FileOutputStream fos = new FileOutputStream(dest);

			// content-length header not set - read the stream a byte at a time
			if (cl == -1) {
				boolean bb = true;
				try {
					while(bb) {
						fos.write(dis.readByte());
					}
				} catch (EOFException eof) {}
			// cool, content-length is set, so read the stream in one chunk
			} else {
				byte[] buf = new byte[cl];
				dis.readFully(buf);
				fos.write(buf);
			}

			fos.flush();
			fos.close();
			dis.close();

			huc.disconnect();
	}

	 public static void main(String args[]) throws IOException {
		 new Download().download("[URL unfurl="true"]https://adwords.google.com",[/URL] "https_example.html");
	 }

}
 
Upvote 0 Downvote
sedj,

That works great! My problem now is that the site I am trying to access requires a cookies enabled browser to retrieve the page I want. Instead I get a warning page asking to enable cookies in my browser. Do you know of the top of your head if it is possible to duplicate browser acceptance of cookies in a java class.

Thanks for your help
 
Upvote 0 Downvote
Not in a *straight* download scenario such as above.

To do this, you would need to use the IO streams to in effect service cookie interaction. Not trivial, but possible ...
 
Upvote 0 Downvote
Sedj,
I tried this application however when I run it I have this message:
java.security.PrivilegedActionException: java.net.ConnectException: Connection refused: connect
Thanks for your help

 
Upvote 0 Downvote
I expect this is because one (or more) of the following :

1) The server you are connecting is refusing your connection.
2) Your firewaill &/OR proxy is refusing you to connect.
3) Your java.security.policy is disallowing you to connect.

--------------------------------------------------
Free Database Connection Pooling Software
 
Upvote 0 Downvote
Sedj

I think my problem comes from my java.security.policy.I don't know how to set this file.I added the line:
permission java.security.AllPermission but I don't know if it is the good solution.
Besides, when I run my application I don't know if I have to use the option -D:
java -Djava.security.policy ....
Thanks for your help
 
Upvote 0 Downvote
The default security policy will allow you to connect to Http/Https servers - have you altered yours since the install (apart from the above) ?

Below are the two default files, which should be in JAVA_HOME/jre/lib/security. You do not need to pass in the policy file argument via "-D" ...

Here is a default java.security file :

Code: 
#
# This is the "master security properties file".
#
# In this file, various security properties are set for use by
# java.security classes. This is where users can statically register
# Cryptography Package Providers ("providers" for short). The term
# "provider" refers to a package or set of packages that supply a
# concrete implementation of a subset of the cryptography aspects of
# the Java Security API. A provider may, for example, implement one or
# more digital signature algorithms or message digest algorithms.
#
# Each provider must implement a subclass of the Provider class.
# To register a provider in this master security properties file,
# specify the Provider subclass name and priority in the format
#
#    security.provider.<n>=<className>
#
# This declares a provider, and specifies its preference
# order n. The preference order is the order in which providers are
# searched for requested algorithms (when no specific provider is
# requested). The order is 1-based; 1 is the most preferred, followed
# by 2, and so on.
#
# <className> must specify the subclass of the Provider class whose
# constructor sets the values of various properties that are required
# for the Java Security API to look up the algorithms or other
# facilities implemented by the provider.
#
# There must be at least one provider specification in java.security.
# There is a default provider that comes standard with the JDK. It
# is called the "SUN" provider, and its Provider subclass
# named Sun appears in the sun.security.provider package. Thus, the
# "SUN" provider is registered via the following:
#
#    security.provider.1=sun.security.provider.Sun
#
# (The number 1 is used for the default provider.)
#
# Note: Statically registered Provider subclasses are instantiated
# when the system is initialized. Providers can be dynamically
# registered instead by calls to either the addProvider or
# insertProviderAt method in the Security class.

#
# List of providers and their preference orders (see above):
#
security.provider.1=sun.security.provider.Sun
security.provider.2=com.sun.net.ssl.internal.ssl.Provider
security.provider.3=com.sun.rsajca.Provider
security.provider.4=com.sun.crypto.provider.SunJCE
security.provider.5=sun.security.jgss.SunProvider

#
# Select the source of seed data for SecureRandom. By default an
# attempt is made to use the entropy gathering device specified by 
# the securerandom.source property. If an exception occurs when
# accessing the URL then the traditional system/thread activity 
# algorithm is used. 
# On Windows systems, the URL file:/dev/random enables use of the
# Microsoft CryptoAPI seed functionality.
#
securerandom.source=file:/dev/random
#
# The entropy gathering device is described as a URL and can 
# also be specified with the property "java.security.egd". For example,
#   -Djava.security.egd=file:/dev/urandom
# Specifying this property will override the securerandom.source setting.

#
# Class to instantiate as the javax.security.auth.login.Configuration
# provider.
#
login.configuration.provider=com.sun.security.auth.login.ConfigFile

#
# Default login configuration file
#
#login.config.url.1=file:${user.home}/.java.login.config

#
# Class to instantiate as the system Policy. This is the name of the class
# that will be used as the Policy object.
#
policy.provider=sun.security.provider.PolicyFile

# The default is to have a single system-wide policy file,
# and a policy file in the user's home directory.
policy.url.1=file:${java.home}/lib/security/java.policy
policy.url.2=file:${user.home}/.java.policy

# whether or not we expand properties in the policy file
# if this is set to false, properties (${...}) will not be expanded in policy
# files.
policy.expandProperties=true

# whether or not we allow an extra policy to be passed on the command line
# with -Djava.security.policy=somefile. Comment out this line to disable
# this feature.
policy.allowSystemProperty=true

# whether or not we look into the IdentityScope for trusted Identities
# when encountering a 1.1 signed JAR file. If the identity is found
# and is trusted, we grant it AllPermission.
policy.ignoreIdentityScope=false

#
# Default keystore type.
#
keystore.type=jks

#
# Class to instantiate as the system scope:
#
system.scope=sun.security.provider.IdentityDatabase

#
# List of comma-separated packages that start with or equal this string
# will cause a security exception to be thrown when
# passed to checkPackageAccess unless the
# corresponding RuntimePermission ("accessClassInPackage."+package) has
# been granted.
package.access=sun.

#
# List of comma-separated packages that start with or equal this string
# will cause a security exception to be thrown when
# passed to checkPackageDefinition unless the
# corresponding RuntimePermission ("defineClassInPackage."+package) has
# been granted.
#
# by default, no packages are restricted for definition, and none of
# the class loaders supplied with the JDK call checkPackageDefinition.
#
#package.definition=

#
# Determines whether this properties file can be appended to
# or overridden on the command line via -Djava.security.properties
#
security.overridePropertiesFile=true

#
# Determines the default key and trust manager factory algorithms for 
# the javax.net.ssl package.
#
ssl.KeyManagerFactory.algorithm=SunX509
ssl.TrustManagerFactory.algorithm=SunX509

#
# Determines the default SSLSocketFactory and SSLServerSocketFactory
# provider implementations for the javax.net.ssl package.  If, due to
# export and/or import regulations, the providers are not allowed to be
# replaced, changing these values will produce non-functional
# SocketFactory or ServerSocketFactory implementations.
#
#ssl.SocketFactory.provider=
#ssl.ServerSocketFactory.provider=

#
# The Java-level namelookup cache policy for successful lookups:
#
# any negative value: caching forever
# any positive value: the number of seconds to cache an address for
# zero: do not cache
#
# default value is forever (FOREVER). For security reasons, this
# caching is made forever when a security manager is set.
#
# NOTE: setting this to anything other than the default value can have
#       serious security implications. Do not set it unless 
#       you are sure you are not exposed to DNS spoofing attack.
#
#networkaddress.cache.ttl=-1 

# The Java-level namelookup cache policy for failed lookups:
#
# any negative value: cache forever
# any positive value: the number of seconds to cache negative lookup results
# zero: do not cache
#
# In some Microsoft Windows networking environments that employ
# the WINS name service in addition to DNS, name service lookups
# that fail may take a noticeably long time to return (approx. 5 seconds).
# For this reason the default caching policy is to maintain these
# results for 10 seconds. 
#
#
networkaddress.cache.negative.ttl=10

and here is the default java.policy file :

Code: 
// Standard extensions get all permissions by default

grant codeBase "file:${java.home}/lib/ext/*" {
	permission java.security.AllPermission;
};

// default permissions granted to all domains

grant { 
	// Allows any thread to stop itself using the java.lang.Thread.stop()
	// method that takes no argument.
	// Note that this permission is granted by default only to remain
	// backwards compatible.
	// It is strongly recommended that you either remove this permission
	// from this policy file or further restrict it to code sources
	// that you specify, because Thread.stop() is potentially unsafe.
	// See "[URL unfurl="true"]http://java.sun.com/notes"[/URL] for more information.
	permission java.lang.RuntimePermission "stopThread";

	// allows anyone to listen on un-privileged ports
	permission java.net.SocketPermission "localhost:1024-", "listen";

	// "standard" properies that can be read by anyone

	permission java.util.PropertyPermission "java.version", "read";
	permission java.util.PropertyPermission "java.vendor", "read";
	permission java.util.PropertyPermission "java.vendor.url", "read";
	permission java.util.PropertyPermission "java.class.version", "read";
	permission java.util.PropertyPermission "os.name", "read";
	permission java.util.PropertyPermission "os.version", "read";
	permission java.util.PropertyPermission "os.arch", "read";
	permission java.util.PropertyPermission "file.separator", "read";
	permission java.util.PropertyPermission "path.separator", "read";
	permission java.util.PropertyPermission "line.separator", "read";

	permission java.util.PropertyPermission "java.specification.version", "read";
	permission java.util.PropertyPermission "java.specification.vendor", "read";
	permission java.util.PropertyPermission "java.specification.name", "read";

	permission java.util.PropertyPermission "java.vm.specification.version", "read";
	permission java.util.PropertyPermission "java.vm.specification.vendor", "read";
	permission java.util.PropertyPermission "java.vm.specification.name", "read";
	permission java.util.PropertyPermission "java.vm.version", "read";
	permission java.util.PropertyPermission "java.vm.vendor", "read";
	permission java.util.PropertyPermission "java.vm.name", "read";
};

--------------------------------------------------
Free Database Connection Pooling Software
 
Upvote 0 Downvote
Sedj,

Thank very munch for your help but I have a new problem when I run my application, I have this message:


java.io.IOException
at sun.net. at com.sun.net.ssl.internal. at plateforme.application.<init>(application.java:74)
at plateforme.application.main(application.java:147)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.Validator Exception: No trusted certificate found
at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SunJSSE_az.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SunJSSE_az.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SunJSSE_ax.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(DashoA6275)

at sun.net. at sun.net. at sun.net. at sun.net. at java.net.URLConnection.getHeaderFieldInt(URLConnection.java:476)
at java.net.URLConnection.getContentLength(URLConnection.java:371)
at com.sun.net.ssl.internal. at plateforme.application.<init>(application.java:73)
... 1 more
Caused by: sun.security.validator.ValidatorException: No trusted certificate found
at sun.security.validator.SimpleValidator.buildTrustedChain(SimpleValidator.java:304)
at sun.security.validator.SimpleValidator.engineValidate(SimpleValidator.java:107)
at sun.security.validator.Validator.validate(Validator.java:202)
at com.sun.net.ssl.internal.ssl.X509TrustManagerImpl.checkServerTrusted(DashoA6275)
at com.sun.net.ssl.internal.ssl.JsseX509TrustManager.checkServerTrusted(DashoA6275)


I use j2sdk1.4.2_04 and I have imported the certificate into the keystore file.
Thanks for your help
 
Upvote 0 Downvote
This was discussed in this thread : thread695-864305 , see if it helps.

--------------------------------------------------
Free Database Connection Pooling Software
 
Upvote 0 Downvote
Status
Not open for further replies.

Similar threads

sarorelasoul
  • Locked
  • Question
HTTP 404 error
Replies
1
Views
44
Diancecht
Diancecht
foxup
  • Locked
  • Question
java - http # of items listed per page
Replies
2
Views
42
foxup
foxup
jamojo
  • Locked
  • Question
Connecting to HTTPS - unable to find valid certification path to reque
Replies
0
Views
34
jamojo
jamojo
sharon3874
  • Locked
  • Question
How to set custom HTTP header for single sign on?
Replies
1
Views
25
Diancecht
Diancecht
OhioSteve
  • Locked
  • Question
get the URL from an HttpServletRequest 1
Replies
2
Views
36
prosper
prosper

Part and Inventory Search

Sponsor

Back
Top