GZip + Blowfish

[coderiyer]

Hello!
I am working on a problem with an existing application used to generate encrypted URLs. The application
1. URL encodes the payload
2. Generates a checksum which is prepended to the payload
3. The result is zipped using GZipOutputStream (for compression)
4. The GZipped string is then encrypted using Blowfish using SunJCE.
5. The above result is Bas64 encoded and URL encoded.

The problem that I am facing is that the program works fine as long as it is called once. But, if it is called repeatedly in a loop, some of the encrypted URLs are invalid. I noticed that the length of the result from the Gzip method varies. If it is a particular length, then the encryption works well otherwise, it gets messed up.
By the way, the payload is a fixed length string.

Here is the GZip code.

private static byte[] zipIt ( String parameterString )
{


try {

ByteArrayOutputStream catcher = new ByteArrayOutputStream();
DelayOutputStream dyos = new DelayOutputStream(catcher);
GZIPOutputStream gzipOut = new GZIPOutputStream( catcher );

byte[] bytesToZip = parameterString.getBytes();

gzipOut.write( bytesToZip, 0, bytesToZip.length );

gzipOut.close();

return catcher.toByteArray();

}
catch ( Exception ioe ) {
ioe.printStackTrace();
return "error".getBytes();
}
}

How do I ensure that the zipped string is always of the same length for the encryption to work?

Thanks in advance!
coderiyer

 

[sedj]

I'm obviously missing something that you mean here :

Consider the below example :

	public void bla(String string) throws IOException{
		ByteArrayOutputStream out = new ByteArrayOutputStream();
		GZIPOutputStream gzOut = new GZIPOutputStream(out);
		gzOut.write(string.getBytes());
		gzOut.finish();
		gzOut.close();
		byte[] gzData = out.toByteArray();
		System.out.println("in : " +string +" , gzip length : " +gzData.length);
	}

 	public static void main(String args[]) throws Exception {
		int i = 0;
		while (i++ < 100)
			new Test().bla("hello");
	}

This outputs the below (note that the gzip'ed data is ALWAYS the same length (as it should be) :

in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25
in : hello , gzip length : 25

.... etc

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

Hi sedj!
Thanks for your response. Let me expand on what I mentioned in my post. It is not that the zip method would generate a different value in one or more of the 100 iterations. But, if you repeat the same 100 iterations multiple times, say repeatedly 10 or 15 times (in essence 1000 or 1500 iterations) some of them turn out to be of different lengths. Since I am able to recreate the problem consistently, could you please try it on a string of length 253 and increase the iterations to say 1000?
Thanks again!
coderiyer

 

[sedj]

Just executed that method 1,000,000 (1 million) times - the length is the same each time.

Input string was : abcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrstuvwxysabcdefghijklmnopqrs

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

Hi sedj! Thanks for your response again.
I verified that the same string gives the same zipped length every time.
What I noticed was that my input string length is the same but it differs in just the timestamp (the input string is a concatenated string of different parameters, with timestamp being one of them which is the only variable here).
I am sorry to bother you but would it be possible for you to try with a string like this? I am also trying different things here but the erroneous encrypted URLs are sporadic.
Please let me know if you are able to recreate different lengths with just a timestamp being different.
Thanks again!
coderiyer

 

[sedj]

Ermmm - if the input data is different - then the output length of the gzip'ed data is also going to be different.

The GZIP algorithm works on the repetition of characters.

For example :

The GZIP length of "aaa" is 23.
The GZIP length of "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" is ALSO 23 (the same, because all the characters are the same).
The GZIP length of "abcde" is 25 (longer than "aaaaa" because there are more different characters).

Different characters == different lengths !

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

Thanks a lot for this piece of information!
I did not know this. Then what are my alternatives?
I need to send this as the input to the Blowfish encryption program which gives an invalid output because of what you have described to be the working of GZIP.
Is there another class that I can use from the java.util.zip package which would get around this problem?
- coderiyer

 

[sedj]

Any zipping algorithm will work as I have indicated - you just cannot specify the desired size of output !

I have no idea why your encryption is failing, because you have not actually specified why it is. My experience in encryption with java is that it does not acutally care what size the input data is.

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

That was my next question to you. I was wondering why the zipped length should have a bearing on the encryption.
I will explain what's happening when the zipped length changes.
The input string length is a constant at 253 characters (though the string itself is not). Now, for the most part, the length of the zipped string is 193 and when fed to the Blowfish program, comes out as an encrypted string of length 200. All these encrypted URLs work. But, certain strings from the gzip method have a length of 191 in which case the output from Blowfish becomes 192. All these URLs fail. What I noticed was that the ones that fail look incomplete (I can tell the ones that are good by looking at the last few characters from the Blowfish output. These end with a %3D).

Thanks,
coderiyer

 

[sedj]

perhaps you should post the code you are using for encryption.

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

public static byte[] encryptSecretKey(byte[] keyBytes, byte[] unencrypted)
{
try {

Provider sunJce = new com.sun.crypto.provider.SunJCE();
Security.addProvider(sunJce);

SecretKeySpec skeySpec = new SecretKeySpec(keyBytes, "Blowfish");

Cipher cipher = Cipher.getInstance("Blowfish");

cipher.init(Cipher.ENCRYPT_MODE, skeySpec);

byte[] myEncrypt = cipher.doFinal(unencrypted);


return myEncrypt;
}
catch (Exception e) {
System.out.println("[BlowFishUtil:encryptSecretKey] - exception : "+e);
e.printStackTrace();
return null;
}
}

keyBytes are from a pre-defined encryption key.
Thanks,
coderiyer

 

[sedj]

I've no idea why your code is breaking - it all seems OK !

Here is some code I wrote ages ago that will show you how to :

Take a string
GZIP it
Encrypt it
Unencrypt it
UnGZIP it
Print out the string

As you will see, it works perfectly

		System.out.println("IN : " +string);
		// Make an encrypt key
		Security.addProvider( new com.sun.crypto.provider.SunJCE() );
		KeyGenerator generator = KeyGenerator.getInstance("Blowfish", "SunJCE" );
		//generate a new random key
		generator.init(56, new SecureRandom());
		Key key = generator.generateKey();


		// GZIP the string
		ByteArrayOutputStream out = new ByteArrayOutputStream();
		GZIPOutputStream gzOut = new GZIPOutputStream(out);
		gzOut.write(string.getBytes());
		gzOut.finish();
		gzOut.close();
		byte[] gzData = out.toByteArray();

		// Encrypt it
		Cipher cipher = Cipher.getInstance("Blowfish");
		cipher.init(Cipher.ENCRYPT_MODE, key);
		byte[] encryptedGZData = cipher.doFinal(gzData);


		// Unencrypt it
		cipher.init(Cipher.DECRYPT_MODE, key);
		byte[] decryptGZData = cipher.doFinal(encryptedGZData);

		// Un-GZIP it
		ByteArrayInputStream in = new ByteArrayInputStream(decryptGZData);
		GZIPInputStream gzIn = new GZIPInputStream(in);
		byte[] buffer = new byte[4096];

		out = new ByteArrayOutputStream();

		while (gzIn.read(buffer, 0, buffer.length) != -1) {
			out.write(buffer);
		}
		gzIn.close();

		// Print out the result
		System.out.println("OUT : " +out.toString().trim());

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

Hi sedj! Thanks for your reply again. There is also a Base64 encoding that's done after the encryption step. And then the result is URL encoded before it is returned. Is there anything in particular that I need to watch out for like encoding type. Anything that you can think of would really be appreciated. I used the CipherOutputstream instead of Cipher.doFinal and also added ECB/PKCS5Padding. Things have improved a little bit but still I am not getting 100% success rate on iterating 20000 times. Still, encrypted strings with length 192 fail whereas the ones with 200 seem to be fine.
coderiyer

 

[sedj]

You say its still failing when using my code ? Or using your code ?
Base64'ing it will make no difference - but why are you URL encoding it ? There are no characters in Base64 that would require encoding. But anyway, this would again have no difference.

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

Your code looks fine though I have not incorporated or tried to run it as is since my code looks familiar. Is there an encoding that could be messing up the final output? BTW, Base64 sometimes appends a "=" and so to be doubly sure URL encoding is done at the end. And the strings that fail do not have a "=" at the end. I am basically trying to brainstorm this with you since I have spent 3 days working on this and running short of ideas. I am the only Java developer here so I do not have anyone else to exchange ideas.
I appreciate your help.
coderiyer

 

[sedj]

Whats your ultimate goal here with the URL encoding ? To encode the URL or the base64 ?

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

Here again are the steps followed in getting the encrypted URL. These were defined 4 years ago before by time. The application is in production except that this particular client is using it differently (calling it in a loop from his program to generate multiples URLs.)
I replaced the Base64 utility (written by a private party) with the one from the Apache Codec project. Still, I am getting mixed results. This is why I am thinking that maybe there is some problem either in the GZIP method or Blowfish or somewhere the data is getting lost in translation. Nothing is jumping out though I read posts on refraining from using getBytes with the default encoding. Could this be the case?

// Steps for creating an encrypted URL:
//
// 1 - Calculate the checksum
// 2 - Concatenate the checksum and payload
// 3 - URL Encode
// 4 - Compress the string using zipIt (GZIP)
// 5 - Encrypt using Blowfish and SunJCE
// 6 - Base 64 encode
// 7 - URL encode
// 8 - Return the complete string
Thanks!
coderiyer

 

[sedj]

Why are you doing a URL encode in step 7 ? Makes no sense to me.

Look - finally - here is some code that does all of what you are doing (without odd step 7) - to PROVE to you that it is YOUR code that is blowing up - not the java encryption, base64, gzip or URL en/decode algorithms.

I have run this code on a loop for 100,000 times - with no problems AT ALL !

Enjoy.

	public void bla(String string) throws Exception{
		System.out.println("IN : " +string);

		//
		//	***IN*** : URL encode, GZIP, Encrypt, Base64
		//


		// URL encode it
		string = URLEncoder.encode(string);
		// Make an encrypt key
		Security.addProvider( new com.sun.crypto.provider.SunJCE() );
		KeyGenerator generator = KeyGenerator.getInstance("Blowfish", "SunJCE" );
		//KeyGenerator generator = KeyGenerator.getInstance("DES", "SunJCE" );
		//generate a new random key
		generator.init(56, new SecureRandom());
		Key key = generator.generateKey();


		// GZIP the string
		ByteArrayOutputStream out = new ByteArrayOutputStream();
		GZIPOutputStream gzOut = new GZIPOutputStream(out);
		gzOut.write(string.getBytes());
		gzOut.finish();
		gzOut.close();
		byte[] gzData = out.toByteArray();

		// Encrypt it
		//Cipher cipher = Cipher.getInstance("DES/ECB/PKCS5Padding");
		Cipher cipher = Cipher.getInstance("Blowfish");
		cipher.init(Cipher.ENCRYPT_MODE, key);
		byte[] encryptedGZData = cipher.doFinal(gzData);

		// Base64 encode it
		sun.misc.BASE64Encoder encoder = new sun.misc.BASE64Encoder();
		String b64Data = encoder.encode(encryptedGZData);



		//
		//	***OUT*** : un-Base64, un-Encrypt, un-GZIP, un-URL encode
		//
		// Un base64 it
		sun.misc.BASE64Decoder decoder = new sun.misc.BASE64Decoder();
		encryptedGZData = decoder.decodeBuffer(b64Data);

		// Unencrypt it
		cipher.init(Cipher.DECRYPT_MODE, key);
		byte[] decryptGZData = cipher.doFinal(encryptedGZData);

		// Un-GZIP it
		ByteArrayInputStream in = new ByteArrayInputStream(decryptGZData);
		GZIPInputStream gzIn = new GZIPInputStream(in);
		byte[] buffer = new byte[4096];

		out = new ByteArrayOutputStream();

		while (gzIn.read(buffer, 0, buffer.length) != -1) {
			out.write(buffer);
		}
		gzIn.close();

		// URL decode it
		string = out.toString().trim();
		string = URLDecoder.decode(string);


		// Print out the result
		System.out.println("OUT : " +string);
       }

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk

 

[coderiyer]

Hi sedj!
Thanks for all your help and time. I will take it from here. It must have been frustrating but I really appreciate for sticking and helping me through it.
-coderiyer

 

[sedj]

Good luck fella, hope you find the solution in the end.

--------------------------------------------------
Free Java/J2EE Database Connection Pooling Software

http://www.primrose.org.uk