Odd compression method for Raster Graphics

[Jeroen Bobbeldijk]

Hi there!

I'm currently trying to decode a PCL file that is captured from an OKI printer.
The full page seems to be printed as a Raster Graphic. I have all the command decoding complete, the only thing I need to finish is the decoding of the Raster Graphics data.
Now comes the odd thing about the captured data, the compression method is "131", I can't find out what this method is. It's not in the PCL documentation (only 0-5). I did some testing and I don't think it's one of the other compression methods. The data looks to be in pairs of 5 bytes. It sends the raster graphics in parts of 4800x64 pixels.
Here's a preview of what a "pair" looks like in hex:
1fffff3a00 (this seems to be an empty pixel row, white lines, the first two rasters are 64x 1fffff3a00, which makes sense as the first part of the page is all white)
So the base data is at least 320 bytes (64 x 5), but the other rasters size depend on the content.
Here's a sample of some other pairs:
7fff917fff
80003f0603
e05f4e03ff
fc0d001fff
ff3a001fff
917f1f8800
1fff92ff1f
87005fff91
000f805f76
00007c0d00
1fff920f1f
87001fffba
031f5f005f
ffb17fe000
1f66003fff
847fff2180
00d003ffff
0007ffff41

I tried a few different approaches, like taking 4 bytes for an int32 and one remainer byte for the pixel value, but this does not seem to be a correct approach.

Anyone has any idea what type of compression this is? None of the existing tools seem to be able to understand the compression method.

 

[Hugh Hood]

Jeroen,

I suspect you are seeing an OKI extension to the standard HP PCL language. It may be undocumented.

For example, Brother adds quite a few extensions in their implementation of PCL.

Concerning their raster graphics compression, they offer:
---------------------------------------------------------------
6.3.8.
Set compression mode
<Esc>*b#M
Identifies the method of compression used to encode the raster image that is to be sent to the printer.

# is 0, 1, 2, 3, 5, 9, 1152 or 1024
# = 0 signifies no compression is to be used
# = 1 indicates that run-length encoding is to be used
# = 2 indicates that TIFF (tagged image file format) encoding is to be used
# = 3 indicates that delta row compression is to be used
# = 5 indicates that adaptive compression is to be used
# = 9 Mode 9 compression
# = 1152 indicates that CCITT G3/G4 data compression is to be used
# = 1024 indicates TIFF format
---------------------------------------------------------------

My quick search of OKI reference materials only shows support for modes 0-5. Obviously, there are more, but you may have to contact OKI to get that info.

 

[Jeroen Bobbeldijk]

Hi Hugh,

Thanks for your reply.
I have searched a lot for documentation on the format, but sadly couldn't find it. I also think it's an OKI extension.

I'm currently in contact with OKI tech support, but they are telling me they don't use compression method 131.
I have done some tests, and it seems that the generic OKI PCL driver uses compression method 0, while the specific OKI device PCL driver uses method 131 (I can reproduce this with the latest driver), sadly, we can't switch to the generic driver. It seems odd to me that they think their driver doesn't use this method. This method seems to be undocumented, even internally.

I hoped someone on here would be able to figure out how this compression works, I have been staring at it for way too long.

 

[jlasman]

Jeroen,

Are you sure that the driver in play is actually PCL5? You seem to be well informed, but it sounds like some proprietary format other than PCL5 like the "Windows Only" printers might use.

jim

Jim Asman

http://www.spectracolorservices.com

 

[Hugh Hood]

Jeroen,

Other than suggest you prepare an uncompressed image (with the normal PCL driver) in order to run it through every known common graphics compression program, and then compare its output for a match to your '131' compression from the special PCL driver, I'll mention that some of the reverse-engineering boys speak of using the command-line tool 'binwalk' to identify unknown compression formats.

It is on GitHub:

Link

This tool does many things. The options I see most often with compression detection are '-be', but it seems to have a general analysis output function as well.

Good luck. I understand completely how you are tired of trying to decipher this. I've had that feeling before on certain projects. Some which I completed; and some which I abandoned.

You may also want to look at the Brother extensions '9', '1152' and '1024'. Perhaps one of those is the '131' OKI uses, albeit with a different name.




Hugh Hood

 

[Jeroen Bobbeldijk]

jlasman,

It's definitely PCL5e, I already created a parser that is able to read the full job, and it follows the PCL 5e spec completely.
The print job also contains the following information:
@PJL COMMENT Product Name : OKI PCL5e Printer Driver
@PJL COMMENT Device Name : OKI B431(PCL)
@PJL ENTER LANGUAGE=PCL
The raster data is also according to the spec, it's just that the data is compressed using a format (131) that's not documented anywhere, at least not in public.

Hugh,

Thanks for the tips! Didn't try these things yet, so that's something I can work with.
I'll report back soon.

 

[jlasman]

Jeroen,

So what happens if you send the file to an HP PCL5 laserjet?

Have you tried to view the file with a PCL viewer such as PCL Works, Swiftview, or CoolUtils?

Jim Asman

http://www.spectracolorservices.com

 

[Hugh Hood]

Jim's suggestion to run your file through a viewer prompted me to take a look through the source code of GhostPCL.

I did find this:

 * Types:
 *
 *     0 - compression mode 0 (no compression), param is size in bytes
 *     1 - compression mode 1 (run length compression), param is size in bytes
 *     2 - compression mode 2 ("Packbits" compression), param is size in bytes
 *     3 - compression mode 3 (delta row compression), param is size in bytes
 *     4 - compression mode 4 (no compression blocks), param is size in bytes
 *     5 - compression mode 5 (adaptive), param is size in bytes
 *     6 - compression mode 5 (ccitt group 3 1d), param is size in bytes
 *     7 - compression mode 5 (ccitt group 3 2d), param is size in bytes
 *     8 - compression mode 5 (ccitt group 4 2d), param is size in bytes
 *     9 - compression mode 9 (modified delta row), param is size in bytes
 *

I'm making a wild guess, of course, but your '131' may be the ccitt group 3 1d referenced in #6 above.

Again,it's a wild guess, but they do share the '31'.

And, because you don't have enough to investigate, here are the supported methods from the Lexmark implementation of PCL:

Parameters:

0 Uncoded (default)
1 Run-Length Encoded
2 Tagged Image File Format (TIFF) Byte
3 Delta Row
5 Adaptive Compression
9 Replacement Delta Row
999 Zlib
1002 Group 4
1003 Group 3 one dimensional
1004 Group 3 two dimensional K=2
1005 Group 3 two dimensional K=4
1006 TIFF word (16 bit)
1007 TIFF double-word (32 bit)
1008 Adaptive Compression (includes TIFF word and TIFF double-word)






Hugh Hood

 

[Jeroen Bobbeldijk]

Jim,

I don't have any other PCL printer, but my guess is that it won't print. A newer OKI printer also prints it just fine though, while that specific device driver does not use 131. I'd happily send you the PCL data if you'd like to try.
I could only find 1 PCL viewer that could view the file properly: EscapeE by RedTitan (

https://www.pclviewer.com/),

they even mention it specifically in the changelog(10.10D: OKI print files using compression mode 131 can now be read), all the others fail. I contacted RedTitan but they do not want to help, even when being paid.

Hugh,

I'll play around with the different compression methods tomorrow. But previously the data didn't look like any of the compression methods.

 

[jlasman]

Jeroen,

Absolutely, send it to me. I have a few viewers etc. You can get the address from the website or try -- jim AT website --

Tek-Tips doesn't like email addresses being displayed for fear of attracting data miners.

It would probably help if you could explain the problem. I gather that you are maybe getting the PCL from elsewhere and need to do something with it. Yes?

jim

Jim Asman

http://www.spectracolorservices.com

 

[Jeroen Bobbeldijk]

Jim,

You have mail
What we are doing, basically: intercept print jobs with a hardware device, read them, conditionally edit them or just pass them along.
We have got this working for multiple protocols (like ESC/POS, XPS and PCL), but this device has some specific PCL which is giving us a lot of headaches.

 

[jlasman]

Jeroen,
Received the file. It does NOT display/print on any of my viewers/printers. First, exactly which printer driver produces this output?

I notice that this is more than just the compression scheme. If you look at the code immediately before the b131m, you see a <esc>*r4800s64t32768A
I appreciate that the image is 4800 pixels wide and I suppose is broken into blocks that are 64 pixels high, but the 32768A start graphics part of it is only a single digit in the manual, typically "1".

It seems to me that without some guidance from Okidata as to what this all means, it will be very difficult if not impossible to reverse engineer this. I would assume they have some sort of developer program who are privy to such info. I will still plug away at this for a time.

jim



Jim Asman

http://www.spectracolorservices.com

 

[Jeroen Bobbeldijk]

Jim,

The driver that produces this output is:

https://www.oki.com/ge/printing/sup...es&productCategory=mono&sku=44566325&lang=ac2

The device that was used is an OKI B431, but I know that the PCL also works fine on OKI B412 (as I also have that one), even though that Driver does not produce such PCL.

I know the 32768A is odd, also because it overflows the range the value is supposed to be in (limit is 32767), so it might be an driver error and should be 1.
I managed to discover that the last byte in the 5 byte chunk is the pixel value byte, when I change it from 00 to FF, I get black lines, when I change it to 0F or F0, I get white/black blocks. So it must be some kind of run-length encoding.

Another interesting find: if you use the first 4 bytes as uint8, and add them together, you get 600 (1F = 31, FF = 255, FF = 255, 3A = 58). This works for the first rows, but not all. 600*8 (8 pixels in pixel value byte) =
4800, which is the raster width. Why they would use 1F FF FF 3A and not FF FF 59 00 is a mystery to me though. So it might mean something else.

 

[Hugh Hood]

Jeroen,

I've got one more possibility for you to consider - you may be seeing ThinPrint.

Thanks to ThinPrint’s adaptive compression method, data volumes when transferring print jobs are reduced by up to 98%. The highly-compressed print data can be processed directly with the ThinPrint Client from the OKI Data printer.

Highest printing speeds thanks to perfect compression of print data streams

If in fact you are seeing Thinprint's 'Advanced Adaptive Compression', which apparently is lossy, you may have to obtain some insider information to get what you need. That may cost your company money, I suspect. I searched ThinPrint's site, but very little technical information is made public.

But, you mention that RedTitan had figured this compression out, so I just don't know.

When you solve this mystery, please pass along what you learned.




Hugh Hood

 

[jlasman]

Jeroen,

It occurred to me that you might go to

http://www.leadtools.com

and see if they have anything that can help you. I purchased a windows "printer driver" from them 15-20 yrs ago or so that would output an image file rather than a printer file. They were way ahead of the curve.

Perhaps they have something that will help you. Just a thought.

Jim Asman

http://www.spectracolorservices.com

 

[Jeroen Bobbeldijk]

Hi Jim,

They don't seem to have anything to help me.
I have given up trying to figure out this compression format myself. I'm still waiting for a reply from OKI.

@Hugh,

It looks like that requires a special driver, so I don't think that's it.

 

[jlasman]

I am suspecting that each of the *b***W lines actually wraps to 64 raster raster lines on the page and that looks pretty obvious for all the white space at the top. It is somewhat reminiscent of the windows only printers that convert all output to giant bitmaps. They did that on cheap printers to put the processing load on the computer CPU and off the far weaker one in the printer. Cheap printers are cheap for a reason. If DansdadUK hadn't passed away I'm sure that he would have had some valuable insights to offer.

Do keep us informed if Okidata comes through with any help. It does seem, though, that the information is deeply buried in a vault somewhere.

Jim Asman

http://www.spectracolorservices.com

 

[Jeroen Bobbeldijk]

After a lot of mailing, calling and waiting, my question went through multiple offices. It ended up in Japan (I guess at OKI HQ).
They decided they do not want to disclose any information about this compression.

 

[Hugh Hood]

Jeroen,

I most certainly give you credit for pushing this are far as you have.

Unfortunately, I have found that "no" is often the easiest answer for some of the larger companies. Even when I'm willing to pay for information.

Obviously RedTitan either obtained the information, or reversed it out.

Was 'binwalk' any help in deciphering the compression format?




Hugh Hood