COBOL performance benchmark

[jkilthau]

I need a COBOL application that can provide my client with a performance benchmark between interpreted code and machine code (something close to machine code actually).

I need something that will perform lots of reads, writes and mathmatical processes which will provide a fairly balanced evaluation of overall performance. It will be compiled to both p-code and object for comparison.

Of course I can write this application, but I'm inherently lazy and and am hoping that, by some oft chance, somebody here might know the whereabouts of such a program

Thanks for any help.

 

[Crox]

Hi,

There is a file COBVSC.ZIP from Computer Associates that is meant to compare the performance between CA-REALIA and C.

It contains COBOL and C source with look-alike functionality. Perhaps that COBOL source is what you need.

If you cannot find it, mail me and I'll send it to you, but promish to tell us about the results.

Regards,

Crox

 

[jkilthau]

Thanks Crox. I'd like to have a look at it, but was unable to locate it through Google, and a search of the CA site. Would you be so kind as to send me a copy? I'll definately share the results (there will be more than one platform and COBOL vendor comparison).

Thanks much
jkilthau@scif.com

 

[Crox]

It is on its way!

Regards,

Crox

 

[jkilthau]

Sorry Crox but I never got the file. Is it huge?

Please try to send it to jkilthau@pacbell.net and cc:
it to jkilthau@scif.com.

Much appreciation and Happy Holidays,
Jim

 

[Crox]

Hi,

it is not huge. I send it now again to the two addresses you gave me here.

Regards,

Crox

 

[Thane]

Crox,

I'd be interested in that same program. Can you email it to me? - T H A N E @ S O F T W A R E S I M P L E . C O M

(Collapse the spaces, obviously).

Thanks

 

[Crox]

FYI:


This is an automatically generated Delivery Status Notification.

Delivery to the following recipients failed.

jkilthau@hobh.scif.com

 

[k5tm]

I can probably predict the outcome.

If your benchmark is all IO statements, the ratio between interpreted and machine code will be close to 1:1. If your benchmark is all computation, the ratio will be somewhere between 3:1 and 10:1, depending on the quality of the interpreted code and the quality of the machine code; about 5:1 is typical these days. I have found the ratio between interpretation and machine code to be consistent in several situations, not just COBOL.

While you are benchmarking, make sure you understand the implications/impacts on resources other than time. Will your hardware configuration be able to scale to as many users considering the much larger memory requirements (also a factor of 5:1 to 10:1) of machine code? If your developers are doing most of their testing in interpretive mode (possibly to save the time needed to link, or to use enhanced debugging features available through the interpreter), are your testing suites sufficient to catch possible differences in behavior betweeen the two methods?

Most who go through this exercise come to a common conclusion: execution time is important in only a small fraction of an application.

 

[Crox]

Hi k5tm,

Funny to see that people have all kind of different experiences. In the mainframe environment of one of the biggest banks in the world, performance is an important issue. CPU time is very important. Not only because of the costs but most of the time because of some time-window. During the evening and the night, there are batch systems processing. It is important that they finish in time. Sometimes it is difficult to get all the things done. When tuning, we often find also cpu-consumers.

Can you tell us more about the platforms you measured and perhaps also how you did it?

In the past I compared CA-REALIA COBOL with some popular PC development environments with CPU bound processing. The ratio's where varying with CA-REALIA on the first place and the others on ratio's from 3 - 3000!

Regards,

Crox

 

[k5tm]

Hi, Crox.

Yes, of course in the big-time banking environment you describe, CPU time is important, not only from the perspective of having batch processes complete in time, but also because CPU time on a mainframe can be an expensive resource. But jkilthau poses the question in a way that, combined with his profile and his email domain name, leads me to believe that s/he is concerned with Linux, Unix, and perhaps Windows.

If you read my profile, you will understand that I thoroughly understand the issues of interpretation, especially of COBOL programs. In another lifetime, I evaluated the issues of interpretation of a famous CISC processor instruction set using a RISC processor. This was obviously a compute bound comparison, and I found that the ratios held up quite well with better execution time ratios being obtained by more extravagent use of memory (e.g. just-in-time compilation of small loops).

WRT Realia and other vendors' offerings (including those of my own company, if you know its total history) in machine code generation for less-than-mainframe environments, the tradeoffs stated in my original response are obtained. You certainly can achieve ratios of 3000:1 in highly contrived examples (contrived from the point of view of a typical COBOL application). The purpose of my response was to encourage jkilthau to consider factors other than raw execution time so that his comapny is not faced with additional problems after expending time and resources getting to a machine code implementation, or disappointment because the application is IO bound.

Frankly, in this time of Java (interpretation), HTML (interpretation), and Visual Basic (typically interpretation), but again with deference to the differences in the mainframe world, I would think that you would desire interpreted code to be state-of-the-art! ;-)

Best regards,
Tom

 

[jkilthau]

Thanks beaucoup Crox. I got the package at my home address and will be compiling and trying it out at my clients site.

The code does not look to be completly ANSI85 compliant, so I might have to make some revisions to get it to function with Mico Focus and Acucobol.

As you requested, I WILL post the results here.

Thanks again for taking the time to help,
Jim

 

[jkilthau]

Ok... It took a bit of tweaking to get the program to compile on the two COBOL platforms, but it ran fine. My interest was to compare Acucobol Vs. Micro Focus on the HP/UX 11.i platform. I was especially interested in the results of using the MF feature by which you can generate a .gnt file, which is something close to object code. Acucobol only generates interpreted code. MF also has an option to generate interpreted code (.int) so I ran the test agains all three formats just to be fair. I have to mention that there were a few other programmers on the machine at the time, so this was not a controlled scientifically accurate test. I did run the tests several times and got very similar results, so I think the test is fair.

Although I've never been a fan of Micro Focus and will usually recommend to my clients that the opt for Acucobol due to it's cleaner utilities and fantastic support staff, the MF .gnt code handily outperformed all the p-code.

results; MF gnt
Acu MF int MF gnt "L" server
Control logic tests 2.09 10.37 0.12 0
Display tests 2.33 0.08 0.07 0.13
Data movement tests 0.78 12.65 0.33 0.04
Computational tests 13.94 69.31 1.32 0.17
Table handling tests 16.15 117.2 1.45 0.24
File input-output tests 1.77 1.41 0.65 0.08
Sieve of Eratosthenes 18.96 50.32 0.88 0.11

Oh yeah... the first three columns were run on an older HP "K" class server while the last coulumn is on an "L" class so you can see the difference the newer hardware makes.

These tests do have some bias towards compute power, but I was happy to see that they DO mix in file I-O, table-processing and key access. The test looks very much like what I had started designed for myself before Crox saved me all this work by sharing the code with me

Crox, if I can return the favor, please don't hesistate to call on me.

Thanks to all,
Jim