Testing

[jack1955]

This might become a debate, but I'd like to know what your environments are like and how you do your application testing.

I am consulting (except I would call it contract programming) at my former employers shop. Testing on the AS/400 is a mixed bag. I like to copy specific data, not the whole file into my developer library. Then I can control exactly what happens to that data.

There are 2 testing environments, integration libraries (programs, source, data), and user acceptance test libraries (program, source, data). The problem here is all AS/400 developers use the same libraries as well as the users. One can chase a problem that happened because someone reloaded the data (incorrectly).

A few years ago I bought TestBench for application testing. It's specific to AS/400, but can be used for GUI apps as well. It uses multiple libraries and journals for testing. So each test, tester, developer, and whoever would have data for their testing. However, when I bought the package I told the boss it would be a big deal to populate the data, a lot of work.

I switched to the data warehousing and ebusiness team and off the AS/400. Someone else was responsible for implementing TestBench- actually it was 3 other people. As you can imagine, it was dropped. Now (that I'm back on the AS/400) I feel we all have the same problems as before. Either too much data for testing, or bad data, or not the right data.

I have been writing test scripts for my application bug fixes, but it's a lot of work. There are no test scripts at the job site, except for new programming. These never get updated. No one figured out the difference between temporary project documentatio and permanent application and system documentation.

What's your shop like? What are your testing experiences? How about those developers who know nothing about the application- they make changes, but can't properly test?

Maybe we can learn from each other.

Jack

 

[jellybeenz]

I am the only developer in my organization (previously it was contracted out). If my program performs updates/writes/deletes, I copy the files into my own library which defaults to the top of my library list, then run my application. If it is just an inquiry or report, I test on live data.
On a somewhat related issue, we recently began looking into Visual RPG for GUI apps by which to access AS400 data from the users desktops. The network admin said he does not want me to test on the network, citing "security issues" with testing on "production data". He suggested leasing an AS400 just for testing purposes, not realizing that all testing is currently done on the "production machine" (common practice in AS400 shops). The reaction to this suggestion was a collective eyeroll, as you may imagine. His REAL issue is in giving me any authority to install my applications on the workstations.

 

[khood]

jack1955,

I too am a contract programmer & have several clients. They all have very different ways of handling testing/test environments, usually depending on how big the IT staff is.

The largest company has an as/400 dedicated to testing, with multiple test enviroments. We use Implementer for change control and Mimix for data backup and replication. We have a complete set of files, programs, menus etc. in the test environment(s). Every couple of months we do a "test environment refresh" with current live data.

Another client uses 1 as/400, but a duplicate set of libraries for testing (with a "tst" prefix). In order to use the environment, you are required to login using a test environment ID. Of course, a programmer can easily circumvent this approach.

My third client does not really have a test environment. It is left up to the individual programmer. This also happens to be the smallest shop (2 programmer2). These guys also are the most frugal when it comes to spending any money.

The bottom line is that all of the approaches work, with advantages and disadvantages to each of them. I prefer to have a development box with change-control software, but this also requires a lot of setup & maintenance, and sometimes the CC package itself causes problems.

 

[jack1955]

This is good. A dialog.

OK. We know the basics. Do you create a test plan and test script? Do you test your report programs with no input data to see if the headings print correctly? In some cases at my shop, some reports print a single line- "TOTAL SALES 0" because no one tested with no input. No headings, no selection criteria prints.

Do test with data that will exersize your program or just what is in production today?

Do you rely on users testing? In most cases I have found they have no training and no desire to do the thorough job, even when I provide a test plan and script. Some actually get people at the plants to help test. They are the good ones.

 

[jellybeenz]

When coding in RPG, I almost never use the program cycle so my first heading exception line is normally before the main program loop (only if I want printed output regardless). I know there is a place for the program cycle, but that's just the style I prefer (more visual?), even with simple reports. I usually don't have a headings problem, input or no. Unless it is a large project, I don't bother with a test script. I just tell the users how it works (often I provide user documentation). For interactive apps, I may tell them to try and "break" it in the test environment.
As far as test data, I do try to exercise the programs with alternative data scenarios (if I know about them). That's where it's so useful to have input from the users. They sometimes know more than I do about what the output should be. Often I have to probe a bit and ask about any unusual conditions so that I can program for them. I have found that user testing ability is all over the map, but there is usually at least one or two in an organization who are conscientious and thorough. Also, the more input they have into the process, the happier they are with the finished product.

 

[SuzieW]

Hi Jack

I'll tell you a bit about testing in our shop. We are a big department (probably over 100 AS/400 developers).
We have a DEV box which is partitioned into two - one partition being devoted solely to testing. We have numerous test environments which each contain a copy of the live database (with a representative sample of data). When testing, you sign on using a special user profile which will set your library list to the appropriate test environment. Control data is held in a library common to all test environments and is mirrored from the production machine. If your development involves changes to control data, then you have to isolate the changed control data within your own test environment.

We tend to have separate system test and user acceptance test phases. System testing is done by the IT developers, in accordance with a system test plan which is written based on the design document. User acceptance testing is carried out by a combination of an independent testing team and end-users, who write their own scripts based on the IT design. The user testing is key, as the users don't know the foibles of the system and will test according to how it will be used in real life. IT testing has a tendency to supply "kind" data which may not "exercise" the programs. But both types of testing are essential.

Things can always slip through the net though - as you say, it is easy to forget to test with no input data. Test environments and their data are not always reliable. You may spend time investigating a problem which turns out to be duff control data in your test environment rather than a problem with the software. Volume testing can also be an issue. If you don't test with realistic data volumes, performance problems can be encountered in production. We had this with some Synon generated subfiles which had a low scan limit. This wasn't revealed in testing due to the volumes of data used.

We have also had problems with system date/job date differences. Some of our programs use system date, some use job date. On production, this is not usually an issue, but in test systems, we may roll the date forward and then any number of problems can arise...

All in all, testing can be very frustrating. I could go on about it all day...

 

[jack1955]

Thanks for your replies.

I'm using this message board for several reasons- one is that I want other people's insight to testing and another is that I was out of work for over a year. Since I'm not an employee of the client's, I need some place to vent. I want to email the IT staff almost daily with questions and suggestions, but it's not my place.

The staff is augmented with contract programmers who design the applications, then code, test, implement. The staff is mostly programmers of short duration, relatively inexperienced, with very little mentoring.

I can see and it was also my experience that there are vast differences between small and large shops with regard to testing. While I was unemployed - I had just 3 interviews - 2 of the places had separate QA groups. I would have been just a programmer. I wouldn't have been happy- just employed. I am a jack-of-all-trades. The full SDLC and business knowledge to boot. And that's the way I like it.

 

[jellybeenz]

I much prefer the informality of a smaller IT shop. As an experienced programmer, I prefer the increased latitude and responsibility I receive here as opposed to a large organization. But that also means that when an application doesn't work, the buck stops with me. I can't blame QA for not catching the bugs. Also, I'm involved in the project from start to finish. I interview users about their business needs, create the program specs, code and test, then implement and write both technical and user documentation. Oh, and I also do a lot of operations stuff like system security and OS upgrades, etc. I would hate the restrictions and compartmentalization of a large IT shop. Been there, done that.

 

[gpzcrasher]

SuzieW

Why have a seperate user profile for testing? Just edit your library list. --------------------------------
If it ain't broke, don't fix it!

 

[PeteJohnston]

I'm a contract developer and I have also noticed big differences in the way people implement (or not) a testing strategy. Change control is a good thing (they use Thenon at my current place) if it is a big department since it (usually) stops people trying to change the same code at the same time but it has it's problems when changes are required YESTERDAY to an object being worked on by someone else. A smaller shop can get away without change control since you probably know, or can ask, what everyone is working on.

Good things and bad things in both cases.

 

[SuzieW]

Hi gpzcrasher

Of course, you're right that we could just change the library list, but when the users are testing, they don't have their own profiles on the dev and test boxes, and you can't expect them to manipulate library lists, so they just sign on with the profile and it puts them straight into a menu to emulate what happens on live.

I must say I agree with jellybeenz about the disadvantages of working in a large department - it's much nicer to follow things through from start to finish, then you know you're giving the users what they want.

Suzie

 

[jack1955]

You guys create test scripts and plan for your own testing? I have been doing that because it gives me a place to put notes on future testing, how the program works, what I did to set up the data. Also allows me to pass this information to co-workers. Well, if they share their information with me. Have you noticed that some programmers just don't share? So everyone has to learn the same lessons. It's like program documentation-some just don't do it.