Transformation limitation

[jguoz]

I'm attempting to create a GIANT if.. then... else.... statement in a key expression of a transformation stage. I think there might be a limit to the number of if statements I can put into a single expression. Does anyone know of such a limit? And if so, is there a way to override the limitation with a change in the registry?

Thanks in advance,

jguoz

 

[johnherman]

You could try asking Ascential support if there is a limit, or even search their knowledge base.

-------------------------
The trouble with doing something right the first time is that nobody appreciates how difficult it was - Steven Wright

 

[BocaBurger]

You would have to ask IBM support these days. However, trying to reach a limit sounds like you have to rethink what you are doing.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[johnherman]

Whoa Nelly, you mean IBM finally finished buying Informix?

-------------------------
The trouble with doing something right the first time is that nobody appreciates how difficult it was - Steven Wright

 

[vmcburney]

If your statement is that big you may be better off moving it into a routine where it can be properly tested for all combinations. It will also let you switch to a case statement, which may be easier to manage.

regards
Vincent
An Expert's Guide to WebSphere Information Integration

http://blogs.ittoolbox.com/bi/websphere/

 

[BocaBurger]

Where have you been? It's been 7 months since the final paperwork was completed? Ascential DS is now IBM WebSphere DataStage.

BTW, those Ascential shares you have, use them for note paper now



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[jguoz]

Thanks for your help on this... I moved my code into a routine as a case stement as suggested by vmcburney (That worked out very well).

The next problem I had was that the job would not compile... but after spending sometime with IBM support, it turns out the the job was currupted and I needed to export the thing and then re-import it back to get it to compile.

Now back to the same question, with a twist. Is there a limit on the number of records that can be processed into a transform. Now that I have the code working with test files, I use a more realistic amount of data. The job aborts when I reach 630,000 records.

During normal processing I'll have several million rows being processed thru this transform, so I hope it just something I've done wrong.

I'm going to contact IBM again regarding this issue, but I thought I'd check here first. I'll post my results if IBM gets me that answer before you guys do.

 

[jguoz]

UPDATE *****
IBM support was not able to give me an answer that resolved our memory issue. After verifying that our configuration should support the number of transaction that I need to process, they suggested adding a sort before my aggregation step to help with the process.

Well I could not get the sort to work properly. Then after having a review with the "team", we came up with the solution that seems to work for us.

I've added tables on the Unix box that will be used between each of the stages. Since the data is going to table space, it doesn't effect the memory usage at all.

I also added an order by clause in the selection tab of the output going to the aggregation stage. That seems to have streamlined that process too. No sort stage needed now.

Hope my pain helps someone out there.

 

[vmcburney]

I've had a problem with the server edition aggregation stage in a much older release. In one project I abandoned it entirely and pushed my data into universe tables and used group by select statements from those tables. It may be that on unsorted data the aggregation stage has to wait for every row to be input before it can output the results, but on sorted it data it can output rows as it reaches the end of each unique aggregation key combination.