Datastage - How to process several files

[tempra]

Hi there. I'd like to see the opinion of the best DS developers of this forum. I have to develop a ds job to read a folder that receives several flat files of same format in daily basis. So, I'm just wondering the best way to aproach this, I mean, wether to implement 2 jobs, one to read the folder content (loop) and other to process the file passed by parameter or to write a external program (shell, c) that merges the files into one and then process it in a job.

Thanks in advance for the attention.

All the best,
Melquior

 

[johnherman]

As I think about your question, it becomes clear to me that the decision should be based on matters other than what you have stated. For instance, is it OK to load one file if another from the same day errors out? Is partial information (only one of two files loads correctly) valuable or is it an all or nothing situation?

From a maintenance point of view and considering the KISS principle, merging the files using uniz cp or Dos/Windows copy before loading is clearly the choice. However, this option does not allow for partial loading except through explicit Data Stage processing in the job.

-------------------------
The reasonable man adapts himself to the world. The unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw

 

[BocaBurger]

The IBM product Websphere DataStage TX might be what you need. It can trigger the engine when a file is created and process it realtime. It can also be set up to trigger and wait for later files, if you need a batch type process, or it can just trigger on a time basis and get all the file in a folder as needed.



BocaBurger
<===========================||////////////////|0
The pen is mightier than the sword, but the sword hurts more!

 

[tempra]

Thank you guys for the replys. It seems that I will have to code everything as it's unlikely that my company will upgrade the ds 7.0 in use.

Cheers,
Melquior