Large Flat Files Problem

[kpmINFO]

Hi everyone,

IN our environment we a very large flat file as the source and we are trying to take necessary steps in order to optimize the loading ....In this case cud any1 go over implementation issues , performance related problems which may arise and also how to handle a few junk characters which come in between , plz let me know your ideas would be greatful ...
thanks in advance for your responses ......

 

[blom0344]

Your question is way to vague...
How big are we talking? What type of cleaning is involved?
What database are we talking about.?

More details please...

T. Blom
Information analyst
tbl@shimano-eu.com

 

[kpmINFO]

HI Thanxs for prompt reply ,

The scenario is like this we have very large flat files coming from different , our team is trying to do some performance optimization with respect to usage of flat files and we are evaluating several different steps , so one of more feasible ideas was to generate several partition of this flat files and then perform i.e something like incremental extraction ..possible solution ..also we are getting very large number of junk characters say lname,fname,middlename and say middlename is empty i mean how should we go about cleansing this junk characters and if anyone can post some sample cripts for pre processing data , it would be gr8 ..thanks for the inputs
bye
KPMINFO