Data Scrubbing 30 Gigs of Flat Files

[underwun]

Looking for a product that can help clean up 30 Gig archived flat files and pull the stuff that matters...any suggestions?

 

[srinathmk]

Hi underwun,

If it is on UNIX or similar platform, try the option of developing scripts for scurbbing flat files using a language called AWK with Shell script. This is really fast. And also AWK is pretty simple. Good luck.

Thanks and Regards,
Srinath M.K

 

[dwtech]

If it is a one time work then you are probably better off writing C/Perl/awk scripts.

If it is a regular scenario which has to run as batch job then check out Ab Initio.

http://www.abinitio.com/

This tool is expensive but is tailormade to carry out the complex transforms and handle very large data volumes. Can do 30 gigs in 10-30 mins depending upon the cleaning required.

Be Diligent!
DWTECH

 

[gleman]

Ab Initio is like using a sledge hammer to swat a fly if you're just trying to cleanup up a 30 gig file. Many people use ETL tools to manage data quality, just like many people use a hammer instead of a wrench to loosen a bolt. It'll get the job done, but....

Greg Leman
Metagenix, Inc.

http://www.metagenix.com