Taking out certain text out of a file

[jpor]

Hi all,

Currently I have been given the task to error check a file generated via an Informix Database. Basically if fields 6 to 11 (Seprated by ';') are blank then I need to strip these lines to a file. But leave any un-erronous lines in the original file. Here is some example data:

1).. This part is correct and does not needed stripping out.

1;0045;NORTHAMPTON;09/12/2004;mrs;marge;simpson;1;Some Street;Springfield;;SP11A
A;marge.simpson@simpsons.com;01270555111;;0794646464;09/12/2004;639639639639639;
Some MakeModel;;Poor;TMobile;K1089;K1121;;;;;;;Absolutely goosed;N;;ABC1;DEF2;10
/12/2004;

2).. This part is incorrect as 'marge' is missing.

2;0046;NORTHAMPTON;09/12/2004;mrs;;simpson;1;Some Street;Springfield;;SP11A
A;marge.simpson@simpsons.com;01270555111;;0794646464;09/12/2004;639639639639639;
Some MakeModel;;Poor;TMobile;K1089;K1121;;;;;;;Absolutely goosed;N;;ABC1;DEF2;10
/12/2004;

3).. This file is incorrect as both 'marge' and 'simpson' are missing.

2;0046;NORTHAMPTON;09/12/2004;mrs;;;1;Some Street;Springfield;;SP11A
A;marge.simpson@simpsons.com;01270555111;;0794646464;09/12/2004;639639639639639;
Some MakeModel;;Poor;TMobile;K1089;K1121;;;;;;;Absolutely goosed;N;;ABC1;DEF2;10
/12/2004;

Does anyone have any ideas?

Thanks.

( "To become Wise, first you must ask Questions&quot

 

[ericbrunson]

Here's some pretty straightforward python to do it.

import csv

for row in csv.reader( open( 'inputfile.txt' ) ):
    # python arrays are zero based
    if ( row[5] and row[6] and ... and row[11] ):
         print ';'.join( row )

 

[ericbrunson]

Oops... it's only slightly complicated because the default delimiter for a csv is a comma. This will work:

import csv

p = csv.reader( open( 'inputfile.txt' ) )
p.dialect.delimiter = ';'

for row in p:
    if ( row[5] and row[6] and ... and row[11] ):
         print ';'.join( row )

 

[jpor]

Ah forgot to mention. I am Using AIX 4.3.3 ML9. Not sure if I have python installed. I do have perl.



( "To become Wise, first you must ask Questions&quot

 

[ericbrunson]

Perl sucks.

 

[jpor]

Okay. I'll try the python. Thanks for your contribution ericbrunson.


( "To become Wise, first you must ask Questions&quot

 

[ericbrunson]

Careful of the line wraps put in by tek-tips:

#!/bin/ksh

IFS=;

while read field1 field2 ... fieldx < yourfile.txt
do 
   if [[ "$field6" != "" && "$field7" != "" && ... && "$field12" != "" ]] 
   then
       print "$field1;$field1;...;$fieldx"
   fi
done

 

[ericbrunson]

awk -F\; '{ if ( $6 != "" && ... && $12 != "" ) print }' < yourinput.txt

 

[ericbrunson]

The python is the most flexible, because it would handle input like:

field1;field2;"field3enclosedbyquotes";field4;"field5;has some random;delimiters";field6

 

[jpor]

Thanks for your input ericbrunson. But will your scripts/programs remove the offending lines that have blank fields from the original file? and place them into another one?



( "To become Wise, first you must ask Questions&quot

 

[ericbrunson]

It will place them in a separate file, but you can just write that file back to the original. Anything that claims to modify the file for you, is in actuality, doing just that. It may reconstruct the file in memory, but that's hardly more than semantics. Even vim makes temp files.

 

[xmb]

Cool csv python module, and python itself..

awk '{ for (i=6; i<=11; i++) if (! $i) next; print }'

. Mac for productivity
.. Linux for developement
... Windows for solitaire

 

[xmb]

Plus a -F\; <where is my wakeup call>