Removing records from csv file 1

[ianicr]

I have a csv file of about 140000 lines of address and telephone data. I've got another file of 50000 phone numbers that need to be removed from the first file. I've tried doing a grep -v -f file2 file1 >file3 but this takes forever. Is there an easier way to do it other than splitting the second file into smaller chunks and then greping with the smaller files?

 

[vgersh99]

sample records from both files, pls

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[CaKiwi]

This awk program is not tested and assumes that the records in the second file contain only a phone number. Change the phfld variable to the field in the first file containing the phone number.

BEGIN {
FS = &quot;,&quot;
while ((getline < &quot;file2&quot >0) a[$0] = 1
phfld = 1
}
{
if (!a[$phfld]) print
}

CaKiwi

&quot;I love mankind, it's people I can't stand&quot; - Linus Van Pelt

 

[ianicr]

Sample records are as follows:
file1:
Mr,J,Bird,439 Stevens Road,Atecks Green,DERBY,,DE12 4GK,01332999999

The other file is just a file of numbers like:
01332123456
01332987654

And sorry i'm not familiar with awk. Do I just save to a file and do awk -f programname filename?

 

[CaKiwi]

yes

CaKiwi

&quot;I love mankind, it's people I can't stand&quot; - Linus Van Pelt

 

[CaKiwi]

For you sample data, phfld should be 9 or if the phone number is always the last field, you could use

if (!a[$NF]) print



CaKiwi

&quot;I love mankind, it's people I can't stand&quot; - Linus Van Pelt

 

[ianicr]

Thanks the awk worked great!

 

[ianicr]

Just a quick question. Is there a way in awk using the above script to echo the records it has removed into another file?

I'm very new to awk but it seems like an excellent utility to learn. Does anyone knows of any good awk tutorials/books?

 

[CaKiwi]

To write the records removed into another file, add an else to the if, like so
{
if (!a[$NF]) print
else print > &quot;file3&quot;
}

I use the O'Reilly book &quot;Sed & awk&quot; by Dougherty and Robbins. There are a lot of resources online including the gawk manual at

http://www.gnu.org/software/gawk/manual/gawk-3.1.1/gawk.html

CaKiwi

&quot;I love mankind, it's people I can't stand&quot; - Linus Van Pelt