Script using ngrep looking for matches...

[Cybex1]

I needed run ngrep against some pcap files for domain names contained in a text file. I was able to accomplish this with the following;

for i in `ls *.pcap`; do <domains.txt xargs -I % ngrep -t -i -W single -q % -I $i >>output.txt; done​

However, it turns out the requirements were not explained clearly enough. Turns out the input file for the domain names is not just a text file with domain names but a tab delimited file with like 20 fields. So I need to bring in all the fields as something like $1, $2, $3, etc. This is needed in order to loop the search routine against $5 and then to write the output file with all the data from the input file when a match is made. Because the output my need to change, I would like to make each field a variable so I can make it look however I need to.

 

[Cybex1]

Guess I should state that the pcap is filtered for UDP port 53, DNS traffic only.

 

[LKBrwnDBA]

You could use awk:

$IFS="\t"
$ awk '{for(i=1;i<=NF;++i) print "Fld"i" is "$i;}' mydata.csv

----------------------------------------------------------------------------
The person who says it can't be done should not interrupt the person doing it. -- Chinese proverb

 

[Cybex1]

LKBrwnDBA,

Thanks! That was helpful but I am still having issues...


I searched the forum and did find some previous posts that were close but not close enough to solve my issue. I needed run ngrep against some pcap files for domain names contained in a text file. The pcap files are filtered for DNS traffic on port 53. I was able to accomplish this with the following;
for i in `ls *.pcap`; do <domains.txt xargs -I % ngrep -t -i -W single -q % -I $i >>output.txt; done

However, it turns out the requirements were not explained clearly enough. Turns out the input file for the domain names is not just a text file with domain names but a tab delimited file with like 20 fields. Up front I am running bash 2 and can't upgrade to 4 for multidimensional associative arrays. That said, I need to bring in all the fields as something like $F1, $F2, $F3, etc. This is needed in order to loop the search routine against $F5 and then to write the output file with all the data from the input file when a match is made. Because the output my need to change, I would like to make each field a variable so I can make it look however I need to. I am thinking AWK could be a solution but I am not sure how to implement it. I got this to work but I am not sure if it is the best or cleanest way to do it. Keep in mind this is just testing and the output of the two variables to the terminal will be replaced with a formatted report with all the variables used in its context.
for i in `ls test*.pcap`; do <~/Desktop/attrib.txt xargs -I % ngrep -t -W single -q % -I $i; done| awk -F"\t" '{print $6, $4, $25}'



So this is where I am at... I have the script returning the column data as variables via AWK. However, in the process of getting the data in the spreadsheet for the matches, I no longer have access to the pcap data. Of course they have requested now that the output look like the example below, so now I need the time and date for the matches from the pcap and I have no idea how to pull it out again without re-running again from the beginning and matching line n in the first output to line n in the second output, which is a huge waste of cycles.
Here is what I have so far...

for i in `ls /media/KINGSTON/test*.pcap`; do </media/KINGSTON/attrib.txt xargs -I % ngrep -W single -qltttiI -q % -I $i; done| IFS='\t'; awk '{print "A match was detected for the " $6 " domain name. " $2;}' /media/KINGSTON/attrib.txt

I need to have both data sources available as variables for each line of matches so I can write the finds out in the report. Some data is needed from each data source when a match is found...

The attrib.txt file looks like thisIt's tab delimited and the numbers are just there to show that there are many fields, the real file has data but I don't have the real file...)
-------------------------------------------------------------
12/03/11 John Smith three four five app.Country-Dogs.net seven eight nine ten
01/23/12 Frank Hussian three four five countrydogs.createsend.com seven eight nine ten
11/10/11 Heather Grayson three four five otracking.com seven eight nine ten
03/22/12 John Smith three four five google.com seven eight nine ten
-------------------------------------------------------------

From the example data I would like the text report to look like below. The first line has parenthesis that show the variable and source where the data in the report would have come from. Example; the “app.Country-Dogs.net” would be the sixth variable via AWK from the first line of the Attrib.txt file.

REPORT.TXT
-----------------------------------------------------------
A match was detected for the app.Country-Dogs.net($6-Attrib.txt) domain name. This domain was added on 12/03/11($1-Attrib.txt) by John Smith($3-Attrib.txt):
Date: 2012-05-18 03:38($1-test.pcap), Source: 192.168.6.91.60531($2-test.pcap), Destination: 192.168.6.1.domain: 62046
($3-test.pcap), Domain: app.country-dogs.net($4-test.pcap)
Date: 2012-05-18 03:38, Source: 192.168.6.91.55058, Destination: 192.168.6.1.domain: 33360, Domain: app.country-dogs.net

A match was detected for the countrydogs.createsend.com domain name. This domain was added on 01/23/12 by Frank Hussian:
Date: 2012-05-18 03:38, Source: 192.168.6.91.41364, Destination: 192.168.6.1.domain: 63516, Domain: countrydogs.createsend.com

A match was detected for the otracking.com domain name. This domain was added on 11/10/11 by Heather Grayson:
Date: 2012-05-18 03:38, Source: 192.168.6.91.57413, Destination: 192.168.6.1.domain: 56461, Domain: otracking.com
Date: 2012-05-18 03:38, Source: 192.168.6.91.43630, Destination: 192.168.6.1.domain: 41693, Domain: otracking.com

A match was detected for the google.com domain name. This domain was added on 03/22/12 by John Smith:

Date: 2012-05-18 03:38, Source: 192.168.6.91.44533, Destination: 192.168.6.1.domain: 40106, Domain: news.google.com
Date: 2012-05-18 03:38, Source: 192.168.6.91.35386, Destination: 192.168.6.1.domain: 10974, Domain: aps.google.com
-----------------------------------------------------------

 

[Cybex1]

Well crap.... Ok, one part of my script was giving me what I told it to but that wasn't what I wanted... This is the only code that currently works:

for i in `ls /media/KINGSTON/test*.pcap`; do awk -F"\t" '{print $5}' </media/KINGSTON/attrib.txt | xargs -I % ngrep -W single -qltttiI -q % -I $i; done

That outputs:

input: /media/KINGSTON/test.pcap
match: app.Country-Dogs.net

U 2012/05/18 03:38:12.374223 192.168.6.91:55058 -> 192.168.6.1:53 .P...........app.country-dogs.net.....

U 2012/05/18 03:38:12.376540 192.168.6.1:53 -> 192.168.6.91:55058 .P...........app.country-dogs.net.............7@.....B........P|...ns2.inmotionhosting.com.........P|...ns.F.g......7<..J|...B......7<..F'..
input: /media/KINGSTON/test.pcap
match: countrydogs.createsend.com

U 2012/05/18 03:38:12.377545 192.168.6.91:41364 -> 192.168.6.1:53 .............countrydogs.createsend.com.....

U 2012/05/18 03:38:12.379876 192.168.6.1:53 -> 192.168.6.91:41364 .............countrydogs.createsend.com..................~...............~..........P....ns1..........P....ns3..........P....ns0..........P....ns5..........P....ns2..........P....ns4..........P....^..
input: /media/KINGSTON/test.pcap
match: otracking.com

U 2012/05/18 03:38:11.392590 192.168.6.91:57413 -> 192.168.6.1:53 .............otracking.com.....

U 2012/05/18 03:38:11.395475 192.168.6.1:53 -> 192.168.6.91:57413 .............otracking.com..................)R..............ns02.domaincontrol...............ns01.@

U 2012/05/18 03:38:11.470465 192.168.6.91:43630 -> 192.168.6.1:53 .............otracking.com.....

U 2012/05/18 03:38:11.473939 192.168.6.1:53 -> 192.168.6.91:43630 .............otracking.com..................)R..............ns01.domaincontrol...............ns02.@
input: /media/KINGSTON/test.pcap
match: google.com

U 2012/05/18 03:38:04.003398 192.168.6.91:44533 -> 192.168.6.1:53 .............news.google.com.....

U 2012/05/18 03:38:04.056097 192.168.6.1:53 -> 192.168.6.91:44533 .............news.google.com.............2....news.l...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...............ns2...............ns3...............ns1...............ns4..

U 2012/05/18 03:38:12.379049 192.168.6.91:35386 -> 192.168.6.1:53 *............maps.google.com.....

U 2012/05/18 03:38:12.381696 192.168.6.1:53 -> 192.168.6.91:35386 *............maps.google.com.............2....maps.l...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...-.......,..J}...............ns2...............ns3...............ns1...............ns4..
input: /media/KINGSTON/test.pcap
match: loser.com

Now I need to try and clean that up to this:

input: /media/KINGSTON/test.pcap
match: app.Country-Dogs.net
2012/05/18 03:38:12 192.168.6.91:55058 192.168.6.1:53 app.country-dogs.net

input: /media/KINGSTON/test.pcap
match: countrydogs.createsend.com
2012/05/18 03:38:12 192.168.6.91:41364 192.168.6.1:53 countrydogs.createsend.com

input: /media/KINGSTON/test.pcap
match: otracking.com
2012/05/18 03:38:11 192.168.6.91:57413 192.168.6.1:53 otracking.com
2012/05/18 03:38:11 192.168.6.91:43630 192.168.6.1:53 otracking.com

input: /media/KINGSTON/test.pcap
match: google.com
2012/05/18 03:38:04 192.168.6.91:44533 192.168.6.1:53 news.google.com
2012/05/18 03:38:12 192.168.6.91:35386 192.168.6.1:53 maps.google.com

input: /media/KINGSTON/test.pcap
match: loser.com

Any SED guru's out there that can offer any ideas on how to clean up sections of the original output, it would be greatly appreciated!

 

[Cybex1]

With the SED stuff I came up with this...

for i in `ls /media/KINGSTON/test*.pcap`; do awk -F"\t" '{print $5}' </media/KINGSTON/attrib.txt | xargs -I % ngrep -W single -qltttiI -q % -I $i; done | sed -e 's/ -> / /g' -e 's/.[0-9][0-9][0-9][0-9][0-9][0-9]//g' -e 's/U //g' -e 's/\.\.\.\.\.//g' -e '/\.\.\.\./d' -e '/^$/d' -e '/input/{x;p;x;}'

However, up front I need to be able to replace ".P...........", ".............", or "*............", etc... That is 13 characters. I am guessing it could be any letter upper or lower, number, or a period. I thought this would work but it did not.

-e 's/\.[A-Z]\.{1,13}//g'

 

[FlorianAwk]

The following command over the log file you gave works fine for me:

sed '/^$/N;s/\n//;s/^input/\ninput/;s/\.\.\([^\.]\(\.\{0,1\}[^\.]\{1,\}\)\{1,\}\).*/ \1/;/^U/{s/U \([^\.]*\)[^ ]* \([^ ]*\) -> \([^ ]*\) [^ ]* /\1 \2 \3 /}' pcap.txt

Result:

input: /media/KINGSTON/test.pcap
match: app.Country-Dogs.net
2012/05/18 03:38:12 192.168.6.91:55058 192.168.6.1:53 app.country-dogs.net
2012/05/18 03:38:12 192.168.6.1:53 192.168.6.91:55058 app.country-dogs.net

input: /media/KINGSTON/test.pcap
match: countrydogs.createsend.com
2012/05/18 03:38:12 192.168.6.91:41364 192.168.6.1:53 countrydogs.createsend.com
2012/05/18 03:38:12 192.168.6.1:53 192.168.6.91:41364 countrydogs.createsend.com

input: /media/KINGSTON/test.pcap
match: otracking.com
2012/05/18 03:38:11 192.168.6.91:57413 192.168.6.1:53 otracking.com
2012/05/18 03:38:11 192.168.6.1:53 192.168.6.91:57413 otracking.com
2012/05/18 03:38:11 192.168.6.91:43630 192.168.6.1:53 otracking.com
2012/05/18 03:38:11 192.168.6.1:53 192.168.6.91:43630 otracking.com

input: /media/KINGSTON/test.pcap
match: google.com
2012/05/18 03:38:04 192.168.6.91:44533 192.168.6.1:53 news.google.com
2012/05/18 03:38:04 192.168.6.1:53 192.168.6.91:44533 news.google.com
2012/05/18 03:38:12 192.168.6.91:35386 192.168.6.1:53 maps.google.com
2012/05/18 03:38:12 192.168.6.1:53 192.168.6.91:35386 maps.google.com

input: /media/KINGSTON/test.pcap
match: loser.com