log files n command line

[hyperdaz]

looking for the best way to search through 30 days worth of logs files..

zgrep '192.1.2.' 20060802.access.log.web |grep 'deny' > /root/test3

this is what i have been using for one file but im looking for 30days worth of file and 3 different ip address ranges...

any ideas.

 

[geirendre]

First, I'm not a script expert
but maby a short script like this would work:

[highlight]
#!/bin/bash
cd /var/log/
for i in 192.*
do
cat $i | grep 'deny' >> /root/test4
done
[/highlight]

It searches trough all files starting with 192. for the word
deny, and puts all occurances in the file /root/test4.
Not sure if it's something like this You need?
It might need expansions and nested loop, depending on your logfiles.

 

[geirendre]

But if You do this kind of log-searches on a regular basis, I would advice you to set up a syslog-server.
I have just put up such a server with Debian, MySQL, syslog-ng, Apache2, PHP, PhpMyAdmin and Php-syslog-ng.
It gatters syslog-events from many hosts into one SQL-base, that I can search trough with a web-interface (Php-syslog-ng)
werry nice

You can see a demo here:

http://www.phpwizardry.com/php-syslog-ng.php

 

[lgarner]

I do this frequently, but not usually for a fixed date range, but geirendre has the right idea.

for a in 192.168.1.2 192.168.1.3 192.168.1.4;do grep $a 200606??.access.log.web > $a.txt; done

Substitute your addresses for the ones shown, and alter the filespec appropriately.

 

[SamBones]

The command [tt]egrep[/tt] can look for more than one pattern at the same time. Something like...

egrep "192.168.1.2|192.168.1.3|192.168.1.4" 200606??.access.log.web | fgrep deny > deny.log

 

[hyperdaz]

I missed one important point they are all in different folders

20060615 20060621 20060627 20060703 20060709 20060715 20060721 20060727 20060802 20060808 20060814
20060616 20060622 20060628 20060704 20060710 20060716 20060722 20060728 20060803 20060809
20060617 20060623 20060629 20060705 20060711 20060717 20060723 20060729 20060804 20060810
20060618 20060624 20060630 20060706 20060712 20060718 20060724 20060730 20060805 20060811
20060619 20060625 20060701 20060707 20060713 20060719 20060725 20060731 20060806 20060812
20060620 20060626 20060702 20060708 20060714 20060720 20060726 20060801 20060807 20060813

egrep string looks the most usefull.

the syslog idea is good but the problem is its not my network just have to play with what im given etc.

 

[hyperdaz]

egrep wont go into each folder will it??

 

[SamBones]

If it's just one level down, this would do it...

egrep "192.168.1.2|192.168.1.3|192.168.1.4" 200608??/* | fgrep deny > deny.log

If it's down a number of levels, some variation of this...

find 200608* -type f -exec egrep "192.168.1.2|192.168.1.3|192.168.1.4" {} \; -print

I just found in the man page that [tt]grep[/tt] can do the same with the "[tt]-E[/tt]" switch...

grep -E "192.168.1.2|192.168.1.3|192.168.1.4" 200608??/* | fgrep deny > deny.log

If it's just a single character changing, this could work too...

grep "192.168.1.[234]" 200608??/* | fgrep deny > deny.log

Try the [tt]man[/tt] pages for [tt]grep[/tt], [tt]egrep[/tt], [tt]fgrep[/tt], and [tt]find[/tt]. It's all in there.

 

[geirendre]

Ok, so a litle rewrite of my humble script might do:

#!/bin/bash
cd /var/log/
for i in 2006*
do
cat $i | grep 'deny' >> /root/$i.txt 2>/dev/null
done
#END

should parse trough each file saving lines with 'deny' to a file named after the folder and with a .txt extension.
e.g. 20060615.txt 20060621.txt 20060627.txt etc...

 

[SamBones]

Useless use of [tt]cat[/tt].

#!/bin/bash
cd /var/log/
for i in 2006*
 do
  grep 'deny' $i >> /root/$i.txt 2>/dev/null
 done
#END

 

[geirendre]

Told U I'm no script expert ;-)