Unix Script for realtime log monitoring and performing tasks

[sur2rock]

Hi Guys,

Need help on how can i write a unix script which does below

1. Realtime monitoring of huge realtime updating log
2. get a sepecific keyword on log - Do below

save a particular string it will be a number in that log line eg. new mas:400 recieved : we keep 400​

search the next line of logs which will have mas:400 and look for more keywords​

search that keyword in a database and forget 400 if the keyword is not in DB​

finally we get mas:400 stored:#1234​

we forget 400 now and proceed with next​

3. the log will keep getting new numbers and we need to make sure we check all of them and notify if there is a error

I am thinking of reading logs , making a qeueu and holding them until its done

Please share your ideas on how we can best achive the above task

 

[ChrisHirst]

http://www.adp-gmbh.ch/perl/scripts/webalizer.html

Chris.

Indifference will be the downfall of mankind, but who cares?
Time flies like an arrow, however, fruit flies like a banana.
Webmaster Forum

 

[SamBones]

Look at Logstash for pulling from a big log. Or even multiple big logs. Send it to something like Elasticsearch to be able to access and analyse it. Maybe add something like RabbitMQ in there to queue the log data if it's coming in fast or from multiple servers. Also look at Splunk maybe. There are many other tools out there besides these.

There are a lot of open source log management and analysis tools out there. Some can handle insane amounts of log data.

And if you have a budget, there are commercial tools too.

 

[sur2rock]

thanks for the response guys - i was thinking more of writing on our own

 

[Annihilannic]

I wrote a similar script in Perl to filter HP-UX syslogs. Basically it does something similar to a 'tail -f' on the log file, matches substrings and invokes various commands or logic based on what is matched.

It's too specific to our requirements to share here, but yours sounds equally specific so you would need to implement it from scratch really. But if you search for 'perl' and 'tail' you'll find some examples for how to do that and build it up from there.

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]