Help with script logic for real time log monitoring 2

[AnotherAlan]

Hi All,

I have to write something that will monitor a logfile in real time, check for network latency and email if these conditions are met. I have a working script that performs the conditional element (thanks to Feherke) but I am now struggling with the logic of how to ensure that only new events are raised and how to do the real time monitoring. I'm not a developer, only a frustrated admin who likes to write scripts, so apologies for my ignorance.

My problem:
My current script parses the entire logfile, so will raise duplicates.
It runs on an ad-hoc basis i.e. only when called by command line or cron, but I would like it to be monitoring the logs at all times.

My though patterns so far:
Use tail -f, I tried piping this into an awk statement but it didn't work...probably my fault.
Use a while loop, but hard to figure out a condition of when to break from the script and restart without missing some lines in the logs. i.e Tail -f piped to awk then into a log, watch for the log to be non-zero bytes, extract data and email.
Use fgrep and a "check" log to remove lines already parsed. I've used this before and it works, my worry is the size of these logs and the system overhead if the script is running continuously.

I've spent 48 hours toiling with this and am now going around in circles. Any pointers to put me back on track would be very much appreciated.

Thanks
Alan

 

[feherke]

Hi

As I understood, you want the script to
[ul]
[li]read from a data file the timestamp of the last seen log line[/li]
[li]search for that line in the log file[/li]
[li]process the lines following the given line[/li]
[li]wait for new lines and process them immediately as they appear in the log[/li]
[li]when interrupted, write to a data file the timestamp from the last log line before exiting[/li]
[/ul]
Is that correct ?

Feherke.

http://free.rootshell.be/~feherke/

 

[AnotherAlan]

Hi Feherke,

That is correct. The awk statement you helped me with yesterday I want to use to monitor a network log, this will alert me immediately there are any latency issues. So the script needs to be running continuously and looking at each line of the logfile as it comes in. I wrote a script that works for ad-hoc queries (but on the entire log), but would really like to have this as real time as possible.
Problem is how do I do this.

Thanks Feherke, your help is appreciated.

Regards
Alan

 

[PHV]

You may use something like [tt]system(sendmail ...)[/tt] in your awk program.

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886

 

[feherke]

Hi

Consider this a pseudo code, as I tested it only partially :

[navy]last[/navy][teal]=[/teal][green][i]"$(< "[/i][/green][navy]$lastfile[/navy][green][i]" )"[/i][/green]

[teal]([/teal]
  sed -n [green][i]"/^$last/,[/i][/green][lime][i]\$[/i][/lime][green][i]p"[/i][/green] [green][i]"$logfile"[/i][/green]
  tail -f -n [purple]0[/purple] [green][i]"$logfile"[/i][/green]
[teal])[/teal] [teal]|[/teal] [b]while[/b] [COLOR=chocolate]read[/color] str[teal];[/teal] [b]do[/b]
  echo [green][i]"$str"[/i][/green] [teal]|[/teal] cut -d [green][i]' '[/i][/green] -f [purple]1[/purple]-[purple]2[/purple] [teal]>[/teal] [green][i]"$lastfile"[/i][/green]

  [gray]# just a theoretic processing based on what I understood[/gray]
  [navy]took[/navy][teal]=[/teal][green][i]"$( echo "[/i][/green][navy]$str[/navy][green][i]" | sed -n 's/.* took [/i][/green][lime][i]\(\S\+\)[/i][/lime][green][i].*/[/i][/green][lime][i]\1[/i][/lime][green][i]/p' )"[/i][/green]
  [teal](([/teal] took [teal]>[/teal] [purple]1000[/purple] [teal]))[/teal] [teal]&&[/teal] echo [green][i]"took too much : $took"[/i][/green]
  [gray]#[/gray]
[b]done[/b]

Feherke.

http://free.rootshell.be/~feherke/

 

[AnotherAlan]

Hi,

Thanks Feherke, this looks really good, I can't test it yet but will be able to tomorrow.

Much appreciate all the help.

PHV, thanks, that is a good idea and will save on another mail process being used.

Regards
Alan

 

[AnotherAlan]

Feherke, this works like a dream, thanks again for your much appreciated brainpower.

Regards
Alan