HELP - I need to write a script to read a log file

[morgaann]

I am new to scripting and I really could use some help. I have a log file that is constantly getting appended to every night and the output looks something like this;

05/29/03 01:54:01 nsrd: /dev/rmt/4ubn Eject operation in progress
05/29/03 01:54:10 nsrd: /dev/rmt/1ubn Mount operation in progress
05/29/03 01:54:22 nsrd: media event cleared: confirmed mount of NTBelz.033 on /dev/rmt/1ubn
05/29/03 01:54:28 nsrd: media notice: dlt7000 tape bak2.NTmfs3.065 on /dev/rmt/7ubn is full
05/29/03 01:54:29 nsrd: media notice: dlt7000 tape bak2.NTmfs3.065 used 50 GB of 35 GB capacity
05/29/03 01:54:45 nsrd: media info: loading volume bak2.NT.C.502 into /dev/rmt/4ubn
5/06/03 06:23:03 nsrd: media event cleared: confirmed mount of bak2.NTmfs4.010 on /dev/rmt/1ubn
05/06/03 06:30:08 nsrd: media notice: dlt7000 tape bak2.NTmfs3.C.062 on /dev/rmt/2ubn is full
05/06/03 06:30:08 nsrd: media notice: dlt7000 tape bak2.NTmfs3.C.062 used 49 GB of 35 GB capacity
05/06/03 06:30:24 nsrd: media info: verification of volume "bak2.NTmfs3.C.062", volid 2931725057 succeeded.
05/06/03 06:30:40 nsrd: write completion notice: Writing to volume bak2.NTmfs3.C.062 complete
05/06/03 23:58:12 nsrd: media notice: dlt7000 tape MSEXCH.376 on /dev/rmt/0ubn is full
05/06/03 23:58:12 nsrd: media notice: dlt7000 tape MSEXCH.376 used 41 GB of 35 GB capacity
05/06/03 23:59:13 nsrd: media info: verification of volume "MSEXCH.376", volid 2818370561 succeeded.
05/06/03 23:59:27 nsrd: write completion notice: Writing to volume MSEXCH.376 complete


I need the script to read this log file and find any line that contains "is full" and has today's date. Once I have those lines I need to get $8 (eg: bak2.NTmfs3.C.062, MSEXCH.376, bak2.NTmfs3.065) field and only save the field name that contains "NTmfs3" with no "C". So from this example I want the field "bak2.NTmfs2.065" to be saved into a filename for later use and I need this file appended with any other field it finds that meets the criteria.
Here's what I got so far. I figure I need to put it in a while loop because this log is constantly being written to and I need to catch the "is full" lines as they come in. The problem with what I've written is that when I get the value for $NAME(ie; NTmfs4) I want to print the full field (bak2.NTmfs4.060) and save it to a file. That is just not working out.



#!/bin/sh -x

LOGDIR=/nsr/logs/daemon.log
TODAY=`date +%m/%d/%y`
TAPEDIR=/var/tmp/extreme_cloning


RUN=true
while [ $RUN ]
do
Volume=`tail -1 $LOGDIR | grep "is full$" | awk '{print $8}'`
NAME=`tail -1 $LOGDIR |grep "is full$" | awk '{print $8}' | cut -d "." -f2 | awk '{print $1}'`
LINE=`tail -1 $LOGDIR | grep "is full$" | awk '{print $8}' | awk 'BEGIN {FS="."} {print NF}'`
if [ "$LINE" = "3" ] && [ "$NAME" = "NTmfs4" ]; then
#echo the Volume name containing the $NAME and append to $TAPEDIR/tapes
else
echo "Finish"
exit
fi
done


Any help is greatly appreciated.

 

[Ygor]

To read a file as it is being appended to you would use tail -f and just leave the script running in the background...

#!/usr/bin/ksh
tail -f $LOGFILE | while read Line
do
if [[ $Line = *'is full' ]]
then
Volume=$(print $Line|cut -d' ' -f8)
eval set $(print $Volume|tr . ' ')
if [[ "$3" = 'C' ]]
then
print $1.$2.$4
else
print $Volume
fi
fi
done >> $TAPEDIR/tapes

 

[morgaann]

Your example works great except for one part. I only need to capture Volume names that contain "NTmfs3" with no ".C" (eg:bak2.NTmfs3.065) and append those to the tapes file. Is there an easy way to do that??

Thanks.

 

[Ygor]

If I understand you correctly, then I think that you want....

#!/usr/bin/ksh
tail -f $LOGFILE | while read Line
do
if [[ $Line = *'is full' ]]
then
Volume=$(print $Line|cut -d' ' -f8)
eval set $(print $Volume|tr . ' ')
if [[ "$2" = 'NTmfs3' ]] && [[ "$3" != 'C' ]]
then
print $Volume >> $TAPEDIR/tapes
fi
fi
done

 

[morgaann]

From what I am able to see so far, it seems to work. My only other question I need to run this using the bourne shell, is there any difference?

Thanks again for your help

 

[PHV]

Try this:

#!/bin/sh
tail -f $LOGFILE | while read Line; do
  case "$Line" in "*is full*")
    Volume=`echo "$Line"|cut -d' ' -f8`
    eval set -- `echo "$Volume"|tr . ' '`
    [ "$2" = "NTmfs3" ] && [ "$3" != "3" ] &&
      echo "$Volume" >> $TAPEDIR/tapes
  esac
done

Hope This Help
PH.

 

[morgaann]

PH,
I tried your suggestion and it just goes into an endless loop. Could it be syntax related? I've never used the case command before so I wouldn't know where to begin debugging.

Thanks.

 

[PHV]

The endless loop is caused by the

 tail -f

command.

Hope This Help
PH.

 

[morgaann]

I thought that was the problem, so I changed the tail -f command to a cat of the logfile and I get the same problem.

Thanks.

 

[PHV]

Try with a

 set -x

command above the loop to see what is wrong.
Sorry for the typo:

[ "$2" = "NTmfs3" ] && [ "$3" != "C" ] &&
                                  ^

Hope This Help
PH.

 

[morgaann]

I ran the script with the -x and all I get is "read Line" scrolling across the screen until I ctrl C out of it.

Thanks.

 

[PHV]

can you post the entire script ?
BTW the LOGFILE variable contains the real file name ?

Hope This Help
PH.

 

[Ygor]

A version to grep for todays date only (no tail -f)...

#!/bin/sh
Date=`date "+^%m/%d/%y"`
grep $Date $LOGFILE | grep "is full$" | while read Line
do
Volume=`echo "$Line"|cut -d' ' -f8`
eval set -- `echo "$Volume"|tr . ' '`
if [ "$2" = "NTmfs3" ] && [ "$3" != "3" ]
then
echo "$Volume" >> $TAPEDIR/tapes
fi
done

 

[morgaann]

i am trying to get this script that Ygor posted above running from cron (it works great from command line) and it doesn't seem to be working. First I tried the script that contains the "tail -f" command and nothing happens. The process is running but no files are generated. I also tried removing the tail and using grep but that doesn't work properly either because it doesn't continuously run.

Any ideas would be appreciated.

Thanks.
AM

 

[PHV]

Can you post the script and the crontabs entry ?

Hope This Help
PH.

 

[morgaann]

#!/bin/sh

LOGFILE=/nsr/logs/daemon.log
TODAY=`date +%m/%d/%y`
TAPEDIR=/var/tmp/extreme_cloning
SCRIPTS=/var/tmp/Scripts
Date=`date "+^%m/%d/%y"`

#grep $TODAY $LOGFILE | grep "is full$" | while read Line
tail -f $LOGFILE | grep "is full$" | while read Line
#cat $LOGFILE | grep $TODAY | grep "is full$" | while read Line
do
Volume=`echo "$Line"|cut -d' ' -f8`
eval set -- `echo "$Volume"|tr . ' '`
if [ "$2" = "NTmfs3" ] || [ "$2" = "NTmfs4" ] && [ "$3" != "C" ]

fi
done



Here is also the cron entry,
0 20 * * * /var/tmp/Scripts/Fulltapechk.sh &

Thanks for your help.

 

[PHV]

> if [ "$2" = "NTmfs3" ] || [ "$2" = "NTmfs4" ] && [ "$3" != "C" ]
>
> fi
The missing

 then

sentence is a copy/paste mistake ?

Hope This Help
PH.

 

[morgaann]

Oops - sorry - here you go.
#!/bin/sh

LOGFILE=/nsr/logs/daemon.log
TODAY=`date +%m/%d/%y`
#TODAY=07/23/03
TAPEDIR=/var/tmp/extreme_cloning
SCRIPTS=/var/tmp/Scripts
Date=`date "+^%m/%d/%y"`

#grep $TODAY $LOGFILE | grep "is full$" | while read Line
tail -f $LOGFILE | grep "is full$" | while read Line
#cat $LOGFILE | grep $TODAY | grep "is full$" | while read Line
do
Volume=`echo "$Line"|cut -d' ' -f8`
eval set -- `echo "$Volume"|tr . ' '`
if [ "$2" = "NTmfs3" ] || [ "$2" = "NTmfs4" ] && [ "$3" != "C" ]
then
$SCRIPTS/ssidlist_unix $Volume

fi
done

The ssidlist_unix script takes the value of $Volume puts and extenstion on it and stores it in a directory (eg, $Volume = bak2.NTmfs3.000 turns into bak2.NTmfs3.000.cpy and is stored in /var/tmp/extreme_cloning/bak2.NTmfs3.000.cpy


Hope this is helpful

Thanks.

 

[Ygor]

It looks like you're still using the tail -f version instead of the version with grep $Date $LOGFILE

 

[morgaann]

I also tried the version with grep $Date $LOGFILE and put it in cron and it wouldn't work either.