help with a script which is monitoring appended data to a log file

[sparkle85]

Hello,

I need some help with the following script:

if ! [ -f ${PATH}/myfile.txt ];then
  echo $(date +%Y%m%d"_"%H%M%S)": Nu am gasit fisierul ${PATH}/myfile.txt"
  ps -fxu pin | grep "/usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log$" | awk '{system("kill "$2)}'
  cat ${LOG_PATH}/x.log | sed -n -e '/LONG/{x;1!p;g;;}' -e h > ${PATH}/myfile.txt
  
  /usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log | sed -n -e '/LONG/{x;1!p;g;;}' -e h >> ${PATH}/myfile.txt &
fi
date1=$(tail -1 ${PATH}/myfile.txt | cut -d" " -f2-5)
if [ -f exec_long.date ];then
  date2=$(cat exec_long.date)
else
  date2=$date1
  echo $date1 > exec_long.date
fi
if ! [ "$date1" = "$date2" ];then
        ${CAL_PATH}/${APP_NAME}
        sleep 1
        echo $date1 > exec_long.date
fi

I have a continuously growing log file (x.log) in which i have to look for certain lines that contain "Long". The line above each line containing the word "Long" it contains a time stamp. I want to extract each line containing the time stamp into myfile.txt and check the difference between time stamps. Whenever there is a difference i need to run another script (${CAL_PATH}/${APP_NAME}), then sleep 1, then continue searching.

Lines with "Long" do not appear continuously, but in blasts. The script runs fine until the first pause encountered. Starting with the first pause, tail -f doesn't write in myfile.txt anymore.

Can someone help me understand why "tail -f" it stops writing into myfile.txt? Or does someone know an alternative to "tail -f" of achieving the initial scope of the script?

I'll appreciate any help.
Thanks!

 

[dkyrtata]

I was trying to do the same thing last year (under HPUX-11). It would appear that "tail -f" does not work for this purpose. In the end I had to replace the "tail -f" with "tail -100". Of course it was not as easy as that. Firstly, I had to create a loop which would sleep for a several seconds between calls to "tail -100". Then I had to sift out the records that had already been examined from the previous iteration. That was not too difficult since the log file had a timestamp for every record. But as you can see, the solution was not as clean as "tail -f" would have been.

Sorry I could not be of more help, but I would be interested if anybody has any better solutions.

 

[feherke]

Hi

Can someone help me understand why "tail -f" it stops writing into myfile.txt?

I am sure [tt]tail[/tt] not stops writing. Probably the next command, to which [tt]tail[/tt]'s output is piped caches its output. With [tt]sed[/tt] and POSIX [tt]awk[/tt] there is nothing to do against that. In [tt]gawk[/tt] calling the [tt]fflush()[/tt] function will solve the problem. If not available, use [tt]perl[/tt] and set [tt]$|=1;[/tt] in it.


Feherke.

http://free.rootshell.be/~feherke/

 

[sparkle85]

Interesting... I am working on the same OS.
I don't even know if your script might help me, given the fact that the data appended to the log file comes in blasts which have variable number of lines, but still, can i ask u to post your sollution?... It may be inspiring .

Thanks for your reply!

 

[sparkle85]

I am sure tail not stops writing. Probably the next command, to which tail's output is piped caches its output. With sed and POSIX awk there is nothing to do against that. In gawk calling the fflush() function will solve the problem. If not available, use perl and set $|=1; in it.

Thanks for your advice... u had a point there.

As i don't have gawk installed on my system, and fflush is not defined in my awk installation... i read system function might be similar (if it is used in a proper way)

So, I tried changing the tail command line with:

/usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log | awk '/HPDCB_OP_ADVICE_OF_CHARGE/{system("print "x"");};{x=$0;}' >> ${PATH}/myfile.txt &

or

/usr/local/coreutils/bin/tail -f ${LOG_PATH}/x.log | awk '/HPDCB_OP_ADVICE_OF_CHARGE/{print x;};{x=$0;system("")}' >> ${PATH}/myfile.txt &

When trying them from outside the script they both work properly. When running the script i encounter another problem. After searching in the already existing log file for the pattern, awk prints out the correct lines in myfile.txt and then the script stopes running...!!!

At the begining awk wouldn't print continuously in myfile.txt but the script kept running. Now is different.

Hope u understood what i tried to explain...

I am not used with fflush and system functions. Is there a problem with the way i used system function? Is this causing my new problem?


Thanks in advance...

 

[feherke]

Hi

Hope u understood what i tried to explain...

Sorry, I am not able to follow what happens there, neither in your code, nor in the explanation. :-(


Feherke.

http://free.rootshell.be/~feherke/

 

[dkyrtata]

Here is the relevant part of the script I have which simulates a "tail -f"

The first 19 characters of my log file (SYS_LOG_FILE) contains a datetime stamp formatted as "YYYY-MM-DD_HH24:MI:SS". So a simple comparison of timestamps helps identify which records have already been examined.



last_rec=$(tail -1 $SYS_LOG_FILE)
tail_cmd='new_recs=$(tail -100 $SYS_LOG_FILE | egrep -i "$search_pattern")'

start_time=$(date +%Y-%m-%d_%H:%M:%S)

while [ "$last_rec" != "" ]; do
elapsed_seconds=$(date_interval "$start_time" "$(date +$UNIX_FN_DTTM_FMT)" "ISEC" "$UNIX_FN_DTTM_FMT2")
[ "$elapsed_seconds" -gt 1800 ] && break # Break out of loop if nothing found in log file after 30 minutes

eval "$tail_cmd" # Get last 100 records (or less) after filtering out unwanted ones

print_cnt=-1 # No new records displayed yet

tempchar=$(echo "\001") # Character to temporarily replace spaces in SYS_LOG_FILE
# records for for-loop to retrieve 1 line at a time

for new_rec in $(echo "$new_recs" | sed "s/ /$tempchar/g"); do
new_rec=$(echo "$new_rec" | sed "s/$tempchar/ /g") # restore original spaces

typeset -L19 new_rec_dttm="$new_rec" # Get datetime of log entry
typeset -L19 last_rec_dttm="$last_rec" # Get datetime of last log entry displayed

if [[ "$print_cnt" -ge 0 || "$new_rec_dttm" > "$last_rec_dttm" ]];then
echo "$new_rec" # Display next record
last_rec="$new_rec" # Save last record diplayed

[ "$print_cnt" -eq -1 ] && print_cnt=1 || let print_cnt=$print_cnt+1 # Track number of "next" records
# diplayed in this loop

[[ -f "$SW_JOBS_DIR/${jobname:-$cmdline_pattern}" && $(echo "$last_rec" | egrep " (EXIT_JOB|ABEND) ") != "" ]] && break 2
elif [ "$new_rec" = "$last_rec" ]; then # Record that was last displayed has been
print_cnt=0 # found. Therefore display subsequent records
fi
done # for new_rec in $(echo . . .)

[ "$print_cnt" -le 0 ] && sleep 7 # sleep if no new records were found
done # while