Writing to a file that is in use by a process 1

[toddyl]

Hi,

I have a third party process running on a unix box that writes to a log file. The log file is automatically updated every 30 seconds with various stats etc. This file is monitored by another application to see if various errors occur and raised alerts if any are found.

I want to test this to ensure the appropriate alerts are generated but am having an issue. An alert should be generated if a line contains the word "Error". So what I am doing is: echo "Error: This is a test" >> process.log. I can then do a grep -i Error process.log and see my error. However no alert is generated. Then when the automatic update is due to happen if I run the grep -i Error process.log command again I can no longer see the message I echoed into the file.

I have done a tail -f of the process.log also and can see my Error message after I echo it in, however, when the automatic update occurs it appears that my message gets overwritten but the lines above where it is remain the same.

I have tried the same using cat to redirect the message from file into the log but with the same results.

Why is this happening? Does anyone know how I can force this message to stay in the log file?

Thanks,

Toddyl

 

[elgrandeperro]

Is it using syslog or its own logger?

From the way it works, it probably is its own custom logger. The logger has the file open, because if it simply appended and closed the fileyour stuff would be there. Instead, it has a file pointer to where it last wrote. So when you add some bits, it simply overwrites them from where it last wrote.
If there was a way to tell the logger to reopen the log file after your write, that would probably solve the issue.

 

[toddyl]

Its using its own logger. Unfortunately I have no idea how this works or how to adjust the logger. From what you are saying it looks as if I cannot simply insert my error message into the log but will instead have to find a way or emulating and error so that one appears in the log.

Thanks.

 

[motoslide]

Disclaimer: This might NOT work!

Since the log file in question is currently in use by the application, you might be able to move it to a new name, and have your monitoring application read the old file with your "Error" test line.

Like so:

# mv process.log process.log.real
# cp process.log.real process.log
# echo "Error: This is a test" >> process.log

This could be problematic and might cause the sending application to error. Before you do the first command, see which process has the file open using the fuser command. Then, after the move, make sure the same process shows to have the "process.log.real" file open.

Also, after you've established that your test works, make sure you put the original file back into place.



"Proof that there is intelligent life in Oregon. Well, Life anyway.

 

[elgrandeperro]

If the file is truly open and it doesn't do a close/reopen, the logger would still log to process.log.real even though the name has changed. You would need to stop/start the logging processes, and if you could figure that out you wouldn't need the mv/cp anyway, just stop, update, start.

 

[victorv]

hi,
think to be inside process that writes the log;

if the program would sound as:
while
begin
open file for append
write log
close
sleep 30
end

probaly you will see your lines. but if it does an open
concurrent at beginning of program, and using service as
ftell and fseek keeps in mind the position of its pointer
at the previous write, it writes current row, exacltly at the position where you have written your row.
To test this, try to write a very long string/rows, and see
if it merges with the program written.

However, you cannot modify the strategy of the program of the process.

If in the log file there is a sort of timestamp, you can write, parallely, your own, with the same timestamp format,
and periodically merge and sort them in a 3rd file.

If there is not time stamp, I see a more complex solution:
you can count (at the moment your program run, choosing a sleeping phase of the process)
the number of lines or the size of the primary file and prepost in your file this information.
The merging program, when process both files, scans 2nd file, reads the counter, reads from the 1st file the numbers of lines of chars, write them on the 3rd file,
writes your line, and reads anothe line from 2nd file.

A bit complicated, but this is important for you, you have found as pass your weekend !

ciao
vittorio

 

[SamBones]

The program writing the log has probably opened this file for writing and is keeping it open, not opening and closing when it needs to write. This means it has it's own pointer into the file for where it writes next (structure resulting from a successful [tt]fopen()[/tt]). That means when you append to the file using ">>", you use the end of file pointer from the file's inode. That doesn't change the program's end of file pointer, so it's next write will overwrite what you appended.

You goal is to test your monitoring process. You can test that part without actually looking for the string "Error". Find another "word" that occurs just occasionally and test your monitoring for that. That will let you work out your monitoring code. Then, when you are satisfied it works, change it to look for "Error". To confirm it works, either wait for a real error to occur, or see if there's a way you can induce an error that will be logged.

At least that's how I would approach it.

 

[toddyl]

I have no access or insight into the logging tool so am unable to change that. The only option available to me is to induce/replicate the error so that the Error log eventually contains the message and my monitoring script picks this up.

Thanks for the feedback -- its great to get an insight into how the logging works.