C "tail" Functionality 4

[ae2k]

I'm attempting to create a C routine that mimics the behavior of "tail -f". The idea is to monitor a log file, and have a function called every time a new line is detected in the log (passing it the new line). This log file could possibly be written many times per second, so speed is a factor. Any help would be appreciated, thanks!

 

[guggach]

why rewrite 'tail' ?
use it, it's better as any prog you will write.

don't forget, RTFMP guggach

 

[ae2k]

As I said I'm creating a routine... I'm not intending to recreate tail per se.

The actual goal of the project is to multicast specific log messages to clients on the network. So far the line parsing and multicast server/client are complete and working. The only thing that remains to be done is detect when that file is written, and pass those lines to the existing architecture.

 

[Salem]

Use stat() to work out when the file has changed (in the previous second).
If it has, read it until you hit EOF again, and process each line as you want to.
Then sleep for a second.

Rinse and repeat as necessary.

--

 

[dchoulette]

I would use sleep, fread, feof and clearerr in a loop.

See thread205-859361

--------------------

Denis

 

[ae2k]

Thanks for the ideas, I'll try to see what I can come up with and get back to you how it went.

 

[ae2k]

I tried this, and it *almost* works:

char buf[301];

while(1) {

fread(buf, 301, 1, f_read);

if (feof(f_read)) {
clearerr(f_read);
} else {
printf("%s\n",buf);
}

sleep(1);
}

The output is sometimes a single line and other times split with a newline... something like:

Jul 4 00:01:10 xxx.xxx.xxx 816336: .Jul 4 00:01:08.558: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to up

Jul 4 00:01:10 xxx.xxx.xxx 816337: .Jul

4 00:01:09.262: %LINK-3-UPDOWN: Interface FastEthernet0/14, changed state to down

Can anyone give me a suggestion? Thanks!

 

[ae2k]

Nevermind I got it working using fgets(), to stop on the endlines and EOFs:

char buf[301];

// Start reading end of log file
fseek(f_read, 0, SEEK_END);

while(1) {

fgets(buf, 301, f_read);

if (feof(f_read)) {
clearerr(f_read);
} else {
printf(buf);
}

}

 

[dchoulette]

You forgot the sleep (only needed is feof is true not after every read).
You should print every time the fgets function read some data and not only if eof is false.
You could even make a loop on fgets.
You should check ferror to stop if a problem occured (I miss this one in my previous post).
The only way fgets can return NULL is if ferror or feof is set: if you checked for error with ferror, no more need to check for eof, just clearerr and sleep.

So I got the following code:

char buf[301];

while(1) {

        while (fgets(buf, 301, f_read)) {
             printf(buf);
        }

        if (ferror(f_read)) {
           perror("Error while reading file");
           exit(1);
        }

        /* here feof is true */
        clearerr(f_read);
        sleep(1);
}

--------------------

Denis

 

[TonyOne1]

Have a look at logtail in

http://ftp.cerias.purdue.edu/pub/tools/unix/logutils/logcheck/logcheck-1.1.tar.gz

 

[ae2k]

Your code works really well for a minute or two dchoulette, but always ends up terminating on a "Segmentation Fault". Any ideas what might be wrong?

 

[newmangj]

The problem with some of the previous code is that it doesn't cope when the log file is truncated (perhpas by being relegated). The code below is something I wrote to monitor a log file and report on any lines containing one of a number of search words.

You should be able to extract just the logfile monitoring part of the code.

Hope it helps.

/***********************************************************

PROGRAM : intelli_tail
LANGUAGE : C
AUTHOR : Gavin Newman
DATE : 25th October, 2002
PLATFORM : Solaris 2.8
COMPILER : gcc
RAISON D'ETRE

This program will perform a tail operation on the file specified on
command line and will only output lines containing the target words

The file position is stored in a file intelli_tail.filename.lpos
where filename is the file being monitored.

The seach words are stored in lines in the file
intelli_tail.filename.search, each set of words being checked

Example lines are as follows

ready
ready now

The first searches for lines with the word ready on it.
The second searches fir lines with bot words (in any order) in it.
Lines beginning with # are comments

These words are loaded into a memory based linked list for speed,
when the file is modified a SIGHUP sent to this application will
cause the in-memory list to be flushed and rebuilt.

I might improve the pattern matching later on.....

-------------------------------------------------------------------
MAINTENANCE HISTORY

*******************************************************************/

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>

#define BUFFLEN 2048
#define TRUE 1
#define FALSE 0

#define SLEEP 5

void show_usage(char * app);
long get_last_fpos(char *);
long get_file_length(char *);
void set_last_fpos(char *);
void sig_shutdown(int);
void sig_reload_patterns(int);
void load_patterns(void);
void unload_patterns(void);
void walk_patterns(void);
int pattern_match(char *);
int write_pid(void);

FILE * fptr_in;
char sz_target[2048];

/* This structure provides a linked list of pointers to
words in memory. Each line in the pattern file has a separate
linked list of these nodes, one per word, which runs from a
pointer in the node structure linked list. */
struct _word_node
{
char * ptr_word; /* Points to memory containing word */
struct _word_node * ptr_next; /* Points to next of these nodes */
};

/* This structure is used in a linked list, one per line in the
search pattern file, it is used to provide a root for each linked
list of word anchor nodes (see above) */
struct _node
{
struct _word_node * ptr_word_node; /* Pts to first in wordnode list */
struct _node * ptr_next; /* Points to next of these */
};

struct _node * ptr_first=NULL; /* Ptr to first node in list */

int main(int argc, char * argv[])
{
long last_pos;
long currpos;
long filelen;
char sz_buffer[BUFFLEN+1];
struct stat statbuff;

/* Check that we were supplied with input */
if(argc!=2)
show_usage(argv[0]);

if(write_pid()==FALSE)
return FALSE;
strcpy(sz_target,argv[1]);

load_patterns();

/* Set signal handling routines */
signal(SIGINT,sig_shutdown);
signal(SIGTERM,sig_shutdown);
signal(SIGHUP,sig_reload_patterns);

/* Get the last place we were up to */
last_pos=get_last_fpos(sz_target);
/* Loop forever (or until we get an external signal) */
while(1)
{
/* Get length of file now */
filelen=get_file_length(sz_target);

/* The file is now shorter than our last read position
restart at the beginning of the file */
if(filelen<last_pos)
{
printf("**** FILE HAS BEEN TRUNCATED. RESTART AT BEGINNING ****\n");
last_pos=0;
}
/* Check for file growth or truncation */
if(filelen==last_pos)
{
/* No addition to the file so sleep for a while */
sleep(SLEEP);
continue;
}

fptr_in=fopen(sz_target,"r");
/* Set file position to last position */
fseek(fptr_in,last_pos,SEEK_SET);
/* Read to EOF */
while(fgets(sz_buffer,BUFFLEN,fptr_in)!=NULL)
{
/* Check for match in search patterns */
if(pattern_match(sz_buffer)==TRUE)
printf("%s",sz_buffer);
}
/* Update file position marker then sleep */
last_pos=ftell(fptr_in);
fclose(fptr_in);
sleep(SLEEP);
}
return 0;
}

/* Just print out information re this program then exit */
void show_usage(char * sz_app)
{
printf("==================================================\n");
printf("USAGE : %s target_file_path\n\n",sz_app);
printf("The program will then extract the latest records\n");
printf("from the target file and output them. In the event\n");
printf("the target file is truncated then all records from\n");
printf("the start of the file will be output\n");
printf("==================================================\n");
exit(1);
}

/* Return the contents of the file intelli_tail.filename.lpos */
/* Return 0 if this file is not found */
long get_last_fpos(char * sz_target)
{
char * ptr_target;
char * ptr;
int hfile;
static long offset;
char sz_lastfile[2048];

/* Build the file name used to hold the last position data */
/* The file is called intelli_tail.filename.lpos */
ptr_target=(char *)malloc(strlen(sz_target)+1);
strcpy(ptr_target,sz_target);
if((ptr=strrchr(ptr_target,'/'))!=NULL)
*(ptr++)=0x00;
else
ptr=ptr_target;
sprintf(sz_lastfile,"intelli_tail.%s.lpos",ptr);
free(ptr_target);

if((hfile=open(sz_lastfile,O_RDONLY))==0)
{
hfile=open(sz_lastfile,O_WRONLY|O_CREAT|O_TRUNC,0666);
offset=0;
}
else
read(hfile,&offset,sizeof(long));

close(hfile);
return offset;
}

long get_file_length(char * sz_target)
{
long currpos;
long endpos;
struct stat statbuff;

if(stat(sz_target,&statbuff)==-1)
return 0;
else
return statbuff.st_size;
}

/* Write the current file position to our lpos file so we
know where we are up to next time we run. */
void set_last_fpos(char * sz_target)
{
char * ptr_target;
char * ptr;
int hfile;
long offset;
char sz_lastfile[2048];

/* Build the file name used to hold the last position data */
/* The file is called intelli_tail.filename.lpos */
ptr_target=(char *)malloc(strlen(sz_target)+1);
strcpy(ptr_target,sz_target);
if((ptr=strrchr(ptr_target,'/'))!=NULL)
*(ptr++)=0x00;
else
ptr=ptr_target;
sprintf(sz_lastfile,"intelli_tail.%s.lpos",ptr);
free(ptr_target);

hfile=open(sz_lastfile,O_WRONLY|O_CREAT|O_TRUNC,0666);

offset=get_file_length(sz_target);

write(hfile,&offset,sizeof(long));
close(hfile);
return;
}

/* This function is called as a result of a signal, it allows
the program to close gracefully. */
void sig_shutdown(int signum)
{
printf("Exiting on signal %d \n",signum);
unload_patterns();
set_last_fpos(sz_target);
fclose(fptr_in);
unlink("intelli_tail.pid");
exit(0);
}

/* Reload the match word list in memory */
void sig_reload_patterns(int signum)
{
printf("Configuration reload requested - signal %d\n",signum);
unload_patterns();
load_patterns();
signal(SIGHUP,sig_reload_patterns);
}

/* Load contents of search pattern file into memory based linked
lists */
void load_patterns()
{
char * ptr_target;
char * ptr;
char sz_patterns[2048];
char sz_buffer[2048];
FILE * fptr;
char sz_delimiter[]=" ";
struct _node * ptr_current_node;
struct _word_node * ptr_current_word_node;
char * ptr_word;

/* Build the file name
The file is called intelli_tail.filename.search */
ptr_target=(char *)malloc(strlen(sz_target)+1);
strcpy(ptr_target,sz_target);
if((ptr=strrchr(ptr_target,'/'))!=NULL)
*(ptr++)=0x00;
else
ptr=ptr_target;
sprintf(sz_patterns,"intelli_tail.%s.search",ptr);
free(ptr_target);
if((fptr=fopen(sz_patterns,"r"))==NULL)
{
printf("Could not open %s - patterns not loaded\n",sz_patterns);
exit(1);
}
while(fgets(sz_buffer,2047,fptr)!=NULL)
{
/* Ignore comments */
if(sz_buffer[0]=='#')
continue;
/* Strip trailing newline */
if((ptr=strchr(sz_buffer,'\n'))!=NULL)
*ptr=0x00;
/* Check for runt line */
if(strlen(sz_buffer)==0)
break;

/* Allocate NODE structure */
ptr_current_node=malloc(sizeof(struct _node));
ptr_current_node->ptr_next=ptr_first;
ptr_first=ptr_current_node;
ptr_current_node->ptr_word_node=NULL;

ptr=strtok(sz_buffer,sz_delimiter);
while(ptr!=NULL)
{
ptr_current_word_node=malloc(sizeof(struct _word_node));
ptr_current_word_node->ptr_next=ptr_current_node->ptr_word_node;
ptr_current_node->ptr_word_node=ptr_current_word_node;
ptr_current_word_node->ptr_word=malloc(strlen(ptr)+1);
strcpy(ptr_current_word_node->ptr_word,ptr);

ptr=strtok(NULL,sz_delimiter);
}
}
fclose(fptr);
}

/* Unload linked lists from memory */
void unload_patterns()
{
struct _node * ptr_temp_node;
struct _word_node * ptr_temp_word_node;
struct _node * ptr_current_node;
struct _word_node * ptr_current_word_node;

if(ptr_first==NULL)
return;

/* Walk primary nodes */
ptr_current_node=ptr_first;
while(ptr_current_node!=NULL)
{
/* Walk word nodes */
ptr_current_word_node=ptr_current_node->ptr_word_node;
while(ptr_current_word_node!=NULL)
{
free(ptr_current_word_node->ptr_word);
ptr_temp_word_node=ptr_current_word_node->ptr_next;
free(ptr_current_word_node);
ptr_current_word_node=ptr_temp_word_node;
}
ptr_temp_node=ptr_current_node->ptr_next;
free(ptr_current_node);
ptr_current_node=ptr_temp_node;
}
ptr_first=NULL;
}

/* Not used at present */
void walk_patterns()
{
struct _node * ptr_current_node;
struct _word_node * ptr_current_word_node;

if(ptr_first==NULL)
return;

/* Walk primary nodes */
ptr_current_node=ptr_first;
while(ptr_current_node!=NULL)
{
printf("At node %X\n",ptr_current_node);
printf(" Word node = %X\n",ptr_current_node->ptr_word_node);
printf(" Next node = %X\n",ptr_current_node->ptr_next);

/* Walk word nodes */
ptr_current_word_node=ptr_current_node->ptr_word_node;
while(ptr_current_word_node!=NULL)
{
printf("At word node %X\n",ptr_current_word_node);
printf(" Word = %X",ptr_current_word_node->ptr_word);
printf(" %s\n",ptr_current_word_node->ptr_word);
printf(" Next = %X\n",ptr_current_word_node->ptr_next);
ptr_current_word_node=ptr_current_word_node->ptr_next;
}

ptr_current_node=ptr_current_node->ptr_next;
}
}

/* Look for matches between this string and the contents of
the pattern tree. Returns TRUE is matched, FALSE if not */
int pattern_match(char * sz_text)
{
struct _node * ptr_current_node;
struct _word_node * ptr_current_word_node;
int matched=TRUE;

if(ptr_first==NULL)
return TRUE;

/* Walk primary nodes */
ptr_current_node=ptr_first;
while(ptr_current_node!=NULL)
{
/* For each major node we reset this to TRUE
We then go through each word for this node, if
we have a no-match we set it to FALSE. At the
end of each major node if the field is set to
TRUE then all words were matched! */
matched=TRUE;

/* Walk word nodes */
ptr_current_word_node=ptr_current_node->ptr_word_node;
while(ptr_current_word_node!=NULL)
{
if(strstr(sz_text,ptr_current_word_node->ptr_word)==NULL)
matched=FALSE;
ptr_current_word_node=ptr_current_word_node->ptr_next;
}
if(matched==TRUE)
return TRUE;
ptr_current_node=ptr_current_node->ptr_next;
}
return FALSE;
}

int write_pid()
{
FILE * fptr;
char sz_pid[10];

if((fptr=fopen("intelli_tail.pid","r"))!=NULL)
{
fgets(sz_pid,9,fptr);
printf("intelli_file is already running or has an\n");
printf("orphan intelli_file.pid file left over\n");
printf("Refer to PID=%s\n",sz_pid);
fclose(fptr);
return FALSE;
}

if((fptr=fopen("intelli_tail.pid","w"))==NULL)
return FALSE;
fprintf(fptr,"%d",getpid());
fclose(fptr);
return TRUE;
}

 

[Salem]

[tt][ignore]

[/ignore][/tt] tags please when posting code.

--

 

[ae2k]

Wow excellent code newmangj... I can't believe how much less overhead your method has than what I was using previously. It still consumes slightly more resources than tail, but is incredibly more efficient than the err/clearerr method. Not only that, but detection of truncation was something I hadn't even thought of. Even your search function is better than the one I was using, so I might just end up using your code right out of the box Don't worry, credit will be given where credit is due

My only gripe would be that the first run starts from the beginning of the file and quickly parses all of the contents before resuming the "tail" functionality. Once it has the position it resumes from the one stored in the file... but what if you have it turned off for a while and then turn it back on? Then you have to wait for all of the entries while it was off to process.

I'm not worried about changing this, it shouldn't be difficult. I was just wondering why you chose this implementation. Thanks again!

 

[newmangj]

ae2k

Thanks for the feedback - feel free to use the code as you need. I wrote the code such as you describe (ie that it runs from the start of the file in the absence of the position file) because that suited my needs at the time.

I needed to make sure that I did not miss any records in the file that might arrive whilst the program is not running.

If this is a problem for you then you or is not needed then you would not need to implement the position recording file, you could just start reading the file from the current position (use ftell and fseek to find the end of the file) and store the position in memory.

Cheers - Gavin