Webalizer and logging

[stevenriz]

I am currently diagnosing a logging problem on a webserver hosting many vhosts. For some reason all logging stopped on a certain day and I am trying to figure out why. Not knowing about Webalizer at all, I am going at it step by step.

Now I am able to run webalizer on one domain and have it read the hist by shutting off incremental and rebuilding the db (I assume). Now there is no new data in there. So I am "guessing" there is some other program that logs the data other then webalizer that I need to look at. Is it simply an Apache thing that logs access to each vhost? Where would I look for the program or programs that populate the access.log files? Thanks!!

 

[lgarner]

Webalizer doesn't log any data at all, it just analyzes and presents it. Apache does its own logging as configured in httpd.conf.

 

[smah]

The 'traditional' way Webalizer & Apache are set up with multiple virtual hosts is that each virtual host creates it's own access log. Webalizer is then set to run as a cron job. Are you sure that Apache's logging stopped; or did webalizer just stop analyzing it?

 

[stevenriz]

I am looking into that very question right now, it "looks" like the apache logs have stopped if that is possible. I did run webalizer for one particular domain and all it's hits ended back in November so that is why I believe the Apache logging has stopped....

 

[Kiehlster]

Make sure you have most of these lines in your apache config. If you don't have virtual hosts, then don't worry about the last bit. However, make sure you have everything else.

LoadModule log_config_module modules/mod_log_config.so

LogLevel warn

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%u - %t - %{Referer}i -> %U" referer

CustomLog logs/access_log combined
ErrorLog logs/error_log

# Log virtual hosts individually
<VirtualHost *:80>
	ServerName [URL unfurl="true"]www.example.com[/URL]
	CustomLog logs/example.com_access_log combined
	ErrorLog logs/example.com_error_log
</VirtualHost>

Also, make sure your webalizer config is set up properly for either general or virtual host-specific logs.

# for general access

LogFile		/path/to/apache/logs/access_log
OutputDir	/path/to/webalizer/webpage/output
HistoryName	/path/to/webalizer/history.hist
Incremental	yes
IncrementalName	/path/to/webalaizer/apache.current

# for specific domains

LogFile		/path/to/apache/logs/example.com_access_log
OutputDir	/path/to/example.com/webalizer/output
HistoryName	/path/to/webalizer/example.com.hist
Incremental	yes
IncrementalName	/path/to/webalaizer/example.com.current

You'll have to change the paths according to where apache stores its logs and according to where you want webalizer to store history info and such. And if you're running on windows, you'll have to specify the drive letter. I can't really help you further without knowing how your have apache configured, how you have webalizer configured, and what options you're using when executing webalizer.

Steve Kiehl
Web Page Designer - Nanovox Productions
Fantasy Artist - Zeadi

 

[stevenriz]

gotcha, thanks Steve. I will let you all know what I find.
steve

 

[roycrom]

I had a problem when i had a lot of virtual hosts all writing to there own logfile.

I actually run out of file descriptors or inodes.

I can't remember exactly what this caused apache to do but it could explain why your setup stopped logging.

------------------------------------------
Somethings come from nothing, nothing seems to come from somethings - SFA - Guerilla

roycrom

 

[stevenriz]

wow, how many vhosts did you have? I would think you would only need one httpd.conf file per host.

Anyway, I am noticing that there are no httpd.conf files in the httpd/conf directories in each vhost. Well, there are however, samplehttpd.conf files. I am wondering if I just copy them to httpd.con and edit them as necessary, if it will work.... I will try it and report back. Weigh in if you have any other suggestions. Thanks!!
Steve

 

[stevenriz]

I am now noticing that in /etc/httpd there is conf and conf.d.

/etc/httpd/conf has httpd.conf where all the VirtualHost stuff at the bottom of the file is commented out. There are over 100 virtual hosts currently running on this box!! How is that possible?!

/etc/httpd/conf.d has a vhosts.conf file where the VirtualHost stuff is commented out there too!!

I am trying to piece it all together, I am just afraid of taking down the 100+ hosts that are on this machine.

Any hints? Thanks all!
Steve

 

[stevenriz]

I also failed to mention that there is a /home/vhost1 /home/vhost2 etc for each of the 100+ virtual hosts. Under each of these directories are these directories ** backup httpd mail stats

- under httpd is a conf and a httpdocs directory
- under conf is a httpd.conf file
- stats is where the usage logging data used to be logged.... it has all stopped on Nov 1 2004 for all vhosts so all that data is stagnant.

Let me know what else you think I should try!

 

[stevenriz]

I think I may have found the problem. We might be setup for mass virtual hosting. First of all, I don't see the virtual domains configured in the main /etc/httpd/conf/httpd.conf file, second of all I found this in the Apache manual....
===========
"The basic idea or mass virtual hosting is to replace all of the static <VirtualHost> configuration with a mechanism that works it out dynamically. This has a number of advantages:" Your configuration file is smaller so Apache starts faster and uses less memory. Adding virtual hosts is simply a matter of creating the appropriate directories in the filesystem and entries in the DNS - you don't need to reconfigure or restart Apache.

I find this to be a significant and crucial statment.....

The main disadvantage is that you cannot have a different log file for each virtual host; however if you have very many virtual hosts then doing this is dubious anyway because it eats file descriptors. It is better to log to a pipe or a fifo and arrange for the process at the other end to distribute the logs to the customers (it can also accumulate statistics, etc.).

thoughts?

 

[Kiehlster]

mass virtual hosting is rather handy, but you end up with like one log file for all the virtual hosts that conform to the mass virtual hosting rules.

I actually have a combination of both mass and individual virtual hosts at the moment. I use my massive one for domains that don't need special options, and I have a httpd.conf file for each of my other virtual hosts that need specific directives such as rewrite rules and such. You could cut down on descriptors by logging stuff to a pipe and having the pipe log to a sql database. That's one thing I've always wanted to do, but am too lazy to deal with it. And I have plenty of descriptors anyway. I just let all the logs rotate to log.4 and then once in a while I move all the .4's to another folder on a different drive and tar them up to save space. You could also just have the .4's delete after a little while since webalizer takes care of history without having to keep around any old log files.

Anyway, are you having trouble with log files that seem to be stopping after a period of time? or are you having trouble with something else? Also how big are your log files when they stop logging?

Steve Kiehl
Web Page Designer - Nanovox Productions
Fantasy Artist - Zeadi

 

[stevenriz]

Thanks Steve. What I think happened is as the number of virtual hosts grew, the previous web team converted the separate log files to the mass virtual hosts config and removed all of them from the main httpd.conf file. I found the mail log file and it seems to be working, I just need to figure out how to get webalizer to see it. Any help there would be appreciated.

Now, is it safe to assume that if I add some virtual hosts to the main httpd.conf file, those will start logging separately?? If this is the case, will we need to restart Apache?

Thanks for all your help!!
Steve

 

[Kiehlster]

Yes, if you add new virtual hosts, they will log separately, and yes you'll have to restart apache, but it will only be off for less than a minute anyway. Best restarting it overnight or early morning when your traffic is low.

As far as telling webalizer where your log is, that should be changable in your webalizer.conf files by setting the value for 'LogFile' to the full path of your log file. Then you just run webalizer every few hours.

Steve Kiehl
Web Page Designer - Nanovox Productions
Fantasy Artist - Zeadi