mattquantic
Programmer
Hi is it possible to have a unix server automatically import the web log files into a database?
M@)
M@)
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations strongm on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
Automatically import the web logs into the database
- Thread starter mattquantic
- Start date
- Status
fredericofonseca
IS-IT--Management
Yes.
But it is not the fact that is Unix that matters.
You need to have a application doing that, either the one that creates the log, or another one that reads the log and inserts into the DB.
Firstly I would search the documentation supplied with your Web Server software, and see if it has the capability of logging automatically to a DB.
Second I would search the web ( is a good place) for anything related to your particular web server software and log files. It is possible that there is a piece of software already doing that.
Regards
Frederico Fonseca
SysSoft Integrated Ltd
But it is not the fact that is Unix that matters.
You need to have a application doing that, either the one that creates the log, or another one that reads the log and inserts into the DB.
Firstly I would search the documentation supplied with your Web Server software, and see if it has the capability of logging automatically to a DB.
Second I would search the web ( is a good place) for anything related to your particular web server software and log files. It is possible that there is a piece of software already doing that.
Regards
Frederico Fonseca
SysSoft Integrated Ltd
If you are using apache, the httpd.conf file will tell you the format of the log file.
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here. Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
# CustomLog /var/log/httpd/access_log common
CustomLog logs/access_log combined
You can use awk to parse this and create sql insert statements.
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here. Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
# CustomLog /var/log/httpd/access_log common
CustomLog logs/access_log combined
You can use awk to parse this and create sql insert statements.
ericbrunson
Technical User
We still don't know if you're using apache, if you are...
Have you looked at mod_log_sql?
Either that or;
CustomLog "|/some/filter/script"
will use your script as a processing filter. I could write a python script that would insert your logs into mysql in, I'm going to say... about 12 lines of code.
Have you looked at mod_log_sql?
Either that or;
CustomLog "|/some/filter/script"
will use your script as a processing filter. I could write a python script that would insert your logs into mysql in, I'm going to say... about 12 lines of code.
- Status
Similar threads