Formating log file processing the file line by line.

[elgo]

I have a web server log file like this.

10.144.6.146 - - [20/Oct/2004:00:00:17 - 0700] "GET /images/grey_dot.gif HTTP/1.1" 304 - 304 - - - 451 120 572 1 20 0

I want to assign each field in to a variable and format the records and prepare a report in readable format. The log file has thousands of lines. Some of the fileds I want to do some calculations

I know there are plenty of web analysis tool available, but I wan to prepare a custom report. The script should process one line at a time.

 

[PHV]

man awk

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ222-2244

 

[elgo]

man awk does not say anything about how to assign each value in to a variable.

 

[elgo]

man awk does not say anything about how to assign the values to variables.

 

[PHV]

awk is well suited to generate the report itself.

 

[mrn]

You could do

col1`=`cat filename | awk '{print $1}'`
col2=`=`cat filename | awk '{print $2}'`

or just do it all on one line

cat filename | awk '{print $1, $3, $2, $7}'




Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant.

 

[vgersh99]

UUOC!

awk '{print $1, $3, $2, $7}' filename

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[elgo]

mrn,

Your first solution would print all the records in the specified field. I want to assign the value one record at a time and assign the value to a variable and then go to the next line. Basically it shoudl process line by line.

vgersh99,

I want to assign the valu to a variable. Your solution will print all the records from teh specified field. I have already tried these two soutions.

 

[vgersh99]

you can either do it in:

ksh

while read f1 f2 f3 trash
do
    echo "f1->[${f1}] f2->[${f2}] f3->[${f3}]
    ....... the rest of your formating/reporting
done < myLogFile.log

awk

{
   ... work with the already addressable fields: $1, $2..$NF
}

What reporting requirements do you have based on your sample input? Provide a sample [AND simple] report.

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[elgo]

Ok, Here is another example: I have the following in one line. All the line will have the same format of records.

machinename:username:028579:companyname:0:1098066159:1098079781:1098079936:0:0:155:152:2:0.00000
0:0:0:0:0:143725:19231:0:0.000000:0:0:0:0:0:0:defaultdepartment:none:1:0:154.000000:28.893298:0.000000:
arch=glinux,cputype=ia32,normal:0.000000:none:415993856.000000

I want to take each field seperately and scramble them and format them in a most appropriate and required format. say

field3,field1,field2,field6,field7,field8,filed32,field40 and so on. Some of the fields like field6,7,8 is epoch time, I need to convert to readable format. I want to calculate the difference between field 6,7 and 8, do some calculation and put the result in another column. I do not want to just get all the records in one field and put in one column, which is the standard behavior of awk or any other utility.

I guess I have explained the problem better.

 

[vgersh99]

'awk' is your friend for this.
as a starter to do "field3,field1,field2,field6,field7,field8,filed32,field40"

nawk -f elgo.awk myLogFile.log

here's elgo.awk

BEGIN {
  FS=":"
  OFS=","
}
{
   print $3, $1, $2, $6,$7, $8, $32, $40
}

The rest of the requirements are left as an exercise for the poster - do 'man nawk' first.

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[elgo]

vgersh99,

Your solution is ok to just print the fields. But how to assign each field value to a variable. say get the field1 value of line one and apply to A, and field 2 value of line one to B and so on. I want to process one line at a time. Your solution will just print all the specified field. I am trying to do this within a for loop.

 

[vgersh99]

no 'looping' is required - awk will read ONE line at time. The fields will be split based on the value of FS.

As I demonstrated my sample code, you can access fields with using the following syntax: $fieldNumber.

Using your example:

variableA=$1
variableB=$2

You can do you 'manupilations' using variable names now.

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[elgo]

vgersh99,

Now my script is working. But there is another problem. How to to ignore space within the line. When I take one line for processing it takes first part upto space one line and the relaining part as another line. How to overcome this problem.

Your help is appreciated.

 

[vgersh99]

your fields are separated by ':' - set your FieldSeparator (in awk) to ":"

FS=":"

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+