Extract out of Log file 1

mmiah1982 · Nov 6, 2012

Hello

I have a log file which I'm looking to extract certain lines containing keywords so far I have managed to produce the script below which extracts any line containg words EventEstablished and Primay into a text file output as below

11/06/12 09:54:04.549 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:08.752 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:11.330 Trc 04542 EventEstablished sent to [3280] (00002a29 Primary Telephony Server 170.20.10.11:4193)
11/06/12 09:54:11.330 Trc 04542 EventEstablished sent to [3280] (00002a29 Primary Telephony Server 170.20.10.11:4193)
11/06/12 09:54:12.705 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:12.986 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:13.033 Trc 04542 EventEstablished sent to [1560] (00002816 Primary Telephony Server 170.20.10.11:4140)
11/06/12 09:54:15.158 Trc 04542 EventEstablished sent to [3280] (00002a29 Primary Telephony Server 170.20.10.11:4193)

The script I use is below:

# Usage: perl Find.pl infile outfile
use strict;
use warnings;

my ($qfn_in, $qfn_out) = @ARGV;

open(my $fh_in, '<', $qfn_in)
or die("Unable to read file \"$qfn_in\": $!\n");
open(my $fh_out, '>', $qfn_out)
or die("Unable to create file \"$qfn_out\": $!\n");

while (<$fh_in>) {

if (/EventEstablished/ and /Portrait/) {
print $fh_out $_;
}
}

Now this is a very basic script I've been trying to enhance it but can't work out quite how yet but I'd like to add the following

1) Ability to add additional lines to extract and include in the output file when additional keywords are availiable, e.g. to also extract lines containing words RequestAgentLogin received

2) Add a header to the output file

3) Display Output file to include only cetain values out of the extracted line and strip down only IP address

i.e. what I would like the output file to produce is something like

Date Time Event IP Address
11/06/12 09:54:04.549 EventEstablished 170.20.10.11
11/06/12 09:54:08.752 EventEstablished 170.20.10.11
11/06/12 09:54:11.330 EventEstablished 170.20.10.41
11/06/12 09:58:11.330 RequestAgentLogin 170.20.10.28

Any help would be greatly appreciated

Thanks

M

feherke · Nov 6, 2012

Hi

Personally I would prefer to write it shorter as :

Code:

perl -ne '[b]print[/b] [b]if[/b][green][i]/EventEstablished/&&/Primary/[/i][/green]' infile outfile

Which enhanced with the requested functionalities would look like this :

Code:

perl -nle 'BEGIN[teal]{[/teal][navy]@l[/navy][teal]=[/teal][b]qw[/b][teal]{[/teal]EventEstablished RequestAgentLogin[teal]}[/teal][teal];[/teal][b]print[/b][green][i]"Date Time Event IP Address"[/i][/green][teal]}[/teal][b]print[/b][green][i]"$1 $2 $3"[/i][/green][b]if[/b][green][i]/Primary/&&/^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/[/i][/green][teal]&&[/teal][navy]$2[/navy][teal]~~[/teal][navy]@l[/navy]' infile outfile

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

mmiah1982 · Nov 6, 2012

Hi Feherke,

Thanks for you reply, I'm just getting to grips with perl so my code is more than likely far from perfect

Ideally I'd like to put everything within the perl file so all i pass through the command line is the name of my perlscript and the input and output files.

I've attempted to run the code you've kindly provided but encountered some issues

Get an error message "Cant Find String terminator "" anywhere before EOF at -e line 1"

Is there anyway to incorporate your enhanced code within a perl script and run the script?

Your code: perl -nle 'BEGIN{@l=qw{EventEstablished RequestAgentLogin};print"Date Time Event IP Address"}print"$1 $2 $3"if/Primary/&&/^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/&&$2~~@l' infile outfile

Does this extract all lines containing words EventEstablised or RequestAgentLogin?
If I wanted to include all lines EventEstablished and Primary or RequestAgentLogin how can this be done via the script?

Thanks again

feherke · Nov 6, 2012

Hi

mmiah1982 said:
Get an error message "Cant Find String terminator "" anywhere before EOF at -e line 1"

Interesting. Please specify your operating system, shell/command interpreter and Perl version.

mmiah1982 said:
Does this extract all lines containing words EventEstablised or RequestAgentLogin?

Actually extracts this lines which contain "Primary" anywhere and contain something looking as IP address as their 7^th or later word and the 5^th word is either "EventEstablised" or "RequestAgentLogin".

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

mmiah1982 · Nov 6, 2012

Hi

Interesting. Please specify your operating system, shell/command interpreter and Perl version.
-- Windows XP, Perl v5.10.1[/b]

Actually extracts this lines which contain "Primary" anywhere and contain something looking as IP address as their 7th or later word and the 5th word is either "EventEstablised" or "RequestAgentLogin".
----Looking to extract lines containing words EventEstablished and Primary and also lines contaning RequestAgentLogin
(there are other lines within log file just containing word Primary which i dont need, only when it also has EventEstablished aswell i'm interested in)

Thanks Again

M

feherke · Nov 6, 2012

Hi

mmiah1982 said:
Windows XP, Perl v5.10.1

Well, that changes everything.

mmiah1982 said:
Looking to extract lines containing words EventEstablished and Primary and also lines contaning RequestAgentLogin

You mean the presence of "Primary" is required only for lines containing "EventEstablished" but not necessary for lines containing "RequestAgentLogin" ?

A complex sample input containing both needed and unneeded cases would really help.

Then I would use this code :

Code:

[b]use[/b] strict[teal];[/teal]
[b]use[/b] warnings[teal];[/teal]

[b]print[/b] [green][i]"Date Time Event IP Address\n"[/i][/green][teal];[/teal]

[b]while[/b] [teal]([/teal][green][i]<>[/i][/green][teal])[/teal] [teal]{[/teal]
  [b]print[/b] [green][i]"$1 $2 $3\n"[/i][/green] [b]if[/b] [teal](([/teal][green][i]/EventEstablished/ && /Primary/) || /RequestAgentLogin/) && /^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/[/i][/green]
[teal]}[/teal]

Like this :

Code:

perl Find.pl infile > outfile

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

mmiah1982 · Nov 6, 2012

Thanks Feherke!

I think that's almost exactly what I'm after!!! One additional tweak is at the moment the input file is the name of a single file that you choose, if the input was a list of files in a directory how would i read this in?

So the script would be in the directory containing all the log files which contain the file names in the same format

i.e

Tlog_06102012_1540.log
Tlog_06102012_1440.log
Tlog_06102012_1340.log
Tlog_06102012_1240.log
Tlog_06102012_1130.log
Tlog_05102012_0940.log
etc........

So I need my command to be perl Find.pl Tlog*.lo > OutputFile

So I can better understand your code, do you mind adding in some commentry on your regular expression so I can work out how you selected which columns to show on the output, and also how you stripped off just the IP address?

print "$1 $2 $3\n" if ((/EventEstablished/ && /Primary/) || /RequestAgentLogin/) && /^(\S+ \S+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/

Thanks Again Really Appreciate your help!
M

feherke · Nov 6, 2012

Hi

mmiah1982 said:
So I need my command to be perl Find.pl Tlog*.lo[red]g[/red] > OutputFile

Well, on Unix-like operating systems that would be the exact command to run.

To make the same work on Windows too, you need to add the globbing to the script :

Code:

[b]use[/b] strict[teal];[/teal]
[b]use[/b] warnings[teal];[/teal]

[b]print[/b] [green][i]"date time event ip address\n"[/i][/green][teal];[/teal]

[highlight][navy]@argv[/navy] [teal]=[/teal] [green][i]<@argv>[/i][/green][teal];[/teal][/highlight]

[b]while[/b] [teal]([/teal][green][i]<>[/i][/green][teal])[/teal] [teal]{[/teal]
  [b]print[/b] [green][i]"$1 $2 $3\n"[/i][/green] [b]if[/b] [teal](([/teal][green][i]/eventestablished/ && /primary/) || /requestagentlogin/) && /^(\s+ \s+) \w+ \w+ (\w+) .* (\d{1,3}(?:\.\d{1,3}){3})/[/i][/green]
[teal]}[/teal]

[small][maroon]Warning[/maroon] The above code was not tested on Windows. It works on Linux and the documentation mentions no reason why it wound not work on Windows.[/small]

[pre]
[maroon],--- starts with[/maroon]
[maroon]|[/maroon] [red],-- a non-space character[/red]
[maroon]|[/maroon] [red]|[/red][purple],-- previous item 1 or more times[/purple]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia],-- a word character[/fuchsia]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia]|[/fuchsia] [green],-- any character[/green]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia]|[/fuchsia] [green]|[/green][lime],-- previous item 0 or more times[/lime]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia]|[/fuchsia] [green]|[/green][lime]|[/lime] [olive],-- a digit character[/olive]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia]|[/fuchsia] [green]|[/green][lime]|[/lime] [olive]|[/olive] [navy],-- previous item 1 up to 3 times[/navy]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia]|[/fuchsia] [green]|[/green][lime]|[/lime] [olive]|[/olive] [navy]|[/navy] [blue],-- non-capturing group[/blue]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia]|[/fuchsia] [green]|[/green][lime]|[/lime] [olive]|[/olive] [navy]|[/navy] [blue]|[/blue] [teal],-- a literal . character[/teal]
[maroon]|[/maroon] [red]|[/red][purple]|[/purple] [fuchsia]|[/fuchsia] [green]|[/green][lime]|[/lime] [olive]|[/olive] [navy]_|_[/navy] [blue]|[/blue] [teal]|[/teal] [aqua],-- previous item 3 times[/aqua]
[maroon]|[/maroon] [red]/\[/red][purple]|[/purple] [fuchsia]/\[/fuchsia] [green]|[/green][lime]|[/lime] [olive]/\[/olive][navy]/ \[/navy] [blue]/\[/blue][teal]/\[/teal] [aqua]/ \[/aqua]
/[maroon]^[/maroon][highlight #fcc]([red]\S[/red][purple]+[/purple] [red]\S[/red][purple]+[/purple])[/highlight] [fuchsia]\w[/fuchsia][purple]+[/purple] [fuchsia]\w[/fuchsia][purple]+[/purple] [highlight #cfc]([fuchsia]\w[/fuchsia][purple]+[/purple])[/highlight] [green].[/green][lime]*[/lime] [highlight #ccf]([olive]\d[/olive][navy]{1,3}[/navy]([blue]?:[/blue][teal]\.[/teal][olive]\d[/olive][navy]{1,3}[/navy])[aqua]{3}[/aqua])[/highlight]/
[silver]\_______/ \_/ \______________________/[/silver]
[highlight #fcc]$1[/highlight] [highlight #cfc]$2[/highlight] [highlight #ccf]$3[/highlight]
[silver] \______________|__________________/[/silver]
captured group
[/pre]

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

mmiah1982 · Nov 6, 2012

Thanks Again for the explanation

Adding Globbing didn't seem to work

get following error Global symbol @argv requires explicit package name at Find.pl

M

feherke · Nov 6, 2012

Hi

Oops. No idea how I managed to post that mess. ( Was a long day yesterday. )

@ARGV should be written in uppercase :

Code:

[navy]@ARGV[/navy] [teal]=[/teal] [green][i]<@ARGV>[/i][/green][teal];[/teal]

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

mmiah1982 · Nov 7, 2012

Perfect! I can't thank you enough for all your help on this, even my noob questions!

Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Extract out of Log file 1

mmiah1982

MIS

feherke

Programmer

mmiah1982

MIS

feherke

Programmer

mmiah1982

MIS

feherke

Programmer

mmiah1982

MIS

feherke

Programmer

mmiah1982

MIS

feherke

Programmer

mmiah1982

MIS

Similar threads

Part and Inventory Search

Sponsor