extracting values from multiple lines into a seperate output file

[mmiah1982]

Hi,

I’m using perl on windows v5.10.1 and would like to extract out of a log file extract below to produce output as below, does anyone know how to extract the items highlighted in red? The format of the log file is pretty much consistent throughout:

Log extract

CCurClgA=901922494724 MetaCode(true)=META_UNKNOWN'
11/07/12 15:49:05.448 Trc 04541 RequestReleaseCall received from [8220] (0001238c 123103 172.46.713.123:1966)
message RequestReleaseCall
AttributeReferenceID 219
AttributeConnID 00c10218203ee739
AttributeThisDN '82123345'
11/07/12 15:49:05.448 Int 04543 Interaction message "RequestReleaseCall" received from 8220 ("123103")
@15:49:05.4480 [gctmi] Distributing request RequestReleaseCall
@15:49:05.4480 [tout] Party [00c10218208ee739:89123103,sc,tDN,rDST,lINT] (reqReleaseCall)

Output I’d like to pull from log file

Date Time Event IP ConnID DN
11/07/12 15:49:05.448 RequestReleaseCall 172.46.713.123 00c10218203ee739 82123345


Any help would be appreciated greatly 

Thanks

 

[rharsh]

What have you tried? What are you having problems with?

 

[mmiah1982]

This is what I have so far, the bit I struggle with mostly is extracting the lines below my initial line to get the values I'm interested in I think there's also some issue with my syntax?

use strict;
use warnings;

print " Date\tTime\tEvent\tIP Address\tConnID\tDN\n ";

@ARGV = <@ARGV>;

while (<>) {

if
(/^([\d\/]+)\s([\d:\.]+)\[Trc\]\s\d+\s(RequestReleaseCall)\s[recieved]\s[from]\s\[d+\]\s\\d+c\s\d+\s([^\:]+)/){print "$1 $2 $3 $4 \n"}

}


Cheers!

 

[feherke]

Hi

Yes, beside some [green]missing[/green] and [red]extra[/red] characters there are also some pointless escapes :

[gray]# before[/gray]
/^([\d\/]+)\s([\d:[COLOR=orange]\[/color].]+)[red]\[[/red]Trc[red]\][/red]\s\d+\s(RequestReleaseCall)\s[red][[/red]recieved[red]][/red]\s[red][[/red]from[red]][/red]\s\[d+\]\s\([red]:[/red]\d+c\s\d+\s([^[COLOR=orange]\[/color]:]+)/

[gray]# after[/gray]
/^([\d\/]+)\s([\d:.]+)[green]\s[/green]Trc\s\d+\s(RequestReleaseCall)\sreceived\sfrom\s\[\d+\]\s\(\d+c\s\d+\s([^:]+)/

Regarding extracting pieces from the other lines too, just add more similar regular expressions and [tt]print[/tt]s. Just spare the [tt]\n[/tt] from the end of [tt]print[/tt]ed strings, excepting the last one.


Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[mmiah1982]

Thanks Ferkhe,

"Regarding extracting pieces from the other lines too, just add more similar regular expressions and prints. Just spare the \n from the end of printed strings, excepting the last one."

My log file has many instances of the word AttributeConnID and AttributeThis DN, it is only when they occur 3 and 4 lines below the word RequestReleaseCall is when I'm interested in extracted will the above work in this case? Will it not pull all the other matches that aren't 3&4 lines below RequestReleaseCall keyword

11/07/12 15:49:05.448 Trc 04541 RequestReleaseCall received from [8220] (0001238c 123103 172.46.713.123:1966)
message RequestReleaseCall
AttributeReferenceID 219
AttributeConnID 00c10218203ee739 <------- 3 lines below RequestReleaseCall
AttributeThisDN '82123345' <--------4 lines below RequestReleaseCall
11/07/12 15:49:05.448 Int 04543 Interaction message "RequestReleaseCall" received from 8220 ("123103")
@15:49:05.4480 [gctmi] Distributing request RequestReleaseCall

Thanks

 

[feherke]

Hi

Well, the information about repeated AttributeConnID and AttributeThisDN lines was missing in your original post. Further unspecified details :
[ul]
[li]Is there a single RequestReleaseCall line or can be more ?[/li]
[li]If there are more RequestReleaseCall lines, only the first has to be extracted or all ?[/li]
[li]Are the lines always in RequestReleaseCall, AttributeConnID, AttributeThisDN order ?[/li]
[li]Are there always AttributeConnID and AttributeThisDN lines at 3-4 line distance after the RequestReleaseCall lines ?[/li]
[/ul]

If you want check the distance between the matched lines, use a counter : initialize it when RequestReleaseCall line is found, increment it on every line, check it when AttributeConnID and AttributeThisDN lines are found.

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[mmiah1982]

Thanks

I do seem to have left some key details out apologies

Is there a single RequestReleaseCall line or can be more ? <---- Single on a line but many instances which I'd like to extract
If there are more RequestReleaseCall lines, only the first has to be extracted or all ? <--- All

Are the lines always in RequestReleaseCall, AttributeConnID, AttributeThisDN order ? <--- Yes
Are there always AttributeConnID and AttributeThisDN lines at 3-4 line distance after the RequestReleaseCall lines ? <-- Yes

Thanks

 

[feherke]

Hi

Well, if you can always count on the presence of each line, then you can do it without counting the lines. Just collect the needed pieces and output them when all 6 found :

[b]use[/b] strict[teal];[/teal]
[b]use[/b] warnings[teal];[/teal]

[b]print[/b] [green][i]"Date\tTime\tEvent\tIP Address\tConnID\tDN\n"[/i][/green][teal];[/teal]

[navy]@ARGV[/navy] [teal]=[/teal] [green][i]<@ARGV>[/i][/green][teal];[/teal]

[b]my[/b] [navy]@out[/navy][teal];[/teal]
[b]while[/b] [teal]([/teal][green][i]<>[/i][/green][teal])[/teal] [teal]{[/teal]
  [navy]@out[/navy] [teal]=[/teal] [teal]([/teal][navy]$1[/navy][teal],[/teal][navy]$2[/navy][teal],[/teal][navy]$3[/navy][teal],[/teal][navy]$4[/navy][teal])[/teal] [b]if[/b] [green][i]/^([\d\/]+) ([\d:.]+) Trc \d+ (RequestReleaseCall) received from \[\d+\] \(\d+c \d+ ([^:]+)/[/i][/green][teal];[/teal]
  [b]push[/b] [navy]@out[/navy][teal],[/teal] [navy]$1[/navy] [b]if[/b] [green][i]/^AttributeConnID (.+)/[/i][/green][teal];[/teal]
  [b]push[/b] [navy]@out[/navy][teal],[/teal] [navy]$1[/navy] [b]if[/b] [green][i]/^AttributeThisDN '(.+)'/[/i][/green][teal];[/teal]

  [b]print[/b] [b]join[/b][teal]([/teal][green][i]"\t"[/i][/green][teal],[/teal] [navy]@out[/navy][teal]),[/teal] [green][i]"\n"[/i][/green] [b]if[/b] scalar [navy]@out[/navy] [teal]==[/teal] [purple]6[/purple][teal];[/teal]
[teal]}[/teal]

( Note that using [tt]\s[/tt] instead of space makes the regular expression harder to read. We used to use [tt]\s[/tt] in our responses just in case the tab and space characters were messed up on their way through the forum's web interface when posted. If you are sure the words in the parsed text are always separated by spaces, better write just spaces in the regular expression. )


Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[mmiah1982]

Thanks that's brilliant almost gives it what I need, one thing though for each result found it's returning the row 6 times? so for the extract of the log file above it returns the following, the same line repeated 6 times?

Date Time Event IP Address ConnID DN
11/07/12 15:49:05.448 RequestReleaseCall 172.46.713.123 00c10218203ee739 82123345
11/07/12 15:49:05.448 RequestReleaseCall 172.46.713.123 00c10218203ee739 82123345
11/07/12 15:49:05.448 RequestReleaseCall 172.46.713.123 00c10218203ee739 82123345
11/07/12 15:49:05.448 RequestReleaseCall 172.46.713.123 00c10218203ee739 82123345
11/07/12 15:49:05.448 RequestReleaseCall 172.46.713.123 00c10218203ee739 82123345
11/07/12 15:49:05.448 RequestReleaseCall 172.46.713.123 00c10218203ee739 82123345

Out of interest how does the push @out work?

push @out, $1 if /^AttributeConnID (.+)/;


Cheers

 

[feherke]

Hi

or each result found it's returning the row 6 times?

Oops. Seems my test input file was too simple. A quick fix, I hope :

[b]use[/b] strict[teal];[/teal]
[b]use[/b] warnings[teal];[/teal]

[b]print[/b] [green][i]"Date\tTime\tEvent\tIP Address\tConnID\tDN\n"[/i][/green][teal];[/teal]

[navy]@ARGV[/navy] [teal]=[/teal] [green][i]<@ARGV>[/i][/green][teal];[/teal]

[b]my[/b] [navy]@out[/navy][teal];[/teal]
[b]while[/b] [teal]([/teal][green][i]<>[/i][/green][teal])[/teal] [teal]{[/teal]
  [navy]@out[/navy] [teal]=[/teal] [teal]([/teal][navy]$1[/navy][teal],[/teal][navy]$2[/navy][teal],[/teal][navy]$3[/navy][teal],[/teal][navy]$4[/navy][teal])[/teal] [b]if[/b] [green][i]/^([\d\/]+) ([\d:.]+) Trc \d+ (RequestReleaseCall) received from \[\d+\] \(\d+c \d+ ([^:]+)/[/i][/green][teal];[/teal]
  [b]push[/b] [navy]@out[/navy][teal],[/teal] [navy]$1[/navy] [b]if[/b] [green][i]/^AttributeConnID (.+)/[/i][/green][teal];[/teal]
  [b]if[/b] [teal]([/teal][green][i]/^AttributeThisDN '(.+)'/[/i][/green][teal])[/teal] [teal]{[/teal]
    [b]push[/b] [navy]@out[/navy][teal],[/teal] [navy]$1[/navy][teal];[/teal]
    [b]print[/b] [b]join[/b][teal]([/teal][green][i]"\t"[/i][/green][teal],[/teal] [navy]@out[/navy][teal]),[/teal] [green][i]"\n"[/i][/green] [b]if[/b] scalar [navy]@out[/navy] [teal]==[/teal] [purple]6[/purple][teal];[/teal]
  [teal]}[/teal]
[teal]}[/teal]

Out of interest how does the push @out work?

[tt]push[/tt] appends the specified value(s) ( 2[sup]nd[/sup] and following parameters ) to the end of given array ( 1[sup]st[/sup] parameter ).

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[mmiah1982]

Thats done the trick! Brill Thanks for your help