Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

Register Log in
  • Congratulations gkittelson on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Find lines with same field and compare different fields

Status
Not open for further replies.
willdev

willdev

Technical User
Jun 6, 2003
3
US
All,

need to compare the lines in a file and match a sending with a sent for a message id, then calculate the total time to send.

The lines of the file looks like this:
04/08/14:14:59:56.024:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000497:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8D8BHF7M6B DELIVERY-ID=JBH8D8BHF7M6R1KAR1H RECIPIENT=12345654 URL=http://xxxxxxxxxx:80/MMSJBH8D8BHF7M6B TRANSACTION-ID=JBH8D8BHF7M6B
04/08/14:15:00:01.067:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000497:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8D8BHF7M6B DELIVERY-ID=JBH8D8BHF7M6R1KAR1H RECIPIENT=123412321 URL=http://xxxxxxxxx:80/MMSJBH8D8BHF7M6B TRANSACTION-ID=JBH8D8BHF7M6B
04/08/14:15:00:26.021:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000498:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8ENBEG1MAB DELIVERY-ID=JBH8ENBEG1MAR1KAR1H RECIPIENT=123412343 URL=http://xxxxxxxxx:80/MMSJBH8ENBEG1MAB TRANSACTION-ID=JBH8ENBEG1MAB
04/08/14:15:00:27.070:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000498:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8ENBEG1MAB DELIVERY-ID=JBH8ENBEG1MAR1KAR1H RECIPIENT=12341234 URL=http://xxxxxxxxxx:80/MMSJBH8ENBEG1MAB TRANSACTION-ID=JBH8ENBEG1MAB
04/08/14:15:00:56.039:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000499:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8E2B6Q4UQB DELIVERY-ID=JBH8E2B6Q4UQR1KAR1H RECIPIENT=123454321 URL=http://xxxxxxxxxx:80/MMSJBH8E2B6Q4UQB TRANSACTION-ID=JBH8E2B6Q4UQB
04/08/14:15:00:56.047:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000499:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8E2B6Q4UQB DELIVERY-ID=JBH8E2B6Q4UQR1KAR1H RECIPIENT=12345432 URL=http://xxxxxxxxx:80/MMSJBH8E2B6Q4UQB TRANSACTION-ID=JBH8E2B6Q4UQB
04/08/14:15:01:06.034:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000500:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8E7BTCVQQB DELIVERY-ID=JBH8E7BTCVQQR1KAR1H RECIPIENT=123452432 URL=http://xxxxxxxxx:80/MMSJBH8E7BTCVQQB TRANSACTION-ID=JBH8E7BTCVQQB
04/08/14:15:01:16.019:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000501:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8FCBEG1MAB DELIVERY-ID=JBH8FCBEG1MAR1KAR1H RECIPIENT=123456354 URL=http://xxxxxxxx:80/MMSJBH8FCBEG1MAB TRANSACTION-ID=JBH8FCBEG1MAB
04/08/14:15:01:16.029:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000501:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8FCBEG1MAB DELIVERY-ID=JBH8FCBEG1MAR1KAR1H RECIPIENT=123456543 URL=http://xxxxxxxxx:80/MMSJBH8FCBEG1MAB TRANSACTION-ID=JBH8FCBEG1MAB
04/08/14:15:01:17.002:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000500:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8E7BTCVQQB DELIVERY-ID=JBH8E7BTCVQQR1KAR1H RECIPIENT=12345334 URL=http://xxxxxxxx:80/MMSJBH8E7BTCVQQB TRANSACTION-ID=JBH8E7BTCVQQB
04/08/14:15:02:06.028:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000502:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8F1BQDPPUB DELIVERY-ID=JBH8F1BQDPPUR1KAR1H RECIPIENT=1234345 URL=http://xxxxxxxxx:80/MMSJBH8F1BQDPPUB TRANSACTION-ID=JBH8F1BQDPPUB
04/08/14:15:02:06.037:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000502:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8F1BQDPPUB DELIVERY-ID=JBH8F1BQDPPUR1KAR1H RECIPIENT=123423456 URL=http://xxxxxxxxx:80/MMSJBH8F1BQDPPUB TRANSACTION-ID=JBH8F1BQDPPUB
04/08/14:15:02:06.071:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000503:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8F1DFEM72B DELIVERY-ID=JBH8F1DFEM72R1KAR1H RECIPIENT=12345345 URL=http://xxxxxxxxx:80/MMSJBH8F1DFEM72B TRANSACTION-ID=JBH8F1DFEM72B
04/08/14:15:02:06.082:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000503:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8F1DFEM72B DELIVERY-ID=JBH8F1DFEM72R1KAR1H RECIPIENT=12345672 URL=http://xxxxxxxxxx:80/MMSJBH8F1DFEM72B TRANSACTION-ID=JBH8F1DFEM72B
04/08/14:15:03:36.029:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000504:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8HABRVAQAB DELIVERY-ID=JBH8HABRVAQAR1KAR1H RECIPIENT=1234565 URL=http://xxxxxxx:80/MMSJBH8HABRVAQAB TRANSACTION-ID=JBH8HABRVAQAB
04/08/14:15:03:36.044:E://MM1/SENT/NOTIFICATION:mpebbe1:28458:000000504:mm1g_far_streamf.c(404):MESSAGE-ID=JBH8HABRVAQAB DELIVERY-ID=JBH8HABRVAQAR1KAR1H RECIPIENT=123456 URL=http://xxxxxxxx:80/MMSJBH8HABRVAQAB TRANSACTION-ID=JBH8HABRVAQAB
04/08/14:15:05:26.028:E://MM1/SENDING/NOTIFICATION:mpebbe1:28458:000000505:mm1g_far_streamf.c(386):MESSAGE-ID=JBH8ITBOV4PEB DELIVERY-ID=JBH8ITBOV4PER1KAR1H RECIPIENT=123456784 URL=http://xxxxxxx/MMSJBH8ITBOV4PEB TRANSACTION-ID=JBH8ITBOV4PEB

Sorry for the length. I need to find message ID and then calculate how many SENDINGs for the messageid before a SENT is shown, and time between first SENDING and SENT. And then due a summary of 0, 1, 2, 3 and 4 retries.

I am trying fgrep/awk and sed. But can't seem to get it right.

Any ideas, even for a portion would be gratefully appreciated.

Cheers,
John

 
Sort by date Sort by votes
See if this gets you started. It doesn't handle messages which cross midnight.

BEGIN{FS=":"}
{
tim = $2*3600+$3*60+$4
id = $9+0
}
/SENDING/{
if (!a[id]) b[id]=tim
a[id]++
}
/SENT/ {
print "id = ",id, "retries = ", a[id], "time = ",(tim-b[id])
delete a[id]
}

CaKiwi
 
Upvote 0 Downvote
Thanks so much CaKiwi! This is a great start.

Here is the output I get now. But it shows many double entries, so I think it is not matching from first SENDING to last SENT and counting the SENDINGS in the middle of these start and stop events.

MESSAGE-ID=JBH8D8BHF7M6B retries= 1 time= 5.043
MESSAGE-ID=JBH8ENBEG1MAB retries= 1 time= 1.049
MESSAGE-ID=JBH8E2B6Q4UQB retries= 1 time= 0.008
MESSAGE-ID=JBH8FCBEG1MAB retries= 2 time= 9.995

Notice that the message ending in ...MAB is shown twice.

I am using your code (thanks again) with $9 changed to $11.

A few questions to help me understand:
1.) What does (!a[id]) do?
2.) What is b[id]?

Kindest Regards,
John
 
Upvote 0 Downvote

!a[id] means "not a[id]" which is true if a[id] is zero, has never been defined or has been deleted.

b[id] holds the time of the first SENDING record.

You may need to split $11 to get the id, something like (untested)

split ($11,/ =/,c)
id = c[2]

instead of id = $11


CaKiwi
 
Upvote 0 Downvote
It is split ($11, c, "=") where the syntax is
(field, array, Field seperator)

This helped tremendously in my output.

But I still have duplicates at the end of the out put. I guess my next task is to sort through the log files to see if this is there. (Don't really want to scan 20,000 line file)

Cheers,
John
 
Upvote 0 Downvote
Status
Not open for further replies.

Similar threads

visitnag
  • Locked
  • Question
Hi I have a file with following
Replies
2
Views
215
visitnag
visitnag
marm1
  • Locked
  • Question
Display consumer/service line with smallest time value
Replies
12
Views
332
marm1
marm1
FrancisMMM
  • Locked
  • Question
awk parsing is getting slower and stalled :(
Replies
1
Views
201
FrancisMMM
FrancisMMM
johngiggs
  • Locked
  • Question
How do I print from a specific string to a new line with a matching pattern
Replies
3
Views
235
mikrom
mikrom
judkil
  • Locked
  • Question
AWK move lines of a file down
Replies
3
Views
203
judkil
judkil

Part and Inventory Search

Sponsor

Back
Top