Acceptable packet loss ?

[fonejack]

Looking at the voice quality stats on the 3300 and I show sporadic E2T G729 calls/sequence numbers with packet loss. 1 here 4 there. Have also had complaints about short periods of garbled voice quality. Should I be seeing any packet loss on calls that are QOSed and priority qued? Running MCD 4.0 on all 3300 in the network.

 

[Supernova99]

Check the engineering guidelines

There will always be some packet loss and jitter but you should not experience more that 0.5% packet loss or 20ms jitter

I will put the EG on Mediafire when it is back inservice so you can download it and check out the details

Share what you know - Learn what you don't

 

[IrwinMFletcher]

Can you describe more the "sporadic E2T G729 calls/sequence numbers with packet loss"? Are you seeing specific logs or are you looking at the e2tshow output?

 

[fonejack]

Sure, I am seeing this in both the ESM under voice quality stats and directly on the E2T with the e2tshow command. I also see underruns on the calls that are showing packet loss.

Below is the output of the e2tshow command

-> e2tshow
e2tshow

Found an E2tProxySp. Will run e2tproxyShow as well
-------------------------------- E2tProxySp Status --------------------
Local E2T has 64 Audio ports, and 128 T38 E2T Listening ports. Is Connected to E2tProxySp. Uses HDLC for transcoding.

E2T configured to use superchannel for transcoding.

MCC Channel Range (Audio) 192 - 255
Virtual MCC Channel Range (T38) 256 - 383
-------------------------------- E2T Stats ---
WED SEP 08 11:01:05 2010


Mitel Dynamic Jitter Buffer
RTC Local Connected
TDM codec is U_LAW_CODEC -- North America
Total Calls Setup 776964 (0 of which were/are T.38)
E2T crypto/clear/T.38 Channels active 7/2/0 (high 19/3/0)
DSP Transcoders currently active 9 (high 20)
Local Transcoders currently active 0 (high 0)
Pass: UDP port not in voice range 413588
Info: Ch with multiple src addresses 0
Pass: ICMP_ECHOREQUEST 262330
Pass: ICMP_REDIRECT 0
Pass: ICMP_UNREACH 2053
Drop: ICMP_UNREACH_PORT UDP 0
Pass: ICMP_ECHOREPLY 741129
Pass: ICMP_SRC_QUENCH 0
Pass: icmp_other 294
Drop: Invalid DTMF digit 0
Drop: Rx UDP audio on Closed Ch 38833
Drop: wrong g711 codec 82806
Drop: codec not g729 997
Drop: codec g729 bad length 0
Drop: codec g711 bad length 19
Drop: Rxed unexpected xcoded packets 0
Calls with Jitter > 150ms 81
Calls with Packet Loss > 2% 2363
RTP Sequence errors 642148
RTP Seq total missing pkts 757521
RTP Seq Packets out of order 67
RTP Seq False Packet Loss 0
RTP Seq False Out of Order 0
Host dst mod by icmp redirect 0
DTMF RTP Pkts Rx 1315
DTMF digits received 206
DTMF no inter digit gap 0
DTMF added pkts for min len 27
DTMF overflow 0
DTMF Duplicate E=1 0
DTMF Digits without E=1 flag 0
DTMF pkts digit starve 2
zFree Errors 0
VQ Stats msgs dropped 0
Drop: flood 838

Ethernet level Dropped Packet Report

Icmp :
Current credits 200(Lowest=188) Max 200 Inc 40 Dropped 0


Tcp SYN:
Current credits 100(Lowest=93) Max 100 Inc 20 Dropped 0


Tcp RST:
Current credits 100(Lowest=0) Max 100 Inc 20 Dropped 838

Last dropped TCP-RST: MAC s=08:00:0f:40:2e:c2 d=1d:d0:2f:e4:48:20 IP len=40 s=10.29.6.171 d=10.29.32.2 TCP sport=49500 dport=2024

UDP :
Current credits 4000(Lowest=3965) Max 4000 Inc 400 Dropped 0


ICMP Fragments discarded 0
Total packets discarded 838

EOS Detection (Active\Req\Reports\Det) 0\42\26\26
EOS event table overflow count 0
aggregate EED counters:
eedInitiated 776964
eedSuccessful 767491
eedAbandoned 9209
eedFailed 264
eedDecryptErrorMustWaitForModify 0
eedUnknownPtWaitForModify 0
eedDataCallWaitForModify 0
eedDtfmWaitForModify 0
eedMustWaitForModify 265
eedControlFsmStateMismatch 0
eedVoiceFsmStateMismatch 0
decryptError 0
encryptError 0
invalidE2tContext 0


Currently Active Channels


C=198 G729 Rx=49103 20ms(cfg=20ms) TxOk=98237 TxErr=0, EOS=0 T38=0 [PktLoss=16 maxBurst=16 OoO=0] [MaxJtr=1ms JB Ovr=0 Undr=36 Depth 20ms Hi=530ms] NetDly=18.14ms 10.2.32.2[50112] DTMF_RFC crypted

C=222 G729 Rx=2815 20ms(cfg=20ms) TxOk=5632 TxErr=0, EOS=0 T38=0 [PktLoss=1 maxBurst=1 OoO=0] [MaxJtr=37ms JB Ovr=0 Undr=15 Depth 30ms Hi=790ms] NetDly=21.02ms 10.17.32.2[50004] DTMF_RFC crypted


C=232 G729 Rx=373 20ms(cfg=20ms) TxOk=759 TxErr=0, EOS=0 T38=0 [PktLoss=5 maxBurst=1 OoO=0] [MaxJtr=2ms JB Ovr=0 Undr=11 Depth 20ms Hi=790ms] NetDly=31.77ms 10.21.32.2[50032] DTMF_RFC crypted


-------------------------------- T2E Stats ---
WED SEP 08 11:01:05 2010


Total T2E Calls Setup 681423 (0 of which were/are T38)
T2E crypto/clear/T.38 Channels active 7/2/0(high 19/3/0)
DSP Transcoders active 9 (high 20)
Local Transcoders active 0 (high 0)
T2E rx inactive ch 177370
T2E cleanup counter 475
T2E rx invalid ch 0
T2E rx inactive super ch 0
T2E rx_fail 0
T2E MccAlloc fail 0
T2E MccFree fail 0
T2E Pkts Dropped during JB Tone 1912
E2T Pkts Dropped during JB Tone 3307
Eth Tx Fail out of resources 0
MccSendHdlcVoiceBuffer fails 0
DspSendVoiceBuffer fails 0
MccAddVoiceBufferToTxQ fails 0
MccAddVoiceBufferToBdRing fails 0
Invalid DTMF Inject state 0
DTMF Inject Start requests 0
DTMF Inject Stop requests 0
loopback channels 6957
loopback inactive ch 0
DSP Transcoder Err 0
Last DSP Transcoder Hrd Err 0x0
Cipher Changes 0
Key Changes 474

Currently Active Channels


C=198 982sec rxOk=98267 txOk=49133 20ms txFail=0 udpPort=50012 Phone=10.2.32.2[50112] ip=G729 DTMF_RFC crypted

C=222 56sec rxOk=5658 txOk=2829 20ms txFail=0 udpPort=50060 Phone=10.17.32.2[50004] ip=G729 DTMF_RFC crypted

C=232 6sec rxOk=680 txOk=340 20ms txFail=0 udpPort=50080 Phone=10.21.32.2[50032] ip=G729 DTMF_RFC crypted




E2t Command Summary
e2tclear - clear the e2t counters
Stream_PrintTables - dump the channel tables
e2thistory - hourly call history
e2tshow - e2t counters
e2tshow 1 - e2t counters summary
e2tmsgtrace = 1 - swLog all Proxy<->E2T messages (=0 to disable)
e2tmsgtrace = 2 - swLog Proxy<->E2T messages except Heartbeat (=0 to disable)
value = 0 = 0x0
->

 

[IrwinMFletcher]

You need to be careful when interpreting these results as there are possibilities for false-positive triggers due to renegotiating of streams.

The fact that a call can have a highwater mark of 790ms and yet a depth of only 20ms, to me means that there was a jump in the stream. Sniff one of the calls and you'll probably see a change in the stream at the start of the call.

 

[IrwinMFletcher]

p.s. just curious, what's the uptime on this system?

 

[fonejack]

Uptime is 304 days. Looking through the maintenance info logs I see the following E2t mesages. Not sure if these should be addressed? Thanks

Log Number Date Time Source Description
4439
2010/Sep/08
10:24:21
E2tSp
IP Network Dropped 3.27 percent of Packets from phone 10.17.32.2
Total Rx: 19934 Lost=674 maxLossBurst=654
1 in a row: 1
2 in a row: 1
3-5 in a row: 0
6-10 in a row: 0
11 or more: 2
4438
2010/Sep/08
10:23:00
E2tSp
IP Network Dropped 7.18 percent of Packets from phone 10.17.32.2
Total Rx: 8665 Lost=670 maxLossBurst=654
1 in a row: 0
2 in a row: 0
3-5 in a row: 0
6-10 in a row: 0
11 or more: 2
4431
2010/Sep/08
10:22:24
E2tSp
IP Network Dropped 6.18 percent of Packets from phone 10.17.32.2
Total Rx: 258 Lost=17 maxLossBurst=17
1 in a row: 0
2 in a row: 0
3-5 in a row: 0
6-10 in a row: 0
11 or more: 1

 

[IrwinMFletcher]

2500 calls/day, not bad, I was just surprised at the 750K calls.

For the first call, looks like 1 big chuck of dropped packets (654). Notice the same for the second call (654 packets at once). So either something caused the system to 'hang' for 13+ seconds, causing the network buffers to overflow, or there was a block on the network. I believe that you would see a log is all of the network clusters were exhausted on the system, so if you don't see that, then it's likely the later.

The last call is too short to care (5 secs), skews the stats.

Only definitive way to know is to sniff.

 

[Supernova99]

Engineering Guidelines attached

Share what you know - Learn what you don't

 

[slapin]

Make sure that you are not exceeding QOS priority queue limits. In this case the routers will drop chunks of packets which don't fit into the limit. Those packets may belong to different calls.