G-450 Gateways Randomly Resetting 1

[iggy1952]

We are running CM 6.3 on our CM core servers.

Recently we were advised by our BP to upgrade the firmware on our G-450 media gateways to 39.12.0 in preparation for upgrading to CM 7.13. Since the firmware update several of the G-450 media gateways have been randomly resetting with no pattern.

Is it possible the firmware update is causing this problem.

iggy1952

 

[kyle555]

never seen gateways reset to be honest. could be power related/power supply/maybe analog ports with weird voltage.

others could chime in to the contrary, but I've never really dealt with spontaneous gateway resets.

if you 'show rec' and 'show logging file content' you'll see the recovery parameters and when the h248 link to cm was lost. Gateways will reset after 'total search' time in minutes if they can't get to a CM. Maybe that's playing a part?

 

[keops7]

Hi,

Can I ask if you can share with us the GW log? It would be helpful to find the issue.

Thanks.

 

[iggy1952]

keops7

Here is a log from one of the gateways that is resetting:

arbavg450-014(super)# show logging file content

<190>Aug 12 10:54:31 arbavg450 -NoTag: -NoUTC 2018 315 1 mediagateway.g450 | 0 BOOT MESSAGE[BOOT-Informational: Booting from bank A with firmware version 39.12.0, ID=9f4d02ca2cbf19084ef7ef1b31bbdf31

<190>Aug 12 10:54:31 arbavg450 -NoTag: -NoUTC 2018 315 1 mediagateway.g450 | 0 warmStart[BOOT-Informational: System boot up from warm reset, ID=9f4d02ca2cbf19084ef7ef1b31bbdf31

<188> arbavg450 -NoTag: -NoUTC505 1 mediagateway.g450 | 0 UPGRADEAUTHFILE[BOOT-Warning: This Gateway Image version supports a newer version of authentication file. It is strongly recommended to update the Authentication File to match the Media Gateway release in order to benefit from enhancements to the Authentication File., ID=9f4d02ca2cbf19084ef7ef1b31bbdf31

<187>Aug 12 10:51:05 arbavg450 -NoTag: -NoUTC 2018 165 1 mediagateway.g450 | 0 GWG-STAMAJNO[VOICE-Error: keepAliveFailed() - Close H248 socket, ID=9f4d02ca2cbf19084ef7ef1b31bbdf31

<190>Aug 10 08:49:25 arbavg450 -NoTag: -NoUTC 2018 670 1 mediagateway.g450 | 0 BOOT MESSAGE[BOOT-Informational: Booting from bank A with firmware version 39.12.0, ID=9f4d02ca2cbf19084ef7ef1b31bbdf31




iggy1952

 

[kyle555]

yeah, so do a show rec to see the recovery timers. And more of those logs! You're looking like warm restarts 3 minutes losing CM registration.

Do a display alarms in CM for the date/time/hour in question on that media gateway and see what you get there. I think you're seeing a normal reboot for lack of registration.

 

[dialtonebone]

show int"

does it show any errors? Are your duplex/speed settings matched? I know a firmware update shouldn't affect these things, but maybe a prior issue is being exacerbated? I'm not discounting that the firmware could possibly be the issue, but that's more reliably verified by Avaya support.

 

[iggy1952]

kyle555

Here are some more logs from the G-450 gateway

0000000041 08/12-10:51:05.00 MgFw#:39.12.0 CCI-STACRINO-01281 REBOOT from Recove ryEngineUtil
0000000040 08/10-08:45:59.00 MgFw#:39.12.0 CCI-STACRINO-01281 REBOOT from Recove ryEngineUtil
0000000039 08/08-06:36:56.00 MgFw#:39.12.0 CCI-STACRINO-01281 REBOOT from Recove ryEngineUtil
0000000038 08/06-04:29:59.00 MgFw#:39.12.0 CCI-STACRINO-01281 REBOOT from Recove ryEngineUtil
0000000037 08/04-02:10:09.00 MgFw#:39.12.0 CCI-STACRINO-01281 REBOOT from Recove ryEngineUtil
0000000036 08/02-00:02:02.00 MgFw#:39.12.0 CCI-STACRINO-01281 REBOOT from Recove

CM log
20180812:105105333:4810301:hmm(15282):MED:[ MTCEVT ERR type=0321 lname=5400 pn1=00000807 pn2=0000000e aux=71445eb4 rc=0]
20180812:105105333:4810302:hmm(15282):MED:[ MTCEVT ERR type=0301 lname=1424 pn1=00000000 pn2=0000000e aux=00000000 rc=0]
20180812:105105337:4810304:mdm(15312):MED:[ MTCEVT ALM type=0003 lname=1424 pname=0000000e cbusy=0000 filt=w]



iggy1952

 

[kyle555]

still don't see the recovery timers. Seems like you've got corresponding alarms on CM to indicate the gateway came unregistered.
Following the timestamps, your gateway rebooted itself at 1054 while it was seemingly unregistered from CM - so I'd look into that.

 

[iggy1952]

kyle555,

Gateway recovery timers:

RECOVERY TIMES
--------------------------------
Primary Search : 1
Total Search : 30
Transition Point: 1

CALL CONTROLLER STATUS
-------------------------------------------
Registered : YES
Active Controller : 192.168.32.35
Controller SW Version : R016x.03.0.124.0
H248 Link Status : UP
H248 Link Encryption : PTLS
H248 Link Error Code : 0x0

PRIMARY MGC HOST, Primary Search Time : 1 min(s)
IPv4 Address IPv6 Address
-------------------- ----------------------------------------------
192.168.32.35 -- Not Available --

SECONDARY MGC HOST
IPv4 Address IPv6 Address
-------------------- ----------------------------------------------
192.168.24.33 -- Not Available --
192.168.56.34 -- Not Available --
-- Not Available -- -- Not Available --

sls disabled


iggy1952

 

[kyle555]

does it try registering to the other 2 secondary MGC HOSTs? Logs would indicate that it should be trying. If not, maybe you do have a firmware bug, but unless its really obscure, I'd expect you can find it in release notes

 

[iggy1952]

kyle555,

The gateway do not try registering to the secondary MGC hosts. The logs show a reset and registering back to the PROCR only.

We recently updated the firmware to 39.12.0 at the recommendation of our Avaya BP in preparation for upgrading from 6.3 to 7.1. They advised no knowledge of issues with the newer firmware version running with CM 6.3.



iggy1952

 

[dialtonebone]

Could you downgrade just one of these back to the original firmware (change the boot-bank) and see if the problem persists?

 

[iggy1952]

randycarroll

Yes. I can downgrade one of the gateways and monitor for a couple of days.

iggy1952

 

[dialtonebone]

Cool - I know it doesn't fix the exact issue at hand but it helps with problem isolation.

If the gateway becomes stable then it might be worth asking Avaya to verify if its a issue of the CMs being mismatched, or if you will end up having this trouble even after upgrading.

 

[iggy1952]

randycaroll,

I have rolled back the G-450 firmware version to 38.21.1 and will monitor for resets. The gateways have reset every 2 to 3 days at various times during the day.

I will post our results.

iggy1952

 

[dialtonebone]

Cool - if these also start resetting, please the top 50-100 lines or so of a "show logg file cont" from the gateway CLI.

 

[iggy1952]

randycaroll,

Wireshark traces show spurious retransmissions between the media gateway and CM core server before the warm reset.

Avaya feels this is a network issue.

iggy1952

 

[iggy1952]

randycaroll,

We rolled back one of the media gateways to firmware version 38.21.1 and the resets have continued about every 2 days.

iggy1952

 

[Saiyan656]

I had the same issue when adding RAM board to second slot which will cause gateway to reboot.
Warning:
Do not insert an additional RAM card into the second slot on the G450 2.x and greater.

https://downloads.avaya.com/css/P8/documents/100059276

 

[iggy1952]

Saiyan656,

Thank you for the information.

The gateways continue to reset every 2 days at random times. Avaya says its a network issue after reviewing the CM and gateway logs. We can find no hardware issues with any of the 3 gateways that reset. The H.248 keep alive links keeping dropping and Wireshark traces show a TCP spurious retransmission but we see no packet drops. Our WAN team sees no drops on the WAN links.

We have a Wireshark trace on one of the gateways and the core CM to capture for the next reset. We have 3 other G-450 gateways with the same firmware version that are not having this issue. Some other sources indicate we could have a flaky core router.

I will advise results of Wireshark trace at next reset.

iggy1952