G450 AVP 7.0 Insertion Failure on S8300E

[PhonesAllDay]

I have two G450's Firmware upgraded and booted on both banks as follows: 36.18.0 -> 37.41.0 ->38.21.2 -> 39.20.0
Loading AVP 7.0, once EULA is accepted, G450 faults with Insertion Failure.
The other G450 is in production with S8300E with AVP version 7.1.3, (upgraded first to 7.0, then 7.1.2...) was restarted several times during power outages and showing Insertion Failure.
Re-seating, downgrading/upgrading G450 firmware, restarting, changing boot banks, re-loading AVP, nothing works and this can be repeated every time. Show MM, the board is "Initializing"
Connecting directly to AVP via services port works just fine.

Avaya sent a document that this is a known issue with fix scheduled in AVP release 7.1.4

Their document states: "same issue in the lab on S8300 D and E on AVP 7.1.0, 7.1.2, 7.1.3 and 7.1.3.2 and on gateway firmware G450 39.12.0, G430 38.20.1, G450 on 38.21.1. I have tired on on other combinations".

Avaya said this is a workaround which I haven't got to work successfully yet: kill -9 `pidof watchd` && /etc/init.d/watchd start

I'm wondering if anyone else is experiencing this, or found a fix?

Thanks,
PhonesAllDay
(and AllNight)

 

[fondog2]

Just curious did you agree to activate EASG after upgrading the firmware? I found that once you activate it you also lose the option to reset the GW PSWD through console using the back door PSWD. I've personally never had an issue installing the AVP on any 8300 D/E other than it taking forever. Makes me wonder if this is causing the issue.

 

[PhonesAllDay]

Yes the insertion failure error comes just after I connect to AVP services port and accept EULA. If you never had issues installing AVP, then you must be some type of super human.

 

[fondog2]

No not super human, but I've done hundreds. As long as the info you add to the kick start is correct they load every time. So to clarify you configured the GW, upgraded the firmware then installed the 8300. I know you said that Avaya stated this is a known issue but a couple of others I've spoke to as well as me have never experienced this issue. If you have a spare GW try installing in it. Now that I think of it I did run into a crazy issue with a couple of GW's that after configuring through the wizard it lost the default GW and wouldn't register and continued to spit out all kinds of crazy errors. Just one other thing to check.

 

[PhonesAllDay]

I have two G450 upgraded to 39.20.0 With Board Insertion Failure. I configure G450, Install AVP, Connect to AVP, scroll and accept the agreement. Once that is completed, the G450 returns the board insertion failure and not able to connect to AVP through the Gateway. As stated above, I tried AVP 7.0, 7.1.3, and several G450 firmware versions all with the same results. I have the AVP working good in another, (much older) G450 on same firmware release. I can't find how to resolve or prevent these errors.


Avaya sent the following info below, I wasn't able to get the workaround to work.


kill -9 `pidof watchd` && /etc/init.d/watchd start


[AVP-750] s8300 not recognized in gateway and applications Created: 10/Jan/19 Updated: 16/Apr/19

Status: In Progress
Project: Appliance Virtualization Platform

Component/s: AvayaVIB

Affects Version/s: 7.1.3.2

Fix Version/s: 8.1.0.0, 7.1.3.4


Type: Bug Priority: P2 - Give High Attention
Reporter:
Assignee:

Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links: Split
Child AVP-775
[SPLIT:8.1.0.0] s8300 not recognized ... Open
Child AVP-776
[SPLIT:7.1.3.4] s8300 not recognized ... Open

Severity: 3-Medium
Found By: External Customer (Worst) - Production
Confirmed Products: CP::Aura Core::Avaya Aura® Appliance Virtualization Platform-7.1.3.2 (P1602)
Problem Type: Breakage of Feature
Found In Release: 7.1.3.2

Target Release: 8.1.0.0, 7.1.3.4

Story Points: 5
Customer Ticket: 1-6O0B1AQ
Sprint: AVP/AVPU/US/ASDT/CMP Sprint 84, AVP/AVPU/US/ASDT/CMP Sprint 85, AVP/AVPU/US/ASDT/CMP Sprint 86
Customer Visible Symptom: --Replace this Note--

This field may be visible to external Avaya customers. When editing this field, be aware of any sensitive information.

--Replace this Note--
Minimum Config: --Replace this Note--

This field may be visible to external Avaya customers. When editing this field, be aware of any sensitive information.

--Replace this Note--

Description
Customer reported that in CM the "list config media-gateway x" command should incorrect data for S8300 D and E boards on AVP 7.1.3.2.0 and G450 firmware 39.12.0.
list configuration media-gateway 1
SYSTEM CONFIGURATION
Board Assigned Ports
Number Board Type Code Vintage u=unassigned t=tti p=psa
001V1 ICC MM S8300X HW00 FW000
001V2 DCP MM MM712AP HW04 FW015 01 u u u u u u u
001V3 ANA MM MM714AP HW05 FW073 01 u u u 05 u u u
001V9 MG-ANNOUNCEMENT VMM-ANN 01 02 03 04 05 06 07 08
09 10 11 12 13 14 15 16
And on the MG:
fresno-g430-001(super)# show mm
MEDIA MODULE DESCRIPTION: v1
-------------------------------------------------
Type : ICC
Description : ICC Media Module
Serial Number : UNKNOWN
HW Vintage : 0
HW Suffix : X
FW Version : 0
No. of ports : 2
Faults : No Fault Messages
I have this same issue in the lab on S8300 D and E on AVP 7.1.0, 7.1.2, 7.1.3 and 7.1.3.2 and on gateway firmware G450 39.12.0, G430 38.20.1, G450 on 38.21.1. I have tired on on other combinations.


Comments
Comment by [ 15/Feb/19 ]
Hi),
I checked in AVP 7.1.3.3 on S8300D card, card details as below.
~ # cat /proc/cajun
cajun_major=3
cajun_sc_irq=7
cajun_base=8160
cajun_slot=1
cajun_sc_count=1
cajun_bailout=80
cajun_fw_vintage=1
cajun_board_suffix=D
cajun_board_serial_number=11WZ04424064

in GW, G450 it is shown as,
solutionpune-SLS(super)# show mm 6
MEDIA MODULE DESCRIPTION: v6
-------------------------------------------------
Type : MM712
Description : DCP Media Module
Serial Number : 081615201628
HW Vintage : 7
HW Suffix : A
FW Version : 14
No. of ports : 8
Faults : No Fault Messages


This is an MGC controlled Media Module, check
the MGC for additional status information.


solutionpune-SLS(super)# show mm 1
MEDIA MODULE DESCRIPTION: v1
-------------------------------------------------
Type : ICC
Description : ICC Media Module
Serial Number : UNKNOWN
HW Vintage : 0
HW Suffix : X
FW Version : 0
No. of ports : 2
Faults : No Fault Messages


This is an MGC controlled Media Module, check
the MGC for additional status information.

May I know since which build, fields are displayed wrong. I do not have any history on it.
Comment by ) [ 06/Mar/19 ]
CPLD driver creates /proc/cajun file with hardware card details and watchd process reads these entries and sends it to Media gateway via heartbeat.
While booting up, watchd starts first and then CPLD drivers is loaded.
# grep -e "watchd " -e "cpld" syslog.log
2019-03-06T19:43:03Z root: init Running watchd start
2019-03-06T19:43:08Z root: init Running 90.cplduser.sh

2019-03-06T19:43:09Z cplduser: cplduser: starting vmk_MgmtUserBegin

watchd thread exits when /proc/cajun file is absent. To fix this behavior, init sequence has to be reordered. As VMware ESXi take cares of startup watchd on every reboot and we have no control of it. So to fix this, either we should have way in watchd to wait till CPLD initializes or we have to restart watchd again after CPLD initializes.

Workaround -
1. Kill watchd process and start it again,
kill -9 `pidof watchd` && /etc/init.d/watchd start

 

[fondog2]

That is strange. I did see on the support site that doing the nvram init did fix one of these.

https://support.avaya.com/public/index?page=content&id=SOLN285478&group=UG_PUBLIC

This was another about they replaced the 8300.

https://support.avaya.com/public/index?page=content&id=SOLN308350&group=UG_PUBLIC

 

[PhonesAllDay]

I have seen both of those support documents.
There must be some combination of G450/S8300 hardware or software/firmware versions that won't work together.

I swapped the two S8300's and they both are working at the moment in the new G450's. (This is after removing/re-inserting several times that didn't work).
One thing I did differently on the G450 that was giving me the most trouble is I upgraded the AVP to 7.1.3 via the S8300 services port. Just about to deploy Utility Services via G450 LAN port and so far I'm not seeing the error.

 

[fondog2]

That could be the issue is the services port installation. On all the servers I bring up I only use the services port to deploy the AVP. Once that has finished and I except the EULA then I flip it to the network. That's 8300, DL360's or the Dell servers. I also found that it is faster to deploy to the current version than go through upgrading. I've caught errors on some upgrades. Glad you fixed it and good idea to swap the 8300's.