MC32s intermittent reboots (warm and cold) 1

[Tman45]

I have a question about MC32s PBX link resets on a CPPM Geo-redundant core system running Contact Center 5.5 ... what would likely cause them? Bad Hardware? Software Corruption? Other? We have all the latest and greatest patches (MGCs AH19, newest DEP for cores, firmwares for our phones, etc.) Here is our latest reset:

If anyone out there can give a description on what is actually occuring based on these messages it would be appreciated!

Here is the biggest downside to this happening - our PBX seems to get 'locked up' on this down 'locked' MC32s and will continue to try each agent in a linear fashion - well this down card shows idle and the DSPs idle in the switch but the DSPs are unusable 'locked' so each agent that the PBX tries to send the call to gets placed in Not Ready state and sometimes this issue clears on its own.

Any ideas?

[498] 06/9/2008 11:36:20 LOG0001 VLAYER: VAPI msp heartbeat failed device 2 retVal 0xfffef634, REBOOTING card
[497] 06/9/2008 11:32:55 LOG0003 VGW: vapiRespCallbackConfig: Free list is empty
[496] 06/9/2008 11:32:55 LOG0003 VGW: vapiReqIdGet timed-out on semTake
[495] 06/9/2008 11:32:50 LOG0004 VGW: channel 18 is closed by pendingTimeStamp timer times out. dspMode: Closing [494] 06/9/2008 11:32:50 LOG0003 VGW: vapiRespCallbackConfig: Free list is empty
[493] 06/9/2008 11:32:50 LOG0003 VGW: vapiReqIdGet timed-out on semTake
[492] 06/9/2008 11:32:45 LOG0004 VGW: channel 21 is closed by pendingTimeStamp timer times out. dspMode: Closing

I can also post more of the log messaging as above from the MC32S, but it is mostly lines 497 thru 492 (with varying channels) repeatedly until the reboot ...

My thought is that if these DSPs are being 'closed' shouldn't the switch see them as disabled and try another card ... its almost like there isn't logic in place to LET GO of this scenario and proceed on to a valid device.

 

[Tman45]

Anyone have MC32s expertise? I still don't have an answer except that Nortel may have a bug. Anyone else have an opinion? Thanks!

 

[tnphoneman]

What about the media cards.... Do they have the latest loadware and peps. These have nothing to do with the MGC.

Signature===========================================

http://www.greatwhitetechnologies.com

 

[Tman45]

Yep, they have the Nortel latest, but apparently Nortel has decided that the AH19 patch that has been applied to the MGCs needs to apply to our MC32s as well (MC32s design team is working with MGC design team). That is the 'bug' I was referring to - just wasn't sure if anyone had a similar reboot issue or could decipher anything further from the log I posted. Thought perhaps it is not a patch issue and something else. I have a theory about PacketCable's media gateway protocol
At least we have the Agent Not Ready patch in as of 11-25 because our DSPs were going offline in our trunk groups and somehow Contact Center would place each agent in Not Ready as it would get stuck on a DSP which fakes CC into thinking the agent is not available.

 

[MagnaRGP]

I have a similar problem including sets getting locked out, but this is the log that I get.
ITG Exception Handler
Exception: 16
Task : "tRTP" (0x027426b0)
Program Counter: 0x00ae11f8
Valid Flag : 0x00000007
Valid Flag : 0x00000007
ARM registers:
r0 = 0x2895fd9d r1 = 0x2895fd9d
r2 = 0x00000002 r3 = 0xfffffdba
r4 = 0x00000001 r5 = 0x00000001
r6 = 0x00000002 r7 = 0x0000000b
r8 = 0x00000001 r8 = 0x00000000
r10 = 0x00000000 r11 = 0x0000002a
r12 = 0x02742406 r13 = 0x02741f24
r14 = 0x004fd1ec r15 = 0x00ae11f8

 

[phones1]

please prt the part nuber of the VGMC, prt the level of the software and the firmware of the card.
Ive had some very strange issues between incompatibilitis of FW & SW ong both AB & AA cards

Remember if it doesn't work hit it harder

Scott UK

 

[Tman45]

Is this what you want phones1?
If I am missing something it would help me if you could instruct me as to which load and command. Thanks!

I find it EXTREMELY interesting that you point out an incompatibility issue is possible as here are our part numbers:

9 MGCs are:
NTDW65AAE5 Rls 3

All MC32s are:
NTDW60BBE5 Rls 1

2 MGCs (40 and 44)are:
NTDW60BAE5 Rls 3

I think what you will say next is that we are NOT supposed to have AA version firmware on our DBL1 and DBL2 where we have a MGC that is BA version ...

I also just caught something that our MGC 64 has DSP2AA12 on both, but I think that is our ONLY MGC with two 32 port daughterboards giving us 64 DSPs for that gateway.

REQ iss

VERSION 4021
RELEASE 5
ISSUE 00 W +
IDLE_SET_DISPLAY OTC

IPMG TYPE CSP/SW MSP APP FPGA BOOT DBL1 DBL2

4 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
8 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
12 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
16 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
20 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
24 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
28 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
32 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
36 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 N/A
40 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
44 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
52 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12
56 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 N/A
60 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 N/A
64 0 MGC AH19 AA07 AA05 AA13 AH09 DSP2AA12 DSP2AA12
68 0 MGC AH19 AA07 AA05 AA13 AH09 DSP1AA10 DSP2AA12

REQ ****

 

[MeridianT]

first i must tell u that u get wrong answer cause mc32 its mc32 ok ?
u cant load the loadware of mgc to the mc32 its bullshit it would never work its different card and the dsp hd&ld not the same as mc32 have .

well i can advice u to the few steps :

1.go to the mc32 by cli and format the flash by typing few commands .
2.after u did it go to the element manager then ip network go to node line and make edit to your node .
3.roll down to cards then open the +
put there the mac address of the elan and the ip address of the elan and tlan of the card u design before.
4. make sure u connect at the backplan of the mc32 slot the special connector and conect the elan and tlan to the switch.
5. now when everything connected and u still in the EM press submit not the SS will do the job and also make reset to the card .
6.important ! once u install the ss its come with firware of mc32 so u do everything from the EM .

if its work now u win if not try to replace the flash on the mc32 i have found few of the empty and i download all the software from nortel web site .

 

[Tman45]

MeridianT -

Yes, I think the correct answer is that we need a patch for our MC32S (which is similiar to what the AH19 patch fixes on the MGC cards) So, Nortel MGC design is working with Nortel MC32S design to figure out what code changes are needed to keep the MC32S from rebooting (as AH19 patch for the MGCs is working properly for us).

Sorry if that was a bit confusing in my post.

I would try to perform some of your steps, but it seems that you want me to basically wipe out the card and reinstall it. I am not familiar enough with the MC32S to attempt such steps, but we will make note of it in case the patch does not work.

Thank you for your input.

 

[MeridianT]

well no u dont need to take out the flash im saying just in case that the ss is couldnt load the software .

usually when u are doing it from ss the ss install the software and the right loadware then he also make the reboot and the card is register by him self of course remember to go to ld 14 and program the 32 channels so when u make reboot u can see in ld 117 by stat serv the channels register and that the card is working well .