ICP 3300 ESM unresponsive

[KurpeusLondon]

Hi guys,

I've one ICP 3300 that's his Web management interface unresponsive. I would be tempted to hard reboot it but I learnt by myself you should never do that (A previous attempts corrupted the DBs and blanked the controller config).

I believe you can reboot it using the VXworks command line but I would rather do that in last resort. (Not sure of the outcome.)


I tried to access it from the second cluster member but no joy. Also the cluster alert group is showing a Major alarm. No details about it.

Anything else I could do ? restarting a service or so ?

Would someone has any documentation about the command line ?

Cheers,

 

[LoopyLou]

You might be able to restart the web interface service but I think you need to reboot.

The single biggest problem with communications is the illusion that it has taken place.

 

[CharlesBr]

in the command line you could simply type "reboot", of course it'll reset the whole thing but it should not clear the DB unless the DBMS flag is off..

 

[james1982]

We have had this previosuly.

manually restarting the voicemail resolved it for us

use putty or a secure telnet tool and login to the 3300 IP using RAW, port 2002

login using normal username and password

you will be prompted with a ->

to restart voicemail without rebooting the system enter

iPVM_Stop

iPVM_Start

 

[KurpeusLondon]

Hmm that's interresting


I tried to call the VM huntgroup. It keeps ringing without picking up the call.

Using RAW doesn't seem to go anywhere neither. I'm being prompted for username and password and then nothing.

Can I do this iPVM_Stop from the console access ?

Thanks,

 

[james1982]

dont think you can, it may be SSH avtually not RAW

 

[james1982]

you should see, after iPVM_Stop press "enter", wait for the "iPVM Stopped" Message, then do the iPVM_Start

login: system
system
Password:


Shell restarted
-> iPVM_Stop
iPVM_Stop


iPVM stopped.

value = 0 = 0x0
-> iPVM_Start
iPVM_Start
Starting iPVM Version 10.0.1.2

value = 0 = 0x0

 

[KurpeusLondon]

Unfortunatelly I don't go that far.

I got the prompt:
login: system
Password:

and when I press enter nothing happens. If I wait, the window eventually crashes and then I need to wait for the tcp timeout before being able to attempt another connection.

I might have to go for the reboot then.

 

[Supernova99]

you are in the sx2000 telnet session
you need to use putty RAW on port 2002

Just type reboot at the > prompt

Share what you know - Learn what you don't

 

[KurpeusLondon]

Supermova, I used putty/raw port 2002 when I tried to log in.


I had to console in and I tried to reboot it but nothing happened.

I waited 10 mins and still nothing happening. I wasn't neither prompted for anything

-> reboot

While I was waiting I kept having these frightening messages:

0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): disk cache error: device 1d7d33a8 block 459 errno c0003, disk read failed
0x579d7c0 (tAtaCallLogger): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x579d7c0 (tAtaCallLogger): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x579d7c0 (tAtaCallLogger): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x579d7c0 (tAtaCallLogger): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x579d7c0 (tAtaCallLogger): disk cache error: device 1d7d33a8 block 15 errno c0003, disk read failed
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): disk cache error: device 1d7d33a8 block 17 errno c0003, disk read failed

As nothing else happened after another 10 mins I tried Ctrl+C

It looks like it killed few processes:

1a14118 vxTaskEntry +68 : shell ()
254374 shell +1c4: 254394 ()
25453c shell +38c: execute ()
2546a8 execute +d0 : yyparse ()
181417c yyparse +6e0: 181255c ()
18126e8 yystart +984: reboot ()
17fed4c reboot +4c : ftpdDelete ()
511f0 ftpdDelete +68 : taskPrioritySet ()
tShell restarted.

But still no joy. I kept waiting and seeing these messages.

0x5796c80 (SIPAUX_Service): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x5796c80 (SIPAUX_Service): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x5796c80 (SIPAUX_Service): disk cache error: device 1d7d33a8 block 459 errno c0003, disk read failed
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): read DMA err: dma status=0x21 regValue=0x0a01 semStatus=-1 PCI Status=0x00 aSTATUS=0xd0
0x3d54f80 (Emu68k_ST): disk cache error: device 1d7d33a8 block 16 errno c0003, disk read failed


I had to hard reset it. It has come back up fine, no DB corrupted.

From the second cluster member, it can see the cluster lost connectivity with this member around midnight and "came back up" 4 min later. Normally I wouldn't pay attention to that as we have a site to site vpn between the two cluster members and occasionnaly the vpn drop for few min but our management plateform has picked up network alerts eversince midnight.

I wonder if the controller had crashed or that some of it services became unresponsive (For sure http service was down. Snmp service went down for 1.5 hour, then came back up for another hour and finally stayed down for 10hours). It doesn't look like the telephony features were affected at the begenning(except for the embedded voicemail ) but it's hard to say as all my trunks are somehow resilients)

I don't like these error messages but the bootup messages displayed no errors after I hard resetted it. Any thoughts about this ?

 

[Supernova99]

These are hard drive access errors so it looks like the reboot cleared these

Share what you know - Learn what you don't