3300 ICP Stale Tasks

[DV21]

I have a 3300 ICP that keeps rebooting because of a critical error due to stale tasks. I did successfully reload the 3300_icp.tar file from backup. Still has the same issue. I don't know how to find the log to tell me which tasks are failing. I can only get to the VxWorks prompt to manage anything.

After reading some other posts I am suspicious that the problem was caused by the 5340's we connected to the system before we started having problems.

The symptoms started with loss of dial tone, you could still call, a reboot resolved the problem once, but when it happened again the system started rebooting itself constantly. We got another 3300 and I added our hard drive with the database backup and ICP modules ,but still have the same problem, any suggestions?

Thanks

 

[IrwinMFletcher]

The software/maintenance logs should indicate what task is stale.

 

[DV21]

How do I get to the logs when it doesn't boot? Can I get to them from VxWorks prompt?

 

[DV21]

Looks like the task with the issue might be "E2T_VPKtFwd" Now what to do with it...

 

[DoubleUT]

The file that Mitel want to investigate this if you like is a file called pstwlog.db2 (or .db)
Make an FTP connection with the system, this is working as soon as VXworks is up.
You can get it of by typing:
cd "/db/database/files"
Enter that and it should bring you to where the file is located.
Copy the file and the reason for the stale task might be in there.

 

[DV21]

Thank you, I'll give it a try.

 

[IrwinMFletcher]

Sorry, I mis-read, I thought you had the logs from the previous failures.

It depends at what stage of the boot process it is at whether or not you can get some sort of logs. Trying to ftp them off before it reboots is an option as DoubleUT suggested. Other than that, post the last 10 lines or so from the shell before the reboot.

 

[DV21]

Oddly the first time I tried to boot it today after the weekend it got far enough to bring up the web page which I copied some of the logs and alarm details to a text file. That time it didn't reboot but the critical light kept blinking.

At that point I thought the critical alarm might be because it wasn't connected to the digital SX-2000 system, so I placed it back in the rack connected the fiber cables and booted it up again, this time it failed saying CPU hogging.

It appears to me that it errors inconsistently at different stages in the boot process, making it more difficult to pinpoint the issue(s). The first file is from the web page logs, the last two are from the following two boot processes.
************************************************************


Log Number: 919
Severity: Error
Date: 2011/Aug/28
Time: 23:20:16
Source: OPS Alarms DSA
Description: ForwardUDTMessageToOPS()->Cannot connect to UDTHANDLER
File Name and Line Number: ESM_OPSAlarms_DSA.cpp;1611

Log Number: 915
Severity: Error
Date: 2011/Aug/28
Time: 23:19:02
Source: AL -- IPVMAP
Description: ipvm_portcb(): DSP Port could not be obtained
File Name and Line Number: ipvm_devices.cpp;822

Log Number: 914
Severity: Error
Date: 2011/Aug/28
Time: 23:19:02
Source: DSP_API
Description: GetPort() - g_freeDspPortListSem == NULL
File Name and Line Number: CDspSP.cpp;933

Alarm Details

Category Name Alarm Level Total Unavailable% Minor Threshold Major Threshold Critical Threshold
Lines Major 174 27 >0 20 100
Per msg link Critical 1 100 - >0 100
Receivers Minor 60 20 >0 25 100
SFT Zones Critical 1 100 - >0 100
************************************************************

Clock Synchronisation SUCCESSFULLY started.

Task StartupHwRst being deleted while ACTIVE
Voice Mail Starting in 2 minutes.
Starting iPVM Version 7.30.02


hogMonTask CPU hogging detected @ MON AUG 29 00:27:29 2011
will get a windview trace

/db/spylog/ev290027.wvr will contain 15 seconds of Object and System Libraries Events (WV_CLASS_3)

alt usage: wvSpy duration_sec, log_name, event_class
<event class> 1=Context Switch 2=Task State Trans 3=Sys Lib Events

****************** Windview Tools ********************

TickTockInit(period in seconds) to enable system time in Wind View trace
TickTockKill()to disable system time in Wind View trace
0x3683f80 (hogMon): wvOn: wvBufId 161353264 wvUpPathId 161220944.

Done

hogMonTask CPU hogging detected @ MON AUG 29 00:27:44 2011
will get a windview trace

/db/spylog/ev290027.wvr will contain 15 seconds of Object and System Libraries Events (WV_CLASS_3)

alt usage: wvSpy duration_sec, log_name, event_class
<event class> 1=Context Switch 2=Task State Trans 3=Sys Lib Events

****************** Windview Tools ********************
****************************************************************

Clock Synchronisation SUCCESSFULLY started.

Voice Mail Starting in 2 minutes.

data access
Exception current instruction address: 0x0137f0a4
Machine Status Register: 0x0000b032
Data Access Register: 0x040c8000
Condition Register: 0x20000045
Data storage interrupt Register: 0x0000b032
Task: 0x366f0c0 "tViPER_IP"
OSA::CriticalSectionAcquire: Failed to take critical section.
- task suspended

ignore dup exception

data access
Exception current instruction address: 0x0137f0a4
Machine Status Register: 0x0000b032
Data Access Register: 0x040c8000
Condition Register: 0x20000045
Data storage interrupt Register: 0x0000b032


Thank you for your help

 

[IrwinMFletcher]

Looks like your DSP SP failed to start. What type of a system is it? If you have additional DSP modules, try to remove those and reboot. If they are part of the midplane, they you may be hosed.

 

[DV21]

Our system has two of Mitel p/n 50002979, it would make a lot of sense. We purchased another 3300 controller, but it didn't come with those modules. I tried booting it with one or the other but if the config needs both, then I need another. Any way to tell which mmc slot is giving the problem to narrow it down?

On a side note- where is it's configuration saved? On the hard drive? Or does it have an on board flash?

 

[IrwinMFletcher]

It would only need one at the most to boot, maybe none, again, depending on the controller type you have.

Look for some error logs earlier in the boot process from the ones that you've shown. They may show the slot number.

There is no configuration file, it auto-detects and boots everything.