System restart last two days

[1norteltech]

For unknown reasons to me, our OPT. 61c rel 25.40 has restarted the last two days. This is the hst when it happened today.

INI000 0000002F 18 49 02 00000000 00000000 00000000 00000000 1 00000087
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ac, Addr=0x13123c80, SSW=0x105
%
% DSET000 DOWN LOAD 0 0 0 0 0 0 0 26837
%
% INI009
% ACDR ACTIVATED
%
% ERR5245 15 4 0 0 0
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ba, Addr=0x13123c80, SSW=0x105
%
% AML: 0 EST CONFIRM TIME: 18:49:20 ACTV_ST: EST TRGT_ST: EST
%
% CSA003 0 18:49:22 11/08/2009
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ba, Addr=0x13123c80, SSW=0x105
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x5018690, Addr=0x13123c00, SSW=0x85
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x5018690, Addr=0x13123c00, SSW=0x85
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x501869e, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x501869e, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x501869e, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x501869e, Addr=0x13123c80, SSW=0x105
%
% DCH1014 0 003FE0A5 00000106
%
% DCH: 0 EST CONFIRM TIME: 18:49:50 11/08/2009
%
% DCH: 10 EST CONFIRM TIME: 18:49:52 11/08/2009
%
% DCH: 3 EST CONFIRM TIME: 18:49:52 11/08/2009
%
% DCH: 19 EST CONFIRM TIME: 18:49:52 11/08/2009
%
% PRI227 DCH: 3 DATA: 0000000C
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ac, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ac, Addr=0x13123c80, SSW=0x105
%
% PRI225 DCH: 19 DATA: 0000000B
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ba, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ba, Addr=0x13123c80, SSW=0x105
%
% AUD000
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x5018690, Addr=0x13123c00, SSW=0x85
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x5018690, Addr=0x13123c00, SSW=0x85
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x501869e, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x501869e, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x501869e, Addr=0x13123c80, SSW=0x105
%
% BERR705
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ac, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ac, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ba, Addr=0x13123c80, SSW=0x105
%
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ba, Addr=0x13123c80, SSW=0x105
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% PRI225 DCH: 3 DATA: 0000000B
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% DTC001
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% BERR511 HI EXC 1: Restart of Task "tSL1" (0x8d125f0) was averted
% Unknown address 0x13123c80 has been remapped
% Number of remapped addresses: 1
% Total
%
% PRI225 DCH: 19 DATA: 0000000B
%
% SRPT782 RST 1: WARM START IN PROGRESS - Reason 15
%
% SRPT782 RST 1: WARM START IN PROGRESS - Reason 15
%
% TTY #10 LOGGED IN 18:53 11/8/2009
%
% SRPT731 OS 1: Warm Start
% Release: x112540b
% Created: Monday July 29 13:41:23 EDT 2002
% Last Cold Start: 0/0/00 0:01:50 (from '/flash/flashos')
%
%
% SRPT751 INI 1: starting INI on side 1 due to System Restart
% Previous INI: side 1 at 10/ 8/09 17:53:42
% INIs since switch-over (11/ 8/09 00:09:15): 1, 1
%
%
% SRPT752 INI 1: INI completed in 17 seconds
%

Anyone have any ideas?

 

[johnpoole]

nothing post ini shows a problem.. what does your screen look like just prior to the ini? that rls was mem short.. anything that uses to much ram can cause an ini... sets built and not plugged in.. numbers (scr) that ring on a dozen sets... any errors that repeat have to be taken care of... i had one back then that would die due to err080's all day.. (16's plugged in and 08's programed)..

i would test my standby cpu, scpu and repeat.. test all conf/tds cards, cni's etc... REMOVE all unused software... t1s or d's that were left in... a switch that age tends to have a lot of left in trash... (all test need to run on a low traffic timeslot)..

remember when your switch was new, we used 286 pc's with 20 meg hard drives.. if your not going to upgrade, you better do perfect maintance.. i saw a 61 last year running rls 12 still working at a no change cust...

john poole
bellsouth business
columbia,sc

 

[Tman45]

Well ... it would be nice if you had some history before this. I think you can go into PDT and somehow retrieve it?

I almost didn't catch this but you have an INI 009 (pry because the square text in front of it

% DSET000 DOWN LOAD 0 0 0 0 0 0 0 26837

% INI009

INI0009 xx Network faults seen from the standby CPU, where xx is the maintenance
display
code (see HEX). This is a minor alarm. This message only appears if the standby CPU may be used but with a
degradation of network or I/O access as shown by comparing INI0002 with
INI0008, INI0003 with INI0009, and INI0007 with INI0010.
Action:
Severity: Critical. Critical to Monitor: Yes. SNMP trap: Yes

 

[1norteltech]

John - We get alot of err80's when there is a storm brewing, which there was when these ini's occured. All the sets are correct, but are in a separate building. I believe it to be a grounding issue with the building, but no one can find anything. I will do the test you recommened when the traffic slows. I do not have any unused software. I removed all the unused T's D's Trunks, acd's ect. long ago.

Tman45 - There was nothing of note in the prior history. That's why I didn't include it, but here it is.
% TIM168 14:00 11/8/2009 CPU 1
%
% DTC001
%
% AUD000
%
% TIM168 14:23 11/8/2009 CPU 1
%
% AUD017 00000000 0000FFFE 0000FFDE
%
% AUD017 00000000 0000FFDE 0000FFFE
%
% TFS501 0 1 0
%
% AUD000
%
% DTC001
%
% TIM168 14:38 11/8/2009 CPU 1
%
% DTC001
%
% AUD000
%
% DTC001
%
% TIM168 15:00 11/8/2009 CPU 1
%
% DTC001
%
% AUD000
%
% DTC001
%
% PRI218 DCH: 10 DATA: 00000009
%
% TIM168 15:29 11/8/2009 CPU 1
%
% DTC001
%
% AUD000
%
% DTC001
%
% TIM168 16:00 11/8/2009 CPU 1
%
% DTC001
%
% AUD000
%
% DTC001
%
% DTC001
%
% TIM168 16:41 11/8/2009 CPU 1
%
% PRI218 DCH: 10 DATA: 00000009
%
% AUD000
%
% DTC001
%
% TIM168 17:00 11/8/2009 CPU 1
%
% DTC001
%
% TIM168 17:15 11/8/2009 CPU 1
%
% PRI218 DCH: 10 DATA: 00000009
%
% AUD000
%
% DTC001
%
% DTC001
%
% AUD000
%
% DTC001
%
% TIM168 18:00 11/8/2009 CPU 1
%
% DTC001
%
% AUD000
%
% DTC001
%
% DTC001
%
% AUD000
% INI000 0000002F 18 49 02 00000000 00000000 00000000 00000000 1 00000087
% BERR705 EXC 1: Bus Error in Task "tSL1" (0x8d125f0)
% SR=0x3010, PC=0x50186ac, Addr=0x13123c80, SSW=0x105
%
continues in first post.

 

[trvlr1]

Grounding is always a good place to start, but another place you can check quickly is your UPS if you have one. Make sure the batteries are holding a charge and are capable of supporting the PBX in the case of a loss or surge of power in the event of a storm, however short. Battery maintenance and replacement is an easy thing to forget and also an easy thing to cut out of the budget ....!

 

[1norteltech]

I actually just had the batteries tested last week, but thanks.

 

[allenmac]

Anytime you have digital or analog sets in different buildings from the main switch, proper grounding is paramount.

Improper grounds will cause havoc in your switch.

 

[1norteltech]

I agree. We had a grounding survey done awhile back and all recommendations were followed. There are 3 sets of fused blocks, one in the switch room, one in the IDF where it leaves the building, and one where it enters the other building. All have digital or analog fuses and all are grounded in each room. I've had it checked again and am told the grounding is good. But I still see errors when there is a bad lightning storm...go figure.

 

[allenmac]

Sounds like you are definitely getting electrical interference. Are your trunks copper & T1/PRI also grounded?

When I was running the streets, we used to have a ton of problems due to lightning coming in through the trunks, and after installing grounded blocks on the trunks, 99% of those failures went away.

I was really impressed with the blocks from Oneac. They had 66-type blocks for copper trunks, and a large 'biscuit' type pass-through jack for T1/PRI circuits. Both were very easy to install.

 

[1norteltech]

All T1's and trunks come in on fiber.

 

[trvlr1]

Going back to the UPS for a sec .... After having your batteries checked, have you made sure the unit has not been left in bypass, meaning no batteries supporting your equipment, just power directly from the wall?

 

[1norteltech]

Is not in bypass, only power from the wall.

 

[1norteltech]

cpu, cmdu, cni, conf tests ok. did notice it was running on core 0, but clock was running on 1. switched it to 0.