BCM450 in continual reboot cycle - caused by UPS

[pkellow]

BCM450 in continual reboot cycle.

Various LEDs indicated booting, however would settle on
Top LED = orange solid
Bottom LED = solid green
These indicate kernal mode (8 seconds) then safe OS.
Then reboot again.

Briefly I could PING 10.10.11.1 through the OAM port.

Any effort to boot to Main OS (using serial connection) would fail (including both boot to Main OS and transition to Main OS), and BCM would boot and always return to safe mode.

All MBMs flashing up like they are receiving power. No symptoms of a power problem whatsoever.

Putty indicated following (this is just an extract trying to force to Main OS, however similar messages seen just on power cycle):

<BEGIN>
You have selected Transition to Main OS.
Do you wish to proceed? [Y/N]? : yUnlocking flash...Done
Erasing flash...Done
Writing environment to /dev/mtd3...Done
Locking ...DoneMain OS
Swapping to mainos
Entering read_request...
mainos
kjournald starting. Commit interval 5 seconds
EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,26), internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Unlocking flash...Done
Erasing flash...Done
Writing environment to /dev/mtd2...Done
Locking ...DoneUnlocking flash...Done
Warning: DQ5 raised while erase operation was in progress, but erase completed OK
Erasing flash...Done
Writing environment to /dev/mtd3...Done
Locking ...Donereset_request=mainos
Entering mainos_chroot...
umount: /dev/sdb3: not mounted
Stopping portmap...
Stopping portmapper: [ OK ]

Stopping sshd...
[ OK ]
Stopping network...
Shutting down interface eth1: [ OK ]

Shutting down interface eth2: [ OK ]

Shutting down loopback interface: [ OK ]

stopping syslog...
Shutting down kernel logger: [ OK ]

Shutting down system logger: [ OK ]

Unloading modules...
Removing kernel loadable modules, if possible and if unused.output from nn_gpiol driver exiting
output from nn_gpiol driver exiting
kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,19), internal journal
EXT3-fs: mounted filesystem with ordered data mode.
modprobe: modprobe: Can't open dependencies file /lib/modules/2.4.22-ncgl-11.45.1.0/modules.dep (No such file or directory)


INIT: version 2.84 booting

Welcome to NCGL NCGL 2005 (cjdavola)
Press 'I' to enter interactive startup.
Mounting proc filesystem: [ OK ]

Unmounting initrd: umount: /initrd: device is busy
[FAILED]

Setting hostname GOGPBCM450: [ OK ]

Your system appears to have shut down uncleanly

Press Y within 5 seconds to force file system integrity check...
Press Y within 4 seconds to force file system integrity check...
Press Y within 3 seconds to force file system integrity check...
Press Y within 2 seconds to force file system integrity check...
Press Y within 1 seconds to force file system integrity check...
Activating swap partitions: [ OK ]

Finding module dependencies: depmod: Can't open /lib/modules/2.4.22-ncgl-11.45.1.0/modules.dep for writing
[FAILED]

Mounting local filesystems: [ OK ]

Enabling swap space: [ OK ]
modprobe: Can't open dependencies file /lib/modules/2.4.22-ncgl-11.45.1.0/modules.dep (No such file or directory)
/sbin/ldconfig: /nn/lib/libcppunit-1.12.so.1 is not a symbolic link

INIT: Entering runlevel: 3

Entering non-interactive startup
Starting init_cleanup: [ OK ]

Starting system logger: [ OK ]

Starting kernel logger: [ OK ]

Starting nn_modules: [ OK ]

Setting network parameters: [ OK ]

Bringing up loopback interface: [ OK ]

Bringing up interface eth0: [ OK ]

Bringing up interface eth1: [ OK ]

Bringing up interface eth2: [ OK ]

Bringing up interface eth3: [ OK ]

Bringing up interface eth4: [ OK ]

Bringing up interface eth5: [ OK ]

Bringing up interface eth6: [ OK ]

modprobe: Can't open dependencies file /lib/modules/2.4.22-ncgl-11.45.1.0/modules.dep (No such file or directory)
Starting network_test: [ OK ]

Starting dhcpd: [ OK ]

Starting efs: [ OK ]

Initializing random number generator: [ OK ]

Starting xinetd: [ OK ]

[ OK ]
Starting UPS monitoring:[ OK ]

Starting DiaLogger: [ OK ]

Starting Pdrd: [ OK ]

Starting enigma: [ OK ]

Starting Pdrd_db_init: [ OK ]

Starting hc_sanity.450.1: [ OK ]

Starting invmgr: Checking dependencies for invmgr.
[FAILED]
<END>

Most notable is failure to start invmgr (I assume inventory manager). I assume the BCM was trying to power up the MBMs and running into a problem, causing the invmgr service to stall, so forcing the BCM into safe mode.

Despite changing the customer's HDD (thinking there was a corruption) and changing the BFT, still problem occurred. I had a replacement unit shipped to site, powered it on (using a different power lead from the comms cabinet) using the customers' BFT & HDD, and all okay. This suggested problem with the PSU / CIF / MBM backplane.

However on rack mounting the BCM450, and using the customer's existing power chord, I had more problems. Traced the power lead to a UPS. Replaced power chord with one straight to raw mains, and all problems resolved.

I.e. Faulty UPS or faulty UPS cable was the root cause of all problems.

Where I ordinarily associate UPS to be a BCM's best friend, this has been a painful lesson that a faulty one can equally be their worst enemy causing system wide BCM problems.

Whenever I have a reboot problem in the future, even if the BCM looks like it is receiving power to all MBMs etc, if I trace the power lead to UPS, first thing I will do is go straight to raw mains and see if this resolves.

Hope this helps any other engineers in a similar situation.

regards
paul

 

[snowman50]

The bcm 450s i have had an issue with as you describe continous boot up loop, was related to the BCM power supply.
Changing the power unit resolved the issue.

The systems were conneced directly to the mains.

Thats good info and may be something to look out for if my customers use UPS.

 

[talkunity1]

Out of curiosity, what kind of UPS was the BCM450 connected to? A "line-interactive" UPS that uses a relay for transfer and may or may not have a switchable transformer scheme for minor undervoltage/overvoltage correction? Or a "true online" design that powers the load via the inverter all the time regardless of mains existence? Does the Customer have considerable electrical noise at their facility (i.e. large motors, welding equipment, etc.)?

The spec sheet on the power supply used in the BCM450 indicates it is very much like a commodity PC-style power supply, and they are very inexpensive in retail distribution. Has anyone else had BCM450 power supply issues in general, requiring replacement, especially with recently installed systems?

 

[gberger]

Same here. A defective power supply was causing a Bcm450R1 to reboot continuously.Replacing it fixed the issue.

 

[DigitelD]

I have also seen the power supply issue several times. I have also had to replace the hard drive as well as the power supply in some.

SHK Certified (School of Hard Knocks)
NCSS, ATSP/IP

 

[pkellow]

Hi, sorry for not responding, the email notification went to my spam folder. I reinterate, I temporarily floor mounted the replacement BCM using raw mains and all was perfect. I then rack mounted it and used the old power lead running through the UPS which reintroduced the problem.

Sorry, I don't have any information on the UPS, as we don't maintain it. There were no active motors nearby that would account for anything.

regards
paul

 

[pkellow]

Referring to my previous note, I would have suspected a PSU problem with the BCM itself, had both the original and replacement BCMs (with different power leads) not suffered problems through the UPS. The only commonality here was the UPS.

regards
paul

 

[pkellow]

sorry guys, turns out it was just the PSU in the BCM that had failed.