Assistance with BCM450 Troubleshoot - Boot Loop - FPGA Configuration Error...

[rhol.bll]

Hey folks. Looking for some assistance in troubleshooting an issue with an old BCM450 still in production at one of our sites. This BCM450 suddenly rebooted itself one morning and got stuck in a boot loop (initial observation).

At about 9AM into the workday, I got the notification that it had gone down. I checked the alarms, which are emailed, and there was no warning. Last issue was a bad password signing into the system via BEM weeks ago.

History: This unit had issues randomly rebooting about four years ago. Turned out the PSU was the culprit. We pulled it, tested it, and it failed. We ordered a replacement PSU and it's been solid since.

Considering the last issue, I immediately pulled the PSU and tested it with an ATX PSU tester on a load, it tested fine.

Since the PSU doesn't seem to be the issue, I put it back in and booted up the system with access to the console. I'm seeing a good POST and it's reading the HDD. However, when it gets to the FPGA configuration, it fails. I have a multi-image HDD, a replacement HDD, but it's R6, and a clone of this drive. So far, replacing the drive with its clone has the same result. Trying the R6 disk also has the same result. We have not tested with the multi-image HDD yet.


EDIT - Boot Loop Video:

We have current weekly backups up to 8/12/2024. The system is set to reboot itself every week in the evening (1AM) after a backup (11PM).

BFT Tage: BCM450 BFT NTC01050

Original System HDD: LBNNTMEND8N37G NTC03101SXE6 01 N0202621 BCM450 R5 Programmed Hard Drive (Seagate Barracuda 7200RPM SATA 80GB - OEM - came with the system).

We have an BCM450 R6 Replacement HDD (160GB) and a Multi-Image BCM450 HDD (250GB).

I have ordered a replacement PSU just in case.


Any assistance or nudge in the right direction for next steps would be appreciated. Below is what I'm seeing in the console and what BEM has recorded for our system (BEM About):


For reference, here's what I'm seeing in the console:


BCM U-Boot 1.2.2 (Jul 8 2010 - 14:37:07), Build: hudson-BCM_5_Sustaining_M450-2 9

CPU: 8567_E, Version: 1.1, (0x807d0111)
Core: E500, Version: 2.2, (0x80210022)
Clock Configuration:
CPU:1333 MHz, CCB: 533 MHz,
DDR: 266 MHz, LBC: 33 MHz
QE: 533 MHz
L1: D-cache 32 kB enabled
I-cache 32 kB enabled
Board: BMB450 R1.0
I2C: ready
DRAM: Initializing
DDR: 512 MB
POST: running...
POST memory PASSED
L2 cache 512KB: enabled
Extended Memory test: Passed
FLASH: 8 MB
Reboot Reason: Power Recycle
Configuring Mac address from NsSecSi PROM
PCI: first=0 last=0
PCIe: LTTSM=0x16
PCIe: first=1 last=1
In: serial
Out: serial
Err: serial
Net: FSL UEC1, FSL UEC0, FSL UEC2, FSL UEC3, FSL UEC4, FSL UEC5, FSL UEC6, FSL UEC7
IDE: Bus 0: OK
Device 0: Model: ST31000524NS Firm: 130C Ser#: 9WK54CTC
Type: Hard Disk
Capacity: 131071.9 MB = 127.9 GB (268435455 x 512)
Device 1: not available
start USB...
USB: EHCI pci controller (1131, 1562) found @(0:1:2)
RTRY/TRDY timeout set to 00000000 from 00008000
EHCI regs address 0x80002020
OHCI pci controller (1131, 1561) found @(0:1:0)
OHCI regs address 0x80000000
scanning bus for devices... 2 USB Device(s) found
scanning bus for storage devices...
probing USB device 0...
probing USB device 1...
1 Storage Device(s) found
reading m450_fpga.image

124284 bytes read from ide ,finished....
Image Name: M450 MAIN: rel 91
Created: 2008-09-05 14:15:40 UTC
Image Type: PowerPC Unknown OS Firmware (gzip compressed)
Data Size: 124220 Bytes = 121.3 kB
Load Address: 00010000
Entry Point: 00000000
Verifying Checksum ... OK
Uncompressing fpga Firmware ... OK


Configuring The FPGA Device .................................................... ................................................................................ ................................................................................ ................................................................................ ............................
Test failed: FPGA Done not clear clocking 16 more times


Test failed: FPGA Done still not clear


Tried to write 480131 bytes...

ERROR: WriteConfigDataToFPGA()
Failed at download fpga image
Fatal Error Occurred
Hit any key to stop autoboot: 16
=>



For reference, the BEM About:

Nortel Business Element Manager Version 60.3.0
View [URL='

https://www.tek-tips.com/file://C:\...er\BusinessElementManager\bin/../README.html'

]README[/URL]
View [URL='

https://www.tek-tips.com/file://C:\...r\BusinessElementManager\bin/../LICENSE.html'

]License Agreement[/URL]
Copyright (c) 2005-2024 [URL='

http://www.nortel.com'

]Nortel[/URL] All rights reserved.

Installed Cartridges:
BCM 50R5 Element Manager Cartridge 5.0.0 (BCM50R5.aa94-0)

0.1.0 (JDM version:603b02_01 supports BES1xx/2xx and BES1xxx/2xxx devices)

BCM 450R5 Element Manager Cartridge 5.0.0 (BCM450R5.ab62-0)

BCM 50R6 Element Manager Cartridge 6.0.0 (BCM50R6.ab47-0)

Technical Support: http://www.nortel.com/support

Java Version: 1.6.0_07
Java Vendor: Sun Microsystems Inc.
Java Home: C:\Program Files\Nortel\Business Element Manager\jre
Java Class Version: 50.0
Java Class Path: C:\Program Files\Nortel\Business Element Manager\BusinessElementManager\lib\patch.zp;C:\Program Files\Nortel\Business Element Manager\BusinessElementManager\bin\run.jar;C:\Program Files\Nortel\Business Element Manager\BusinessElementManager\bin;C:\Program Files\Nortel\Business Element Manager\BusinessElementManager\server\default\conf;C:\Program Files\Nortel\Business Element Manager\BusinessElementManager\bin\lax.jar;
Operating System: Windows Vista
System Architecture: x86
Operating System Version: 6.2
Build ID: bcmsusemfw60aa07

 

[pccsipoffice]

I am not particularly familiar with these but I would say. Get your self grounded out with a wrist strap.
Spray air onto the board to make sure no debris. Next reseat different sockets and see if you have luck that way.
Hope this helps.

 

[Tendivski]

I too managed a couple of BCMs until a few years ago. You have done a good job narrowing down the problem. The error iwriting to the FPGA with multiple boot images points to the FPGA hardware. I think a spare BMC unit from eBay is the next troubleshooting and repair step, considering they are available quite cheap.

 

[curlycord]

Yes, purchase another 450 anyway, just in case needed in a hurry.

Do you have a list of patches installed?
There was one that addresses this FPGA.

Also make sure connections for the Interface card (small board between PS/BFT/ModBay) are good, or replace it.

 

[curlycord]

I see you are using and old version of BEM, BEM works with Windows 10 or 11.
[URL='

https://curlycord.com/files/temp/EM-62.0.3.zip'

]Download BEM here[/URL]

You can use the latest Java too but you need to do the security fix (I have this in the FAQ's too):
Edit the java.security file.
You may have to do this for every time java is updated, you can disable updates in java console if need.
Only necessary if you cannot attempt the above fixes at the time.

-Do NOT use Windows Explorer to do this.
-In Windows navigate to your Notepad icon and right click it's icon and select "Run as Administrator" (Win 10 users select More 1st).
-Using notepad select File then Select Open and navigate to Program Files/Java/YourVersion#/lib/security or 64BIT users "Program Files (x86)/Java/YourVersion#/lib/security
-In the far lower right hand side change "txt files" to "all files"
-Open java.security.
-Use the CTL + F command to start FIND
-Search for MD5 (ignore lines with # or Example at the beginning of the line) and remove these two references: MD5 (2 locations) and MD5withRSA (1 location), (Update Sept 2021) as for the line that has MD5with RSA you also need to remove TLSv1, TLSv1.1, as seen here : jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
Ignore the rest further down the page where you will see 3 in a row.
-click file/save file

 

[rhol.bll]

First, thank you all for all the replies and recommendations.

Update:

I had preemptively ordered a replacement R5 BFT from a telecom reseller on the cheap, same model, mostly as a crapshoot. That arrived yesterday afternoon. I also took the liberty of cloning the original HDD to an SSD, and one extra SSD clone for safe keeping. With all of that on hand, I got to work.

The refurbished BFT had a bad DIMM, so I swapped that with the one in the old BFT and that did the trick. I fired up the system with the replacement BFT and the cloned SSD, it all booted up without issue. I saw the FPGA configured successfully and saw some indication that the file system had been updated. I signed in with our credentials and I was presented with a menu with some choices to reboot into different modes.

Attached is "TEST 02 - BCM450 Boot with Ref. BFT and Cloned SSD - 08222024.txt", the entire console output on the first boot.

I rebooted to Main OS (option 1) then disconnected the null modem cable and hopped on to BEM. I have network access to the BCM, the phones came up, and I got to testing what features we had and did not have afterwards.

So far, general telephony is working. All handsets came up, we have internal extension to extension calling, and we have inbound/outbound calling. I immediately went to check for keycodes, and of course, those are all gone. We have no voicemail access or management (callers can leave voicemail) and our digital trunk to another site is not working. We dial using a 4-digit DN.

Here's the empty keycode listing:



Note: I did not perform a restore from backup. This all spun up from the cloned SSD. I'm unsure if I should do so before making any further changes.


Tendivski,

Good call on the board replacement.


Curlycord,

I have no documentation on the system prior to this trouble, no list of patches, and no configurations, just system backups. Here's what I'm seeing from BEM now:





I am going through my predecessor's email and files to find any order information regarding what features we purchased this with. I don't mind re-licensing the unit, but it's not knowing what we need that's the issue now.

Also, thank you for the updated BEM software, I'll get that installed and configured. I'm familiar with the JRE MD5 hack for older Java applications, thanks.

At this point, I'm looking for what features we had, how to acquire or restore keycodes, and wondering if restoring from backup is necessary at this point (I don't believe it is).


Any further advice would be much appreciated. Thanks again everyone.

 

[curlycord]

I will send you a PM on the keycodes since it's tied to the BFT.

As for patches, spread the NAME column further apart so we can see the version number, then take a screen shot.

 

[rhol.bll]

I will send you a PM on the keycodes since it's tied to the BFT.

As for patches, spread the NAME column further apart so we can see the version number, then take a screen shot.

I'll check your PM momentarily. Thank you.

The full version numbers are visible in their own "Version" column to the right of the "Name" column. I've spread them further apart on the Software Update History pane and there's no hidden information. See below:

 

[rhol.bll]

Update.

I did find the original configuration report. It has a list of Keycode features we're missing. Now begins this next adventure of acquiring these and applying them.

 

[curlycord]

Thanks, those patches are out of date so I can supply those too, 4 of them.
And yes, my blind for not seeing the bloody version column.

 

[Firebird Scrambler]

I've had a few BCM450 systems where they randomly lost power. We found no reason why it happened as mentioned above.

By pure chance, the fault was on the tiny circuit board that sits in the middle of the chassis that connects all the other devices such as the Motherboard, modules and power supply.

This small card has many soldered connections and we discovered that the BCM got hot, causing one of the soldered pins to melt causing a high resistance and this shut down the phone system.

Once it cooled down, the system would power up again.

 

[curlycord]

The OP must have sorted this out as it's been 10 days now without an update.

 

[rhol.bll]

I've had a few BCM450 systems where they randomly lost power. We found no reason why it happened as mentioned above.

By pure chance, the fault was on the tiny circuit board that sits in the middle of the chassis that connects all the other devices such as the Motherboard, modules and power supply.

This small card has many soldered connections and we discovered that the BCM got hot, causing one of the soldered pins to melt causing a high resistance and this shut down the phone system.

Once it cooled down, the system would power up again.

Firebird,

Apologies, fell off of this issue and onto other issues. Circling back now. We're between finishing off this repair or replacing the whole system. Hence the pause in circling back.

As for what happened, the FPGA on the BFT was toast. It would fail its checksum test during POST over and over, refusing to boot. We replaced the BFT with a refurb and that brought us online. We've been "getting by" since.

 

[rhol.bll]

The OP must have sorted this out as it's been 10 days now without an update.

curlycord,

Followed up on your last PM. Thanks.