Nortel 4.5 SIP Major problem

[NZNewbie]

We have recommissioned a OPT 11 Branch Office 6mnths after the site was evacuated due to earthquake. The site has SIP trunk tie lines. After the was put back into production the hard drive failed on the Sig Server. This was replaced and then we noticed errors. We patched for this and within a week we started loosing the SIP. what we found is this:

NRS shows the site as registered
LD 117 STAT SERV shows all links up
LD 96 STAT DCH 30 shows D Channel down
Can not browse to the sig server via IP network
Can not browse to the sig server via crossover cable
Can browse to the sig server via serial port
Reset of Call Server no effect.
Reset of Sig Server will resolve
Able to ping Elan / TLan of local and Main Office from serial of Sig Server
Can ping ELAN / TLAN of Branch office from Main Office
RDP Logs show no other events as a build up other than the system is working then PRI224 PRI226 errors and its not working there is no build up of errors.

The last effort we did was Deleted the current node - 301 and saved this change to remove any possibility of the existing file corrupting a new setup.
Powered all devices down (taking the opportunity to inspect main processor card for any living or dead furry animals - all looked well feed and happy).
Replaced MC32 card and with a blank flash card again to avoid any previous contamination.
Setup new HUB for all Elan cables and thus removing the Elan off the customer's switches, again hopefully avoiding any potential issues with this kind of setup. Re-cabled linking cables in a less than temporary fashion and labelled accordingly.
Installed new Signal Server (removing old Signal server though this is still in the server room just in case).
Powered all devices back up and ensured local Extns and PSTN operation all working fine.
Connected to new signal server and rebuilt the node 301. Saved and transferred. Once re-established connectivity, then loaded the correct level of patching to both the MC32 card and the signal server.
Tested OK

Then the next day it failed again. It is failing at least 2 ~ 3 times a day and with no fixed pattern. to date we have done the following:

Changed Sig Server Hard Drive x2
Changed Sig Server x 3
Patched Sig Server
Patched MC32 card
Patched Call Server
Removed all patches from the Sig Server
Patched to match Tauranga Branch Office
Repatched to latest DEP List
Swapped MC32 card +Flash Card and Patched
Swapped Cables
Separated the E LAN from the YPG Network
Removed Sig Server off UPS
Put Sig Server back onto UPS

We are beyond grasping at straws here. Any clues???

 

[KillianK]

Maybe have check the data switch that option 11 is connected to, run some stats on the ports for errors? Change out the patch cables maybe? or do you think this problem is more to do with the option 11?

 

[NZNewbie]

Tried that Killian.... absolutly nothing, as a matter of fact when we loose the SIP network shows everything up and happy and no errors.

We may have foud it using Wireshark. We detected that another MAC Address was trying to log onto the TLAN and the NODE IP. Problem is we havent been able to isolate where the MAC Address is coming from. its an INTEL MAC Address but is definatly differant to the TLAN NIC which is also an INTEL MAC address. We think it might be someone with a laptop who has the same IP Addresses set up for 1st / 2nd choice for a home or private network with DHCP as 3rd choice. These people are in and out of the office a lot which is why it happens at random times.

If we prove it is this I'll post it incase anyone else in the world has the same problem occur, which is very unlikely as stuff like this only ever seems to happen to me.

 

[mugen112]

Nortel CS1000 SIP was really buggy up until around Release 5.5....good luck with getting it to run on R4.5!!! Once you get it working just don't breathe on the switch...

http://voipdesigner.blogspot.com/

Jason Black, Senior Design Specialist

 

[NZNewbie]

Thanks Mugen...

The instal set upo in 2007 went perfect with Main office 5 Branch Offices and 6 SRGs and since then we have had no problems at all... Just since we tried recommissioning this site after earthquake we started having problems... all other sites are OK

We are going to change the TLAN and NODE IP addresses and see what happens