NCP500 crashed

[keypulse]

Hi

I recently moved an NCP500, upgraded from V4 to V8, added a DSP and moved onto SIP (previously on BRI). I had converted the old database using PBX Unified V7.8.2.0, removed the SD card from the NCP and wrote V8.02.04 firmware over the card, replaced card, switched to initialise, powered on the NCP, switched to Run/Normal when Run light stabilised. Then logged in, skipped the wizards and wrote the new database and activation key to the card.

All seemed fine apart from a few glitches with the SIP (router issues, managed by provider), then the company went into coronavirus lockdown and all their DDIs were diverted on the SIP platform to their mobiles.

A few people returned to work this week and the lines were playing up again. I could only access the NCP remotely and noticed in the log an error 122 - Virtual card start up error. I remotely performed a reboot and the system stopped responding after that.

I will go to site tomorrow and have built a new database from scratch, again using PBX Unified V7.8.2.0. I have formatted an SD card using the SD format tool from the dealer site and written V8.02.04 to it. The card formatted as FAT

Is there anything I'm missing here? I'm really hoping it's just a database or SD card corruption (and not the MPR card broken). I think I may have formatted the original Panasonic SD card using Windows (not the proper tool) during the upgrade which may have caused this.

Are my PBX Unified/Firmware versions compatible? Has anyone had an MPR card die on them before and if that's happened is there some way of transferring the activation keys from the dead MPR to a replacement one?

Thanks...

 

[obtsystems]

Do you have sip extension card installed and did you change the sip port on the sip trunk port

Do you register or is it a peer trunk with the sip provider

 

[keypulse]

Hi obtsystems

No V-SIPEXT card is installed. The router is managed by the provider (Gamma) and their port mapping 5060<>35060 wouldn't work properly so I thought it easier all round to just change the SIP client port on the V-SIPGW to 5060

The trunks register via static IP, there is no Authentication Password or Registrar Server

 

[obtsystems]

Since the changes and reboot the system is the sip truck card not coming into service or failing to start

 

[keypulse]

The SIP trunk card comes into service fine. I cleared the system this morning and put the new database onto it, carried out tests and spent some time making changes. Everything was working fine till 5 minutes after I left site (I'm back there now) when one-way speech started again on the SIP trunks on incoming and outgoing calls (outside party can hear people on site, people on site cannot hear outside called/calling parties).

There is nothing in the error log. Not sure where to go from here.

 

[keypulse]

Taking the V-SIPGW card in and out of service does nothing. While this is happening (1 way speech on all trunks) I am able to ping the MPR but not the DSP. After a reset I can then ping the DSP again but I don't know how long before it goes wrong again.

 

[obtsystems]

Sounds like an IP is clashing with the dsp.

Make sure it is not in the dhcp pool range

If it clashes it can stop working until it rebooted

Other thing is to be sure port 5060 is locked to gamma ip. You could be getting bombarded with requests.

Mirror the Maintence port and do s wireshark trace to see what’s happening

 

[kizo]

maybe some of attached it helps you

 

[keypulse]

Thanks both.

obtsystems; the guy who looks after the IT has reserved me a range outside of the DHCP pool. That doesn't stop anyone changing a device to static but I've seen no evidence of this (yet!). This is a Gamma managed router on a Gamma Assured service so everything should be locked to their IP (and mine for maintenance). The service has been stable for around 32 hours now, Wireshark captures will be the next step if it falls over again.

kizo; thanks for the documents. Can confirm I changed the SIP client port to 5060 and Register Ability to Disable as in the first pdf.
The Word document I have seen before but the NAT is done differently:
WAN 5060 to MPR 5060 UDP (but should this also be TCP?)
WAN 16000-16511 to DSP 16000-16511 UDP
WAN 5300 to MPR 5300 TCP & UDP (maintenance)
The last 3 documents I'll keep as they'll be useful but in this case calls work perfectly for some time, until they don't and the DSP port seems to lock up.

Thanks again for your replies.

 

[obtsystems]

Is there anything in the error logs

5060 is udp only as is the 16000 range

 

[kizo]

After changing the SIP parameters, the PBX should be reset with Utility/Reset PBX commands

 

[keypulse]

obtsystems; nothing in the error log since the reboots on Monday.
kizo; it's been rebooted many times!

Well it seems to have been stable for over 2 days now but I doubt this is resolved yet so I'll just keep a close eye on it. Thanks for the suggestions both.

 

[obtsystems]

Hopefully it was something causing a storm on the switches and has been removed

Best of luck, I don’t like something just starting to work. But sometimes it just happens

 

[keypulse]

This morning, after a whole week it started acting up again! One-way audio (cannot hear outside party). I have been placing test calls and they were working up to late last night.

The customer reset the Gamma router, this time this made no difference. We made a log of failed calls for Gamma to look at.

Logging into the NCP remotely: There was nothing in the NCP's error log. I restarted the VIPGW16 card, this produced 391 *10101 to *10106 (SIP channels 1 to 6) "Data Link established" logs, but the calls were still bad.

I then rebooted the NCP, this fixed the calls and produced a 2 10000 "System Restart" plus the 6 x "Data Link established" logs as above.

I took PRTSIPC files from the NCP before and after the reboot. They don't seem to show any errors but they only seem to show traffic on the MPR IP address, nothing for the DSP IP (even when calls are working).

I've been assured by the IT guy that there is nothing that can be clashing IP wise but I'm starting to question this again. What's the best way to run a capture to find this out?

The network is 192.168.16.0/24
The site has 18 x DPT and 8 x IPT phones
DHCP router (for PCs etc.) is 192.168.16.1
I have 192.168.16.160 - 192.168.16.180 reserved for telephony (outside of DCHP range)
MPR is 192.168.16.160
DSP is 192.168.16.161
Phones are 192.168.16.162 - 192.168.16.169
Gamma router is 192.168.16.180
The customer is plugging some PCs through the IP phones, due to lack of cabling. 6 of the IP phones are in a separate building served by a Line-of-sight link. Were it not for this I'd have just put all the telephony on a separate network by now

The IP phones all plug into network switches as does the Gamma router and the NCP500. I can bring a switch that does port mirroring. Can I sit this switch between the NCP and the existing switch, set those two ports on my switch to "Mirror" and plug a laptop running Wireshark into a third port on my switch set to "Monitor", or is there a better way to capture?

Thanks!

 

[kizo]

Which DSP model do you have?
Attached is Panasonics instruction for Gamma ( for example).
Connect to PBX with Programmer code *PCCKXTDA* and your Password.
Go to Utility , click "ASCII mode" at the bottom, set command LANMON and follow instruction.
MNT port becames a Mirror port of LAN port

 

[keypulse]

Hi Kizo.

Thanks for the info. It's a DSP4
The trunks were fine since the reboot on Monday. Yesterday I went to site and we changed the addresses on the DSP, the MPR and the Gamma router yesterday on the off-chance the problem was an intermittent IP clash but the problem happened again overnight, no audio on calls this morning.

I took captures through the monitor port throughout the day and looking back through some of them, on the ones taken not long before I left site there are pages and pages of ARP broadcasts concerning a duplicate IP that isn't even on the subnet. Some ZyXEL WAPs were installed at the site at the same time the PBX was moved onto SIP, along with new switches.

Is it possible for traffic like that to knock the DSP card out?

 

[obtsystems]

What IP address are you using on the system

What is that ip, is it a switch?

 

[keypulse]

What IP address are you using on the system

What is that ip, is it a switch?

MPR 192.168.16.170
DSP 192.168.16.171
Gamma router 192.168.16.172

If you mean the duplicate IP (192.168.1.2) I'm still waiting for their IT guy to get back to me on that but I think it likely the new ZyXEL WAPS...

Introduction
To set the NWA/WAC to be managed by an AP controller in a different subnet or change between
management modes, use the AC (AP Controller) D iscove r y screen (see Section 5.4 on page 66).
Table 3 NWA/WAC Management Mode Comparison
MANAGEMENT MODE
Standalone AP
Managed AP
DEFAULT IP ADDRESS
UPLOAD FIRMWARE VIA
Dynamic or
Static (192.168.1.2)

https://fccid.io/I88NWA5123-AC/User-Manual/Users-Manual-1-2784271

 

[obtsystems]

If you filter the wireshark traces to the dsp what do you see. That is a broadcast and probably not affecting you as it is not system or gateway address

 

[keypulse]

If you filter the wireshark traces to the dsp what do you see

Sorry for the slow reply. I replaced the DSP last week, all was fine for 4 days then the DSP locked up again on Tuesday. When this happens the SIP trunks can be used, but only one-way audio. Same with the IP phones. DSP IP doesn't respond to ping. A reboot has it up and running again, till next time it happens. So replacing the DSP made no difference and I guess at least that's ruled out.

Here are some examples from the Wireshark trace filtered to the DSP. The capture was taken when everything was working. 88.215.51.228 is Gamma, 192.168.16.171 is the DSP and 192.168.16.162 is an IP phone.

Frame 1: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits) on interface en0, id 0
    Interface id: 0 (en0)
    Encapsulation type: Ethernet (1)
    Arrival Time: May 13, 2020 15:56:13.382647000 GMT Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1589381773.382647000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 214 bytes (1712 bits)
    Capture Length: 214 bytes (1712 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:data]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: Cisco_64:70:8c (c4:f7:d5:64:70:8c), Dst: Panasoni_c6:17:b2 (00:80:f0:c6:17:b2)
    Destination: Panasoni_c6:17:b2 (00:80:f0:c6:17:b2)
    Source: Cisco_64:70:8c (c4:f7:d5:64:70:8c)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 88.215.51.228, Dst: 192.168.16.171
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xb8 (DSCP: EF PHB, ECN: Not-ECT)
    Total Length: 200
    Identification: 0x06d2 (1746)
    Flags: 0x4000, Don't fragment
    Fragment offset: 0
    Time to live: 248
    Protocol: UDP (17)
    Header checksum: 0x1c8c [validation disabled]
    [Header checksum status: Unverified]
    Source: 88.215.51.228
    Destination: 192.168.16.171
User Datagram Protocol, Src Port: 6014, Dst Port: 16000
    Source Port: 6014
    Destination Port: 16000
    Length: 180
    Checksum: 0xea3c [unverified]
    [Checksum Status: Unverified]
    [Stream index: 0]
    [Timestamps]
Data (172 bytes)
    Data: 80080ae47449c3b01071ba11d5d5d5d5d5d5d5d5d5d5d5d5…
    [Length: 172]

Frame 78664: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits) on interface en0, id 0
    Interface id: 0 (en0)
    Encapsulation type: Ethernet (1)
    Arrival Time: May 13, 2020 16:02:28.069893000 GMT Daylight Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1589382148.069893000 seconds
    [Time delta from previous captured frame: 0.000007000 seconds]
    [Time delta from previous displayed frame: 0.000007000 seconds]
    [Time since reference or first frame: 374.687246000 seconds]
    Frame Number: 78664
    Frame Length: 214 bytes (1712 bits)
    Capture Length: 214 bytes (1712 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:rtp]
    [Coloring Rule Name: UDP]
    [Coloring Rule String: udp]
Ethernet II, Src: Panasoni_c6:17:b2 (00:80:f0:c6:17:b2), Dst: Panasoni_d4:74:d8 (00:80:f0:d4:74:d8)
    Destination: Panasoni_d4:74:d8 (00:80:f0:d4:74:d8)
    Source: Panasoni_c6:17:b2 (00:80:f0:c6:17:b2)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.16.171, Dst: 192.168.16.162
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
    Total Length: 200
    Identification: 0x1e59 (7769)
    Flags: 0x4000, Don't fragment
    Fragment offset: 0
    Time to live: 128
    Protocol: UDP (17)
    Header checksum: 0x392e [validation disabled]
    [Header checksum status: Unverified]
    Source: 192.168.16.171
    Destination: 192.168.16.162
User Datagram Protocol, Src Port: 12064, Dst Port: 8000
    Source Port: 12064
    Destination Port: 8000
    Length: 180
    Checksum: 0x174a [unverified]
    [Checksum Status: Unverified]
    [Stream index: 188]
    [Timestamps]
Real-Time Transport Protocol