500v2 9.1 - SIP Trunks broke overnight - No Incoming Calls

[AvantiWade]

Hello All,

I am having issues with my IP Office that started sometime yesterday. The system is running on r9.1 on a 500v2 attached to an app server with voicemail pro and One X Portal. The system is only running 3 users currently as we use it as a test bed until we are ready to migrate. One of the users commented that we haven't had any incoming calls today so I made a test call in and was directed to our Disaster Recover number that's setup with our SIP provider. Outbound calls work fine, I can see the SIP options messages between our provider and my system working fine but when I recieve and incoming call its just a stream of INVITE's that are not answered by the system. I verifed in the System Status Audit Trail that no changes were made related to the trunks or incoming call routes. I verifed no changes were made on the firewall, and temporarily put an allow all rule towards the 500v2 to verify the firewall wasn't blocking anything. I compared a known good config from the initial setup to my current config and found no changes related to ICR, Trunks, etc. I am at a loss for what the issue could be. I sent my SIP Provider an email requesting assistance with all of the same information listed above and the below snip from the SysMonitor. Does anyone have any input related to these SIP Messages? I can't see anything out of the ordinary but I am not fluent in SIP...

16:06:19 1308731mS SIP Rx: UDP SIPPROVIDER:5060 -> MYPUBLICIP:5060
INVITE sip:MYWORKNUMBER@MYPUBLICIP SIP/2.0
Record-Route: <sip:SIPPROVIDER;lr>
Record-Route: <sip:SIPPROVIDER;lr>
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK44ad.b446e0021ca14cc90030907c8d0b8432.0
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK44ad.edfba0e5c71f18b9b0e0f30328baf535.0
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK44ad.fec3d7e2.0
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK04Ba8c01281169a1d5c
From: "MYCELLPHONE " <sip:+MYCELLPHONENUMBER@4.55.11.163:5060;isup-oli=62>;tag=gK0454993b
To: <sip:+MY10DIGITNUMBER@SIPPROVIDER:5060>
Call-ID: 788841060_23144897@4.55.11.163
CSeq: 30709 INVITE
Max-Forwards: 14
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MYNAME " <sip:+MYCELLNUMBER@4.55.11.163:5060>
P-Asserted-Identity: "MYNAME " <sip:+MYCELLNUMBER@4.55.11.163:5060>
Supported: 100rel
Content-Length: 303
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 7848 29192 IN IP4 4.55.11.163
s=SIP Media Capabilities
c=IN IP4 4.55.11.130
t=0 0
m=audio 6136 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20
16:06:19 1308736mS SIP Call Rx: phone
INVITE sip:MYWORKNUMBER@MYPUBLICIP SIP/2.0
Record-Route: <sip:SIPPROVIDER;lr>
Record-Route: <sip:SIPPROVIDER;lr>
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK44ad.b446e0021ca14cc90030907c8d0b8432.0
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK44ad.edfba0e5c71f18b9b0e0f30328baf535.0
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK44ad.fec3d7e2.0
Via: SIP/2.0/UDP SIPPROVIDER:5060;branch=z9hG4bK04Ba8c01281169a1d5c
From: "MYNAME " <sip:+MYCELLNUMBER@4.55.11.163:5060;isup-oli=62>;tag=gK0454993b
To: <sip:+MYWORKNUMBER@SIPPROVIDER:5060>
Call-ID: 788841060_23144897@4.55.11.163
CSeq: 30709 INVITE
Max-Forwards: 14
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MYNAME " <sip:+MYCELLNUMBER@4.55.11.163:5060>
P-Asserted-Identity: "MYNAME " <sip:+MYCELLNUMBER@4.55.11.163:5060>
Supported: 100rel
Content-Length: 303
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 7848 29192 IN IP4 4.55.11.163
s=SIP Media Capabilities
c=IN IP4 4.55.11.130
t=0 0
m=audio 6136 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20

 

[AvantiWade]

To clarify my setup:


SIP Line 17
Group 18 18

Local URI - *
Contact - *
Display Name - *
PAI - Internal Data

Incoming Call Routes for all DID's have Line Group 18 selected and the DID on the Incoming Number line.

There is a shortcode setup for 9N; , Dial, N"@sipprovider.com" targeting Line Group ID 18.


This has worked for months and suddenly stopped working.

 

[janni78]

You should have a TX message on the incoming SIP INVITE but it's not seen here.

You could post a default monitor trace and that might give a clue on what's happening.

"Trying is the first step to failure..." - Homer

 

[AvantiWade]

Here is an entire trace. I defaulted the settings for SysMonitor, there never is a TX from my system...

19:31:53 13643815mS H323Evt: Recv: RegistrationRequest 10.10.200.10; Endpoints registered: 10; Endpoints in registration: 0
19:31:57 13647812mS SIP Rx: UDP 66.23.190.100:5060 -> 500v2PUBLICIP:5060
INVITE sip:1MYNUMBER@500v2PUBLICIP SIP/2.0
Record-Route: <sip:66.23.190.100;lr>
Record-Route: <sip:209.193.79.10;lr>
Via: SIP/2.0/UDP 209.193.79.80:5060;branch=z9hG4bKabd8.5640c26957a79e1508aadea281f55f97.0
Via: SIP/2.0/UDP 209.193.79.10:5060;branch=z9hG4bKabd8.2b244fd9012cf7dab221e244782fd6a3.0
Via: SIP/2.0/UDP 66.23.129.10:5060;branch=z9hG4bKabd8.5e8700af4e91d25e95cf96dd90722d6f.0
Via: SIP/2.0/UDP 4.55.39.95:5060;branch=z9hG4bK04B329a5b0808aa6c24
From: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060;isup-oli=62>;tag=gK047550c6
To: <sip:+1MYNUMBER@66.23.129.10:5060>
Call-ID: 654583039_42919169@4.55.39.95
CSeq: 10029 INVITE
Max-Forwards: 15
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
P-Asserted-Identity: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
Supported: 100rel
Content-Length: 300
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 25094 389 IN IP4 4.55.39.95
s=SIP Media Capabilities
c=IN IP4 4.55.39.66
t=0 0
m=audio 5532 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20
19:31:58 13648299mS SIP Rx: UDP 66.23.190.100:5060 -> 500v2PUBLICIP:5060
INVITE sip:1MYNUMBER@500v2PUBLICIP SIP/2.0
Record-Route: <sip:66.23.190.100;lr>
Record-Route: <sip:209.193.79.10;lr>
Via: SIP/2.0/UDP 209.193.79.80:5060;branch=z9hG4bKabd8.5640c26957a79e1508aadea281f55f97.0
Via: SIP/2.0/UDP 209.193.79.10:5060;branch=z9hG4bKabd8.2b244fd9012cf7dab221e244782fd6a3.0
Via: SIP/2.0/UDP 66.23.129.10:5060;branch=z9hG4bKabd8.5e8700af4e91d25e95cf96dd90722d6f.0
Via: SIP/2.0/UDP 4.55.39.95:5060;branch=z9hG4bK04B329a5b0808aa6c24
From: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060;isup-oli=62>;tag=gK047550c6
To: <sip:+1MYNUMBER@66.23.129.10:5060>
Call-ID: 654583039_42919169@4.55.39.95
CSeq: 10029 INVITE
Max-Forwards: 15
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
P-Asserted-Identity: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
Supported: 100rel
Content-Length: 300
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 25094 389 IN IP4 4.55.39.95
s=SIP Media Capabilities
c=IN IP4 4.55.39.66
t=0 0
m=audio 5532 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20
19:31:59 13649299mS SIP Rx: UDP 66.23.190.100:5060 -> 500v2PUBLICIP:5060
INVITE sip:1MYNUMBER@500v2PUBLICIP SIP/2.0
Record-Route: <sip:66.23.190.100;lr>
Record-Route: <sip:209.193.79.10;lr>
Via: SIP/2.0/UDP 209.193.79.80:5060;branch=z9hG4bKabd8.5640c26957a79e1508aadea281f55f97.0
Via: SIP/2.0/UDP 209.193.79.10:5060;branch=z9hG4bKabd8.2b244fd9012cf7dab221e244782fd6a3.0
Via: SIP/2.0/UDP 66.23.129.10:5060;branch=z9hG4bKabd8.5e8700af4e91d25e95cf96dd90722d6f.0
Via: SIP/2.0/UDP 4.55.39.95:5060;branch=z9hG4bK04B329a5b0808aa6c24
From: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060;isup-oli=62>;tag=gK047550c6
To: <sip:+1MYNUMBER@66.23.129.10:5060>
Call-ID: 654583039_42919169@4.55.39.95
CSeq: 10029 INVITE
Max-Forwards: 15
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
P-Asserted-Identity: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
Supported: 100rel
Content-Length: 300
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 25094 389 IN IP4 4.55.39.95
s=SIP Media Capabilities
c=IN IP4 4.55.39.66
t=0 0
m=audio 5532 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20
19:32:00 13650819mS SIP Rx: UDP 66.23.129.253:5060 -> 500v2PUBLICIP:5060
INVITE sip:1MYNUMBER@500v2PUBLICIP SIP/2.0
Record-Route: <sip:66.23.129.253;lr=on>
Record-Route: <sip:66.23.129.236;lr;ftag=gK047550c6>
Via: SIP/2.0/UDP 66.23.129.253:5060;branch=z9hG4bKabd8.66472225ffe913d6ebe19e144d164987.0
Via: SIP/2.0/UDP 66.23.129.236:5060;received=10.153.1.17;branch=z9hG4bKabd8.2072febe59035258d46f26cdb25c31f1.0
Via: SIP/2.0/UDP 66.23.129.10:5060;received=10.153.3.101;branch=z9hG4bKabd8.5e8700af4e91d25e95cf96dd90722d6f.1
Via: SIP/2.0/UDP 4.55.39.95:5060;branch=z9hG4bK04B329a5b0808aa6c24
From: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060;isup-oli=62>;tag=gK047550c6
To: <sip:+1MYNUMBER@66.23.129.10:5060>
Call-ID: 654583039_42919169@4.55.39.95
CSeq: 10029 INVITE
Max-Forwards: 16
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
P-Asserted-Identity: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
Supported: 100rel
Content-Length: 300
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 25094 389 IN IP4 4.55.39.95
s=SIP Media Capabilities
c=IN IP4 4.55.39.66
t=0 0
m=audio 5532 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20
19:32:01 13651281mS SIP Rx: UDP 66.23.129.253:5060 -> 500v2PUBLICIP:5060
INVITE sip:1MYNUMBER@500v2PUBLICIP SIP/2.0
Record-Route: <sip:66.23.129.253;lr=on>
Record-Route: <sip:66.23.129.236;lr;ftag=gK047550c6>
Via: SIP/2.0/UDP 66.23.129.253:5060;branch=z9hG4bKabd8.66472225ffe913d6ebe19e144d164987.0
Via: SIP/2.0/UDP 66.23.129.236:5060;received=10.153.1.17;branch=z9hG4bKabd8.2072febe59035258d46f26cdb25c31f1.0
Via: SIP/2.0/UDP 66.23.129.10:5060;received=10.153.3.101;branch=z9hG4bKabd8.5e8700af4e91d25e95cf96dd90722d6f.1
Via: SIP/2.0/UDP 4.55.39.95:5060;branch=z9hG4bK04B329a5b0808aa6c24
From: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060;isup-oli=62>;tag=gK047550c6
To: <sip:+1MYNUMBER@66.23.129.10:5060>
Call-ID: 654583039_42919169@4.55.39.95
CSeq: 10029 INVITE
Max-Forwards: 16
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
P-Asserted-Identity: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
Supported: 100rel
Content-Length: 300
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 25094 389 IN IP4 4.55.39.95
s=SIP Media Capabilities
c=IN IP4 4.55.39.66
t=0 0
m=audio 5532 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20
19:32:02 13652283mS SIP Rx: UDP 66.23.129.253:5060 -> 500v2PUBLICIP:5060
INVITE sip:1MYNUMBER@500v2PUBLICIP SIP/2.0
Record-Route: <sip:66.23.129.253;lr=on>
Record-Route: <sip:66.23.129.236;lr;ftag=gK047550c6>
Via: SIP/2.0/UDP 66.23.129.253:5060;branch=z9hG4bKabd8.66472225ffe913d6ebe19e144d164987.0
Via: SIP/2.0/UDP 66.23.129.236:5060;received=10.153.1.17;branch=z9hG4bKabd8.2072febe59035258d46f26cdb25c31f1.0
Via: SIP/2.0/UDP 66.23.129.10:5060;received=10.153.3.101;branch=z9hG4bKabd8.5e8700af4e91d25e95cf96dd90722d6f.1
Via: SIP/2.0/UDP 4.55.39.95:5060;branch=z9hG4bK04B329a5b0808aa6c24
From: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060;isup-oli=62>;tag=gK047550c6
To: <sip:+1MYNUMBER@66.23.129.10:5060>
Call-ID: 654583039_42919169@4.55.39.95
CSeq: 10029 INVITE
Max-Forwards: 16
Allow: INVITE,ACK,CANCEL,BYE,REFER,SUBSCRIBE,PRACK,UPDATE
Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed
Contact: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
P-Asserted-Identity: "MY NAME " <sip:+1MYCELLPHONE@4.55.39.95:5060>
Supported: 100rel
Content-Length: 300
Content-Disposition: session; handling=required
Content-Type: application/sdp

v=0
o=Sonus_UAC 25094 389 IN IP4 4.55.39.95
s=SIP Media Capabilities
c=IN IP4 4.55.39.66
t=0 0
m=audio 5532 RTP/AVP 0 8 18 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sendrecv
a=maxptime:20

 

[janni78]

Don't show much, I assume you have UDP and a LAN interface set under Transport on the SIP line.

What settings do you have in Network Topology under System on that interface?

"Trying is the first step to failure..." - Homer

 

[AvantiWade]

Thanks for your help Janni. You are correct, on Transport its set to UDP 5060 on LAN2. LAN2 is statically assigned to a public IP address.

On the Network Topology tab for LAN2 I am set to Open Internet, Public IP address is set to one of my Static Public IP's. Ports are set to 5060. No STUN servers setup.

That LAN2 interface is hung off of a DMZ interface on our sonicwall that is setup for Transparent L3 splice so I can put servers in there with public IP's and then pass the appropriate ports through the firewall to them negating NAT issues.

I'm really hung up on the system not responding to the INVITES. I can watch monitor while calling from my cell and watch it trace through and the second it stops I hear my Disaster Recovery number start ringing.

I'm wondering if it has to do with the Association Method under SIP Line - Advanced. According to the settings:

​

The match criteria used for each line can be varied. The search for a line match for an incoming request is done

​

against each line in turn using each lines Association Method. The order of line matching uses the configured Line

​

Number settings until a match occurs. If no match occurs the request is ignored.
This certainly looks like what is occurring.


And right after I finished typing the above a rep from my sip provider called and started troubleshooting with me. She had me change from sipprovider.com in our ITSP Proxy Address to 3 of their IP Addresses separated by commas. I made a test call towards the system and it WORKED! So it looks like I've got some DNS issues. I'm going to test pointing it towards a different DNS server and see what happens. The provider uses DNS SRV records pointing at the 3 servers that I statically assigned which had been resolving fine up until yesterday'ish so I'm not sure what happened...

 

[janni78]

Depends how they have configured it, but if they change the DNS record when they wanna switch servers IPO will still use the old IP since it has resolved it and stored it in cache so you're better of using IPs.

A reboot should have fixed it so if that didn't help as well then it might be you DNS that cached the IP.
They should have a short TTL on the DNS record so it get checked more often.

"Trying is the first step to failure..." - Homer

 

[AvantiWade]

It's our DNS Server. I just pointed the IP Office towards our ISP's dns and 8.8.8.8 . Both resolved the name correctly and inbound calls came in fine.

It's a Windows 2012r2 Essentials server running our AD and DNS that was recently stood up. I'm still new to administering AD and DNS so its likely something I did or haven't done. Will post back with the fix once I figure it out.

 

[janni78]

I think the problem is their DNS, you should check the TTL on their DNS-record.

I had similar issues with a SIP-provider here that just changes their DNS record as a fallback solution, not really ideal.
The IP Office have already resolved the address so I don't think it will try again until it's rebooted.

"Trying is the first step to failure..." - Homer

 

[AvantiWade]

I just checked and their TTL is set for 5 minutes. I also had restarted the 500v2 several times through the day when this issue was occurring so it should have re resolved the address . Its got to be something related to our internal DNS server.

This company has a similar setup to what you are talking about. They have 3 servers in their SRV records with different priorities/weights. Since the IP Office is supposed to be able to handle SRV records I pointed it at their main address per their install doc's. I may switch it and put the 3 servers dns hostname or even their IP's in to prevent this issue in the future.