VPN phones no longer working with Comcast upgrade 1

[vz58]

Recently had three 9620L VPN phones (out of a total of 5) that stop responding - getting the discover IP message on the phones, when the remote clients changed their network from Century Link to Comcast. As it turns out this seems to be a Comcast network concern because I can duplicate the same problem in my office using Comcast and the tech company I'm working with has tested the same on various other locations using Comcast. The phones work like they should on Verizon, Lumos, Century Links, etc but not Comcast. Catalyst doesn't have any ideas and haven't had much luck with Comcast yet and unfortunately I don't have the funds in the kitty to escalate to Avaya teir 2, although I might be forced into it to keep the customer. Any rate, two of the phones are working properly from different networks. Also, all VPN phones use the same Cisco ASA, same basic settings, same DHCP pool, and the same gateway. Any one had this or similar problems and if so what resolve? Avaya IPO is 8.0.44, IP 500v2 with all necessary license and correct default routes, etc.

I'm not seeing anything that really piques my interest in monitor, other than the fact that it recognizes the extn and then promptly moves on. Also, the phone passes through the VPN fine and I can ping it from the VM server and from the IPO system status.

Any help would be appreciated immensely.

Thanks,

 

[nurban]

THANK YOU!!

I am having the same EXACT issue... Remote phones working fine from other networks, but on Comcast, no such luck.

We're authenticating with an ASA 5510, and our IT vendor was able to get a 5610 to register on their network.

We do have a case open with Avaya, for about 3 weeks now. Earlier today they (Avaya) were able to register a 9620L from their network - and with the same settings on 2 different Comcast networks we can not.

Next step is to find someone who doesn't have Comcast and test.

Have you made any headway on this issue?

 

[amriddle01]

This is black and white a Comcast issue, not sure how Avaya can help with it. Comcast are clearly blocking/interfering with the traffic in a way the previous provider wasn't


Avaya Implementation Qualified Professional Specialist Technical Engineer (AIQPSTE)

 

[nurban]

We just tested on Verizon FIOS - and the test was successful.

Comcast seems to be the issue here. Opening a case with Comcast, hoping that Avaya can provide some assistance - Big company to Big company.

 

[vz58]

Well, It's good to know that I'm not the only one having the same Comcast issues - however it's not good that everyone else is having the same issue. We're having the same results with the Cisco ASA where the tunnel builds and authenticates like a charm, but the phone just sits there looking stupid. Both the configs on the IPO and the ASA are correct as we've tested the phone setup on Verizon, Century Link, Ntelos (Lumos Networks) and AT&T and it works like it should. Finding someone at Comcast that has a clue to what's going on is like beating your head against the wall. I'm to the point of sending out some third party SIP handsets to this location. I'd be hard pressed to believe Comcast would block port 5060. If i can get some resolve with Comcast I'll post here for all that have the same problem.

I spoke with one of my IT contacts from another customer of mine and he mentioned that the tunnel could be set up to send clear text instead of encryption. Having built lots of VPNs over the years, but admittedly not particularly well versed on the deep technical aspect of them, could this be a possibility? Logically it would make sense that if Comcast could see the IPO ports 1719, 1720 and the random RTP ports they may have them blocked. I can't see how that would happen with a completely encrypted VPN. Any thoughts?

Thanks - Nurban and Amriddle01 for replying.

 

[vz58]

Just another two cents to throw in. This must be a recent Comcast change within their infrastructure because I've had Comcast in my office for years and all of the remote VPN phones I've configured over the years and tested, have been in the office and connecting to the customer sites prior to shipping them out. So this has to have happened in the last eight or so months I would guess.

Kenny

 

[hairlessupportmonkey]

we have the same dirty issues here with Virgin Media (Cable broadband) Its generally always network related. Do Comcast supply the modems / routers? Perhaps a software upgrade to them has turned off VPN Pass through etc?

ACSS - SME
General Geek

 

[nurban]

Yes on the changes to Comcast's network...

I did get confirmation from Comcast that they recently (4-6 weeks or so ago) pushed a major upgrade through their network. The understanding of the tech that I talked to said it was a firmware upgrade to support IPv6.

I will be following up with them on Friday as I'm out in the field all day day today.

Also - from our monitor traces, it actually looks like some of the data getting through to the system has been truncated or dropped. We see the requests of the phone to register through monitor, but it doesn't appear to be a complete request. Its as if in a csv file a comma is off or something. One of the IT we interact with suspects it might be due to additional packet inspection on the Comcast side, but this is just speculation at this point in time. Will post any additional information as it becomes available.

 

[nurban]

Quick update -
Just got off of the phone with Comcast. They are aware of the issue, and this is a high priority issue for them. There is a known Protocol 50 (IPSEC) issue on one of the core switches between carriers that is suspect. Cisco is also working on this issue with them based on its nature.

 

[Westi]

star for sharing the info

Thanks nurban

Joe W.

FHandw, ACSS (SME), ACIS (SME)

http://convergednetworks.ca

Give a tech a solution and he will be back tomorrow to ask you the next question, teach a tech how to read the manual and he will be able to solve the problems for a life time.

 

[amriddle01]

Lol high priority, it's been at least two weeks, that's enough time to design and produce a new range of routers/switches never mind fix a programming/firmware issue


Avaya Implementation Qualified Professional Specialist Technical Engineer (AIQPSTE)

 

[mood911]

We are experiencing the exact same issue with Comcast users as of last Thursday am. Was anyone able to resolve the issue or did comcast finally push a change? Any help would be appreciated!

 

[nurban]

Sonic Wall works fine... Cisco ASA is still not.
Just tested a new phone. Test Comcast --> Windstream - Phone works fine. Router is SonicWall TZ100 with PSK
Just tested the install I referenced above. Comcast --> Cox Phone isnt working. Router is a Cisco ASA 5510 using Xauth
The Xauth method is working from every other carrier's network without a problem.

 

[mood911]

Thanks Nurban! Regarding your post about protocol 50 issue, did you have ticket open with them? I can't seem to get past level 1, they won't escalate since they don't understand the problem. Maybe if there was an old ticket I could reference it might help convince them I'm not crazy. Thinking of pursuing this with Cisco since we are also terminating into a ASA 5500, any thoughts? Thanks again for the quick reply!!

 

[nurban]

We keep opening tickets, and they get closed without resolution... CR316259700 is one of the latest ones. I've asked the IT company involved to peruse as well. Please update if you find anything else!

 

[mood911]

OK, after several hours no the phone with Cisco and Comcast I have a little more insight into our problem. We are experiencing the issue on one of two ASA's, the difference between them being geography and code version (problem ASA running 8.0 code working ASA 8.2 code). We verified that port 1720 on the IPO seems to be the culprit (unable to even telnet to port 1720 from a windows client) though other ports seem to operate normally (80). Comcast is finally taking this seriously and have escalated this to Level 3, but they still have no insight. Cisco's opinion is this is a comcast issue but recommended upgrading our ASA to 8.2 just in case. I'll be upgrading over the weekend and I'll let you know how it goes.

Nurban, out of curiousity what version of ASA code are you running on the ASA?

 

[nurban]

Checked with the IT vendor -

ASA Version 8.2(1) is what has been loaded all along.

 

[vz58]

Glad to see that someone is at least getting some proactive response with Comcast. I haven't. We know that Comcast is definitely holding most of the guilt with this, but is this also a possible ASA software load issue as well? I can confirm with the IT guys what load they are running as well. If I recall they also opened a ticket with Cisco - what they asked I can't immediately confirm. Although most ports were verified to be opened in the firewall - 1720, 1719.

Nurban - one of your earlier posts mentioned Sonicwall working fine but Cisco ASA not on the same carrier. If I'm following correctly, could this possibly be a combo problem with the Cisco ASA and Comcast? Anyone know of any other VPN issues with Comcast using other appliances - Adtran, Juniper, Netgear, etc?

 

[nurban]

vz58 -

Yes, SonicWall with PSK was working fine, Cisco with Xauth wasn't.
That being said, all of our Comcast service went down today, for about 20 minutes. On a whim, based on the work that mood911 is doing with Comcast, I decided to give it another shot.

Believe it or not - the phone finally came up. Call quality was horrible (Comcast is claiming a major network outage), but this is the first time since I started working on this customer in Feb that we can remotely log in a VPN phone via Comcast. 3 Months wait, not so cool, but glad its finally working and I can close this out!

 

[mood911]

Nurban, glad to hear this is working for you, unfortunately not the case here. After upgrading our second ASA to 8.2.X the problem still exists so it seems like it's the path rather than the ASA code. So we are going to move all VPN phones to second ASA, but I'll keep pushing Comcast (though I have no hope they will do anything to help.) Another thing I noticed is that the Soft Phone clients are also affected so I was able to test PPTP vpn running over the same path as the non-working IPSEC client and it works fine. So problem = Comcast + IPSEC Xauth on ASA = Port 1720 blocked.