VPN packet loss

[jfmays]

I'm dealing with a strange problem that we've been having for some time.

Cisco 7206 running IOS (tm) 7200 Software (C7200-IK9SU2-M), Version 12.3(23). Have a crossover ethernet connection to a linux box on fastethernet4/0. Have a backbone internet connection on ATM2/0.4. Have a crypto map attached to fastethernet0/0 for a VPN to a remote endpoint at another computer. They have another machine past their endpoint of the vpn. The vpn exists so their machine (65.119.118.76) can talk to our machine (24.235.0.25).

Pings between the VPN endpoints, 0% loss. Pings to the machines inside the endpoints, 0% loss. Pings to anywhere through any interface on our vpn endpoint router, 0% loss. No packet less anywhere, UNTIL...

When you ping from the machine on our lan through the vpn to the machine on their lan, anywhere from 5-40% packet loss.

Things I have determined --

When pinging, show crypto ipsec sa shows every packet being encrypted and entering the vpn. From their end, they see every ping packet arrive and get returned. Again, show crypto ipsec sa shows every packet being decrypted coming out of the vpn. It would seem the problem is something related to finally delivery from our endpoint router to the linux box, but only for packets that come out of the vpn.

Any hints would be greatly appreciated. Also, I read a description of a similar problem from someone that they said turned out to be a bug in a deferred version of IOS. Does anyone know anything about that?

 

[North323]

what is your mtu size on the machine on the lan? do you have mtu path discovery enabled on the devices?

 

[jfmays]

crypto isakmp key xxxxxxxxxxxx address 65.119.118.136

crypto map WinnetToSyniverse 20 ipsec-isakmp
description PHL-3845-SS7-VPN router
set peer 65.119.118.136
set transform-set TSI2
match address PHL-3845-SS7-VPN

interface FastEthernet0/0
description Win.Net New Albany CO LAN
ip address 216.24.28.17 255.255.255.248 secondary
ip address 24.235.29.17 255.255.255.248
no ip route-cache cef
ip ospf message-digest-key 1 md5 7 xxxxxxxxxxxxx
duplex full
no cdp enable
crypto map WinnetToSyniverse

ip access-list extended PHL-3845-SS7-VPN
permit ip host 24.235.0.25 host 65.119.118.76

 

[jfmays]

Here you have run square up against ignorance on my part. How do I tell if the mtu path discovery is enabled?

 

[North323]

first question...what is the mtu size on the LAN pc? can you change it to 1260?

second..you would clearly see mtu path discovery enabled in your config. can you install wireshark on your LAN pc and take a look at the packets....they never lie. You will probably see DF packets or Dont Fragment but they need to be to get down that tunnel.

 

[jfmays]

The lan pc is an ubuntu machine. The mtu on the lan pc was 1500, I've now dropped it to 1260, but the problem is still occurring.

To reiterate, the outgoing packets all get encrypted going into the vpn, and decrypted coming out. I can tell that with "show crypto ipsec sa". At one time the lan pc was on a different network segment. At that time I could sniff the lan traffic and could see that the last packets were never coming back across lan to the lan pc. Now it's on a point-to-point crossover ethernet connection directly on the vpn device, but the behavior is the same. I really don't think the lost packets are ever leaving the vpn device. 100 of the packets come out of the vpn, then 5-40% of them simply get lost or dropped or something on the vpn router and never get delivered to the lan pc.

The lan pc interface shows no dropped packets or errors.

 

[brianinms]

You try a different cable?

 

[jfmays]

Yeah, it's a different cable now than when the lan pc was on a different network segment. Is there any way to get dropped ICMP packets to show up in the cisco logs?

 

[North323]

you can turn up logging. but sounds like you are not getting any traffic to that LAN box so you may have a route issue before you can address the packet loss??

 

[jfmays]

Why does it sound that way? When I ping from the lan pc to the computer on the other end of the vpn, as I said, I get 5-40% packet loss, but that means that I do get 60-95% of the packets back. There are other network services running fine on the PC, and in fact communications with the pc on the other end of the VPN work -- albeit with the spottiness you'd expect with a lot of packet loss occuring.

 

[brianinms]

Is your internet connection a DS3 or a MPLS ?

 

[burtsbees]

First off, what happens with pings from the edge router to the vpn connected machine---


computerA---routerA---tunnel---routerB---computerB

From routerA to ComputerA, I mean?

Also, have you tried layer 1, i.e. different crossover cable, different ports, etc.?

You stated..."They have another machine past their endpoint of the vpn." How is this connected? Through a switch? Crossover cable?

In other words, let's eliminate everything else BUT the VPN tunnel. We don't want to overcomplicate until necessary!

burt

 

[jfmays]

First off, what happens with pings from the edge router to the vpn connected machine--- From routerA to ComputerA, I mean?"

I assume you are viewing the A machines as us and the B machines as them. 0% packet loss.

"Also, have you tried layer 1, i.e. different crossover cable, different ports, etc.?"

Yes. We've changed out everything past within A in your diagram, including the port on router A that we talk through. We even located Computer A on a different lan segment that traverses a T1 coming out of router A instead of the ethernet port and have tried a different computer completely for computer A -- same behavior.

"You stated..."They have another machine past their endpoint of the vpn." How is this connected? Through a switch? Crossover cable?"

I really couldn't say. I know we can ping their router (router B) from computer A and get 0% packet loss, but again, pings don't traverse the VPN in that case because the target doesn't match the access list. The only thing I have to ping past that is computer B, and then the packets go through the VPN. That's when the packet loss starts.

They tell us that we're the only place they see this packet loss, they support lots of customers off of computer B. In fact we have another computer off another WAN segment through another VPN that does the exact same thing with them and there is no packet loss. I think that is actually a different router B and computer B on their end, though, in Atlanta, whereas the one for this discussion is in Philly.

"In other words, let's eliminate everything else BUT the VPN tunnel. We don't want to overcomplicate until necessary!"

I understand what you are trying to do and am perfectly happy to answer any questions or perform any tests you want to suggest, but honestly, I did that before I ever posted a question to a cisco VPN forum. The only reason I'm talking here at all is that everything else works. The packets loss seems to relate specifically to packings coming out of our VPN end on router A and heading to computer A. All other traffic through computer A and router A, 0% loss. Changing out hardware makes no difference.

 

[burtsbees]

I was fairly certain that troubleshooting the simple things had been done to this point, just that I know from experience that when one gets to the point of wondering about bugs in the IOS, then a fresh set of eyes may point to something missed in the initial troubleshooting. You understand, and I hope this did not frustrate you. That being said...

As far as how their side is physically connected---it does not matter, if L1 has been troubleshot all the way. Time to analyze traffic/stats.

In their router connecting the Linux box---I would look at things like sh int to look at packet drops, buffers, available bw, loads, etc. to get a rough %age of packets being dropped vs. actual traffic. I would also look at matches of access-lists to try and determine the majority of traffic type when doing the pings. Also, look at sh processes cpu to try and get a %age of cpu usage. I wonder what else it is doing when it is encrypting traffic in the tunnel...

This will eliminate traffic trends---I do understand that all other traffic may be okay, so this is not the likely culprit. At this point, however, if there is SN on the router, I would upgrade the IOS and look at the Bug Toolkit on Cisco.com to look for related TAC cases and/or bugs related to this for your IOS.

/