Troubleshooting sudden drops in TTL

[dozier]

Hello,

I've been trying to troubleshoot a situation where a there is a TCP connection between point A (the client) and point B (the server) periodically failing. This connection traverses a variety of mediums including ethernet segments, point-to-point T1, internet VPN, and MPLS and spans the networks of three separate business entities (not including the internet path). One thing I'm seeing in packet captures is that everything is fine, and then all of the sudden the TTL drops on packets sourced from point A so much that they never make it all the way to point B. I'm trying to figure out why and where this is happening.

Here's a simple layer 3 diagram of the path.

[Point A]->[Cisco branch firewall]->[Cisco core firewall]->[Cisco VPN router]->[Cisco VPN headend]->[Cisco PPS firewall]->[Cisco MPLS router]->??->[Point B]

I have captures running at the 3 Cisco firewalls and it is between the core firewall and PPS firewall that the TTL on these packets suddenly plunges. There are no apparent communications issues, and while I understand the internet is a wildcard, I'm not seeing how anything on the internet could result in the TTL being lowered since they are encrypted IPSEC VPN tunnels for the duration of the internet trip.

Does anyone know how I can view the TTL of these packets on the Cisco routers? Packet debugging (even detailed) does not seem to display the TTL.

Thanks.

 

[Quadratic]

Assuming you are correct that TTL timeout is the issue, is it only in one direction when the problem occurs, or in both? Usually if TTL is changing it means the path is changing, such that more hops are taken. I'd run some trace routes before and during this problem (trace from both directions) to confirm layer 3 pathing. If it takes it to the point of TTL timeout, typically that would be a routing loop somewhere.

CCNP, CCDP, CCIP

 

[dozier]

This issue seems to happen very sporadically and does not last very long, which makes doing traceroutes while it is occurring very problematic.

So, no ideas on how to see the TTL info in a Cisco router IP debug or something similar?

 

[baddos]

You could run wireshark somewhere to see the TTL. The Cisco ASA/PIX firewalls also have a packet capturing feature that you can use to capture the packets as well.

 

[Quadratic]

Lets start from the beginning: what are you seeing that leads you to believe that TTL is hitting 0? When this happens, are there any logs inn any routers regarding routing convergence or link failures? TTL hitting 0 would mean a routing loop.

CCNP, CCDP, CCIP
Core Network Planner, ISP