Adding new GRE tunnel broke all others

[davidarndt]

Hi,

We have a Win2KSP4 WAN interconnected by some Frame Relay and some GRE tunnels over the Internet. Routers are mostly 1750's with some 2600's and a couple 831's.

I added a tunnel from an additional site using a new 831 router and the same destination routers as all the other sites. The clients at that new site had trouble accessing some services on some of the servers in the destination site but could resolve IP just fine. SMB access wasn't possible (couldn't browse a share on the affected server, for example). However, they could browse shares on other servers on the same switch as the one they couldn't.

During the week, I began seeing other remote sites connected via GRE tunnels experience the same problems until, during the weekend, all GRE connected sites had the same problem.

After six hour of troubleshooting, I changed the MTU on one of the client machines to 1430 and it began to work normally. So I changed the MTU on each of the destination tunnels to 1524 and it fixed all the other sites and clients.

1. Why did this stop working everywhere when adding that last tunnel in spite of the fact that it had been reliable for years at all other sites?
2. What are the repercussions of having the tunnel MTU set higher than default? I believe it will fragment all normal sized packets and degrade performance, but I'm not sure exactly.
3. Is there a better (and free) solution?

Thanks very much!
David

 

[ADB100]

On the router Ethernet interfaces have you got 'IP unreachables' enabled or disabled? If its disabled (no ip unreachables) then the router does not send ICMP unreachables back to clients when they attempt large IP datagrams. If its disabled try enabling it.

Andy

 

[davidarndt]

I did, indeed, have IP unreachables enabled.

 

[lambent]

It should be "ICMP type 3 code 4" for the Path MTU discovery.

There're some Cisco documentations about this issue. I believe one of the possible cause is that there're some firewalls between end hosts that block this ICMP type 3 code 4 messages so that PMTUD fails.

Usually I use the following method:

"Use the ip tcp adjust-mss command on the tunnel interfaces so that the router will reduce the TCP MSS value in the TCP SYN packet. This will help the two end hosts (the TCP sender and receiver) to use packets small enough so that PMTUD is not needed"

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_paper09186a00800d6979.shtml

also this one

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_tech_note09186a0080093f1f.shtml

 

[davidarndt]

Thank you, lambent. I believe your speculation is probably right on the money. Any idea why the problem would begin right after adding a new tunnel from a new site?

Anyway, the IOS versions we have don't support the "ip tcp adjust-mss" interface command. We're using 12.0 and 12.1 vintages. We're not under contract so I don't think we can get updates right now. I'll try to contact a vendor and see if that's true. For now, setting the "ip mtu 1524" is working although it's killing the routers that are destinations for several tunnels. Even non-tunneled traffic is slower and somewhat unreliable. I wonder if there's any better workaround than changing the mtu to force fragmentation of normal-sized packets (other than changing the MTU at each of 130 workstations).

Doing the "ip tcp path-mtu-discovery" made the "ip mtu 1524" workaround stop working, by the way. Not sure I understand why that's the case...

 

[lambent]

GRE tunnels over the Internet"

Check with your ISP and see if they just change their mind and start blocking ICMP unreachable messages. Just a guess anyway.

For the issue with "ip tcp path-mtu-discovery", I didn't try that as I always use the "ip tcp adjust-mss" walkaround.