Internal IP interface goes missing, everything else works (except VPN, of course)

[jmkelly]

This is one of the strangest things I've ever seen.
We have a WatchGuard XTM505 firewall providing VPN access through IPSEC tunnels to XTM2x firewalls at about a dozen remote sites. At one site, the XTM22 firewall comes up fine, then stops providing VPN services after 20-40 minutes. Everything else keeps on working: you can access the Internet through the firewall, and if you look at the XTM505 on the other end, the tunnels are actually still up and passing traffic. The firewall's "hostwatch" utility shows connections between hosts on either end of the tunnel. But users can't access anything on the other end of the tunnel, nor can I access the firewal via its internal IP address--it doesn't respond to pings, http requests, or the proprietary Watchguard System Management software.

I've swapped out the hardware and the configuration (using a configuration from a trouble-free XTM22 at another remote site); neither made any difference. Watchguard's Level 2 support is baffled too.

To recap, the parts that maintain the IPSEC SAs stay up; the routing functions of the firewall stay up; the physical interfaces stay up; but after 20 minutes the IP interface stops transmitting data to or from the tunnel, and stops responding to any IP or ICMP requests.

Any ideas?

 

[jmkelly]

This turned out to be a a memory leak bug that WG knows about but doesn't know when it'll get fixed. WG Support told me to do two things: downgrade the OS from 11.6.1 to 11.4.2 (vintage July 2011--and this was after the first-tier support told me to upgrade it to 11.6.1!) and re-do the VPN tunnels so that the tunnel's IP range didn't overlap with the XTM's LAN IP range. That took 16 net/mask combinations, but it did the trick, and was less invasive than the OS downgrade.