Hello everyone-
I have a PIX 506e which has five VPN tunnels set up to my various customers. I use ViceVersa software to do some offsite file synchronizations across these VPNs (ie I am backing up some of my customer's critical files offsite every night across the VPNs). This configuration has been rock solid until recently when I would check the synchronization the following morning and find out that most if not all of the file copies had failed. Typically the error I get is "the semaphore timeout period has expired" which I believe basically means that the something has interrupted the connectivity to the point where the operating system's file copy times out (copies are being performed using UNC paths). All of these VPNs have identical configurations (I believe) but not all of the tunnels are dropping all of the time. I have one that drops just about every single time, one that drops most of the time and one that usually does not drop. The two that usually drop are both T-1s while the one that usually doesn't is a 384k upstream DSL. Obviously this would lead me to believe that it might be a utilization-based issue but the PDM shows the PIX is loafing CPU wise and is using just over half of the available RAM. Plus, this amount of traffic shouldn't even phase a PIX. I used to have another tunnel with 3Mbps up in addition to these and never had a hiccup. As far as I know, nothing has changed in the config nor the environment. I did have one customer that no longer needed the VPN access so I removed the tunnel from the config but of course that had no bearing on the issue at hand.
Standard Internet connectivity has not been a problem. Connections are fast and reliable. We have about 10 machines inside the PIX running on a Comcast business cable connection getting 4Mbps up and usually between 30-35Mbps down. We have other sporadic traffic coming in through the firewall (but not through a tunnel) such as SMTP, RDP and some antivirus software heartbeat traffic. As these VPN file copies are done after the business day (usually starting in the early evening and running through the night), the other traffic is at its lowest point during those times.
Here is where I am:
1) Considering it is happening on multiple VPN tunnels, I believe the issue has to be with my side (as opposed to an issue with the client PIX or connection).
2) I was thinking it could be some intermittent hardware failure in the PIX but I would think I would notice other types of sporadic connectivity failures which I haven't. We occasionally do some very large file downloads (1.8GB) at pretty high speed (3MB/sec, that's bytes not bits) and it has no problems dealing with that level of bandwidth so that leads me to believe hardware is probably not the issue.
3) The VPN drops occur towards the beginning of the file copies rather than towards the end so it is not taking much for it to trigger the issue.
4) All file copies are done from the same server so that eliminates other machines/configs from the equation. Dual Xeons, 3.0Ghz HT with 3GB of RAM. Not really processor bound or memory bound so I don't believe the machine is the issue. Also, if it was, I would expect the tunnel that never drops to drop as well (which it isn't) and I would think that I might see internal dropped file connections and such which I am not.
5) It seems like I can artifically increase the likelihood of keeping the VPN from dropping but running continuous pings across the tunnels while the backups are running. Obviously the pinging is somehow managing to keep the tunnels alive or helping to prevent whatever timeout is occurring. This too leads me to believe that the issue is PIX, connectivity or config bound as opposed to a computer issue.
6) Servers on the other side of the file copy are all managed by me. Basically all Dell PowerEdge servers running Server 2000 or Server 2003 with similar configs, operating practices, etc. Once again, considering it all appears to be related to me, I haven't focused on the remote side of the equation too much but that doesn't mean that a Windows update or network card driver hasn't caused this flaky behavior (since I would have installed the same update to all of my customer's machines that are appropriate). Not likely but...
Anyway, I probably have been too long winded but I wanted to give you as much of the initial picture as possible. I have run debug traces on the VPNs but they really don't appear to be telling me anything I don't already know. Also, SHOW CRYPTO SA doesn't seem to tell me anything juicy either (no significantly high send or receive errors, etc) nor does SHOW INT. Can anyone help me figure out how to take this troubleshooting to the next level and narrow it down further? Before you ask, I don't have a spare 506e to swap in or I would have done so already.
Thanks in advance for any help or advice you may be able to offer.
Bill
I have a PIX 506e which has five VPN tunnels set up to my various customers. I use ViceVersa software to do some offsite file synchronizations across these VPNs (ie I am backing up some of my customer's critical files offsite every night across the VPNs). This configuration has been rock solid until recently when I would check the synchronization the following morning and find out that most if not all of the file copies had failed. Typically the error I get is "the semaphore timeout period has expired" which I believe basically means that the something has interrupted the connectivity to the point where the operating system's file copy times out (copies are being performed using UNC paths). All of these VPNs have identical configurations (I believe) but not all of the tunnels are dropping all of the time. I have one that drops just about every single time, one that drops most of the time and one that usually does not drop. The two that usually drop are both T-1s while the one that usually doesn't is a 384k upstream DSL. Obviously this would lead me to believe that it might be a utilization-based issue but the PDM shows the PIX is loafing CPU wise and is using just over half of the available RAM. Plus, this amount of traffic shouldn't even phase a PIX. I used to have another tunnel with 3Mbps up in addition to these and never had a hiccup. As far as I know, nothing has changed in the config nor the environment. I did have one customer that no longer needed the VPN access so I removed the tunnel from the config but of course that had no bearing on the issue at hand.
Standard Internet connectivity has not been a problem. Connections are fast and reliable. We have about 10 machines inside the PIX running on a Comcast business cable connection getting 4Mbps up and usually between 30-35Mbps down. We have other sporadic traffic coming in through the firewall (but not through a tunnel) such as SMTP, RDP and some antivirus software heartbeat traffic. As these VPN file copies are done after the business day (usually starting in the early evening and running through the night), the other traffic is at its lowest point during those times.
Here is where I am:
1) Considering it is happening on multiple VPN tunnels, I believe the issue has to be with my side (as opposed to an issue with the client PIX or connection).
2) I was thinking it could be some intermittent hardware failure in the PIX but I would think I would notice other types of sporadic connectivity failures which I haven't. We occasionally do some very large file downloads (1.8GB) at pretty high speed (3MB/sec, that's bytes not bits) and it has no problems dealing with that level of bandwidth so that leads me to believe hardware is probably not the issue.
3) The VPN drops occur towards the beginning of the file copies rather than towards the end so it is not taking much for it to trigger the issue.
4) All file copies are done from the same server so that eliminates other machines/configs from the equation. Dual Xeons, 3.0Ghz HT with 3GB of RAM. Not really processor bound or memory bound so I don't believe the machine is the issue. Also, if it was, I would expect the tunnel that never drops to drop as well (which it isn't) and I would think that I might see internal dropped file connections and such which I am not.
5) It seems like I can artifically increase the likelihood of keeping the VPN from dropping but running continuous pings across the tunnels while the backups are running. Obviously the pinging is somehow managing to keep the tunnels alive or helping to prevent whatever timeout is occurring. This too leads me to believe that the issue is PIX, connectivity or config bound as opposed to a computer issue.
6) Servers on the other side of the file copy are all managed by me. Basically all Dell PowerEdge servers running Server 2000 or Server 2003 with similar configs, operating practices, etc. Once again, considering it all appears to be related to me, I haven't focused on the remote side of the equation too much but that doesn't mean that a Windows update or network card driver hasn't caused this flaky behavior (since I would have installed the same update to all of my customer's machines that are appropriate). Not likely but...
Anyway, I probably have been too long winded but I wanted to give you as much of the initial picture as possible. I have run debug traces on the VPNs but they really don't appear to be telling me anything I don't already know. Also, SHOW CRYPTO SA doesn't seem to tell me anything juicy either (no significantly high send or receive errors, etc) nor does SHOW INT. Can anyone help me figure out how to take this troubleshooting to the next level and narrow it down further? Before you ask, I don't have a spare 506e to swap in or I would have done so already.
Thanks in advance for any help or advice you may be able to offer.
Bill
