2-ASA5505 problems: ASA won't boot...VPN tunnels unstable 1

[toosober]

I was recently working with Cisco on a problem our ASAs are having and was told to upgrade from IOS 8.02 to 8.03 as part of the solution. Now, when the ASA reloads it says loading.... boot.... and then just hangs with a flashing cursor??? Anyone know what to do next to get into it to get it to at least boot to the original IOS that was working???

Also I am not sure if I should ask to questions in the same post or post twice, so I will just ask all here and if this is incorrect please let me know.

I have setup 3 site-to-site VPN tunnels across 3Mb DSL links back to our main site which has a 10Mb link. When the tunnels come up and stabilize (after about 10-15 minutes all are connected following a reload at the main site) the tunnels work fine. But throughout the day the tunnels constantly drop and reconnect. Our plans are to add more sites, but I am unwilling to move forward until the current issues are resolved.

Is there some "timer" or something I have missed? Should the tunnels, even if "re-keying", cause service affecting problems. The users are frequently losing connection as the tunnels reset and I have been unable to stabilize them.

I am sure this is not the way the tunnels are to work, but Cisco said everything was configured correctly and it was an ISP issue. I can ping between the sites on the outside interfaces with at least 90%-95% response.

If anyone has any suggestions i am open.

Thanks in advance!

 

[garnetbobcat]

Issue 1:

It sounds like the image that you loaded was corrupt or became corrupt. You should download it again and then transfer it to the ASA again.

Since you can't boot it, you'll need to break into ROMMON and copy the image that way. Follow the directions here:

http://www.cisco.com/en/US/docs/security/asa/asa80/configuration/guide/trouble.html#wp1064632

Issue 2:

It sounds like an "ISP issue" to me if Cisco said things were right. My suggestion is to debug isakmp for a while and see what happens when the connections drop. Maybe there will be some clues in the logs. Problems like that are tricky to pin down.

Matt
CCIE Security

http://www.wr-mem.com

 

[garnetbobcat]

Thinking about my response to Issue 1...

If you didn't erase the old OS image before you loaded the new one, you should be able to just boot the old one.

> boot <image name>.bin

or something like that. Try issuing a "dir" to see what images are on the flash.

Matt
CCIE Security

http://www.wr-mem.com

 

[toosober]

Thanks Matt!!! I was having issues using the rommon method you first posted, but the second post solved my issue. I booted to the previous image, deleted the asa803-k8.bin, downloaded it again, uploaded to the ASA and it booted to it with no problem this time...

Now I am back to my original problem of the tunnel stability...

Thanks again!

 

[toosober]

I should probably add that in the Syslog Messages one of the errors is:
Group = 75.x.x.x,IP=75.x.x.x, IKE lost contact with remote peer, deleting connection (keepalive typePD)


Thanks...

 

[garnetbobcat]

That suggests a connection problem between the sites. The host that is giving you that message is not receiving responses to the Dead-Peer-Detection (DPD) messages it's sending. When it fails to get enough responses, the tunnel is torn down.

Are you seeing the same messages at the same time on the other end?

Also, you might want to check routers in the path (if you control any) and see if something is happening there.

Matt
CCIE Security

http://www.wr-mem.com

 

[toosober]

I did some more research and found that all of the remote sites were pinging each other at 99/100%, but any site pinging the main site would ping anywhere from 58% to 92%. Unless it was afterhours, when there was no traffic, then they were able to ping 100%.
I contacted our ISP and they found a piece of equipment that was hard coded at 1.5Mb so it was limiting our 10Mb connection. This was also why their usage reports, I had requested at the beginning stages of troubleshooting, showed we were barely using the pipe. As soon as they turned it up to 10Mb, the ASAs stopped bouncing, and all the tunnels, 4 now, have been stable and not bounced once. I think the problem has been resolved.

Thought I should post this for future reference?! And let Matt, and Cisco, know they were correct in the thoughts of the ISP being the issue.

Thanks for your help on this Matt!!!

 

[garnetbobcat]

Awesome! I always feel guilty when I tell people "it's an ISP issue" because they are so hard to track down. Good job getting them to really examine the network!

Even if your case with is closed you might drop the engineer a line. That's such a great find. I personally know some TAC engineers who would appreciate hearing about it. :-D

Thanks for letting us know how it got resolved!

By the way, if that 1.5Mb configuration was depriving you of your paid for bandwidth, press for a discount!

Matt
CCIE Security

http://www.wr-mem.com

 

[toosober]

Notifying the TAC engineer was the secong thing I did...