ASA Site to Site VPN dropping packets 1

[toy4two2]

Hi guys,

Today I installed 2 ASA 5505s to replace some EOL Nortel routers. Install went smoothly and pretty much everything worked, VPN established, connectivity out to the Internet outside the tunnel, etc.

However, when pinging across the VPN to the remote site's inside interface (and hosts on the LAN there) I am getting drops sporadically, say every 12 packets, solid for 10, drop 5, then solid again for 15...

Log on both sides doesn't complain about anything, just see normal ICMP builds and teardowns. For troubleshooting I even tried to disable the ASA's "basic threat detection" and opened the firewall to all ip traffic to eliminate that as the issue. Still got the drops. Assuming maybe the processor was over worked I disabled all the debug logging too, still got the drops. I eventually removed the ASAs and called my hosting provider on the other side of the world to plug the Nortel's back in.

Of course the Nortel's worked perfectly, no drops at all.

I am wondering what I can do to troubleshoot this. My gut feeling is a fragmentation / MTU issue on the tunnel. Anyone else experience this?

Configs below.

 

[Supergrrover]

I'll get a better look tomorrow but you should start smaller.
Site to site first. Then add hair-pinning after you have it stable.

Definitely clean up the object groups (lots of duplicates) and acl Nat exempt has redundant entries. The 2nd route outside isn't needed.


Brent
Systems Engineer / Consultant
CCNP, CCSP

 

[toy4two2]

agreed I have a previous config with just site to site tunnel I will post up.

 

[toy4two2]

Here we go, I set this config up in a lab using spare ASAs, it does work, see anything else wrong, I redid the NAT exemptions and redundant route.


ASA Version 8.2(1)
!
hostname branch1
enable password 8Ry2YjIyt7RRXU24 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.200.248 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address <branch1 public ip here> 255.255.255.248
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
ftp mode passive
clock timezone PST -8
clock summer-time PDT recurring
access-list outside_access_in extended permit icmp any any echo-reply
access-list outside_1_cryptomap extended permit ip 192.168.200.0 255.255.255.0 192.168.100.0 255.255.255.0
access-list inside_nat0_outbound extended permit ip 192.168.200.0 255.255.255.0 192.168.100.0 255.255.255.0
pager lines 24
logging enable
logging asdm informational
mtu inside 1500
mtu outside 1500
icmp unreachable rate-limit 1 burst-size 1
icmp permit any inside
icmp permit any outside
no asdm history enable
arp timeout 14400
global (outside) 10 interface
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 10 192.168.200.0 255.255.255.0
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 <branch1 public gateway ip here> 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
aaa authentication http console LOCAL
http server enable
http 192.168.200.0 255.255.255.0 inside
http 192.168.100.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map outside_map 1 match address outside_1_cryptomap
crypto map outside_map 1 set pfs group1
crypto map outside_map 1 set peer <branch2 public ip here>
crypto map outside_map 1 set transform-set ESP-3DES-SHA
crypto map outside_map interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh 192.168.200.0 255.255.255.0 inside
ssh 192.168.100.0 255.255.255.0 inside
ssh timeout 5
console timeout 0
management-access inside

threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
username xxxx password xxxx
tunnel-group <branch2 public ip here> type ipsec-l2l
tunnel-group <branch2 public ip here> ipsec-attributes
pre-shared-key xxxx
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
!
service-policy global_policy global
prompt hostname context
Cryptochecksum:306d46bd1de19cc94128386d34ebbd47
: end

 

[toy4two2]

ASA Version 8.2(1)
!
hostname branch2
enable password 8Ry2YjIyt7RRXU24 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.100.248 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address <branch2 public ip here> 255.255.255.252
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
ftp mode passive
access-list outside_access_in extended permit icmp any any
access-list outside_access_in extended permit udp any any eq tftp
access-list outside_1_cryptomap extended permit ip 192.168.100.0 255.255.255.0 192.168.200.0 255.255.255.0
access-list inside_nat0_outbound extended permit ip 192.168.100.0 255.255.255.0 192.168.200.0 255.255.255.0
pager lines 24
mtu inside 1500
mtu outside 1500
icmp unreachable rate-limit 1 burst-size 1
icmp permit any inside
icmp permit any outside
no asdm history enable
arp timeout 14400
global (outside) 10 interface
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 10 192.168.100.0 255.255.255.0
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 <branch2 public gateway ip here> 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console LOCAL
aaa authentication http console LOCAL
http server enable
http 192.168.100.0 255.255.255.0 inside
http 192.168.200.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map outside_map 1 match address outside_1_cryptomap
crypto map outside_map 1 set pfs group1
crypto map outside_map 1 set peer <branch1 public ip here>
crypto map outside_map 1 set transform-set ESP-3DES-SHA
crypto map outside_map interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh 192.168.100.0 255.255.255.0 inside
ssh 192.168.200.0 255.255.255.0 inside
ssh timeout 5
console timeout 0
management-access inside

threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
username xxxx password xxxx encrypted privilege 15
tunnel-group <branch1 public ip here> type ipsec-l2l
tunnel-group <branch1 public ip here> ipsec-attributes
pre-shared-key xxxx
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
!
service-policy global_policy global
prompt hostname context
Cryptochecksum:982ba1f62e1d90b51331d4a98e59ab1b
: end

 

[toy4two2]

here is another hint. In my lab, the latency on the VPN tunnel is high, 200 ms.

 

[toy4two2]

in my lab on the VPN i even lose packets using the above configs. All I need to do is ping a healthy 1000 byte ping across the tunnel and at the same time load a web page on a web server on the other side of the link. Not much data, and look at the latency.

 

[Supergrrover]

Are there errors on your interfaces? How are you connecting them? Is there a speed/duplex mismatch?

Brent
Systems Engineer / Consultant
CCNP, CCSP

 

[toy4two2]

I went ahead and hard coded all interfaces 100 FULL. Tried the same test, same result.

Lab setup is this:

ASA #1 --CAT5--> 1720 ---Serial Cable--> 1720 --CAT5--> ASA #2

Using configs posted. If I remove the ASA's and the VPN tunnel and just go from PC to 1720-1720 to PC2 no issues.

All ASA's have the same image version. I have 4 ASAs so thinking it was a hardware issue I tried different ones, no luck. Even reloaded the images on them trying to figure it out. I pretty much threw in the towel and put in a TAC request.

 

[Supergrrover]

Let me know what they say. My other ASA had to go out on loan. Is it fragmenting?

Brent
Systems Engineer / Consultant
CCNP, CCSP

 

[toy4two2]

still working with TAC. One thing is the problem happens when our NMS is polling across the VPN connection via SNMP UDP port 161. Log in the ASA is flooded with deny messages and thats when it gets so over worked it starts dropping packets and I even get Scanning drop rate exceeded messages.

I am trying to learn why an SNMP packet from one side of the VPN to the other triggers any ACLs at all, they should bypass the ACL lists.

 

[toy4two2]

configs were fine. All along turned out to be a hub I didn't know about sitting between our data closet and my ASA. Must have been a duplex mismatch. Such an odd issue.