PIX to PIX VPN errors

[fritz101]

I recently installed a PIX to PIX VPN, both using v6.3(3). Ping times are great between hosts on each network, and applications like VNC, telnet, and http work fine. We are having major problems with SQL Server connections, though. Only intermittant connectivity from Enterprise Manager and Query Analyzer is available. Upon a debug crypto ipsec, I get tons of the following errors:

IPSEC(sw_esp_decap): fail antireplay check
IPSEC(cipher_ipsec_request): decap failed for X.X.X.145 -> Y.Y.Y.130

(X is remote public PIX ip, Y is local public PIX ip)

Here is the VPN config portion of my local PIX boxes:

crypto ipsec transform-set tunnelset ah-md5-hmac esp-aes-192 esp-md5-hmac
crypto map tunnelmap 10 ipsec-isakmp
crypto map tunnelmap 10 match address 100
crypto map tunnelmap 10 set peer X.X.X.145
crypto map tunnelmap 10 set transform-set tunnelset
crypto map tunnelmap interface outside
isakmp enable outside
isakmp key ******** address X.X.X.145 netmask 255.255.255.255
isakmp policy 10 authentication pre-share
isakmp policy 10 encryption aes-256
isakmp policy 10 hash md5
isakmp policy 10 group 5
isakmp policy 10 lifetime 86400


Cisco doesn't have any documentation on this error. Anyone have any ideas?


I am not sure if this is related or not, but I also sometimes get the following debug crypto ipsec error:

IPSEC(ipsec_prepare_encap_request): ERROR: unable to fragment packet pktsize=1420, eff_mtu = 1412

thanks
fritz

 

[themut]

What you have is a fragmentation issue, your configuration is fine since you are able to ping and use other services. Try to lower down the MTU size to 1300 or lower on both ends. Depending on the application you may also be able configure it in such a way that you can determine the largest pakcet it can generate. Make sure your application doesn't send jumbo frames.

 

[fritz101]

Lowering the MTU on the outside gave me many more "unable to fragment packet" errors. That wasn't my major issue, though.

The errors below are my big problem. They appear exactly when I try to connect to a remote SQL server.

IPSEC(sw_esp_decap): fail antireplay check
IPSEC(cipher_ipsec_request): decap failed for X.X.X.145 -> Y.Y.Y.130

thanks so far!

 

[marhoul]

Did you get any resolution on this IPSEC(sw_esp_decap): fail antireplay check error?

I ma having similar problems with a PIX-Sonicwall connection.

Thanks,

Mark

 

[fritz101]

Increasing the MTU size helped somewhat (not decreasing), but didn't fix the problem altogether. I eventually had to decrease the level of encryption being used for IPSEC. If you find another way of fixing the problem, let me know!

 

[mbilgrav]

Hi,

Just wondering:
Your transform set say aes-192, and your isakmp encryption says 256.
Isnt this wrong ?

 

[fritz101]

The isakmp encryption is for key-exchange only, whereas the transform set is the encryption algorithm for data transfer. Thanks for your thoughts though....I haven't messed around with it since reducing the encryption somewhat to get rid of the errors.

 

[mbilgrav]

ok, I never seen a config before, where hte two differs, but I guess it is ok to do so.
Just for tests: What happends if you set them both to the same size, does the errors disappear ?

 

[fritz101]

I don't think its related. I had changed my transform set encryption and the errors went away. isakmp is still aes-256

 

[mbilgrav]

Could you be so kind to inform about how the crytpo ipsec transform-set looks like after your changed it ?

I am currios about what you changed, and if you took away the AH, cause, as I understand it now, the errors you recieve are related to AH, so obviously if you take away AH, the error goway aswell.

 

[fritz101]

i just changed the esp-aes-192 to a different (lower) encryption algorithm, didn't take anything out