ASA5505 adding jitter/latency?

[Guest_imported]

I have a newly-set-up ASA5505 that (thankfully!!) replaced our old Linksys with DD-WRT. Loving the stability and the fact it can actually handle the load from our 25 workstations without crashing.

However, I've noticed that pings from outside to the outside interface as well as pings from the inside to an outside IP seem to have a good bit more latency now, even when the network connection (6m/6m) is virtually idle.

Even pinging the router our ISP has put on our side of the DSL connection (for connection monitoring/PPPoE authentication) results in far-higher-than-expected latency.

Our setup is this:
[Internal Network] -- [ASA5505] -- [ISP-provided Cisco 891] -- [G.SHDSL connection] -- [Internet]

When I had the Linksys installed (and it wasn't overloaded), I could ping that Cisco 891 and get response times of 1-2ms. The first router on the far side of the DSL connection was consistently 8-10ms. Pings to seattle.voip.ms, our long-distance provider, were almost perfectly consistently 37ms. (Pretty good latency from Alaska, which is why I chose them!)

Now, after installing the ASA, even with virtually 0 network load, pinging the Cisco 891 results in pings anywhere from 20 to 80ms (average of 40). That's over a 100mbps link and about three feet of Ethernet cable. Pings to the first hop on the other side of the DSL connection range anywhere from 10ms (?!) to 700ms (average of 180). Pings to seattle.voip.ms are 80 to 1350ms (average 570).

Pings to the ASA's inside interface are sub-1ms, so it's nothing on the inside network.

The reading that I've been doing online (i.e

http://www.networking-forum.com/viewtopic.php?f=35&t=7774)

indicates the ASA should add negligible latency, but that seems like it's not the case. I don't have (as far as I know, unless I've screwed the config up!) a super-complicated firewall ruleset, and the ASA's CPU and memory usage is sitting constant (regardless of load) at 10% and 212 (out of 512MB), respectively. I can't think of any reason why so much latency would be added unless I've inadvertently set up some weirdness in the config that's delaying packets.

Also, I'll take this opportunity to mention that I found the thread at

http://www.tek-tips.com/viewthread.cfm?qid=1573769&page=13

very helpful (which is how I found this forum--holy cow, a technical forum with people who are friendly and helpful?!). That's actually what caused me to check the ping--the voice quality still seemed to experience some drop-outs even when the connection was nearly idle.

I know it's a separate topic, but given the ASA's limited QoS capabilities as outlined in that thread, I may need a better solution. I have a couple of Cisco 1841s sitting on a shelf upstairs (got secondhand from a downsizing nonprofit). Would that be a useful/workable option? I assume I'd sit it inline between the ASA (which would perform NAT and firewall) and the ISP-provided Cisco 891, and it would do QoS by the DSCP markings (which the ASA doesn't change). Or would it be better to have the 1841 (assuming it can) do NAT and let the ASA focus solely on firewall protection? In any case, I just thought I'd bring this up in case it dovetails with the issue I'm having with the ASA and ping times.

I've also attached a (hopefully fully!) sanitized config to this post as well as a sample of the ping times to various points below. (I've done them as ping -f below, but I get similar--actually worse--results even without flooding.)

Thanks in advance!

--- [firewall inside interface] ping statistics ---
18954 packets transmitted, 18953 received, 0% packet loss, time 8838ms
rtt min/avg/max/mdev = 0.157/0.458/2.448/0.262 ms, ipg/ewma 0.466/0.451 ms


--- [Cisco 891, this side of the DSL connection] ping statistics ---
490 packets transmitted, 487 received, 0% packet loss, time 7121ms
rtt min/avg/max/mdev = 8.141/23.216/145.276/15.646 ms, pipe 10, ipg/ewma 14.563/24.758 ms

--- [ISP first hop] ping statistics ---
647 packets transmitted, 645 received, 0% packet loss, time 9796ms
rtt min/avg/max/mdev = 8.044/49.733/186.344/33.161 ms, pipe 11, ipg/ewma 15.165/30.324 ms

--- seattle.voip.ms ping statistics ---
760 packets transmitted, 752 received, 1% packet loss, time 11445ms
rtt min/avg/max/mdev = 50.130/87.995/204.962/31.396 ms, pipe 15, ipg/ewma 15.079/78.277 ms

 

[Guest_imported]

Well, whaddya know.

I waited until the office closed and did a system reload just for the heck of it. (Couldn't take it down while we were open.)

Problem solved.

Pings to the 891 are right smack dab at 1.9ms with a deviation of .1ms. Pings to seattle.voip.ms are consistently 45ms (close enough to 37 and with no variance/jitter, so I'm happy).

Only trouble is: how long will it last? If it slows down over the next couple of days again (last reboot was 48 hours ago), then maybe it's a memory issue.

I do have "threat-detection statistics host number-of-rate" set to 3 (24 hours), but I didn't figure that would be a major issue given our relatively light traffic load (25 workstations). Plus, memory usage was sitting tight consistently at just over 200MB (out of 512MB).

Also, before rebooting, I did try changing to threat-detection statistics host number-of-rate 1, but either the statistics cache didn't empty after that or that's not the root of the problem, since it had no effect.

I didn't think to try disabling all threat detection (especially scanning threat detection and scanning threat statistics) before rebooting. If it slows down again, that'll be my next test.

So, I'll monitor things over the next couple of days, but in the meantime, I'm all ears for any ideas.

 

[hairlessupportmonkey]

what version of software is on the asa?

I have found 8.2 the most useful and stable. 8.4 seems to screw up NAT.

ACSS - SME
General Geek

 

[Guest_imported]

8.3.

On my iPhone so apologies for the brevity.

 

[Guest_imported]

OK, that was a short-lived solution. It's happening again--barely 24 hours later.

I tried turning all advanced/scanning threat detection off and all statistics off and let it think for awhile, but it's still giving me:

--- [Cisco 851 on local side of DSL connection] ping statistics ---
1103 packets transmitted, 1103 received, 0% packet loss, time 1102302ms
rtt min/avg/max/mdev = 1.714/295.496/1046.101/343.790 ms, pipe 2

I suppose my next option would be to leave those features off and then reload the device and see if/when the problem develops again. (Maybe it's not something that a live-running config change will cure, even if the problematic services are disabled.)

Of course, I'd appreciate any suggestion in the meantime.

And sorry, that should have been 8.2 above (typo on iPhone):

# sh ver

Cisco Adaptive Security Appliance Software Version 8.2(1)
Device Manager Version 6.2(1)

Compiled on Tue 05-May-09 22:45 by builders
System image file is "disk0:/asa821-k8.bin"
Config file at boot was "startup-config"

* up 1 day 1 hour

Hardware: ASA5505, 512 MB RAM, CPU Geode 500 MHz
Internal ATA Compact Flash, 128MB
BIOS Flash Firmware Hub @ 0xffe00000, 1024KB

Encryption hardware device : Cisco ASA-5505 on-board accelerator (revision 0x0)
Boot microcode : CN1000-MC-BOOT-2.00
SSL/IKE microcode: CNLite-MC-SSLm-PLUS-2.03
IPSec microcode : CNlite-MC-IPSECm-MAIN-2.04

 

[ADB100]

I can't find anything specific in the release notes of 8.2(5) (the latest 8.2 release) regarding fixes for latency, however 8.2(1) was the 1st public release of 8.2 so I would upgrade to at least 8.2(4) and re-test. Probably just put 8.2(5) on.

 

[Guest_imported]

>I suppose my next option would be to leave those features off and then reload the device and see if/when the problem develops again. (Maybe it's not something that a live-running config change will cure, even if the problematic services are disabled.)

I turned them off (only thing enabled is basic threat detection; no statistics and no scanning threat detection) and within a day or two (it's now been four) it's doing it again.

Still:

--- [cisco 891] ping statistics ---
26 packets transmitted, 26 received, 0% packet loss, time 25003ms
rtt min/avg/max/mdev = 9.114/347.001/932.706/330.793 ms

And from ASA:

asa# ping [Cisco 891]
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to [Cisco 891], timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/234/700 ms

1 hop on a 100mbps Ethernet connection...not normal! External pings on the outside interface are similar.

I'm waiting to hear back from our IT consultant I bought the ASA from re: accessing downloads of the software to try an update as suggested. (I'll probably also ask him to look into this, but I'm trying to solve this myself, both to teach myself more and also save us some $... )

 

[baddos]

Have you ruled out the isp provided router from being an issue? I would place a host next to that router and test that. I looked at your config and besides you running an older version software, it looks fine. I would personally remove reverse path checking on the inside, but it's fine the way it is.

 

[Guest_imported]

@baddos: No, I haven't completely ruled out the 891. However, I'll note a couple of things:

1) Never had this issue before the ASA
2) Power cycling the ASA solves the problem (for a day)
3) Power cycling the 891 does not solve the problem

I do want to test another host on the outside of the ASA, but I'll need to work with my ISP to obtain some additional external addresses (no cost with our connection but I do need to fill out some justification for use paperwork, and our network will be renumbered), or I'll need to take down the connection out of hours and substitute a test machine in place of the ISP's 891. Just haven't had a chance to do that yet. Thanks for the idea, though.

I added the ip-verify on both interfaces per a couple of articles I read about securing the ASA. I wasn't sure about the inside interface option, but it was recommended by people far more knowledgeable than I, so I did it. I'd trust someone here engaging me directly far more than an article on the Internet, but that said, there does seem to be a valid reason to have it enabled:

http://www.tek-tips.com/viewthread.cfm?qid=1527745&page=22

>Staticfactory (IS/IT--Management)
4 Feb 09 12:46
The host addresses have started to change dynamically as we set up blocking rules, so we've deduced that a host somewhere has been compromised and is creating the spoofed traffic. Thanks for your help.

Thanks again for the help. Hope to hear back from our consultant today re: new software. Would you also recommend staying in the 8.2 release or moving up to 8.4?

 

[hairlessupportmonkey]

the other option might be MTU size / packet fragmentation

you could try reducing the MTU size on the modem WAN port and see if that helps. I know ciscos can be a bit funny with MTU sizes....

ACSS - SME
General Geek

 

[Guest_imported]

MTU is 1420 per ISP's suggestion. Would you suggest reducing it further?

 

[baddos]

No, it should be set to what the ISP recommends.

Do you see anything in the ASDM log mentioning reverse patch check failures or scanning threats?

 

[Guest_imported]

Nope, just searched through the 1,000-line buffer and didn't see anything recently.

I've been doing some other work on the ASA (VPN stuff) and have been looking at logs off and on all morning and have seen a couple mentions of scanning threats, but only a couple. I did think it a bit weird since I turned scanning threat detection off...

 

[hairlessupportmonkey]

what sort of enviroment is the unit in? does it get really hot? the ASA 5505 runs pretty hot and doesnt like being snuggled up against and underneath other kit... let it breathe!

ACSS - SME
General Geek

 

[Guest_imported]

Hmm. Intriguing idea. It's in a rack in a ~60-degree telco room, but there was something on top of it. It was as warm as any other piece of network equipment--certainly not hot to the touch.

I gave it some room, but it still gets obstinately slow.

I did a reboot of the device last night, which cleared it up for about 16 hours. Although I think that rules out the ISP's router, I still plan on taking that out of the equation after hours on Sunday just for testing...

 

[hairlessupportmonkey]

what else is the firewall doing? vpns? routing? etc etc....

Perhaps there is an issue with the config?

try a factory and reload?

ACSS - SME
General Geek