871 downloads get slower and slower 2

[shaferbus]

I'm far from an IOS expert, so I'm hoping that someone can tell me if I have a configuration problem or a hardware problem.

Our office network is behind a Cisco 871 router connected to a T1, with 512 Kb/s dedicated to internet bandwidth (the rest is for voice lines). Over the past several months, we have been having problems with slow internet downloads. At first, it was just large files (2+ Mb, not THAT large...), and only sometimes. The file starts downloading fine, and then the download speeds drops off to nothing. I know the speed indication in the IE download dialog isn't exactly a scientific measurement, but it would start off at 100+ Kb/s, but after a few hundred Kb, it would start dropping until it reached bytes/sec speeds, and eventually fail. Now it's degraded to the point that web pages with a lot of graphics take a long time to load, and downloading a file of any size (like AV updates) is practically impossible!

If I bypass the router and connect my PC directly to the ISP's WAN connection, the problem disappears.

The weird thing is, internet speed tests (connected through the router) show a download rate that is well within normal parameters for our connection, even if I run it while a simple PDF download has slowed to 90 bytes/sec!

I've replaced cables to and from the router, power cycled everything multiple times, and unplugged everything but the router and my PC, with no effect.

The IOS configuration (below) has not been altered in at least 6 months. Can anyone see anything there that would cause a cumulative problem, or does it sound like the 871 is FUBAR?
Any other suggestions?

Thanks

version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname yourname
!
boot-start-marker
boot-end-marker
!
logging buffered 51200 debugging
logging console critical
enable secret 5 <<password cipher>>
!
aaa new-model
!
!
aaa authentication login default local
aaa authorization exec default local 
aaa authorization network sdm_vpn_group_ml_1 local 
!
aaa session-id common
!
resource policy
!
clock timezone PCTime -5
clock summer-time PCTime date Apr 6 2003 2:00 Oct 26 2003 2:00
ip subnet-zero
no ip source-route
ip cef
no ip dhcp use vrf connected
ip dhcp excluded-address 192.168.16.61 192.168.16.254
!
ip dhcp pool sdm-pool1
   import all
   network 192.168.16.0 255.255.255.0
   dns-server 216.135.95.2 64.132.94.250 
   default-router 192.168.16.1 
!
ip dhcp pool <<server>>
   host 192.168.16.2 255.255.255.0
   hardware-address 00c0.9f10.d8a6
!
ip dhcp pool <<A PC>>
   host 192.168.16.3 255.255.255.0
   hardware-address 0016.41ef.439d
!
ip dhcp pool <<Another PC>>
   host 192.168.16.4 255.255.255.0
   hardware-address 000d.6071.7798
!
ip dhcp pool <<Yet Another PC>>
   host 192.168.16.14 255.255.255.0
   hardware-address 0040.ca36.35b3
!
ip dhcp pool <<Printer>>
   host 192.168.16.13 255.255.255.0
   hardware-address 0000.f0a2.9947
!
ip dhcp pool <<Another Printer>>
   host 192.168.16.30 255.255.255.0
   hardware-address 0880.1fff.22b1
!
ip dhcp pool <<Yet Another Printer>>
   host 192.168.16.60 255.255.255.0
   hardware-address 0000.aaad.9445
!
ip dhcp pool Shop
   host 192.168.16.5 255.255.255.0
   hardware-address 0040.f473.55aa
!
ip dhcp pool maintenance2
   host 192.168.16.6 255.255.255.0
   hardware-address 0009.6bf3.a799
!
ip dhcp pool shopmgr
   host 192.168.16.7 255.255.255.0
   hardware-address 0040.2b4d.cfea
!
ip dhcp pool mezzanine
   host 192.168.16.8 255.255.255.0
   hardware-address 0001.29d3.9cd4
!
!
ip inspect name DEFAULT100 appfw DEFAULT100
ip inspect name DEFAULT100 cuseeme
ip inspect name DEFAULT100 ftp
ip inspect name DEFAULT100 h323
ip inspect name DEFAULT100 icmp
ip inspect name DEFAULT100 netshow
ip inspect name DEFAULT100 rcmd
ip inspect name DEFAULT100 realaudio
ip inspect name DEFAULT100 rtsp
ip inspect name DEFAULT100 esmtp
ip inspect name DEFAULT100 sqlnet
ip inspect name DEFAULT100 streamworks
ip inspect name DEFAULT100 tftp
ip inspect name DEFAULT100 tcp
ip inspect name DEFAULT100 udp
ip inspect name DEFAULT100 vdolive
ip inspect name DEFAULT100 802-11-iapp
ip inspect name DEFAULT100 http
ip inspect name DEFAULT100 https
ip tcp synwait-time 10
no ip bootp server
ip domain name yourdomain.com
ip name-server 216.135.95.2
ip name-server 64.132.94.250
ip name-server 192.168.16.2
ip ssh time-out 60
ip ssh authentication-retries 2
!
appfw policy-name DEFAULT100
  application http
    strict-http action allow
    audit-trail on
!
!
crypto pki trustpoint TP-self-signed-1834174675
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1834174675
 revocation-check none
 rsakeypair TP-self-signed-1834174675
!
!
crypto pki certificate chain TP-self-signed-1834174675
 certificate self-signed 01
  <<All the Certificate Stuff>>
  quit
username admin privilege 15 secret 5 <<Password>>
!
! 
!
crypto isakmp policy 1
 encr 3des
 authentication pre-share
 group 2
crypto isakmp client configuration address-pool local SDM_POOL_1
!
crypto isakmp client configuration group SHAFERVPN
 key <<VPN Key>>
 dns 192.168.16.1 192.168.16.2
 wins 192.168.16.2
 domain <<Our Domain>>
 pool SDM_POOL_1
 acl 105
 include-local-lan
 pfs
 max-users 3
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec df-bit clear
!
crypto dynamic-map SDM_DYNMAP_1 1
 set security-association idle-time 600
 set transform-set ESP-3DES-SHA 
 reverse-route
!
!
crypto map SDM_CMAP_1 isakmp authorization list sdm_vpn_group_ml_1
crypto map SDM_CMAP_1 client configuration address respond
crypto map SDM_CMAP_1 65535 ipsec-isakmp dynamic SDM_DYNMAP_1 
!
!
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
!
interface FastEthernet3
!
interface FastEthernet4
 description $FW_OUTSIDE$$ES_WAN$$ETH-WAN$
 ip address 66.192.xxx.xxx 255.255.255.0
 ip access-group 101 in
 ip verify unicast reverse-path
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip inspect DEFAULT100 out
 ip nat outside
 ip virtual-reassembly
 ip route-cache flow
 duplex auto
 speed auto
 crypto map SDM_CMAP_1
!
interface Vlan1
 description $ETH-SW-LAUNCH$$INTF-INFO-HWIC 4ESW$$ES_LAN$$FW_INSIDE$
 ip address 192.168.16.1 255.255.255.0
 ip access-group 100 in
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 ip nat inside
 ip virtual-reassembly
 ip route-cache flow
 ip tcp adjust-mss 1452
!
ip local pool SDM_POOL_1 192.168.17.50 192.168.17.55
ip classless
ip route 0.0.0.0 0.0.0.0 66.192.43.1
!
no ip http server
ip http authentication local
ip http secure-server
ip nat inside source static tcp 192.168.16.14 80 interface FastEthernet4 80
ip nat inside source route-map SDM_RMAP_1 interface FastEthernet4 overload
ip nat inside source static tcp 192.168.16.2 8082 interface FastEthernet4 8082
ip nat inside source static tcp 192.168.16.2 8085 interface FastEthernet4 8085
!
logging trap debugging
access-list 1 remark INSIDE_IF=Vlan1
access-list 1 remark SDM_ACL Category=2
access-list 1 permit 192.168.16.0 0.0.0.255
access-list 100 remark auto generated by Cisco SDM Express firewall configuration
access-list 100 remark SDM_ACL Category=1
access-list 100 permit tcp any any eq www
access-list 100 remark GoChart Schedule
access-list 100 permit tcp any any eq 8085
access-list 100 remark Permit shafers website
access-list 100 permit tcp any any eq 8082 log
access-list 100 permit udp host 192.168.16.2 eq domain any
access-list 100 permit tcp any any eq 4443
access-list 100 deny   ip 66.192.xxx.0 0.0.0.255 any
access-list 100 deny   ip host 255.255.255.255 any
access-list 100 deny   ip 127.0.0.0 0.255.255.255 any
access-list 100 permit ip any any
access-list 101 remark auto generated by Cisco SDM Express firewall configuration
access-list 101 remark SDM_ACL Category=1
access-list 101 remark Auto generated by SDM for NTP (123) ntp.twtelecom.net
access-list 101 permit udp host 207.250.222.200 eq ntp host 66.192.xxx.xxx eq ntp
access-list 101 permit ip host 192.168.17.50 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.17.51 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.17.52 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.17.53 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.17.54 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.17.55 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.16.50 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.16.51 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.16.52 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.16.53 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.16.54 192.168.16.0 0.0.0.255
access-list 101 permit ip host 192.168.16.55 192.168.16.0 0.0.0.255
access-list 101 permit udp any host 66.192.xxx.xxx eq non500-isakmp
access-list 101 permit udp any host 66.192.xxx.xxx eq isakmp
access-list 101 permit esp any host 66.192.xxx.xxx
access-list 101 permit ahp any host 66.192.xxx.xxx
access-list 101 remark Auto generated by SDM for NTP (123) nist1-ny.witime.net
access-list 101 permit udp host 208.184.49.9 eq ntp host 66.192.xxx.xxx eq ntp
access-list 101 permit udp host 64.132.94.250 eq domain any
access-list 101 permit udp host 216.135.95.2 eq domain any
access-list 101 permit tcp any any eq www
access-list 101 remark GoChart Schedule
access-list 101 permit tcp any any eq 8085
access-list 101 permit tcp any any eq 8082 log
access-list 101 remark SSL
access-list 101 permit tcp any any eq 4443
access-list 101 permit ip 192.168.16.0 0.0.0.255 any
access-list 101 permit icmp any host 66.192.xxx.xxx echo-reply
access-list 101 permit icmp any host 66.192.xxx.xxx time-exceeded
access-list 101 permit icmp any host 66.192.xxx.xxx unreachable
access-list 101 deny   ip 10.0.0.0 0.255.255.255 any
access-list 101 deny   ip 172.16.0.0 0.15.255.255 any
access-list 101 deny   ip 192.168.0.0 0.0.255.255 any
access-list 101 deny   ip 127.0.0.0 0.255.255.255 any
access-list 101 deny   ip host 255.255.255.255 any
access-list 101 deny   ip host 0.0.0.0 any
access-list 101 deny   ip any any
access-list 102 remark SDM_ACL Category=4
access-list 102 permit ip 192.168.16.0 0.0.0.255 any
access-list 103 remark SDM_ACL Category=2
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.17.50
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.17.51
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.17.52
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.17.53
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.17.54
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.17.55
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.16.50
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.16.51
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.16.52
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.16.53
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.16.54
access-list 103 deny   ip 192.168.16.0 0.0.0.255 host 192.168.16.55
access-list 103 permit ip 192.168.16.0 0.0.0.255 any
access-list 104 remark SDM_ACL Category=4
access-list 104 permit ip 192.168.16.0 0.0.0.255 any
access-list 105 remark SDM_ACL Category=4
access-list 105 permit ip 192.168.16.0 0.0.0.255 any
no cdp run
route-map SDM_RMAP_1 permit 1
 match ip address 103
!
!
control-plane
!
banner login ^CCCCCCCAuthorized access only!
 Disconnect IMMEDIATELY if you are not an authorized user!^C
!
line con 0
 no modem enable
 transport output telnet
line aux 0
 transport output telnet
line vty 0 4
 transport input telnet ssh
!
scheduler max-task-time 5000
scheduler allocate 4000 1000
scheduler interval 500
ntp clock-period 17175186
ntp server 207.250.222.200 source FastEthernet4 prefer
end

 

[shaferbus]

baddos - the WAN connection is a 512 kbps channel of a T1.
If I disable IP Inspect and leave the ACL in place, it seems to perform correctly. (I had to add "permit tcp any any eq www" to ACL 101 to permit web traffic with CBAC off).

As for MTU size... good question! I have never done anything with MTU size, so does that mean it's set at the ethernet default of 1500?

I poked around tek-tips about optimizing MTU, and did the ping test thing (is that how I should determine max MTU?) and came up with a value of 1472 being the max size before packets were fragmented.

If I set the MTU setting on the WAN interface, does it have any effect on the VPN tunnel, and settings on the other end?

 

[alphacrasher]

Shaffer,

Interesting thought.

When you compare your download speed at speedtest.net both with an inspection rule and without any inspection rule. Is there any difference in your measured download speeds?

If I understand your problem correctly, you should have a normal download speed running their test without any inspection rule and you probably won't even be able to bring up the flash window to run the test with an inspection rule.

One thought that has crossed my mind is that the method by which your ISP controls your bandwidth might be messing with the 871, in particular it's implementation of inspection rules. The 871, if I am not mistaken, is a level 2 switch, even though it appears as a level 3.

For instance, you are "paying for a T1 connection", but I would bet, if it is a good sized provider, you are simply being given access to a fraction of an optic fiber network if not directly, then via a broadband connection.

Might the issue might be HOW your ISP limits your bandwidth to what you pay for? Is it by intermittently limiting your access to DNS servers, or a more crude approach of simply dropping packets?

In other words, might their methodology interfere severely with the 871 inspection methodology?

If this is the case it might also explain the time dependence of the problem as it occurred with technological upgrades.

 

[baddos]

Yes you will need to set your MTU to 1472 then. It will affect your VPN traffic which would have an even lower MTU. Try setting your MTU on your WAN interface and turning cbac back on and see if that helps.

 

[shaferbus]

Sorry, thought I posted this yesterday...

Changing the MTU size didn't seem to have any effect on the problem. Once I turned IP Inspection on we were back to the same old problems.

Alphacrasher, the speed tests are one of the very puzzling things about this whole situation! ACL's on or off, CBAC on or off - online speed tests from several different sites show normal throughput. (Naturally they are fastest if I bypass the router completely, but that makes sense)

Oh, and the T1 is a real T1 - a big fat 50 pair cable that they had to string for blocks. The ISP's "integrated services unit" box splits off the voice channels to our PBX and the data channel to the router.

 

[baddos]

Personally I wouldn't run ip inspection on this router at all.

If you have to have this feature, try doing this to disable the java applet inspection. This part of the http inspection runs terribly slow.

access-list 51 deny any
no ip inspect name DEFAULT100 http
ip inspect name DEFAULT100 http java-list 51

 

[joeccie]

Try to disable http and https inspection. I have seen this issue many times and disabling this feature always helps. I believe there is a know bug with http/https inspection where is slows down over time.

Thanks,
Joe

http://certificationchat.com

 

[alphacrasher]

Joeccie,

Might you know if this bug is terminal or, can it be solved by paying cisco and downloading an image?

 

[Beamer030608]

Hi I was wondering if you had an update? I am in the process of troubleshooting this same type of error, which appears to be happening at all of our locations setting behind an 871.

 

[shaferbus]

Beamer, I ended up just turning off IP inspection on the router. What IOS version are you running? Both alphacrasher and I are running quite old versions, and I'm wondering if a later one would work (see the Cisco bug I mentioned earlier in the thread).

Ultimately, I bought an 881, because I needed to buy something to be the VPN endpoint at a remote office anyway, so I'll relegate the 871 to that role and use the 881 in our main office in it's place.

 

[Beamer030608]

Well i believe i am running an even older version then you. I wanted to update last year but there was some sort of bug dealing with multiple vlans. The version i am running is c870-advipservicesk9-mz.124-4.t4.bin


As much as I would love to turn off the IP inspection I need it on in-order for this router/firewall to forward requests to a n2h2 web filter.

 

[baddos]

Beamer, make sure you are using the java-list function as I described earlier. I bet it will drastically improve performance.

 

[Beamer030608]

i have decided to post my own thread so I dont risk Hijacking your thread...

 

[Volz]

Doesn't debugging/logging use a lot of router resources? It looks like debug is on and the way I understand it is that debugging will slow router. I am no expert. Just started CCNA studies.

 

[baddos]

logging debugging and debug are two separate things

 

[alphacrasher]

I plunked down the money for access to Cisco's downloads.

It turns out the problem is due to a known problem with the old IOS.

The problem did not go away with then new T24 IOS, but was substantially improved.

They sent me a document that shows with full inspection rules the 871 is only supposed to have a 12.8 Mbit/sec throughput.

This seems rather lame and I can't shake the feeling that the hardware is being deliberately hamstrung by the IOS. Particularly because there seems to be no degradation when using multiple vlans.

 

[shaferbus]

Well, at least that means that we're not crazy!

12.8 MB I can buy, but I wasn't even able to maintain 12.8 KB download speeds with Inspection on with the 12.4(4)T7 IOS.

Are you using the same config?

What I still don't understand is why neither of our routers had a problem until recently. Did Cisco have anything to say about that?