Slow to No Performance CISCO 1841 over T1 Line

[Deadasbetamax]

Hello,

I have setup a CISCO 1841 with the following configuration and experiencing extremely slow connections to websites over our T1 link to the Internet. I just configured this router and was wondering if anyone could find anything in my configuration that would provide me with an idea of what is causing this problem. Version and Config are below

**Show Version**
Cisco IOS Software, 1841 Software (C1841-ADVSECURITYK9-M), Version 12.4(3h), REL
EASE SOFTWARE (fc2)
Technical Support:

http://www.cisco.com/techsupport

Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Wed 25-Jul-07 13:43 by stshen

ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)

Testing uptime is 7 minutes
System returned to ROM by power-on
System image file is "flash:c1841-advsecurityk9-mz.124-3h.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 1841 (revision 7.0) with 234496K/27648K bytes of memory.
Processor board ID FTX1137X098
2 FastEthernet interfaces
1 Serial(sync/async) interface
1 Virtual Private Network (VPN) Module
DRAM configuration is 64 bits wide with parity disabled.
191K bytes of NVRAM.
62720K bytes of ATA CompactFlash (Read/Write)

Configuration register is 0x2102

**Show Config**

!
! Last configuration change at 15:26:24 CDT Wed Oct 10 2007
! NVRAM config last updated at 15:28:02 CDT Wed Oct 10 2007
!
version 12.4
service timestamps debug uptime
service timestamps log datetime msec
service password-encryption
!
hostname Testing
!
boot-start-marker
boot-end-marker
!
logging buffered 4096 debugging
no logging console
!
aaa new-model
!
!
!
aaa session-id common
!
resource policy
!
clock timezone CST -6
clock summer-time CDT recurring
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
ip subnet-zero
ip cef
!
!
ip inspect dns-timeout 15
ip inspect name outbound tcp
ip inspect name outbound udp
ip inspect name outbound ftp
ip inspect name outbound realaudio
ip inspect name outbound smtp
!
!
ip flow-cache timeout active 1
ip domain name testing.com
ip name-server X.X.X.65
!
!
!
** CRYPTOGRAPHY PORTION AND USERNAMES REMOVED **
!
!
!
!
!
interface FastEthernet0/0
description connected to Lan
ip address 10.1.1.1 255.0.0.0
ip nat inside
ip virtual-reassembly
ip route-cache flow
duplex auto
speed auto
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
interface Serial0/0/0
description connected to the Internet
ip address X.X.X.186 255.255.255.252
ip access-group 101 in
ip inspect outbound out
ip nat outside
ip virtual-reassembly
no ip route-cache cef
ip route-cache flow
no ip mroute-cache
clock rate 2000000
no cdp enable
!
ip classless
ip route 0.0.0.0 0.0.0.0 X.X.X.185
ip flow-export source FastEthernet0/0
ip flow-export version 5
ip flow-export destination 10.0.0.152 2055
!
no ip http server
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip nat pool inet_add X.X.X.38 X.X.X.38 netmask 255.255.255.248
ip nat inside source list 10 pool inet_add overload
ip nat inside source static 10.1.1.10 X.X.X.33
ip nat inside source static 10.0.0.218 X.X.X.34
ip nat inside source static 10.0.0.101 X.X.X.35
ip nat inside source static 10.1.1.12 X.X.X.36
!
logging trap debugging
logging 10.0.0.152
access-list 10 permit 10.0.0.0 0.255.255.255
access-list 101 permit tcp any host X.X.X.33 eq 1494
access-list 101 permit tcp any host X.X.X.33 eq www
access-list 101 permit tcp any host X.X.X.33 eq 2598
access-list 101 permit tcp any host X.X.X.33 eq 443
access-list 101 permit tcp any host X.X.X.34 eq www
access-list 101 permit tcp any host X.X.X.34 eq 3000
access-list 101 permit tcp any host X.X.X.34 eq 3001
access-list 101 permit tcp any host X.X.X.34 eq 3003
access-list 101 permit tcp any host X.X.X.34 eq 8800
access-list 101 permit tcp any host X.X.X.186 eq 22
access-list 101 permit tcp any host X.X.X.35 eq www
access-list 101 permit tcp any host X.X.X.35 eq 3000
access-list 101 permit tcp any host X.X.X.35 eq 389
access-list 101 permit tcp any host X.X.X.35 eq smtp
access-list 101 permit tcp any host X.X.X.35 eq pop3
access-list 101 permit tcp any host X.X.X.35 eq 587
access-list 101 permit tcp any host X.X.X.35 eq 143
access-list 101 permit tcp any host X.X.X.35 eq domain
access-list 101 permit tcp any host X.X.X.35 eq 3002
access-list 101 permit tcp any host X.X.X.35 eq 1000
access-list 101 permit tcp any host X.X.X.35 eq 366
access-list 101 permit tcp any host X.X.X.36 eq 1494
access-list 101 permit tcp any host X.X.X.36 eq 2598
access-list 101 permit tcp any host X.X.X.36 eq www
access-list 101 permit tcp any host X.X.X.36 eq 443
access-list 101 permit tcp any host X.X.X.36 eq 3389
access-list 101 deny ip any any
snmp-server community public RO
snmp-server enable traps tty
!
!
control-plane
!
banner motd CC
Unauthorized Access to this system is strictly prohibited!!
!
line con 0
exec-timeout 0 0
privilege level 15
line aux 0
password 7 130E1B14
modem InOut
modem autoconfigure type usr_sportster
transport input all
speed 1200
line vty 0 4
transport input telnet ssh
line vty 5 15
transport input telnet ssh
!
scheduler allocate 3000 1000
end

Any help at this point would be greatly appreciated as I have been over this many times even switching routers to try and solve the issue to no avail.

 

[DanInRaleigh]

*i wonder how much faster it would be with out the IP inspection taking place... not that leaving it off is an option..

http://www.daniel-white.com

 

[Deadasbetamax]

Well before I began using a router with the Advanced Security Features a somewhat similar config ran much faster on a router with the Basic IP config. However not using IP INSPECT is really not an option. In the previous config we pretty much were leaving at least 1 machine wide open in order to get a proxy server with a rather loose port range working properly.

 

[plshlpme]

im running the ip inspect similar to this on a 1760 router with two dsl lines going into it and it runs fine.. this 1841 im sure has the power to run this config no problem..

when the t1 is running slow have you ran a

sh int s0/0/0 to see what kind of load you are pushing?
all it takes is some user or host to suck your t1 bandwidth dry and cause the latency to go through the roof due to queuing and other congestion mechanisms.

i see you running 10.0.0.0/8 on your lan.. how many users do you have using this link?

 

[Deadasbetamax]

Well from traffic monitoring software we are no where near topping out the available bandwidth on the T1 connection. Our LAN is no where near the size necessary for a 10.0.0.0/8 based address scheme. The whole IP scheme was setup up long before I got here by an outside contractor. Its something I have wanted to change for quite sometime, just never had the time.

 

[burtsbees]

Packet sniffer/protocol analyzer time...

Burt

 

[BuckWeet]

Are all protocols slow? Or just ftp, http, etc..


Try multiple things to find if its a cerain thing that is slow.

 

[brianinms]

I would either disable ip cef if you are going to use ip cache flow.

 

[Deadasbetamax]

The only protocol that we have really tested was http. At the time of testings there wasn't a whole lot of other traffic.

 

[Deadasbetamax]

I have tried it with both ip cef both off and on it seems to make no difference.

 

[brianinms]

Try toggling ip cache flow

 

[Deadasbetamax]

So far to the best of my ability it appears to have to do with the access list 101. As soon as it is applied the slow performance with frequent connection losses begins. I remove access-list 101 and performance goes to back to normal. Thing is I just can figure what the issue with the list is. Latest config below.


!
version 12.4
service timestamps debug uptime
service timestamps log datetime msec
service password-encryption
!
hostname Testing
!
boot-start-marker
boot-end-marker
!
logging buffered 4096 debugging
no logging console
!
aaa new-model
!
!
!
aaa session-id common
!
resource policy
!
clock timezone CST -6
clock summer-time CDT recurring
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
ip subnet-zero
ip cef
!
!
ip inspect dns-timeout 15
ip inspect name outbound tcp
ip inspect name outbound udp
ip inspect name outbound ftp
ip inspect name outbound realaudio
ip inspect name outbound smtp
!
!
ip flow-cache timeout active 1
ip domain name testing.com
ip name-server X.X.X.65
!
!
!
!
!
** Users and Crypto Portion Removed **
!
!
!
!
!
interface FastEthernet0/0
description connected to Lan
ip address 10.1.1.1 255.0.0.0
ip nat inside
ip virtual-reassembly
ip route-cache flow
duplex auto
speed auto
!
interface FastEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
interface Serial0/0/0
description connected to the Internet
ip address X.X.X.186 255.255.255.252
ip access-group 101 in
ip inspect outbound out
ip nat outside
ip virtual-reassembly
ip route-cache flow
no ip mroute-cache
no cdp enable
!
ip classless
ip route 0.0.0.0 0.0.0.0 X.X.X.185
ip flow-export source FastEthernet0/0
ip flow-export version 5
ip flow-export destination 10.0.0.152 2055
!
no ip http server
ip http authentication local
no ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip nat pool inet_add X.X.X.38 X.X.X.38 netmask 255.255.255.248
ip nat inside source list 10 pool inet_add overload
ip nat inside source static 10.1.1.10 X.X.X.33
ip nat inside source static 10.0.0.218 X.X.X.34
ip nat inside source static 10.0.0.101 X.X.X.35
ip nat inside source static 10.1.1.12 X.X.X.36
!
logging trap debugging
logging 10.0.0.152
access-list 10 permit 10.0.0.0 0.255.255.255
access-list 101 permit tcp any host X.X.X.33 eq 1494
access-list 101 permit tcp any host X.X.X.33 eq www
access-list 101 permit tcp any host X.X.X.33 eq 2598
access-list 101 permit tcp any host X.X.X.33 eq 443
access-list 101 permit tcp any host X.X.X.34 eq www
access-list 101 permit tcp any host X.X.X.34 eq 3000
access-list 101 permit tcp any host X.X.X.34 eq 3001
access-list 101 permit tcp any host X.X.X.34 eq 3003
access-list 101 permit tcp any host X.X.X.34 eq 8800
access-list 101 permit tcp any host X.X.X.186 eq 22
access-list 101 permit tcp any host X.X.X.35 eq www
access-list 101 permit tcp any host X.X.X.35 eq 3000
access-list 101 permit tcp any host X.X.X.35 eq 389
access-list 101 permit tcp any host X.X.X.35 eq smtp
access-list 101 permit tcp any host X.X.X.35 eq pop3
access-list 101 permit tcp any host X.X.X.35 eq 587
access-list 101 permit tcp any host X.X.X.35 eq 143
access-list 101 permit tcp any host X.X.X.35 eq domain
access-list 101 permit tcp any host X.X.X.35 eq 3002
access-list 101 permit tcp any host X.X.X.35 eq 1000
access-list 101 permit tcp any host X.X.X.35 eq 366
access-list 101 permit tcp any host X.X.X.36 eq 1494
access-list 101 permit tcp any host X.X.X.36 eq 2598
access-list 101 permit tcp any host X.X.X.36 eq www
access-list 101 permit tcp any host X.X.X.36 eq 443
access-list 101 permit tcp any host X.X.X.36 eq 3389
access-list 101 deny ip any any
snmp-server community public RO
snmp-server enable traps tty
!
!
control-plane
!
banner motd CC
Unauthorized Access to this system is strictly prohibited!!
!
line con 0
exec-timeout 0 0
privilege level 15
line aux 0
password 7 130E1B14
modem InOut
modem autoconfigure type usr_sportster
transport input all
speed 19200
line vty 0 4
transport input telnet ssh
line vty 5 15
transport input telnet ssh
!
scheduler allocate 3000 1000
end

 

[burtsbees]

Instead of an acl that allows all those connections (remember, it's not a firewall), why not just use static NAT statements, and deny tcp connections that originate from the outside in just a few statements?
I am assuming that this is a dirty dmz you're protecting with acl 101? If so, you should be using a PIX...
But you could just deny ip from any to the whole dmz block in one statement, and permit ip any any, and use static NAT statements that specify tcp/udp ports to allow users from the outside to connect. As it is, it's got to sift through all those statements until it finds a match.

Burt

 

[Deadasbetamax]

Let me see if I am understanding this correctly. If not please correct me. The access-list 101 that I am attempting to put in to place is too long, which is causing too much use of the routers resources and while having to process the list. You believe that this is the cause of the issues I have been experiencing.

I could understand that. However, at least in my opinion that doesn't seem to be the case. The moment I put the first item in access-list 101 all NAT based web traffic begins to suffer greatly. As far as I can tell once the whole access-list is in place all traffic to statically assigned NAT addresses is fine. At least in my testing I could detect no performance loss. It's just the NAT based traffic that is suffering at least as far as I can tell.

In addition I had an access-list exactly the same as this one running on a CISCO 1721 up until Monday until the when it's flash memory died. I wouldn't think the 1841 would be unable to keep up with tasks which the 1721 could perform. I mean it could be that I am just misinformed. Again if any of this seems to be in error please let me know.

I have been pulling my hair out since Monday afternoon on this particular task and at this point I just want to get it done. This has not been my week.

 

[burtsbees]

So just http? On all 4 servers?
I was saying that about acl 101 because of you saying that it all works when you remove it from the interface...it actually does not make sense, because it goes through 2 statements to reach port 80 on .33 server, for example. If this is the only protocol having trouble, that may just be an indication of that particular protocol (http). You have an acl in place to permit traffic from the internet to those servers via those ports and deny everything else...you stated that you have trouble getting TO the internet.
What is the difference between the 1721 and this router in terms of DRAM, IOS and flash?

Burt

 

[brianinms]

1. An 1841 will smoke a 1721 in performance

2. The acl 101 isnt too long

3. The problem with the acl is that it is blocking all inbound traffic except for what you allowed.

 

[Deadasbetamax]

Then how do I go about letting in traffic over the NAT x.x.x.38. At least from the configuration of the 1721 it was my opinion that the CBAC's ip inspect commands were supposed to let the NAT traffic both in and out? I could go with with an permit any on x.x.x.38. This still leaves me with one problem.

On x.x.x.35 there is a proxy server providing a white-listed based access to the Internet. This proxy server uses a whole slew of ports for Internet connections. I saw a range from 2000 to about 8000 one time while monitoring it's traffic. So the only way to make sure those connections go through would be permit any x.x.x.35. Really don't want to do that.

Am I missing something with ip inspect portion of my config?

 

[brianinms]

Yes, CBAC can inspect traffic both ways, however when you block traffic with an acl then its blocked. What you have to do is configure your ACL to allow the services that you need, then block the ones you dont, then permit ip any any.

http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122cgcr/fsecur_c/ftrafwl/scfcbac.htm

 

[burtsbees]

I would think that if CBAC were protecting the dmz, then you would put the outbound ip inspect and acl 101 inbound on int fa0/0...can you post a topology? It seems you have CBAC going backwards...

Burt

 

[Deadasbetamax]

From the guide you posted
"CBAC creates temporary openings in access lists at firewall interfaces. These openings are created when specified traffic exits your internal network through the firewall. The openings allow returning traffic (that would normally be blocked) and additional data channels to enter your internal network back through the firewall. The traffic is allowed back through the firewall only if it is part of the same session as the original traffic that triggered CBAC when exiting through the firewall."

If this is true, should the commands

ip inspect dns-timeout 15
ip inspect name outbound tcp
ip inspect name outbound udp
ip inspect name outbound ftp
ip inspect name outbound realaudio
ip inspect name outbound smtp

pretty much allow any traffic through? I mean if the TCP and UDP are allowed, doesn't that pretty much any traffic generated from client computers would pass through any restrictions in place using the ACL? Sorry if I'm missing something. It's just that this portion and ACL 101 were taking directly from 1721 router. If it worked there should it not work here? I gotta be missing something.