1841 slow internet, must refresh pages several times.. whats to blame? 1

[nmessick]

I have an 1841 hooked to a bonded T1 router from our ISP. The 1841 is doing normal stuff, subnet routing, NAT and Firewall. For some reason when surfing the web we need to refresh pages several times before they will connect. Once they connect they load up very quickly, but the initial connection takes forever. Any ideas? It is the router, the problem does not happen on the other side of the interface.'


interface FastEthernet0/1
description Internet$FW_OUTSIDE$$ETH-LAN$
ip address 67.XXX.XX.XX 255.255.255.248
ip access-group 106 in
ip verify unicast reverse-path
no ip redirects
no ip unreachables
no ip proxy-arp
ip nbar protocol-discovery
ip inspect SDM_LOW in
ip inspect SDM_LOW out
ip nat outside
ip virtual-reassembly
ip route-cache flow
duplex auto
speed auto
no mop enabled
!

ip inspect name SDM_LOW icmp
ip inspect name SDM_LOW rcmd
ip inspect name SDM_LOW tcp alert on audit-trail on
ip inspect name SDM_LOW udp alert off audit-trail off
ip inspect name SDM_LOW ntp
ip tcp synwait-time 10

 

[North323]

need the whole config

 

[nmessick]

Current configuration : 15721 bytes
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service sequence-numbers
!
hostname E-town
!
boot-start-marker
boot-end-marker
!
security authentication failure rate 3 log
security passwords min-length 6
logging buffered 51200 debugging
logging console critical

!
aaa new-model
ip inspect name SDM_LOW icmp
ip inspect name SDM_LOW rcmd
ip inspect name SDM_LOW tcp alert on audit-trail on
ip inspect name SDM_LOW udp alert off audit-trail off
ip inspect name SDM_LOW ntp
ip tcp synwait-time 10
!
!
ip ips notify SDEE
no ip bootp server
ip name-server xxx.160.xxx.2
ip name-server xx.xxx.xx.5
!
!
ip tcp synwait-time 10
!
!
ip ips notify SDEE
no ip bootp server
ip name-server xxx.160.xxx.2
ip name-server xx.xxx.xx.5
!
!
!
crypto pki trustpoint TP-self-signed-
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate
revocation-check none
rsakeypair TP-self-signed-1438219780
!
crypto pki trustpoint tti
revocation-check crl
rsakeypair tti
!
!
crypto pki certificate chain TP-self-signed-

!
!
class-map match-any voice-signaling
match ip dscp cs3
match ip dscp af31
class-map match-any voice
match ip dscp ef
match ip precedence 5
!
!
policy-map voice-qos
class voice
priority percent 20
class voice-signaling
bandwidth percent 5
class class-default
fair-queue
!
!
!
!
!
interface Null0
no ip unreachables
!
interface FastEthernet0/0
description "Data Subnet"
no ip address
no ip redirects
no ip unreachables
no ip proxy-arp
ip route-cache flow
duplex auto
speed auto
no mop enabled
!
interface FastEthernet0/0.1
description $ETH-LAN$$FW_INSIDE$
encapsulation dot1Q 1 native
ip address xx.xxx.xx.200 255.255.255.0
ip access-group 104 in
no ip redirects
no ip unreachables
no ip proxy-arp
ip inspect SDM_LOW in
ip nat inside
ip virtual-reassembly
no snmp trap link-status
!
interface FastEthernet0/0.2
description $ETH-LAN$$FW_INSIDE$
encapsulation dot1Q 2
ip address xx.xxx.xx.1 255.255.255.0
ip access-group 105 in
ip helper-address xx.xxx.xx.1
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat inside
ip virtual-reassembly
no snmp trap link-status
!
interface FastEthernet0/1
description Internet$FW_OUTSIDE$$ETH-LAN$
ip address xx.xxx.xx.74 255.255.255.248
ip access-group 106 in
ip verify unicast reverse-path
no ip redirects
no ip unreachables
no ip proxy-arp
ip nbar protocol-discovery
ip inspect SDM_LOW in
ip inspect SDM_LOW out
ip nat outside
ip virtual-reassembly
ip route-cache flow
duplex auto
speed auto
no mop enabled
!
interface Serial0/0/0
description "T1 to Abbottstown"$FW_INSIDE$
ip address xx.xxx.xx00.1 255.255.255.252
ip access-group 102 in
ip helper-address 192.158.2.95
no ip redirects
no ip unreachables
no ip proxy-arp
ip nbar protocol-discovery
ip nat inside
ip virtual-reassembly
ip route-cache flow
service-policy output voice-qos
!
ip classless
ip route 0.0.0.0 0.0.0.0 xx.xxx.xx.77
ip route 192.168.1.0 255.255.255.0 xx.xxx.xx00.2
ip route xx.xxx.xx.0 255.255.255.0 xx.xxx.xx00.2
ip route xx.xxx.xx.0 255.255.255.0 xx.xxx.xx.2
!
ip http server
ip http authentication local
ip http secure-server
ip http timeout-policy idle 600 life 86400 requests 10000
ip nat pool Addresses xx.xxx.xx.72 xx.xxx.xx.77 netmask 255.255.255.248
ip nat inside source list ToTheInternet interface FastEthernet0/1 overload
ip nat inside source static tcp xx.xxx.xx.5 25 xx.xxx.xx.73 25 extendable
ip nat inside source static tcp xx.xxx.xx.5 80 xx.xxx.xx.73 80 extendable
ip nat inside source static tcp xx.xxx.xx.5 443 xx.xxx.xx.73 443 extendable
ip nat inside source static tcp xx.xxx.xx.2 80 xx.xxx.xx.74 80 extendable
ip nat inside source static tcp xx.xxx.xx.2 443 xx.xxx.xx.74 443 extendable
ip nat inside source static tcp xx.xxx.xx.2 1723 xx.xxx.xx.74 1723 extendable
ip nat inside source static tcp xx.xxx.xx.201 20 xx.xxx.xx.76 20 extendable
ip nat inside source static tcp xx.xxx.xx.201 21 xx.xxx.xx.76 21 extendable
ip nat inside source static tcp xx.xxx.xx.4 80 xx.xxx.xx.76 80 extendable
!
ip access-list extended ToTheInternet
remark Neils Rule
remark SDM_ACL Category=2
remark AnyAny
permit ip any any
remark Neils Rule
remark SDM_ACL Category=2
remark AnyAny
!
logging trap debugging
access-list 1 remark SDM_ACL Category=2
access-list 1 permit xx.xxx.xx.0 0.0.0.255
access-list 1 permit xx.xxx.xx00.0 0.0.0.3
access-list 100 remark auto generated by SDM firewall configuration
access-list 100 remark SDM_ACL Category=1
access-list 100 permit ip any any
access-list 100 remark auto generated by SDM firewall configuration
access-list 100 remark SDM_ACL Category=1
access-list 101 remark auto generated by SDM firewall configuration
access-list 101 remark SDM_ACL Category=1
access-list 101 deny ip xx.xxx.xx00.0 0.0.0.3 any
access-list 101 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 101 deny ip xx.xxx.xx.72 0.0.0.7 any
access-list 101 deny ip host 255.255.255.255 any
access-list 101 deny ip 127.0.0.0 0.255.255.255 any
access-list 101 permit ip any any
access-list 101 remark auto generated by SDM firewall configuration
access-list 101 remark SDM_ACL Category=1
access-list 102 remark auto generated by SDM firewall configuration
access-list 102 remark SDM_ACL Category=1
access-list 102 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 102 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 102 deny ip xx.xxx.xx.72 0.0.0.7 any
access-list 102 deny ip host 255.255.255.255 any
access-list 102 deny ip 127.0.0.0 0.255.255.255 any
access-list 102 permit ip any any
access-list 102 remark auto generated by SDM firewall configuration
access-list 102 remark SDM_ACL Category=1
access-list 103 remark auto generated by SDM firewall configuration
access-list 103 remark SDM_ACL Category=1
access-list 103 permit udp any host xx.xxx.xx.74 eq isakmp
access-list 103 permit tcp any host xx.xxx.xx.74 eq 50
access-list 103 permit udp any host xx.xxx.xx.74 eq non500-isakmp
access-list 103 permit tcp any host xx.xxx.xx.74 eq 443
access-list 103 permit tcp any host xx.xxx.xx.74 eq www
access-list 103 permit tcp any host xx.xxx.xx.73 eq 443
access-list 103 permit tcp any host xx.xxx.xx.73 eq www
access-list 103 permit tcp any host xx.xxx.xx.73 eq smtp
access-list 103 deny ip xx.xxx.xx00.0 0.0.0.3 any
access-list 103 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 103 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 103 permit udp any eq bootps any eq bootps
access-list 103 permit icmp any host xx.xxx.xx.74 echo-reply
access-list 103 permit icmp any host xx.xxx.xx.74 time-exceeded
access-list 103 permit icmp any host xx.xxx.xx.74 unreachable
access-list 103 deny ip 10.0.0.0 0.255.255.255 any
access-list 103 deny ip 172.16.0.0 0.15.255.255 any
access-list 103 deny ip 192.168.0.0 0.0.255.255 any
access-list 103 deny ip 127.0.0.0 0.255.255.255 any
access-list 103 deny ip host 255.255.255.255 any
access-list 103 deny ip host 0.0.0.0 any
access-list 103 deny ip any any log
access-list 103 remark auto generated by SDM firewall configuration
access-list 103 remark SDM_ACL Category=1
access-list 104 remark auto generated by SDM firewall configuration
access-list 104 remark SDM_ACL Category=1
access-list 104 remark Kevin Warrenty
access-list 104 permit ip any host xxx.35.xx.246
access-list 104 remark SMTP allow for server1
access-list 104 permit tcp host xx.xxx.xx.5 any eq smtp
access-list 104 remark Deny All SMTP from spam bots
access-list 104 deny tcp any any eq smtp
access-list 104 deny ip xx.xxx.xx00.0 0.0.0.3 any
access-list 104 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 104 deny ip xx.xxx.xx.72 0.0.0.7 any
access-list 104 deny ip host 255.255.255.255 any
access-list 104 deny ip 127.0.0.0 0.255.255.255 any
access-list 104 permit ip any any
access-list 104 remark auto generated by SDM firewall configuration
access-list 104 remark SDM_ACL Category=1
access-list 104 remark Kevin Warrenty
access-list 104 remark SMTP allow for server1
access-list 104 remark Deny All SMTP from spam bots
access-list 105 remark auto generated by SDM firewall configuration
access-list 105 remark SDM_ACL Category=1
access-list 105 deny ip xx.xxx.xx00.0 0.0.0.3 any
access-list 105 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 105 deny ip xx.xxx.xx.72 0.0.0.7 any
access-list 105 deny ip host 255.255.255.255 any
access-list 105 deny ip 127.0.0.0 0.255.255.255 any
access-list 105 permit ip any any
access-list 105 remark auto generated by SDM firewall configuration
access-list 105 remark SDM_ACL Category=1
access-list 106 remark auto generated by SDM firewall configuration
access-list 106 remark SDM_ACL Category=1
access-list 106 permit tcp any host xx.xxx.xx.76 eq ftp-data
access-list 106 remark Keivn Warrenty
access-list 106 permit ip host 129.35.117.246 any
access-list 106 remark search appliance
access-list 106 permit tcp any host xx.xxx.xx.76 eq www
access-list 106 remark AS400 FTP
access-list 106 permit tcp any host xx.xxx.xx.76 eq ftp
access-list 106 permit tcp any host xx.xxx.xx.74 eq 1723
access-list 106 remark VPN Port
access-list 106 permit gre any host xx.xxx.xx.74 log
access-list 106 permit udp any host xx.xxx.xx.74 eq isakmp
access-list 106 permit tcp any host xx.xxx.xx.74 eq 50
access-list 106 permit udp any host xx.xxx.xx.74 eq non500-isakmp
access-list 106 permit tcp any host xx.xxx.xx.74 eq 443
access-list 106 permit tcp any host xx.xxx.xx.74 eq www
access-list 106 permit tcp any host xx.xxx.xx.73 eq 443
access-list 106 permit tcp any host xx.xxx.xx.73 eq www
access-list 106 permit tcp any host xx.xxx.xx.73 eq smtp
access-list 106 permit udp host 205.160.192.2 eq domain host xx.xxx.xx.74
access-list 106 deny ip xx.xxx.xx00.0 0.0.0.3 any
access-list 106 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 106 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 106 permit udp any eq bootps any eq bootps
access-list 106 permit icmp any host xx.xxx.xx.74 echo-reply
access-list 106 permit icmp any host xx.xxx.xx.74 time-exceeded
access-list 106 permit icmp any host xx.xxx.xx.74 unreachable
access-list 106 deny ip 10.0.0.0 0.255.255.255 any
access-list 106 deny ip xxx.16.0.0 0.15.255.255 any
access-list 106 deny ip 192.xxx.0.0 0.0.255.255 any
access-list 106 deny ip xxx.0.0.0 0.255.255.255 any
access-list 106 deny ip host 255.255.255.255 any
access-list 106 deny ip host 0.0.0.0 any
access-list 106 deny ip any any log
access-list 106 remark auto generated by SDM firewall configuration
access-list 106 remark SDM_ACL Category=1
access-list 106 remark Keivn Warrenty
access-list 106 remark search appliance
access-list 106 remark AS400 FTP
access-list 106 remark VPN Port
access-list 107 remark auto generated by SDM firewall configuration
access-list 107 remark SDM_ACL Category=1
access-list 107 deny ip xx.xxx.xx00.0 0.0.0.3 any
access-list 107 deny ip xx.xxx.xx.0 0.0.0.255 any
access-list 107 deny ip xx.xxx.xx.72 0.0.0.7 any
access-list 107 deny ip host 255.255.255.255 any
access-list 107 deny ip 127.0.0.0 0.255.255.255 any
access-list 107 permit ip any any
no cdp run
!
!
control-plane
!
banner login !^C
!
line con 0
transport output telnet
line aux 0
transport output telnet
line vty 0 4

transport input telnet
!
scheduler allocate 4000 1000
end

 

[nmessick]

FE0/1 is the interface to the internet.

FE0/0 is spilt into 2 VLANS that do routing, the serial interface is a point to point to another location. There is QOS for VOIP on the internal interfaces.

 

[North323]

two things i wonder
1) you have several routes but only 1 gateway of last resort, those other routes are not being used?
ip route 0.0.0.0 0.0.0.0 xx.xxx.xx.77
ip route 192.168.1.0 255.255.255.0 xx.xxx.xx00.2
ip route xx.xxx.xx.0 255.255.255.0 xx.xxx.xx00.2
ip route xx.xxx.xx.0 255.255.255.0 xx.xxx.xx.2

doesnt 192.168.1.0 fall into 0.0.0.0?

why do you have this? ip tcp synwait-time 10 this affects all tcp connections, including internet. try changing that or removing...default is 30 seconds

 

[North323]

download wireshark and take a packet capture.....

 

[burtsbees]

Initially it sounds like MTU...how many users are we talking? Your router on a stick is a definite bottleneck, especially with an 1841 processor engine doing all the packet processing for the firewall and acl filtering. Try this on the fa0/1

E-town>en
E-town#conf t
E-town(config)#int fa0/1
E-town(config-if)#ip tcp adjust-mss 1452
E-town(config-if)#ip mtu 1492

It looks like with the vlan tagging with router on a stick, the ethernet frames are too big, like 1512 (for 802.1q). You could do an extended ping with extended options and set the df bit to see where the MTU needs to be set.

Other things you could do---don't do router on a stick, ry eliminating one acl at a time, etc. to see if and when things get better. My guess is MTU and the bottleneck with router on a stick, and of course the proc being overloaded with cycles to process the firewall, acl's, etc.

Look at sh int fa0/1, and see how many giant frames might be being dropped.

Burt

 

[North323]

burt...if its mtu, wouldnt that always be an issue and he would never get to the internet?

 

[burtsbees]

Nope---it's really common with ADSL, with ethernet frames being 1500 bytes, and adsl needing 1492 for some reason (cannot remember)...some sites will work fine, others would need constant refreshes. Cisco.com, ironically, is one of these sites! In this case, it looks like the ethernet frames are expected to be 1500 with the T1, but are actually larger with the 802.1q tags...just a guess...we'll see, I suppose...a couple of mss and mtu tweaks will tell the tale.

Now if the df bit were set, then they would have a lot of dropped packets, but still would be able to get to some sites...

Burt

 

[North323]

awesome...good explanation...if the df packet is not set, they could use mtu path discovery?

 

[burtsbees]

Yes, exactly.

Burt

 

[nmessick]

I adjusted the MTU, that did not take care of it.
CPU load never really gets over 15-20%. There is very little traffic going between the VLAN's, the 2nd VLAN is a VOIP traffic segment and all that really goes over that is some signaling and administration.. no browsing, file sharing, etc. We do hammer the internet connection pretty hard internaly, plus we do exchange, webhosting, VPN.


I was thinking it maybe the firewall packet inspection... but if its this.. ACL's etc... shouldn't I see the CPU load being higher?


(with MTU adjusted)
Repeat count [5]: 5000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]: yes
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5000, 100-byte ICMP Echos to 74.125.67.100, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!..!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!
!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!
!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!
..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
.!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!
!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!
!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!.

 

[burtsbees]

Post a sh interface and let's look at some counters, loads, buffers, drops, etc. Also, post a sh access-list and verify the matches to see where the majority of traffic is...

Burt

 

[nmessick]

FastEthernet0/0 is up, line protocol is up
Hardware is Gt96k FE, address is 0017.595e.5734 (bia 0017.595e.5734)
Description: "Data Subnet"
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/525/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 443000 bits/sec, 126 packets/sec
5 minute output rate 801000 bits/sec, 138 packets/sec
207101029 packets input, 1138306940 bytes
Received 6884281 broadcasts, 0 runts, 0 giants, 0 throttles
13378 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
204491111 packets output, 830476776 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
FastEthernet0/0.1 is up, line protocol is up
Hardware is Gt96k FE, address is 0017.595e.5734 (bia 0017.595e.5734)
Description: $ETH-LAN$$FW_INSIDE$
Internet address is 192.168.2.200/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1.
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters never
FastEthernet0/0.2 is up, line protocol is up
Hardware is Gt96k FE, address is 0017.595e.5734 (bia 0017.595e.5734)
Description: $ETH-LAN$$FW_INSIDE$
Internet address is 192.168.4.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 2.
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters never
FastEthernet0/1 is up, line protocol is up
Hardware is Gt96k FE, address is 0017.595e.5735 (bia 0017.595e.5735)
Description: Internet$FW_OUTSIDE$$ETH-LAN$
Internet address is XXXX/29
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/110/0 (size/max/drops/flushes); Total output drops: 109
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 720000 bits/sec, 89 packets/sec
5 minute output rate 288000 bits/sec, 73 packets/sec
111283688 packets input, 2544427114 bytes
Received 30843 broadcasts, 0 runts, 0 giants, 0 throttles
409 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
102491341 packets output, 2828776774 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial0/0/0 is up, line protocol is up
Hardware is GT96K with integrated T1 CSU/DSU
Description: "T1 to Abbottstown"$FW_INSIDE$
Internet address is 192.168.200.1/30
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 14/255, rxload 2/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input 00:00:03, output 00:00:07, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1675
Queueing strategy: Class-based queueing
Output queue: 0/1000/64/1675 (size/max total/threshold/drops)
Conversations 0/22/256 (active/max active/max total)
Reserved Conversations 1/1 (allocated/max allocated)
Available Bandwidth 773 kilobits/sec
5 minute input rate 13000 bits/sec, 27 packets/sec
5 minute output rate 89000 bits/sec, 30 packets/sec
63436963 packets input, 2267964727 bytes, 0 no buffer
Received 278809 broadcasts, 0 runts, 0 giants, 0 throttles
1351 input errors, 1350 CRC, 499 frame, 197 overrun, 0 ignored, 741 abort
66879848 packets output, 895051379 bytes, 0 underruns
0 output errors, 0 collisions, 5 interface resets
0 output buffer failures, 0 output buffers swapped out
3 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

NVI0 is up, line protocol is up
Hardware is NVI
MTU 1514 bytes, BW 10000000 Kbit, DLY 0 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation UNKNOWN, loopback not set
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out



E-town# sh access-list
Standard IP access list 1
10 permit 192.168.4.0, wildcard bits 0.0.0.255
20 permit 192.168.200.0, wildcard bits 0.0.0.3
Extended IP access list 100
10 permit ip any any (1192 matches)
Extended IP access list 101
10 deny ip 192.168.200.0 0.0.0.3 any
20 deny ip 192.168.2.0 0.0.0.255 any
30 deny ip xx.xxx.62.72 0.0.0.7 any
40 deny ip host 255.255.255.255 any
50 deny ip 127.0.0.0 0.255.255.255 any
60 permit ip any any
Extended IP access list 102
10 deny ip 192.168.4.0 0.0.0.255 any
20 deny ip 192.168.2.0 0.0.0.255 any (65 matches)
30 deny ip xx.xxx.62.72 0.0.0.7 any (706118 matches)
40 deny ip host 255.255.255.255 any
50 deny ip 127.0.0.0 0.255.255.255 any
60 permit ip any any (63001339 matches)
Extended IP access list 103
10 permit udp any host xx.xxx.62.74 eq isakmp
20 permit tcp any host xx.xxx.62.74 eq 50
30 permit udp any host xx.xxx.62.74 eq non500-isakmp
40 permit tcp any host xx.xxx.62.74 eq 443
50 permit tcp any host xx.xxx.62.74 eq www
60 permit tcp any host xx.xxx.62.73 eq 443
70 permit tcp any host xx.xxx.62.73 eq www
80 permit tcp any host xx.xxx.62.73 eq smtp
90 deny ip 192.168.200.0 0.0.0.3 any
100 deny ip 192.168.4.0 0.0.0.255 any
110 deny ip 192.168.2.0 0.0.0.255 any
120 permit udp any eq bootps any eq bootps
130 permit icmp any host xx.xxx.62.74 echo-reply
140 permit icmp any host xx.xxx.62.74 time-exceeded
150 permit icmp any host xx.xxx.62.74 unreachable
160 deny ip 10.0.0.0 0.255.255.255 any
170 deny ip 172.16.0.0 0.15.255.255 any
180 deny ip 192.168.0.0 0.0.255.255 any
190 deny ip 127.0.0.0 0.255.255.255 any
200 deny ip host 255.255.255.255 any
210 deny ip host 0.0.0.0 any
220 deny ip any any log
Extended IP access list 104
10 permit ip any host 129.35.117.246 (159296 matches)
20 permit tcp host 192.168.2.5 any eq smtp (1219189 matches)
30 deny tcp any any eq smtp (11886 matches)
40 deny ip 192.168.200.0 0.0.0.3 any
50 deny ip 192.168.4.0 0.0.0.255 any (165 matches)
60 deny ip xx.xxx.62.72 0.0.0.7 any (5958800 matches)
70 deny ip host 255.255.255.255 any
80 deny ip 127.0.0.0 0.255.255.255 any
90 permit ip any any (109050101 matches)
Extended IP access list 105
10 deny ip 192.168.200.0 0.0.0.3 any
20 deny ip 192.168.2.0 0.0.0.255 any (47 matches)
30 deny ip xx.xxx.62.72 0.0.0.7 any (97170 matches)
40 deny ip host 255.255.255.255 any
50 deny ip 127.0.0.0 0.255.255.255 any
60 permit ip any any (71787696 matches)
Extended IP access list 106
10 permit tcp any host xx.xxx.62.76 eq ftp-data
20 permit ip host 129.35.117.246 any (522 matches)
30 permit tcp any host xx.xxx.62.76 eq

http://www (40110

matches)
40 permit tcp any host xx.xxx.62.76 eq ftp (845057 matches)
50 permit tcp any host xx.xxx.62.74 eq 1723 (60393 matches)
60 permit gre any host xx.xxx.62.74 log (561860 matches)
70 permit udp any host xx.xxx.62.74 eq isakmp (443 matches)
80 permit tcp any host xx.xxx.62.74 eq 50
90 permit udp any host xx.xxx.62.74 eq non500-isakmp
100 permit tcp any host xx.xxx.62.74 eq 443 (247824 matches)
110 permit tcp any host xx.xxx.62.74 eq

http://www (17058522

matches)
120 permit tcp any host xx.xxx.62.73 eq 443 (8 matches)
130 permit tcp any host xx.xxx.62.73 eq

http://www (8652

matches)
140 permit tcp any host xx.xxx.62.73 eq smtp (840141 matches)
150 permit udp host 205.160.192.2 eq domain host xx.xxx.62.74 (784 matches)
160 deny ip 192.168.200.0 0.0.0.3 any
170 deny ip 192.168.4.0 0.0.0.255 any
180 deny ip 192.168.2.0 0.0.0.255 any (5 matches)
190 permit udp any eq bootps any eq bootps
200 permit icmp any host xx.xxx.62.74 echo-reply
210 permit icmp any host xx.xxx.62.74 time-exceeded (15149 matches)
220 permit icmp any host xx.xxx.62.74 unreachable (364570 matches)
230 deny ip 10.0.0.0 0.255.255.255 any (22 matches)
240 deny ip 172.16.0.0 0.15.255.255 any (25 matches)
250 deny ip 192.168.0.0 0.0.255.255 any (1157 matches)
260 deny ip 127.0.0.0 0.255.255.255 any
270 deny ip host 255.255.255.255 any
280 deny ip host 0.0.0.0 any
290 deny ip any any log (989918 matches)
Extended IP access list 107
10 deny ip 192.168.200.0 0.0.0.3 any
20 deny ip 192.168.4.0 0.0.0.255 any
30 deny ip xx.xxx.62.72 0.0.0.7 any (1184 matches)
40 deny ip host 255.255.255.255 any
50 deny ip 127.0.0.0 0.255.255.255 any
60 permit ip any any (12715 matches)
Extended IP access list ToTheInternet
10 permit ip any any (7019755 matches)
E-town#

 

[burtsbees]

I see several input errors on the LAN interface and several output errors on the WAN interface, as well as interface resets. Packets were queued for transmission, but never went out for some reason or another, which will reset the interface. You have also had 3 carrier transitions, which is not a big deal. Looks like the problem is coming from packets traveling to the fa0/0 interface (router on a stick). Do you have a layer 3 switch laying around? You could test to see if the bottleneck at the router-on-a-stick is causing the problem. It does not look like actual packet errors (runts, CRC, etc), as those would likely be reported in "sh int fa0/0".

Please post a sh logg, and any syslog you have for the past 24 hours or so. Also, you should really clear the counters every few days...you at LEAST need to

router(config)#clear counters

on all interfaces and let these counters increment for a few days. Myself, I have KRON reset the counters every 3 days and NAT translations every week...just a thought...

In the meantime, you could try and put a different router in this one's place, or place a L3 switch to router the vlans to take the bottleneck away, all this to try and rule out layer 1. How many users are on each vlan?

/

 

[nmessick]

The switch attached to that interface does not show any errors on that port, yet the router does. Would that indicate anything?

There is a very minimal amount of traffic going over these links. 6-8 channels of VOIP and whatever traffic ends up comming over from the point to point in S0/0/0... 1.5 mbit there..? I can't see enough traffic to be causing a bottleneck with the router on a stick config.

sh logg only has stuff on the firewall. Lots of stuff on TCP sessions. Nothing I see thats relevant to the ports.


Output for the last hour since I cleared the logs.

5 minute output rate 770000 bits/sec, 204 packets/sec
738890 packets input, 205147047 bytes
Received 12768 broadcasts, 0 runts, 0 giants, 0 throttles
11 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
788303 packets output, 299166877 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

 

[burtsbees]

Have you tried a different router to rule out the physical interface having problems??? That is what it is starting to look like (odd with no CRC errors, or maybe that fact makes PERFECT sense if it is the interface...).

Also, something that rarely shows anything with an intermittently faulty interface, nevertheless worth a try, is

router(config)#test int fa0/0

/

 

[nmessick]

...weird?

E-town#test int
No IP address for FastEthernet0/0.
Skipping...
Test FastEthernet0/1 [y/n] ? y
... Failed - timeout problem
Test Serial0/0/0 [y/n] ? y
... Failed - timeout problem
No IP address for NVI0.
Skipping...
4 interfaces: 0 passed, 2 failed, 0 skipped, 2 untestable

 

[nmessick]

Alright. I did some more work on this. I snagged a 2801 router and hooked it up to the internet port. At first I thought I had licked the problem, web pages loaded in a snap, but after 10-15 minutes this new router was running just as slow as the old one. What the heck is the problem!?!!? I'm seeing 150-400 open connections in the firewall. I've disabled all the application inspection types... what else could it be?

 

[nmessick]

oh this ticks me off! I think I may have found it. I disabled logging and things sped up substantially. What are the odds that this has been the problem all along?