Cisco Routing Problem

[noviwinger]

Have a Cisco 3640 router that stops routing to one destination. The destination uses 2-T1's and is configured for packet-based loadbalancing.

The symptoms during the routing problem are:
1.) From the ethernet side cannot ping the single destination
2.) inside router - Can ping all destinations.
3.) routing stops randomly
4.) when the routing stops, other destinations on the router still work fine.
5.) router sends the packets to the default route, in our case our internet firewall.

Any assistance you can provide will be most appreciated

 

[dearingkr]

Going to need a sanitized 'sh run'

MCSE CCNA CCDA

 

[noviwinger]

Here is a pretty sanitized verision:

Interfaces that are shutdown are removed:

version 12.2
no service single-slot-reload-enable
service timestamps debug uptime
service timestamps log datetime localtime
no service password-encryption
no service dhcp
!
hostname ROUTER
!
ip subnet-zero
ip cef
!
!
no ip finger
!
ip dhcp-server 172.16.18.24
no ip dhcp-client network-discovery
!
key chain FH
key 1
key-string jbw
!
!
!
interface FastEthernet0/0
description to FH Network
ip address 172.16.18.1 255.255.0.0
ip access-group 115 in
no ip unreachables
ip rip authentication key-chain FH
speed 100
full-duplex
!
interface FastEthernet0/0.100
description VLAN 100 Secured Wireless
encapsulation dot1Q 100
ip address 222.2.2.1 255.255.255.0
ip helper-address 172.16.18.24
!
interface FastEthernet0/0.200
description VLAN 200 Guest Wireless
encapsulation dot1Q 200
ip address 172.28.2.1 255.255.255.0
ip helper-address 172.16.18.24

interface Serial1/1
description PROBLEM-ROUTE-Second T1
ip address 199.16.25.13 255.255.255.252
ip load-sharing per-packet
ip rip authentication key-chain FH
!
interface Serial2/0
description UNAFFECTED ROUTE 1
bandwidth 1216
ip address 199.16.25.25 255.255.255.252
ip rip authentication key-chain FH
!
interface Serial2/1
description PROBLEM-ROUTE-FIRST-T1
bandwidth 1216
ip address 199.16.25.29 255.255.255.252
ip load-sharing per-packet
ip rip authentication key-chain FH
!
interface Serial2/2
description UNAFFECTED ROUTE 2
bandwidth 1216
ip address 199.16.25.33 255.255.255.252
ip rip authentication key-chain FH
!
interface Serial2/3
description UNAFFECTED ROUTE 3
bandwidth 1216
ip address 199.16.25.17 255.255.255.252
ip rip authentication key-chain FH
!
interface Serial3/1
description ISDN-Backup-ROUTE
bandwidth 128
ip address 199.16.25.101 255.255.255.252
ip rip authentication key-chain FH
encapsulation ppp
dialer in-band
dialer map ip 199.16.25.102 name OMITTED
dialer-group 1
no fair-queue
pulse-time 1
ppp authentication chap
!
!
router rip
version 2
redistribute static
passive-interface FastEthernet0/0
network 172.16.0.0
network 199.16.16.0
network 199.16.21.0
network 199.16.20.0
neighbor 199.16.25.14
neighbor 199.16.25.30
neighbor 199.16.25.18
neighbor 199.16.25.102
no auto-summary
!
ip classless
ip route 0.0.0.0 0.0.0.0 172.16.18.254 - DEFAULT ROUTE
OMITED IP ROUTES
ip route 199.16.16.0 255.255.255.0 199.16.25.30 AFFECTED ROUTE
ip route 199.16.16.0 255.255.255.0 199.16.25.14 AFFECTED ROUTE
ip route 199.16.16.0 255.255.255.0 Serial2/1 AFFECTED ROUTE-RECENTLY ADDED AFTER LAST DROP
ip route 199.16.16.0 255.255.255.0 Serial1/1 AFFECTED ROUTE-RECENTLY ADDED AFTER LAST DROP
ip route 199.16.16.0 255.255.255.0 199.16.25.102 200 -ISDN BACKUP ROUTE
ip route 199.16.20.0 255.255.255.0 199.16.25.30
ip route 199.16.20.0 255.255.255.0 199.16.25.14
ip route 199.16.20.0 255.255.255.0 199.16.25.102 200
ip route 199.16.21.0 255.255.255.0 199.16.25.30
ip route 199.16.21.0 255.255.255.0 199.16.25.14
ip route 199.16.21.0 255.255.255.0 199.16.25.102 200
ip route 199.16.25.100 255.255.255.252 Serial3/1
ip route 199.16.26.0 255.255.255.0 199.16.25.30
ip route 199.16.26.0 255.255.255.0 199.16.25.14
ip route 199.16.26.0 255.255.255.0 199.16.25.102 200
no ip http server
!
logging history debugging
logging trap debugging
logging source-interface FastEthernet0/0
logging 172.16.17.18
dialer-list 1 protocol ip permit
terminal-queue entry-retry-interval 1
SNMP OMITED
banner motd 
**********************************
RESTRICTED SYSTEM
AUTHORIZED USERS ONLY
**********************************

!
line con 0
transport input none
line aux 0
line vty 0 4
password netman
login
!
end

 

[dearingkr]

What destination are you having problems with?

I'm assuming 172.16.18.254 is another router/firewall on the LAN segment?
Is it running rip?

Also please post 'sh ip route'


MCSE CCNA CCDA

 

[noviwinger]

Here is the sh ip route while it is working:

Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route

Gateway of last resort is 172.16.18.254 to network 0.0.0.0

S 192.168.12.0/24 [1/0] via 172.16.155.5
S 192.168.29.0/24 [1/0] via 172.16.18.254
S 199.16.26.0/24 [1/0] via 199.16.25.30
[1/0] via 199.16.25.14
C 222.2.2.0/24 is directly connected, FastEthernet0/0.100
S 199.16.27.0/24 is directly connected, Serial2/0
is directly connected
S 192.168.14.0/24 [1/0] via 172.16.18.254
S 192.168.15.0/24 [1/0] via 172.16.155.4
S 192.168.240.0/24 [1/0] via 172.16.18.99
192.168.150.0/27 is subnetted, 6 subnets
S 192.168.150.160 [1/0] via 172.16.21.35
S 192.168.150.128 [1/0] via 172.16.21.30
S 192.168.150.96 [1/0] via 172.16.21.25
S 192.168.150.64 [1/0] via 172.16.21.20
S 192.168.150.32 [1/0] via 172.16.21.15
S 192.168.150.0 [1/0] via 172.16.21.10
199.16.25.0/30 is subnetted, 8 subnets
C 199.16.25.100 is directly connected, Serial3/1
C 199.16.25.12 is directly connected, Serial1/1
C 199.16.25.28 is directly connected, Serial2/1
C 199.16.25.24 is directly connected, Serial2/0
C 199.16.25.16 is directly connected, Serial2/3
C 199.16.25.32 is directly connected, Serial2/2
R 199.16.25.52 [120/1] via 199.16.25.14, 00:00:06, Serial1/1
[120/1] via 199.16.25.30, 00:00:06, Serial2/1
R 199.16.25.48 [120/1] via 199.16.25.14, 00:00:06, Serial1/1
[120/1] via 199.16.25.30, 00:00:06, Serial2/1
S 192.168.110.0/24 [1/0] via 172.16.18.98
192.168.24.0/25 is subnetted, 1 subnets
S 192.168.24.0 [1/0] via 172.16.18.254
S 192.168.111.0/24 [1/0] via 199.16.25.30
S 192.168.77.0/24 [1/0] via 172.16.155.3
S 192.168.10.0/24 [1/0] via 172.16.155.1
S 192.168.27.0/24 [1/0] via 172.16.155.2
172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
S 172.16.20.0/24 [1/0] via 172.16.18.254
C 172.16.0.0/16 is directly connected, FastEthernet0/0
S 172.19.0.0/16 [1/0] via 172.16.18.254
S 172.18.0.0/16 is directly connected, Serial2/3
172.21.0.0/16 is variably subnetted, 2 subnets, 2 masks
S 172.21.10.0/23 [1/0] via 172.16.18.254
S 172.21.12.0/24 [1/0] via 172.16.155.2
172.20.0.0/24 is subnetted, 1 subnets
S 172.20.10.0 [1/0] via 172.16.155.7
172.28.0.0/24 is subnetted, 2 subnets
S 172.28.1.0 is directly connected, FastEthernet0/0.100
C 172.28.2.0 is directly connected, FastEthernet0/0.200
67.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
S 67.70.189.133/32 [1/0] via 172.16.18.254
S 67.0.0.0/8 [1/0] via 172.16.18.254
S 52.0.0.0/8 [1/0] via 172.16.18.254
S 199.16.29.0/24 is directly connected, Serial2/2
is directly connected
S 199.16.18.0/24 [1/0] via 172.16.18.254
S 192.168.98.0/24 [1/0] via 172.16.18.99
S 192.168.201.0/24 [1/0] via 172.16.18.254
S 199.16.16.0/24 is directly connected
is directly connected
is directly connected, Serial2/1
is directly connected, Serial1/1
57.0.0.0/8 is variably subnetted, 11 subnets, 6 masks
S 57.23.36.0/24 [1/0] via 172.16.18.99
S 57.22.164.128/26 [1/0] via 172.16.18.99
S 57.22.157.128/26 [1/0] via 172.16.18.99
S 57.0.0.0/8 [1/0] via 172.16.18.254
S 57.14.78.17/32 [1/0] via 172.16.18.99
S 57.14.78.18/32 [1/0] via 172.16.18.99
S 57.69.0.0/16 [1/0] via 172.16.18.254
S 57.14.71.192/27 [1/0] via 172.16.18.99
S 57.14.71.160/27 [1/0] via 172.16.18.99
S 57.14.74.160/27 [1/0] via 172.16.18.99
S 57.14.1.192/26 [1/0] via 172.16.18.99
10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S 10.0.0.0/8 [1/0] via 172.16.18.254
S 10.5.2.0/24 [1/0] via 172.16.18.99
S 10.48.30.0/24 [1/0] via 172.16.18.254
S 10.205.15.0/25 [1/0] via 172.16.18.254
S 192.168.23.0/24 [1/0] via 172.16.18.254
S 192.168.36.0/24 [1/0] via 172.16.155.6
S 192.168.22.0/24 [1/0] via 172.16.18.99
161.215.0.0/16 is variably subnetted, 2 subnets, 2 masks
S 161.215.133.128/25 [1/0] via 172.16.18.99
S 161.215.44.203/32 [1/0] via 172.16.18.99
S 192.168.34.0/24 [1/0] via 172.16.155.2
S 192.168.0.0/24 [1/0] via 172.16.18.99
S 192.168.254.0/24 [1/0] via 172.16.17.189
S 192.168.50.0/24 [1/0] via 172.16.18.99
S 192.168.1.0/24 [1/0] via 172.16.18.99
S 199.16.20.0/24 [1/0] via 199.16.25.30
[1/0] via 199.16.25.14
S 192.168.49.0/24 [1/0] via 172.16.18.99
S 199.16.21.0/24 [1/0] via 199.16.25.30
[1/0] via 199.16.25.14
S 192.168.3.0/24 [1/0] via 172.16.18.99
S 192.168.48.0/24 [1/0] via 172.16.18.254
S* 0.0.0.0/0 [1/0] via 172.16.18.254

 

[noviwinger]

I almost forgot: yes 172.16.18.254 is our firewall and is the next hop out

 

[dearingkr]

Have a Cisco 3640 router that stops routing to one destination

What destination?

MCSE CCNA CCDA

 

[noviwinger]

The destination is 199.16.16.0 which has 3 routes
199.16.25.14 - T1
199.16.25.30 - T1
199.16.25.102 - ISDN DOD

I just had another outage and I was running a Syslog and capturing IP routing events and found there is a RIP update coming in and flushing the routes and installing a new route to send 199.16.16.0 to 0.0.0.0

Any tips on what is the best way to capture where the rip updates are coming from?

 

[dearingkr]

What other Layer3 devices are on your network?
Don't forget switches.

I assume you're using rip because you have other non-Cisco devices.


MCSE CCNA CCDA

 

[noviwinger]

Most of the switches on my network are 2900xl and 3500xl but we do have a few 2950s. Our core switch is 6506.

I am using RIPv2 because it is what I have inherited. I am looking at alternatives like OSPF, but have not moved forward with it yet.

 

[dearingkr]

The 2900's are Layer2 switches no rip there

The 3500's may be Layer3 depending on which model you have. If you're not using them for routing, them make sure rip is disabled.

Definitely need to check the config of the 6506 and the firewall. They are the most likely culprits.


MCSE CCNA CCDA

 

[noviwinger]

Neither the 6506 nor the firewall have rip enabled.
Also, all our routers are cisco

 

[dearingkr]

What model are your 3500s?

Also, your syslog entry for the rip undate should have given you some kind of source info, at the very least, what interface the update came from.

MCSE CCNA CCDA

 

[noviwinger]

The 3500's are 3512xls. Older units.

Not showing the source of the update was the strange part. The syslog shows "adding 0.0.0.0/0(metric 1) via 0.0.0.0 on FastEthernet0/0 to RIP database. The it follows with redist 0.0.0.0/0 (metric 0) to RIP.

I have another wrinkle that maybe the ultimate cause. I just notived the router at the far end had a uptime of 45 minutes. My syslog data only goes through this morning, configuration error on my part, so I do not have logs from previous days to see the router has done this before.

I will let you know if this is the case.

 

[dearingkr]

the syslog entry appears to be correct, your defaul gateway is 172.16.18.254 via FE0/0


On your 'sho ip route':

S 199.16.16.0/24 is directly connected
is directly connected
is directly connected, Serial2/1
is directly connected, Serial1/1

doesn't make any sense.
that subnet isn't configured on those interfaces.

MCSE CCNA CCDA

 

[noviwinger]

I think we have at least some of it figured out. There were some ripping issues, but the bigger problem is the remote router is rebooting. Some diagnostics on the remote rotuer found the router getting BUS Errors and cold starting. I contacted Cisco and I have a replacement coming today.

 

[dearingkr]

If that's the remote router for the 199.16.16.0 network, it would explain the rip updates when the link goes down and then back up later.


MCSE CCNA CCDA

 

[noviwinger]

It is the 199.16.16.0 network.

Thank you for your help.