eigrp redistribute question 2

[iwanthome]

My network is 3640,7500 and four 6509 msfc interconnected with ethernet vlan 200,like this:
6509 msfc(10.1.200.250)----vlan 200----3640(10.1.200.246)
6509 msfc(10.1.200.251)----vlan 200----7500(10.1.200.248)
6509 msfc(10.1.200.252)----vlan 200
6509 msfc(10.1.200.254)----vlan 200
I have a eigrp's problem.Two external route are changed frequently on cisco router 7500.The two external router
is redistributed on 6509 msfc,and it stays long on 3640.When I show ip eigrp events on one of four msfcs,I can
see "14:16:32.177 Poison squashed: 10.88.254.1/32 reverse" and " 14:16:32.177 Poison squashed: 0.0.0.0/0
reverse".I think maybe it is the reason.But I am in doubt why 3640 is work ok. Can anybody point me how to
troubleshooting this? When and why 6509 msfc "Poison squashed: 10.88.254.1/32 reverse" ? Why 7500's routing table
is changed and 3640 is not?
thanks !
any comment is appreciated!

3640#sh ip route
D EX 10.88.254.1/32 [170/281600] via 10.1.200.254, 1d05h, Ethernet0/0
[170/281600] via 10.1.200.251, 1d05h, Ethernet0/0
[170/281600] via 10.1.200.250, 1d05h, Ethernet0/0
[170/281600] via 10.1.200.252, 1d05h, Ethernet0/0
D*EX 0.0.0.0/0 [170/281600] via 10.1.200.254, 1d05h, Ethernet0/0
[170/281600] via 10.1.200.251, 1d05h, Ethernet0/0
[170/281600] via 10.1.200.250, 1d05h, Ethernet0/0
[170/281600] via 10.1.200.252, 1d05h, Ethernet0/0
3640>sh ip eigrp nei
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 10.1.200.248 Et0/0 13 1w2d 20 450 0 282759
9 10.1.200.252 Et0/0 11 1w2d 16 450 0 10937
8 10.1.200.254 Et0/0 14 1w2d 17 450 0 10932
4 10.1.200.250 Et0/0 12 1w2d 17 450 0 10988
2 10.1.200.251 Et0/0 14 1w2d 19 450 0 10987

7500#sh ip route
D EX 10.88.254.1/32
[170/28160] via 10.1.200.252, 01:13:07, FastEthernet1/1/0
[170/28160] via 10.1.200.250, 01:13:07, FastEthernet1/1/0
[170/28160] via 10.1.200.254, 01:13:07, FastEthernet1/1/0
[170/28160] via 10.1.200.251, 01:13:07, FastEthernet1/1/0
D*EX 0.0.0.0/0 [170/28160] via 10.1.200.252, 01:13:18, FastEthernet1/1/0
[170/28160] via 10.1.200.250, 01:13:18, FastEthernet1/1/0
[170/28160] via 10.1.200.254, 01:13:18, FastEthernet1/1/0
[170/28160] via 10.1.200.251, 01:13:18, FastEthernet1/1/0
7500#sh ip eigrp nei
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq Type
(sec) (ms) Cnt Num
31 10.1.200.251 Fa1/1/0 13 1w2d 1 300 0 10991
29 10.1.200.250 Fa1/1/0 11 1w2d 1 300 0 10992
28 10.1.200.254 Fa1/1/0 13 1w2d 1 300 0 10936
27 10.1.200.252 Fa1/1/0 14 1w2d 1 300 0 10941
25 10.1.200.246 Fa1/1/0 12 1w2d 1 300 0 34149


6509_1#sh ip route
S 10.88.254.1/32 [1/0] via 10.0.8.3
S* 0.0.0.0/0 [1/0] via 10.1.100.248
6509_1>sh ip eigrp nei
IP-EIGRP neighbors for process 1
H Address Interface Hold Uptime SRTT RTO Q Seq Type
(sec) (ms) Cnt Num
5 10.1.200.248 Vl200 11 1w2d 44 594 0 282759
1 10.1.200.246 Vl200 13 1w2d 11 450 0 34145
8 10.1.200.252 Vl200 13 1w2d 8 1012 0 10937
7 10.1.200.254 Vl200 12 1w2d 25 1012 0 10932
9 10.1.200.251 Vl200 13 1w2d 8 1012 0 10937


6509_1>sh ip eigrp events
Event information for AS 18:
1 14:16:32.257 Poison squashed: 10.76.0.0/14 reverse
2 14:16:32.241 Change queue emptied, entries: 1
3 14:16:32.241 Metric set: 10.76.0.0/14 1807360
4 14:16:32.241 Update reason, delay: metric chg 517120
5 14:16:32.241 Update sent, RD: 10.76.0.0/14 1807872
6 14:16:32.241 Route install: 10.76.0.0/14 10.1.200.248
7 14:16:32.241 Route install: 10.76.0.0/14 10.1.200.245
8 14:16:32.241 FC sat rdbmet/succmet: 1807360 1804800
9 14:16:32.241 FC sat nh/ndbmet: 10.1.200.248 1807872
10 14:16:32.241 Find FS: 10.76.0.0/14 1807360
11 14:16:32.241 Rcv update met/succmet: 1807360 1804800
12 14:16:32.241 Rcv update dest/nh: 10.76.0.0/14 10.1.200.248
13 14:16:32.177 Poison squashed: 10.88.254.1/32 reverse
14 14:16:32.177 Poison squashed: 0.0.0.0/0 reverse
15 14:16:32.177 Poison squashed: 10.0.224.0/19 reverse
16 14:16:27.329 Poison squashed: 10.76.0.0/14 reverse
17 14:16:27.225 RDB delete: 10.76.0.0/14 10.1.200.248
18 14:16:27.201 Poison squashed: 10.76.0.0/14 reverse
19 14:16:27.173 Change queue emptied, entries: 1
20 14:16:27.173 Metric set: 10.76.0.0/14 1807872
21 14:16:27.173 Update reason, delay: metric chg 517120
22 14:16:27.173 Update sent, RD: 10.76.0.0/14 1807360
23 14:16:27.173 Route install: 10.76.0.0/14 10.1.200.245
24 14:16:27.173 Send reply: 10.76.0.0/14 10.1.200.248
25 14:16:27.173 FC sat rdbmet/succmet: 1807872 1805312
26 14:16:27.173 FC sat nh/ndbmet: 10.1.200.245 1807360
27 14:16:27.173 Find FS: 10.76.0.0/14 1807360
28 14:16:27.173 Rcv query met/succ met: 4294967295 4294967295
29 14:16:27.173 Rcv query dest/nh: 10.76.0.0/14 10.1.200.248

 

[fmonteiro]

Would you, please, set the "debug eigrp packets" for both the 6509_1 switch and the 7500 router? Also, could you set the debug eigrp neighbors for the 7500 router?

 

[tschouten]

I'm with fmonteiro on this, show an output of those debug options. I will say first thought is posion reverse isn't allowing some of your updates from the MSFC to the 7500. It is basically saying I received update from this interface already, poison reverse dictates I cannot send route changes out the interface because I may be sending faulty information. But I can't be 100% sure until I see what is happening from the debugs.

 

[iwanthome]

I'm so sorry,I can't debug on 6509 or 7500 because they have about 40 eigrp neighbors and they are our procuction enviorement.
I can use all show command if you need.
please help me! thanks!

 

[fmonteiro]

Could you, at least, get a debug ip eigrp notifications for the 7500 ( I am not expecting a hugh output from this command but do it after hours). If this is not possible post a sh ip eigrp events for the 7500.

 

[tschouten]

I looked at this again and think I noticed something I missed before:

Route install: 10.76.0.0/14 10.1.200.245

Where is this route update coming from you don't have this listed anywhere as a router or MSFC. Want to know where 245 is; I think what could be happening here is poison reverse is causing an update error due to mis-configured gateway of last resort on the 6509 or the 7500. It is deleting the route from the routing data base-
17 14:16:27.225 RDB delete: 10.76.0.0/14 10.1.200.248

Look at it from the start, well sort of the start. An update is happening.
3 14:16:32.241 Metric set: 10.76.0.0/14 1807360
4 14:16:32.241 Update reason, delay: metric chg 517120

The routes are being installed.
6 14:16:32.241 Route install: 10.76.0.0/14 10.1.200.248
7 14:16:32.241 Route install: 10.76.0.0/14 10.1.200.245


8 14:16:32.241 FC sat rdbmet/succmet: 1807360 1804800
9 14:16:32.241 FC sat nh/ndbmet: 10.1.200.248 1807872

Searching for Feasible Successor
10 14:16:32.241 Find FS: 10.76.0.0/14 1807360

Receiving Updates
11 14:16:32.241 Rcv update met/succmet: 1807360 1804800
12 14:16:32.241 Rcv update dest/nh: 10.76.0.0/14 10.1.200.248

Poison Reverse stopping updates.
13 14:16:32.177 Poison squashed: 10.88.254.1/32 reverse
14 14:16:32.177 Poison squashed: 0.0.0.0/0 reverse
15 14:16:32.177 Poison squashed: 10.0.224.0/19 reverse
16 14:16:27.329 Poison squashed: 10.76.0.0/14 reverse

MSFC Deleting Route because an update could not be received.

17 14:16:27.225 RDB delete: 10.76.0.0/14 10.1.200.248

(which makes me wonder if you aren't seeing any SIA errors somewhere?)


again it would be more helpful if I could see the 7500's eigrp events to see if what I'm thinking is correct, I could be totaly wrong.

 

[iwanthome]

I investigate this for these two days.Maybe I found the reason.MY topu is actually like this:
6509 msfc(10.1.200.250)-vlan 200-3640(10.1.200.246)-wan 1--r1
6509 msfc(10.1.200.251)-vlan 200-7500(10.1.200.248)-wan 2--r1
6509 msfc(10.1.200.252)-vlan 200
6509 msfc(10.1.200.254)-vlan 200
I think the flowchart is like this:
1. wan 2 is down
2. wan 2 is up
3. r1 will build eigrp neighbor with 7500,and send update to 7500.
4. 7500 receive update and begin DUAL.because r1 is not successor of 7500 except 10.76.0.0(suppose this),so DUAL is local-compution,and 7500 will not send update to other except 10.76.0.0.
5. 7500 will reset sh ip route timeclock,so when I show ip route,all route 's uptime will change frequently because my wan is not stablity.And I show ip eigrp nei,I found there is one neighbor's uptime is same as sh ip route uptime.
So I think this is the reason.and I found this on routing tcp/ip vol 1 p 207:
A router will reassess its list of feasible successors for a route, as described in the last section, any time an
input event occurs. An input event can be:
A change in the cost of a directly connected link
A change in the state (up or down) of a directly connected link
The reception of an update packet---->this is the reason?
The reception of a query packet
The reception of a reply packet

English is not my native language,so I can't express exactly what I think.

thanks!

 

[iwanthome]

something I forget:
I sh log on 7500,I can found wan 2 is down and up,and the time is approved my suppose.

 

[tschouten]

Sounds about right to me

 

[fmonteiro]

You got it! Congratulations!
Is the wan2 flapping? Perhaps you should further investigate what is happening!

 

[iwanthome]

thanks for your two guys help!
And wan2 flapping is out control of me. I will report it to ISP.