CDP showing two of each neighbor 1

[Kawdude]

Equipment:
Cisco 3600 series router

Symptom:
Execute the show cdp neighbor commands and there are two of each neighbor showing on two seperate interfaces.

Analysis:
Incorrect route

Problem:
Routes in the 3600 appears correct.

Question:
Has anyone on Tek-Tips Forums experienced this problem also?

 

[vipergg]

Probably means someone has looped the cables between two different ports , you verify with "show spanning tree blocked ports command. You should not see dual instances if everything is wired correctly. If its sub interfaces then you might see 2 instances if they go to the same switch.

 

[burtsbees]

If the two ports are plugged into the same switch, then of course they will each show an instance of the switch. Please be a bit more specific to exactly what is showing in sh cdp nei, and also---are you doing sh cdp nei detail? What is connected to each of the router ports?

Burt

 

[Kawdude]

Unfortunately the switches are all unmanaged Netgear 524 devices. However the fact that cables may be looped could be the problem. There had been some cable management performed about a month ago and this would be close to the time this problem began to show up. Today the cable arrangement will be checked.

If this isn't the situation more details will be posted.

 

[Kawdude]

I've checked for loops and none existed. However there remains some cable work that needs completed. Once this work is completed then if the problem persists more information will be added.

 

[jneiberger]

Also, I'd like to point out that CDP has nothing to do with routing. CDP is a layer two (datalink) protocol. The routing tables are not relevant.

 

[burtsbees]

Please post a detailed topology, sh cdp nei det outputs, and sh run outputs.

Burt

 

[Kawdude]

Ok, The duplicates devices are gone I'm seeing neighbors as one again. Replaced 80% of the cables in at the MDF and corrected a few connections as well. One of these worked at least for the duplicated device issue. There is a part two to this story and it is that the network has been dropping packets and experiencing severe slowdowns. This began at the same time that the duplicate devices started showing up.

Here is some information:

Show buffers:
Buffer elements:
499 in free list (500 max allowed)
4819977 hits, 0 misses, 0 created

Public buffer pools:
Small buffers, 104 bytes (total 152, permanent 50):
149 in free list (20 min, 150 max allowed)
3773400 hits, 2141 misses, 757 trims, 859 created
1351 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25):
23 in free list (10 min, 150 max allowed)
603170 hits, 870 misses, 174 trims, 174 created
699 failures (0 no memory)
Big buffers, 1524 bytes (total 50, permanent 50):
50 in free list (5 min, 150 max allowed)
72589 hits, 266 misses, 24 trims, 24 created
248 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
10 in free list (0 min, 100 max allowed)
46 hits, 206 misses, 4 trims, 4 created
206 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 206 misses, 4 trims, 4 created
206 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
0 hits, 206 misses, 4 trims, 4 created
206 failures (0 no memory)

Interface buffer pools:
CD2430 I/O buffers, 1524 bytes (total 0, permanent 0):
0 in free list (0 min, 0 max allowed)
0 hits, 0 fallbacks

Header pools:
Header buffers, 0 bytes (total 265, permanent 256):
9 in free list (10 min, 512 max allowed)
253 hits, 3 misses, 0 trims, 9 created
0 failures (0 no memory)
256 max cache size, 256 in cache

Particle Clones:
1024 clones, 0 hits, 0 misses

Public particle pools:
F/S buffers, 256 bytes (total 384, permanent 384):
128 in free list (128 min, 1024 max allowed)
256 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
256 max cache size, 256 in cache
Normal buffers, 1548 bytes (total 512, permanent 512):
384 in free list (128 min, 1024 max allowed)
1179 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
128 max cache size, 128 in cache

Private particle pools:
Serial0/0 buffers, 1548 bytes (total 64, permanent 64):
0 in free list (0 min, 64 max allowed)
64 hits, 0 fallbacks
64 max cache size, 32 in cache
FastEthernet0/0 buffers, 1548 bytes (total 192, permanent 192):
0 in free list (0 min, 192 max allowed)
192 hits, 0 fallbacks
192 max cache size, 128 in cache
FastEthernet0/1 buffers, 1548 bytes (total 192, permanent 192):
0 in free list (0 min, 192 max allowed)
192 hits, 0 fallbacks
192 max cache size, 128 in cache
FastEthernet1/0 buffers, 1548 bytes (total 192, permanent 192):
0 in free list (0 min, 192 max allowed)
192 hits, 0 fallbacks
192 max cache size, 128 in cache
FastEthernet1/1 buffers, 1548 bytes (total 192, permanent 192):
0 in free list (0 min, 192 max allowed)
192 hits, 0 fallbacks
192 max cache size, 128 in cache


Showing IP Traffic:

IP statistics:
Rcvd: 72156181 total, 487662 local destination
0 format errors, 0 checksum errors, 0 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 0 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
Bcast: 315904 received, 16 sent
Mcast: 163400 received, 245331 sent
Sent: 322710 generated, 71297845 forwarded
Drop: 113911 encapsulation failed, 0 unresolved, 0 no adjacency
2 no route, 0 unicast RPF, 0 forced drop

ICMP statistics:
Rcvd: 0 format errors, 0 checksum errors, 0 redirects, 4 unreachable
6394 echo, 0 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 0 other
0 irdp solicitations, 0 irdp advertisements
Sent: 70515 redirects, 32 unreachable, 0 echo, 6339 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 0 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements

UDP statistics:
Rcvd: 340451 total, 7 checksum errors, 340440 no port
Sent: 35095 total, 0 forwarded broadcasts

TCP statistics:
Rcvd: 773 total, 0 checksum errors, 51 no port
Sent: 479 total

Probe statistics:
Rcvd: 0 address requests, 0 address replies
0 proxy name requests, 0 where-is requests, 0 other
Sent: 0 address requests, 0 address replies (0 proxy)
0 proxy name replies, 0 where-is replies

EGP statistics:
Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no listener
Sent: 0 total

IGRP statistics:
Rcvd: 0 total, 0 checksum errors
Sent: 0 total

OSPF statistics:
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks

Sent: 0 total

IP-IGRP2 statistics:
Rcvd: 140052 total
Sent: 210273 total

PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0, Register Stops: 0/0, Hellos: 0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0

IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0

ARP statistics:
Rcvd: 449582 requests, 655 replies, 0 reverse, 0 other
Sent: 96676 requests, 17604 replies (1863 proxy), 0 reverse

Showing Interfaces:

FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 0002.b934.5281 (bia 0002.b934.5281)
Internet address is 128.X.X.X/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 299 drops
5 minute input rate 315000 bits/sec, 146 packets/sec
5 minute output rate 465000 bits/sec, 164 packets/sec
34108639 packets input, 3565162521 bytes
Received 923500 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
33864348 packets output, 3044602336 bytes, 0 underruns
0 output errors, 0 collisions, 5 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial0/0 is down, line protocol is down
Hardware is DSCC4 with integrated T1 CSU/DSU
Internet address is 10.X.X.X/30
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0 (size/max/drops); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/0/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=down DSR=up DTR=up RTS=up CTS=down

FastEthernet0/1 is up, line protocol is up
Hardware is AmdFE, address is 0002.b934.5282 (bia 0002.b934.5282)
Internet address is 100.X.X.X/24
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Half-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:07:29, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 0 drops
5 minute input rate 54000 bits/sec, 26 packets/sec
5 minute output rate 10000 bits/sec, 22 packets/sec
1349848 packets input, 246608738 bytes
Received 178 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
1752561 packets output, 162279302 bytes, 0 underruns
0 output errors, 31003 collisions, 3 interface resets
0 babbles, 0 late collision, 11287 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
FastEthernet1/0 is up, line protocol is up
Hardware is AmdFE, address is 0002.b934.5291 (bia 0002.b934.5291)
Internet address is 192.X.X.X/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 0 drops; input queue 0/75, 309 drops
5 minute input rate 405000 bits/sec, 140 packets/sec
5 minute output rate 299000 bits/sec, 111 packets/sec
35894068 packets input, 179876420 bytes
Received 920063 broadcasts, 0 runts, 0 giants, 0 throttles
113595 input errors, 113595 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
33930709 packets output, 3706662245 bytes, 0 underruns
0 output errors, 0 collisions, 5 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
FastEthernet1/1 is up, line protocol is up
Hardware is AmdFE, address is 0002.b934.5292 (bia 0002.b934.5292)
Internet address is 172.X.X.X/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:06, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Queueing strategy: fifo
Output queue 0/40, 7028 drops; input queue 0/75, 0 drops
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
2055986 packets input, 523782827 bytes
Received 68064 broadcasts, 0 runts, 0 giants, 0 throttles
3611 input errors, 3611 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast
0 input packets with dribble condition detected
2376867 packets output, 1748993525 bytes, 0 underruns
0 output errors, 0 collisions, 55 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out


Seriously I've worked along time now trying to figure out the root cause of this problem. Any ideas?

 

[jneiberger]

If you are seeing the same neighbors on multiple interfaces and you're experiencing slowdowns that began at the same time, it's almost certain that you have a spanning tree problem and you're experiencing a loop in your switched topology.

Do all of your switches have spanning tree enabled?

 

[unclerico]

Wow, look at your CRC errors on your F1/0 and F1/1. High CRC errors with low collisions is caused by excessive noise (i.e. bad cables). Did you make the cables yourself? I would double check whatever is connected to the other side of both of those interfaces. Your F0/1 interface is experiencing a high rate of collisions suggesting a speed/duplex mismatch (I see it is set at 10/half). I know you said that the network is slow now but you never specicified if it was the entire network or just certain subnets hanging off of this router. I do agree with jneilberger that you probably still have a loop somewhere.

I hate all Uppercase... I don't want my groups to seem angry at me all the time! =)
- ColdFlame (vbscript forum)

 

[jneiberger]

On those interfaces with input errors, do you have the speed and duplex manually configured? If so, you may have a duplex mismatch. Set both sides to AUTO, clear the counters, and see if there is a decrease in errors and increase in performance.

 

[Kawdude]

First I want to thank each of you for attempting to help me out here with this. Just know as we move on here this network (these errors) were already in place on my arrival certainly they were not part of my design.

To add some highlites here:

Question:

Do all of your switches have spanning tree enabled?

Answer: This network has Netgear524 unmanaged switches. There are 17 in the MDF and 27 campus wide. There is two Netgear FSM726 both have spanning tree set to on. I've suspected this as a problem but have only had access to the two FSM726's now for two weeks since the password had been lost. I did finally guess the password (Oh! No! I'm becoming a mandatory ethical hacker!) and now have access to both devices. Just added turn off spanning tree on both of the FSM726s to my task list for today!

Question:

Did you make the cables yourself?

Answer:

No all new cables were purchased. One note here would be that the MDF used to look like a peak time visit to the local Pasta Palace. At least 80 percent of the cables have now been replaced at the MDF with new cables.


Adding a note here concerning Interface Fa0/1:

This one connects to another router on another network. The IT department on that side of things confirms that they have the interface back to us set at half duplex. I've known a few mistakes to have been made on that side so also added to my task list is to request that someone visually check the configuration on that side again. Can only help if they were wrong the first time or maybe it was changed by someone it does happen.

Final note is addressing the slow downs:

This has been the most frustrating part so far. One time reports from everyone on the Fa1/0 that everything is crawling. Next time it maybe Fa1/1 that complains. Again my VPN users will complain they cross the Fa0/1 to access applications on that side for the most part. I've not had a complaint from anyone on Fa0/0 about a slowdown be assured it does happen here too as I've experienced the slowdown myself on that subnet along with the other interfaces. The durations and times vary but the times are also close to textbook. 8am during peak login time. 10am when when everyone is crunching away. 2pm/3pm again everyone is crunching also 2nd shift logins occur. 4pm when everyone is wrapping up to go home for the day. This is not all inclusive but average times that complaints have been noted.

 

[burtsbees]

So, at the MDF, you now have patch panels?lol
I worked for a health care company before, and one of their sites' wiring closet was the worst I'd ever seen. Not one cable was hung properly, a few switches hanging (4 port deals) daisy-chained---like 4 in a row, and for the 125 users in the office---one router, 4 4port switches, and four Cisco EtherFast HUBS!!!
Anyway---VPN users complain on a link with collisions...an interface with lots of CRC errors and no collisions---I agree with uncle---layer 1 issue.
Had you considered the router itself? Also, I would clear those interface counters for a fresh look at troubleshooting...

Burt

 

[jneiberger]

We're not worried about fa0/1. That's running at half duplex, so you expect to see collisions. We're more worried about the errors on your full duplex links (1/0 and 1/1). I personally think you have a duplex mismatch on 1/0, but that's just a guess. It's either that or a bad cable.

I agree with Burt about clearing the counters, too.

 

[Kawdude]

There is simply not enough time or space to enter all the data of what has been attempted here to get through this problem.

Good news is in all the swapping out of wires and rearranging cables the duplicate problem which is the original problem has now been resolved. Depending on which of the Netgear524 devices a workstation is assigned to, you will get a diffrent symptom from each switch.

Second set of good news is that I've gotten approval to replace the netgear switches with another solution. So we can now consider this thread as done.

Thanks again for your suggestions and wish that I could have given you the root cause on this however given the structure or a better statement would be the mal-structure of our core it is amazing that we can transmit a single packet successfully.

Moving this network in a forward direction though and hopefully will be able to discuss brighter days with everyone here in the future.