Cisco 2650. system operation may be compromised 1

[abidu]

Good day all,

Recently I encountered an issue with my cisco 2650 Router.

here is the boot info:

Self decompressing the image : ######################################################################################################### ######################### [OK]

Smart Init is enabled
smart init is sizing iomem
ID MEMORY_REQ TYPE
0001C1 0X000B3B80 C2600 Single Fast Ethernet
0X00098670 public buffer pools
0X00211000 public particle pools
TOTAL: 0X0035D1F0

If any of the above Memory Requirements are
"UNKNOWN", you may be using an unsupported
configuration or there is a software problem and
system operation may be compromised.
Rounded IOMEM up to: 4Mb.
Using 3 percent iomem. [4Mb/128Mb]

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706



Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IO3-M), Version 12.3(25), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 28-Jan-08 19:21 by alnguyen

cisco 2650 (MPC860P) processor (revision 0x200) with 126976K/4096K bytes of memory.
Processor board ID JAD0626CTA2 (1399540196)
M860 processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
1 FastEthernet/IEEE 802.3 interface(s)
32K bytes of non-volatile configuration memory.
16384K bytes of processor board System flash (Read/Write)

Slot is empty or does not support clock participate
WIC slot is empty or does not support clock participate


Press RETURN to get started!


The fast ethernet interface is up but unable to ping anything that is connected to it, I am able to see the mac addresses(arp) and also cdp.

Tried to replace the memory and flash and the IOS with no result.

Any ideas will be greatly appreciated.
Thanks in advance,
Alex

 

[vipergg]

Bootup looks normal. If you can see cdp thru the interface its probably not a interface issue . Sounds like a config issue .

 

[burtsbees]

Clear the arp cache and then try. Also, clear counters, and post sh int fa whatever. Post a sh run also.

Burt

 

[abidu]

Here is the info that you requested:

lab-router2#sh int fastEthernet 0/0
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 000a.4183.80a0 (bia 000a.4183.80a0)
Internet address is 192.168.1.3/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:35, output 00:00:03, output hang never
Last clearing of "show interface" counters never
Input queue: 25/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
109 packets input, 11890 bytes
Received 87 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
41 packets output, 3953 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out




lab-router2#sh run
Building configuration...

Current configuration : 672 bytes
!
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname lab-router2
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
no network-clock-participate slot 1
no network-clock-participate wic 0
ip cef
!
!
!
!
no ip domain lookup
!
!
!
!
!
!
interface FastEthernet0/0
ip address 192.168.1.3 255.255.255.0
duplex auto
speed auto
!
!
no ip http server
no ip http secure-server
!
!
control-plane
!
banner motd ^C Do Not Login Restricted ! ^C
!
line con 0
exec-timeout 0 0
password cisco
logging synchronous
login
line aux 0
line vty 0 4
password cisco
logging synchronous
login
!
!
end

+++++++++++++++++++++++++

So, after boot I am able to ping the router and able to see the connected device, also arp shoes me the connected mac address, all good so far.. this lasts 15-20 seconds after which I cannot communicate with the device or router, there is no interface reset nothing that will point to an interface failure or reset.

And what do you thins about this:

Smart Init is enabled
smart init is sizing iomem
ID MEMORY_REQ TYPE
0001C1 0X000B3B80 C2600 Single Fast Ethernet
0X00098670 public buffer pools
0X00211000 public particle pools
TOTAL: 0X0035D1F0

If any of the above Memory Requirements are
"UNKNOWN", you may be using an unsupported
configuration or there is a software problem and
system operation may be compromised.
Rounded IOMEM up to: 4Mb.
Using 3 percent iomem. [4Mb/128Mb]

I have never seen this before and it might have something to to with the issue ?

Thanks,
Alex

 

[burtsbees]

That is just the amount of DRAM the system is using for the buffers of fa0/0. It states that if you see "UNKNOWN", then you can worry, but it gives hex values for everything, so no worries. I would do an extended ping, with extended commands to choose the source interface as fa0/0, and I would do like 5000 repeat count or so, to see if a continuous ping ever fails. It sounds like a bad cable, or at least a questionable cable. What are you pinging, and how exactly is it connected (through a switch, directly with a crossover cable, etc.), and have you tried different devices, different ways of connecting the device, etc.?

Burt

 

[abidu]

This has been tested with different cables and different devices. The weirdest thing is that the ping will fail after exactly ~15 seconds after the router has booted up.

 

[vipergg]

That message is a normal bootup message. whats on the other end of that interface F0/0 . Is there another network device and how is that configured?

 

[abidu]

Tried with a cisco switch, there is no mismatch, also tried with a laptop and PC.

 

[abidu]

I am running of of ideas with what's wrong with the fast Ethernet interface.

Please help!
cheers,
Alex

 

[VinceWhirlwind]

Surely the log tells you what the event is that happens after 15 seconds?

 

[abidu]

Nope, there is no info with debug or any other info that will lead me to believe that the interface is bad. Running out of ideas...

 

[burtsbees]

What exactly is connected to the router? Can you connect a computer directly to the router with a crossover cable and try an extended ping, repeat of 25,000, or connect with a different switch, repeat count of 25,000 on an extended ping? Please

router#clear counters

first before doing anything! Also, do this

access-list 101 permit icmp any any echo log
access-list 101 permit icmp any any echo-reply log
access-list 101 permit icmp any any time-exceeded log
access-list 101 permit icmp any any time-exceeded log
access-list 101 permit icmp any any timestamp-request log
access-list 101 permit icmp any any timestamp-reply log
access-list 101 permit icmp any any traceroute log
access-list 101 permit ip any any
int fa0/0
ip access-group 101 in

then ping int fa0/0 ip address from the node directly connected to the router, or from the switch connected to fa0/0, or a node connected through a switch. Remember to do an extended ping (like Windows, C:\ping 192.168.1.3 -t

Then do
router#sh logg
router#sh access-list 101
and look for matches on sh access-list 101

Also,

router#test interfaces

though if you can ping for 15 seconds, this test should actually be good...

Burt

 

[abidu]

Burt,

Used an laptop, switch and a PC all there with the same result.

Immediately after the router has booted you can see the pings:

This is the order of the commands that I have used during the test:

Reply from 192.168.1.1: bytes=32 time=1613ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time=343ms TTL=255
Reply from 192.168.1.1: bytes=32 time=16ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time=1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255
Reply from 192.168.1.1: bytes=32 time<1ms TTL=255


Router#clear counters
Clear "show interface" counters on all interfaces [confirm]
Router#
*Mar 1 00:00:31.683: %CLEAR-5-COUNTERS: Clear counter on all interfaces by console
Router#sh access-list 101
Extended IP access list 101
permit icmp any any echo log (16 matches)
permit icmp any any echo-reply log
permit icmp any any time-exceeded log
permit icmp any any timestamp-request log
permit icmp any any timestamp-reply log
permit icmp any any traceroute log
permit ip any any (63 matches)
Router#sh logg
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns)
Console logging: level debugging, 15 messages logged
Monitor logging: level debugging, 0 messages logged
Buffer logging: disabled
Logging Exception size (4096 bytes)
Count and timestamp logging messages: disabled
Trap logging: level informational, 18 message lines logged
Router#sh int fa0/0
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 000a.4183.80a0 (bia 000a.4183.80a0)
Internet address is 192.168.1.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:00:14
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 1000 bits/sec, 2 packets/sec
5 minute output rate 0 bits/sec, 1 packets/sec
29 packets input, 3602 bytes
Received 18 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
12 packets output, 874 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Router#test int
Router#test interfaces fa0/0
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Passed
1 interfaces: 1 passed, 0 failed, 0 skipped, 0 untestable

After a few seconds:

Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.
Reply from 192.168.1.50: Destination host unreachable.

After i could not ping the router anymore:


Router#clear counters
Clear "show interface" counters on all interfaces [confirm]
Router#sh ip
*Mar 1 00:01:21.221: %CLEAR-5-COUNTERS: Clear counter on all interfaces by co
Router#sh int fa0/0
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 000a.4183.80a0 (bia 000a.4183.80a0)
Internet address is 192.168.1.1/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:27, output 00:00:01, output hang never
Last clearing of "show interface" counters 00:00:07
Input queue: 17/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
1 packets output, 60 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

Router#sh access-list 101
Extended IP access list 101
permit icmp any any echo log (135 matches)
permit icmp any any echo-reply log
permit icmp any any time-exceeded log
permit icmp any any timestamp-request log
permit icmp any any timestamp-reply log
permit icmp any any traceroute log
permit ip any any (117 matches)
Router#sh logg
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns)
Console logging: level debugging, 17 messages logged
Monitor logging: level debugging, 0 messages logged
Buffer logging: disabled
Logging Exception size (4096 bytes)
Count and timestamp logging messages: disabled
Trap logging: level informational, 20 message lines logged
Router#sh run
Building configuration...

Current configuration : 1047 bytes
!
version 12.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Router
!
!
ip subnet-zero
!
!
!
ip audit notify log
ip audit po max-events 100
!
!
!
voice call carrier capacity active
!
!
!
!
!
!
!
!
!
mta receive maximum-recipients 0
!
!
!
!
interface FastEthernet0/0
ip address 192.168.1.1 255.255.255.0
ip access-group 101 in
duplex auto
speed auto
!
interface Serial0/0
no ip address
shutdown
!
interface Serial1/0
no ip address
shutdown
clockrate 2000000
!
ip classless
ip http server
!
!
access-list 101 permit icmp any any echo log
access-list 101 permit icmp any any echo-reply log
access-list 101 permit icmp any any time-exceeded log
access-list 101 permit icmp any any timestamp-request log
access-list 101 permit icmp any any timestamp-reply log
access-list 101 permit icmp any any traceroute log
access-list 101 permit ip any any
!
call rsvp-sync
!
!
mgcp profile default
!
dial-peer cor custom
!
!
!
!
!
line con 0
line aux 0
line vty 0 4
login
!
!
end

Router#sh ip int br
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 192.168.1.1 YES NVRAM up up
Serial0/0 unassigned YES NVRAM administratively down down
Serial1/0 unassigned YES NVRAM administratively down down
Router#
-------------------------------------------------------

That's about it, as you can see the interface is up.

The strange thing is if i will try to do an flash upgrade thru romon the interface is looking good and i am able to transfer the IOS over to the router.

Cheers,
Alex

 

[burtsbees]

So you connected a laptop/pc DIRECTLY to the router using a crossover as I suggested, bypassing the switch altogether? Also, after the pings fail, ping the interface anyway to see if the acl matches increase and if the log catches the pings. Then test the interface again (test int fa0/0), again after pings fail. Shut the interface down, then bring it back up

router(config)#int fa0/0
router(config-if)#shut
router(config-if)#no shut

then try it again. I had a similar problem on a 2620, and it ended up being the cable...I hope you have tried this...you're not making your own, are you? I tried 3 cables before I knew it was the cable (coworker, actually my boss, made the cables)...

You have to be thorough when testing this! Be as specific as you can with the details, I.E. if you tried a DIRECT CONNECTION with a pc using a CROSSOVER cable, eliminating the switch, etc.

Burt

 

[abidu]

I have tried all possible alternatives, and yes the cable is crossover and brand new from the bag connected directly to the PC and laptop and tried with 3 different switches as well/
shutting down the interface and bring it up will not do the trick. I am able to ping the ip that is setup on the router also test of fa0/0 is success.

Alex

 

[burtsbees]

Here's one last test, possibly a work-around...Set up a loopback interface, and connect the switch and pc to the fa0/0 interface as normal, and see if pings to the loopback interface you set up fail after 15-20 seconds...

lab-router2>en
lab-router2#conf t
lab-router2(config)#int lo0
lab-router2(config-if)#ip add 10.10.1.1 255.255.255.255
lab-router2(config-if)#end
lab-router2#wr

C:\ping 10.10.1.1 -t

Post the results. You also said in the first post...

"Tried to replace the memory and flash and the IOS with no result."

What IOS was there before, and what IOS is on there now?

After this, I would deem the interface as being bad (like the AmdFE chip), and hopefully the loopback can be a work-around. Just for grins, post a sh diag and sh controllers fa0/0, please. Let's see if some registers are blank...

Burt