2651 lock-up - two routers lock up in same hour

[pehi]

Hi - I have two 2651's, both running IOS v 12.3(11). They are configured with EIGRP, OSPF and a couple of classless static routes. The EIGRP vlan-based sub interfaces face each other, the the OSPF and Classless go out over a separate interface to a third router running an older (12.3(8) IOS with no EIGRP. For some reason at approx 0500 each morning both the newer devices lock up solidly within an hour of each other and don't even respond to console input.

Any thoughts welcome

 

[KiscoKid]

I'd do the following:

1. Check for the presence of a Core Dump and, if present, open a TAC case.
2. Review the Bug Toolkit on the CCO to see what relevant bugs for your IOS version may explain what you have seen.
3. It may have been a denial of service attack. Are these devices internet-facing?

 

[castlesmadeofsand]

What are the memory configs and flash size? For OSPF ABR's redistributing EIGRP, you need a lot of power. Backups every AM at 0500?

Tim

 

[pehi]

The wierd thing with this is that even if I remove the EIGRP routing completely, and leave both routers running OSPF only, both of them still lock up almost simultaneously around the same times. There isn't any backup running then, in fact it's the most quiescent part of the day.

 

[DanInRaleigh]

..wow this is really wierd, when you find the solution please come back and post..

....do logs give you any indication what happend..like what is happening right before the lockup..

..do you have a syslog server setup?

http://www.kiwisyslog.com/downloads.php

**Kiwi Syslog Daemon is a freeware Windows Syslog Daemon. It receives, logs, displays and forwards syslog messages and SNMP traps from any syslog or SNMP trap enabled device.**




CCNP,CCSP,MCSE,Sec+,Net+,A+...

 

[castlesmadeofsand]

When you say they lock up at the same time running no eigrp, does that mean you started them back up at the same time? In other words, do they lock up after 1 hour, 5 hours, or is it that no matter what time they go back online, they lock up at 5:00 am? If the latter scenario is accurate, do they work until 4 or 5 am before they lock up, like are they working...working...working---then idle for a given amount of time?
I am asking this to try to find ANYthing that may be common or consistent between the two other than the fact that they lock up. What I would do---anything with a processor and a fan to cool it, I look at first (make sure the fans did not fail). Is the environment where the 2 routers are warmer than anywhere else?
If all else fails, Try to take a known good 2651, configure it exactly as one of the ones in question and put it in place of a locking up one. If it locks up, it is surely doing too much work. Also, if you swap routers, make sure it has the same size RAM and flash if you can.
If nothing comes out of all of this, call the Vatican and have them send over a priest to perform an exorcism.

Tim

 

[pehi]

Hi,
The routers are in separate buildings... it seems that no matter at shat time they are reset, the still lock up around 0500. In fact one night while working late, I even reset one of them at 0430, and it still locked up at 0500... I had a console cable plugged in, and hte only output produced was from the preceding reset.

Vatican it is I think.

 

[KiscoKid]

Router issues are not typically so predictable that at 5am every day, they will lock up like this.

I would suggest you have some kind of issue in the local surroundings of these devices that may be causing them to lose power or suffer from a power surge or something similar.

I only had a funky, semi-similar issue where devices in a comms cabinet kept resetting themselves throughout the day. In the end, it transpired it was the nearby elevator causing the issue - elevators require large amounts of energy and can create power surges.

Have you tried to see what happens if the routers are removed and installed elsewhere? Do they still lockup at 5am?

 

[castlesmadeofsand]

DOMINI DOMINI DOMINI...

Tim

 

[pehi]

What I've decided to do is to include a second router at each site as a test to see if it's the co-existence of EIGRP and OSPF on the same router that's causing the issue. The first router will use OSPF, the second EIGRP. My question now is, do I need to configure a gateway on each interceding catalyst switch for the second router, or can EIGRP supposedly find a way through?