AccessPoint 1000 / Xedia reboots 1

[PEagle]

Hi All,

I get few uncontroled reboots of my AP-1000 router per day.
I use maximum memory (256MB) and connection over 4xE1 to 2 upstream providers (2 full bgp peers with total 230 000 router prefixes and 40 000 paths).

Each time router reboots i check console and see message:

last reboot caused by processor exception
4 - address load/fetch
PC: 0x90480DBC SP: 0x87FE9AD0 EXCEPTION TYPE 0x4
Stacktrace:
(14): 903E8998(903E8BF4)
(13): 9000B2B4(9000B36C)
(12): 900091D0(9000948C)
(11): 90427744(904282F8)
(10): 900094B0(9000A454)
(9): 904918F8(90491954)
(8): 90008000(90008488)
(7): 9000AEB0(9000AEC0)
(6): 9000AED4(9000B0E0)
(5): 9000C490(9000C518)
(4): 9000C56C(9000C5D4)
(3): 9000C56C(9000C5D4)
(2): 9000C56C(9000C5B8)
(1): 904808C0(904808C8)
(0): 904808F8(90480AD0)

I will be gratefull for any help !!!

Piotr

 

[spdom]

Hi,

I also have the same problem.

I have 2 AP 1000 & they reboot randomly.

One has a HSSI 34MB Card & the other has OC3 155MB Card.

Both Units have the same OS.

Lucent Access Point 1000 (hw: 1.6; sw: V2.5.0.R2.2 12/21/2001 19:10:38:00)

 

[spdom]

I have a capture of my error

>
PC: 0x90688784 SP: 0x92C99600 EXCEPTION TYPE 0x2
R0/ZERO=0x0000000000000000 R1/AT =0xFFFFFFFF90C90000
R2/V0 =0x0000000000000000 R3/V1 =0xFFFFFFFF90B7D9E0
R4/A0 =0xFFFFFFFF83FEFC98 R5/A1 =0xFFFFFFFFFFFFFF80
R6/A2 =0x0000000000000000 R7/A3 =0x0000000000000000
R8/T0 =0x0000000000000001 R9/T1 =0x0000000000000000
R10/T2 =0x0000000000000001 R11/T3 =0x0000000000000001
R12/T4 =0x000000000000002B R13/T5 =0x000000000000002D
R14/T6 =0x0000000000000000 R15/T7 =0x0000000000000AC4
R16/S0 =0x0000000000000000 R17/S1 =0x0000000000000000
R18/S2 =0xFFFFFFFF92C99698 R19/S3 =0x0000000000000000
R20/S4 =0xFFFFFFFF92C99708 R21/S5 =0x1122334411223344
R22/S6 =0x1122334411223344 R23/S7 =0x1122334411223344
R24/T8 =0xFFFFFFFF92CB43E2 R25/T9 =0xFFFFFFFF92CB43B8
R26/K0 =0x0000000000000000 R27/K1 =0x0000000000000000
R28/GP =0x1122334411223344 R29/SP =0xFFFFFFFF92C99600
R30/FP =0xFFFFFFFF92C998D8 R31/RA =0xFFFFFFFF9068B368
Stacktrace:
ª

Access Point simple bootstrap version 1.02
built 10/29/1999 11:14:17:00




Access Point full bootstrap version 1.0.R2
built 06/27/2000 17:08:26:00

enter ^C^C to break into prompt.

Starting decompression from 913EB4A8 to 90006000
Decompression done length:B9ECB4 booting OS at 90036000
The HTTP Server daemon was NOT started...


Lucent Access Point 1000 (hw: 1.6; sw: V2.5.0.R2.2 12/21/2001 19:10:38:00)

 

[PEagle]

I will resolve part of this problem. If You have any contact to Lucent support, try to request software revision 2.5.0R.3 for Your router. I also have newest hardware version (4.0).

Lucent Access Point 1000 (hw: 4.0; sw: V2.5.0.R3 12/28/2001 11:21:33:00)


This revision is not public available. If You read release notes, mips code is just little modyfied. Any way, when I start to using it, I find many changes. The most, major change is memory managemet, which give more control to memory allocation and give chance to connect 3 full BGP peers. I load up to 340000 routes with no special problems.
You should also try to change buffers quantum, max forwarding table and arp table size.

I write, that i resolve part of problems, becouse I still have troubles with exporting/sending selected routes to other routers (iBGP). I use filters based on AS-Path. I also know, how to make crash of this router just by enabling ICMP-REDIRECT in high-load.

When I reading your crash dump trace log, i see that is different Exception Type. In my log exception type is 0x04 (memory load/fetch problem). Your exception is 0x02.

I will be gratefull, if you will be interested to continue this thread. I still have open trouble tickets in Lucent Technologies and this contact can probably help you and me. I can send more informations to support. Maybe LT start seriouse investigation.

If you give me your e-mail address (me is porlewicz@teleton.pl). I will send you release notes to 2.5.0.R3 ;-)

I will looking for any contacts to admins, that are familiar with Access Point routers (any types, AP-450,600,1000,1500). I imagine to create working group (f.x. news serwer group), that helps to resolve problems, optimize configurations etc. If you know somebody that are interested this thing, let him know (and me )

Piotr

 

[spdom]

I have upgraded the AP to V2.5.4.4

pSOSystem (202.xxx.xxx.x)
Lucent Access Point 1000 (hw: 1.6; sw: V2.5.0.R4.4 10/27/2002 16:27:12:00)


When it reboots I get the following syslog errors.

2003-04-14 09:44:58 Local7.Debug 202.xxx.xxx.x 2 4/13/2003 18:50:37:00 REBO exception INFO last reboot caused by processor exception<010> 2 - TLB load/fetch<010> PC:90689150 SP:92C17D00
2003-04-14 09:44:58 Local7.Debug 202.xxx.xxx.x 3 4/13/2003 18:50:37:00 CFG INIT INFO The Configuration Management Subsystem is initializing.
2003-04-14 09:44:58 Local7.Debug 202.xxx.xxx.x 4 4/13/2003 18:50:37:51 SNMP Initialized INFO The SNMP Agent is Up and Running.
2003-04-14 09:44:58 Local7.Debug 202.xxx.xxx.x 5 4/13/2003 18:50:41:05 SSL Initialized INFO SSL environment is initialized and ready.
2003-04-14 09:44:58 Local7.Debug 202.xxx.xxx.x 6 4/13/2003 18:50:41:05 HTTPS Initialized INFO HTTP SSL Daemon is initialized and Running.
2003-04-14 09:44:58 Local7.Debug 202.xxx.xxx.x 7 4/13/2003 18:50:41:05 SshD Initialized INFO SecureShell Daemon is initialized and Running.
2003-04-14 09:44:58 Local7.Debug 202.xxx.xxx.x 8 4/13/2003 18:50:41:56 BGP initialize INFO BGP initialized

I have Lucent Support in AU looking into the problem also.

The unit rebooting has a HSSI card.
Syslog receives the following errors. Any idea what they are ??

2003-04-14 09:11:27 Local7.Debug 202.xxx.xxx.x 716 4/13/2003 18:17:19:79 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:12:27 Local7.Debug 202.xxx.xxx.x 717 4/13/2003 18:18:19:85 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:13:27 Local7.Debug 202.xxx.xxx.x 718 4/13/2003 18:19:19:80 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:14:27 Local7.Debug 202.xxx.xxx.x 719 4/13/2003 18:20:19:86 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:15:27 Local7.Debug 202.xxx.xxx.x 720 4/13/2003 18:21:19:81 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:16:27 Local7.Debug 202.xxx.xxx.x 721 4/13/2003 18:22:19:86 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:17:27 Local7.Debug 202.xxx.xxx.x 722 4/13/2003 18:23:19:93 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:18:27 Local7.Debug 202.xxx.xxx.x 723 4/13/2003 18:24:19:89 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:19:27 Local7.Debug 202.xxx.xxx.x 724 4/13/2003 18:25:19:93 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:20:27 Local7.Debug 202.xxx.xxx.x 725 4/13/2003 18:26:19:89 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:21:27 Local7.Debug 202.xxx.xxx.x 726 4/13/2003 18:27:19:94 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:22:27 Local7.Debug 202.xxx.xxx.x 727 4/13/2003 18:28:20:00 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:23:27 Local7.Debug 202.xxx.xxx.x 728 4/13/2003 18:29:19:95 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:24:28 Local7.Debug 202.xxx.xxx.x 729 4/13/2003 18:30:20:00 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:25:28 Local7.Debug 202.xxx.xxx.x 730 4/13/2003 18:31:20:06 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:26:28 Local7.Debug 202.xxx.xxx.x 731 4/13/2003 18:32:20:02 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:27:28 Local7.Debug 202.xxx.xxx.x 732 4/13/2003 18:33:20:07 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:28:28 Local7.Debug 202.xxx.xxx.x 733 4/13/2003 18:34:20:04 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:29:28 Local7.Debug 202.xxx.xxx.x 734 4/13/2003 18:35:20:09 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:30:28 Local7.Debug 202.xxx.xxx.x 735 4/13/2003 18:36:20:14 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:31:28 Local7.Debug 202.xxx.xxx.x 736 4/13/2003 18:37:20:10 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:32:28 Local7.Debug 202.xxx.xxx.x 737 4/13/2003 18:38:20:15 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:33:28 Local7.Debug 202.xxx.xxx.x 738 4/13/2003 18:39:20:10 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:34:28 Local7.Debug 202.xxx.xxx.x 739 4/13/2003 18:40:20:15 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:35:28 Local7.Debug 202.xxx.xxx.x 740 4/13/2003 18:41:20:21 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:36:28 Local7.Debug 202.xxx.xxx.x 741 4/13/2003 18:42:20:16 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32
2003-04-14 09:37:28 Local7.Debug 202.xxx.xxx.x 742 4/13/2003 18:43:20:22 FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32

 

[PEagle]

It seems like some interoperability problem. What kind of system is on second site of Your FR link ?

The 2.5.4.4 is available in Lucent as production release or hot-fix ?

ps. W will start new 1500 box with 3.0.0R1 software. It is much more stable, but needs more memory (minimu 128MB to start system).

Piotr

 

[spdom]

I have tried the following OS on both systems

V2.2.1R.1
V2.5.0.R2.2
V2.5.0R.3
V2.5.0R.4.4

They all have the reboot problem..

It seams a bit strange that I have 3 systems (1 spare) & they all reboot. 1 has a HSSI & the other a OC3 card. I think it may have something to do with the way I am setting up the box.

Can you email me steven@australis.com.au your lucent configs (txt version is best) minus any security stuff & i'll compare them to mine & see if there are any similarities

 

[PEagle]

It just means, that is not hardware failure. It can be system bug or system manufacure serial problem.

In my opinion, errored configuration should not cause system reboot, becouse it potentialy can be very big security problem.

Look at You mail box for my configs.

 

[spdom]

Peter,

I still have not been able to resolve my rebooting issue.

I must admit that since the Blaster virus has been around the router has been rebooting more than ever. I have included my SYSLOG REBO entries, as you can see the 20th Aug the router rebooted more than normal this is when Blaster started.

I found a MaxTNT user having a crashing problem due to the virus; do you think that it could be related to what we are experiencing?

Cheers
Steven.


4 5/20/2003 17:52:53:080 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
6 5/30/2003 01:22:48:005 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
10 6/5/2003 04:38:22:053 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 6/6/2003 17:28:35:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 6/7/2003 09:11:08:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 6/7/2003 21:59:21:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
12 6/10/2003 23:18:55:080 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 6/12/2003 07:52:28:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 6/13/2003 11:42:21:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
10 6/18/2003 19:14:35:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
6 6/23/2003 06:23:29:005 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
14 6/25/2003 13:38:43:005 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
12 6/26/2003 23:41:37:080 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 6/27/2003 04:12:10:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
30 7/3/2003 00:11:27:003 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
40 7/13/2003 19:33:45:025 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
8 7/18/2003 12:13:39:030 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 7/20/2003 04:44:16:000 REBO userReset INFO last reboot caused by user reset
6 7/27/2003 07:10:11:005 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
4 7/30/2003 02:57:44:080 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 7/30/2003 19:35:17:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
18 8/3/2003 07:33:32:050 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
2 8/4/2003 10:29:05:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
10 8/9/2003 23:54:00:051 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
6 8/12/2003 04:56:55:005 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
52 8/20/2003 04:12:53:055 REBO powerLoss INFO last reboot caused by power loss
52 8/20/2003 04:38:52:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
52 8/20/2003 05:09:52:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e8cc
52 8/20/2003 05:15:11:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
52 8/20/2003 05:20:31:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
52 8/20/2003 05:56:10:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
52 8/20/2003 09:34:29:030 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974
52 8/21/2003 04:53:47:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
52 8/21/2003 05:04:46:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
52 8/21/2003 05:14:06:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
52 8/22/2003 05:34:25:055 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9039e658
2 8/22/2003 15:26:38:000 REBO exception INFO last reboot caused by processor exception<010> type 2 - TLB load/fetch<010> pc = 9055e974

> Hi,
> I know this isn't the place, but any MAX TNT users out there seeing weird card failures begining with the onslaught of MSBlast? I saw a news.com article about it... however I can't find any more info. Anyone know of any active ascend / lucent tnt mailing lists?
Sean

> Article Text:
In addition, network administrators reported on a ewsgroup that telecommunications equipment maker Lucent Technologies' TNT MAX network gateway crashed due to some interaction with traffic created by the MSBlast worms. A representative for the company confirmed that Lucent was
investigating the issue, but couldn't supply details.

---------------------

This problem is actually caused by the &quot;good&quot; blaster worm nachi
> Nachi pings a host before it trys to spread so it doesn't waist its time on non-existent hosts. The problem is that each one of those pings generates an arp request and with such a high number of pings MAX TNT boxes can't handle the high number of arp request and lock up or reboot
> The ping has a specific signature, 92byes all AA as the content, that you can create a policy map for Cisco has an article on how to block Nachi ICMP traffic on your inbound router interface

http://www.cisco.com/warp/public/707/cisco-sn-20030820-nachi.shtml

 

[petercheung]

hi all,

I am using AP1000 (hw:4.1;sw:V2.5.0.R1), facing the problem or random reboot too. More, the ip.1 interface will up-down randomly. I lost the contact of the lucent maintenance , and would like know where can find the sw upgrade?

thx..

..peter
From Hong Kong

 

[PEagle]

Hi All,

1.
I receive annousment from Lucent Techical Suport, that new version of APOS for models AP-1000 and AP-450 are available.
The new version has number 2.5.0R.5.1 and has some fixes to memory management, especialy for forwarding profiles and ICMP redirection. In all previouse version is is possible to create configuration, that overload memory buffers and in results cause in box reboot (with panic).

2.
Peter, try to find somewhere new software. You can also try to change Your memory allocation. In version 2.5 driver buffers modification is available.

Piotr

 

[spdom]

Hi everyone,

I'm still getting these problems, I'm working with Lucent in Australia to resolve the problem.

Does anyone else have Cisco Routers that connect to the Lucent AP.
If so, do the Cisco Routers run Cisco Discovery Protocol (CDP)

Does anyone else get an error like this
FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32

 

[grazman]

The
FR dlRcvNotUI INFO frame-relay.1 - dl rcv packet dropped, not UI frame, 32

message is a setting that needs to be changed on the other end. Proxy arp needs to be set differently on the other end.

if it s a cisco router at the other end and the config reads

no ip proxy-arp

then

#configure terminal
(config)#interface Ser
(config)#interface Serial 3/3.500
(config-subif)#ip pr
(config-subif)#ip proxy-arp

save/write the change to the cisco, it'll probably go away.

 

[grazman]

Reboots have been seen due to a bug in the router code. Had something to do with NTP port scanning.

Upgrade to the latest release

v2.5.0.R5.QVPN

See if that helps.

 

[grazman]

Do you have the HTTP server enabled? If so, disable it (requires a reboot after doing so).

http://www.securitytracker.com/alerts/2002/Jul/1004867.html

 

[BITlucent]

What we have done in the interim is to switch off the http management console , since doing that the router has experienced uptimes of more than 1 hour that is used to.
It has currently been up for a while.


Description = Lucent Access Point 1000 (hw: 4.0; sw: V2.5.0.R3.QV PN 12/26/2001 16:35:21:00)
Time Since Last Boot = 7340658 (20 hours 23 min 26 secs)

Before doing this , we never ever had a issue or problem with the web management console in the 1st place and we've switched it off also one time before for the same problem and it did not solve the issue of frequent hourly reboots then ( 12 months previously )

 

[teartear]

Same problem with our AP-1000.

To stop the web server:

>config services web-server-desired-status down

>reset

I certainly hope this stops the frequent reboots. Customer calls were getting more and more hostile. Thanks for this thread - I wish I had found it sooner.

The web based UI was nice for those folks in training or scared of the CLI. It's a shame that it's gone, but that java stuff was getting old anyway.

Will Lucent respond with a free fix or is it time to scrap the AP-1000 and upgrade?

 

[petercheung]

>PEagle , I want get the latest version too, however my company need I present some proof / release notes about this problem. Where can I got this?

Lucent in HK tech support is rare and expensive., I wonder will which is the software 2.5.0 R?

>spdom , I am connecting ap1000 with cisco 7501.

 

[MotleyC]

Has anyone solved these problems? We have 2 AP450's which get similar stack dumps and then reboot. This have been going on for several months and we have not been able to find a solution. We are running software version 2.5R3 and I know 2.5R6 is the newest but we do not have access to it. If someone has access to the newest software please I would GREATLY appreciate it.