Bus error crashes, need help interpreting... 2

[americanmcneil]

I have a Cisco 7200 VXR that has recently started to crash randomly. Not wry often but often enough to cause me concern. After some digging i found that it is a bus error causing the crash. After doing some more digging on the Cisco website I did a "show context" and it brought up the error that caused the crash and the memory location. Now, I am not a registered Cisco customer, just a registered guest user so I cannot use the Output Interpreter tool on the website. Can anyone help me interpret the results? If so, I will post the output I get from the "show context" here.

Scott "Thrown to the Wolves" McNeil

http://www.epproach.net

 

[americanmcneil]

Actually, I'll go ahead and post it now...

System was restarted by bus error at PC 0x60FBC420, address 0x38 at 14:27:33 EST Tue Mar 4 2008
7200 Software (C7200-IK9O3S-M), Version 12.3(3i), RELEASE SOFTWARE (fc1)
Compiled Fri 12-Aug-05 18:48 by ssearch
Image text-base: 0x60008954, data-base: 0x61E5E000


Stack trace from system failure:
FP: 0x643B5320, RA: 0x60FBC420
FP: 0x643B5340, RA: 0x60FB2B9C
FP: 0x643B5358, RA: 0x60FBAD20
FP: 0x643B5370, RA: 0x60FB4E5C
FP: 0x643B53B8, RA: 0x60FAC520
FP: 0x643B53E8, RA: 0x60FAC82C
FP: 0x643B5450, RA: 0x60FACDFC

Fault History Buffer:
7200 Software (C7200-IK9O3S-M), Version 12.3(3i), RELEASE SOFTWARE (fc1)
Compiled Fri 12-Aug-05 18:48 by ssearch
Signal = 10, Code = 0xC, Uptime 3d22h
$0 : 00000000, AT : 63100000, v0 : 00000006, v1 : 00000006
a0 : 00000000, a1 : 0000FF00, a2 : 00000000, a3 : 143262BC
t0 : 00000020, t1 : 3400FF01, t2 : 3400C100, t3 : FFFF00FF
t4 : 607DD4F0, t5 : 6444D340, t6 : 6444D33C, t7 : 6444D338
s0 : 636539D8, s1 : 00000000, s2 : 643AC6B4, s3 : 00000001
s4 : 63610000, s5 : 643B5408, s6 : 643B5410, s7 : 63500000
t8 : 0D0D0D0D, t9 : 00000000, k0 : 3040D001, k1 : 00000040
gp : 63109028, sp : 643B5320, s8 : 00000000, ra : 60FB2B9C
EPC : 60FBC420, SREG : 3400FF03, Cause : 0000000C
Error EPC : 920808A1, BadVaddr : 0000003

Now, according to the website the interpreter will help me determine if the error memory location is a software issue or a hardware issue. Any help would be greatly appreciated.

Scott "Thrown to the Wolves" McNeil

http://www.epproach.net

 

[burtsbees]

The following hyperlinks are headings for the supported commands that you
submitted. Each hyperlink takes you to the relevant section within the analysis results.

ROUTER CONSOLE MESSAGE Analysis
STACK DECODE Analysis

Back to top
ROUTER CONSOLE MESSAGE NOTIFICATIONS (if any)

ERROR: The system encounters a 'Bus Error' when the processor tries to access
a memory location that either does not exist (indicating a software issue) or does
not respond properly (indicating a hardware issue).
TRY THIS:
1. For 68000 Processor based platforms, carry out the following steps:
Consider the output:
"System restarted by bus error at PC 0x30EE546, address 0xBB4C4".
Here the system is trying to access the address "0xBB4C4"
a. Based on the address accessed by the router when the 'Bus Error' occurred,
use the "show region" command to determine the memory location to which
the address corresponds. If the address reported by the 'Bus Error' does
not fall within the ranges displayed in the show region output, it implies
that the router was trying to access an invalid address. This in turn indicates
a Cisco IOS Software problem. Use the Output Interpreter to decode the output
of the "show stacks" command to identify the software bug that is causing
the bus error.
b. If the address falls within one of the ranges in the "show region" output,
it implies that the router was accessing a valid memory location, but the
hardware corresponding to that address did not respond properly. This indicates
a hardware problem. Find the memory address that the system is trying to
access (for example in the above output, it is 0xBB4C4), and also the range
of memory addresses between which this address falls. The hardware issue
is most likely related to the corresponding component. If a new router has
been installed, or if the router has been moved from one location to another,
the memory chips may have become loose. It is recommended that you re-seat
or firmly push the memory chips into the slot.
For more information, see Troubleshooting Bus Error Crashes.
2. For the RISC Processor based platforms, use the Output Interpreter Tool to decode
the output of the "show stacks" command and identify the Cisco IOS Software
bug that is causing the bus error.
REFERENCE: For more information on 'Bus Error' crashes, see
TAC Case Collection - System is in a boot loop, a bus error exception
TAC Case Collection - Router reloads due to a bus error
Troubleshooting Bus Error Crashes

REFERENCE: For more information, see:
Troubleshooting Router Crashes
Less Common Types of System Crashes

Back to top
STACK DECODE NOTIFICATIONS (if any)

Note: Understanding the 'Fixed In Version' field - Sometimes the fixed-in version
specified may not be available for download from the IOS Upgrade Planner. This
is because, bug fixes are incorporated into software versions which have not been
regression tested. These builds are called interim images and will have a "."
in the build number found between the parenthesis, for example 12.3(8.1). The bug
fix will be in the next available image, and all later images. So if a bug is fixed
in 12.3(8.1), the bug fix is present in 12.3(9), 12.3(10), and so on.
REFERENCE: For more information on how IOS images are named, see Cisco IOS White
Paper.

The failure was caused by a software defect.
Note that this is a bus error crash
and can also be hardware related.
Please read Troubleshooting Bus Error Crashes
for more details,
especially if no bug from the list seems to match your issue.

The stack trace decoded symbols are:
ipnat_destroy_all_seq_delta_pairs
ipnat_destroy_all_seq_delta_pairs
ipnat_destroy_sip_appl_data
ipnat_delete_appl_data
ipnat_delete_entry
ipnat_age_protocol_entries
ipnat_ager_timers
ipnat_ager
r4k_level_table
ipnat_destroy_sip_appl_data
ipnat_destroy_all_seq_delta_pairs

Possible bug matches are listed below. Bugs with a score of .90 or more
are the most likely candidates:


Score Bugid Status Fixed In Duplicate Title
0.93 CSCed65315 R 12.3(9b) 12.3(9.10)T 12.3(9.10) None Bus error @ ipnat_destroy_all_seq_delta_pairs.
0.65 CSCeb12516 D CSCdz14900 nrp crash at ipnat_destroy_all_seq_delta_pairs
0.65 CSCec58210 D 12.3(0.1)PI1 CSCed65315 Bus error at ipnat_apply_seqdelta
0.53 CSCee63116 D CSCed65315 NRP2 crashes at %ALIGN-1-FATAL:Illegal access to a low address



Burt

 

[burtsbees]

Here are the bugs...

A)Bug ID CSCed65315

Bus error @ ipnat_destroy_all_seq_delta_pairs.
Symptoms: A Cisco router that runs Cisco IOS Release 12.3(5a) may reload
because of a bus error. The output of the show version
command may show the following:

System returned to ROM by bus error at PC 0xXXXXXXXX, address 0xYYYYYYYY

Conditions: The symptom may be observed when IP NAT is configured.

Workaround: Enter the no ip nat service sip tcp port
5060 command and the no ip nat service sip udp port
5060 command.

The following link provides general information about bus errors:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800cdd51.shtml

Status
Fixed (Resolved)

Severity
2



Product
Cisco IOS software

Technology


1st Found-In
12.3(5b)M
Known Affected Versions This link will launch a new window.


Fixed-In
12.3(9.10)M
12.3(9.10)T
12.3(9b)M

Component(s)
nat

Regression
Y
Related Bugs
Bus error at ipnat_apply_seqdelta
Symptom: A router running IOS version 122-19 may reload wth a bus error. Conditions: Nat is configured. Workaround: There is no known workaround at this time.
NRP2 crashes at %ALIGN-1-FATAL:Illegal access to a low address
Symptom: A Cisco 6400 NRP2 , upgraded to c6400r2sp-g4p5-mz.123-9 , may experience a condition where the NRP raqndomly crashes and displays the following error message on the console output: %ALIGN-1-FATAL: Illegal access to a low address addr=0x38, pc=0x60DB8834, ra=0x60DAEA54, sp=0x629540D8 Unexpected exception, CPU signal 10, PC = 0x60DB8834 -Traceback= 60DB8834 60DAEA54 $0 : 00000000, AT : 61430000, v0 : 00000006, v1 : 00000006 a0 : 00000000, a1 : 0000FF00, a2 : 00000000, a3 : 01703D58 t0 : 00000038, t1 : 3400FF01, t2 : 00001782, t3 : FFFF00FF t4 : 00001700, t5 : 00000000, t6 : FFFFC0FF, t7 : 00000000 s0 : 62A12AB8, s1 : 00000000, s2 : 62DDB034, s3 : 00000001 s4 : 61520000, s5 : 629541C8, s6 : 629541D0, s7 : 61510000 t8 : 62367558, t9 : 000012E9, k0 : 30400801, k1 : 00300000 gp : 6143BF88, sp : 629540D8, s8 : 00000000, ra : 60DAEA54 EPC : 60DB8834, ErrorEPC : 602EC450, SREG : 3400FF03 MDLO : 00000000, MDHI : 00000000, BadVaddr : 00000038 Cause 0100000C (Code 0x3): TLB (store) exception NRP2-CP 00000202 / 00000000 System Bootstrap, Version 12.0(20000930:035957) [pgettner-r36 101], DEVELOPMENT SOFTWARE Copyright (c) 1994-2000 by cisco Systems, Inc. Composite Reset Reason = (0xa) Level 1 Watch Dog time out Level 2 Watch Dog time out Conditions: Occurs randomly after upgrade to this release. Workaround: None

B)Bug ID CSCeb12516

Software forced crash ipnat_l4_udp_sip_fixup
Symptom:
A router experience a software forced reload.

Conditions:
The router is configured for NAT SIP which is enabled by default.

Workaround:
turn off NAT SIP using the following commands:

no ip nat service sip tcp port 5060
no ip nat service sip udp port 5060

Before CSCdy13584, this workaround does not work. Status
Fixed (Resolved)

Severity
3



Product
Cisco IOS software

Technology


1st Found-In
12.2(15)B
12.2(11)T
12.3M
Known Affected Versions This link will launch a new window.


Fixed-In
12.2(13)T12
12.3(2)T9
12.3(3.1)T
12.3(7)XI
12.3(2.3)B
12.3(0.5)B3a
12.2(4)YA11
12.3(2.3)M

Component(s)
nat

Regression
Y
Related Bugs
Bus error crash at ip_feature_fastswitch
Symptom: A Cisco 3725 router may reload unexpectedly with a bus error under high traffic conditions Conditions: High traffic (6-7MB) triggers it. Workaround: None
nrp crash at ipnat_destroy_all_seq_delta_pairs
Customer's NRP crashes periodically due to nat processes.
NPE-G1 restarts by bus error at dequeue
cisco 7206VXR NPE-G1 with c7200-is-mz.122-16.B.bin crashes with bus error There is no known workaround yet
NPE-G1 restarts by bus error at turbo_extended_check
cisco 7206VXR NPE-G1 with c7200-is-mz.122-16.B.bin crashes with bus error There is no known workaround yet
Crashes were found in ARF with nat_sip regression testing
Router with IOS 12.2(15)T05 crashed with nat_sip regression test. The crash does not happen consistantly. Workaround: turn off NAT SIP ALG will aviod the crash.
Crash after the IOS upgrade PC 0x3C36CC
An IAD2420 may crash during normal processing of calls. Cause is unknown. There is no workaround.
c7200: Memory corruption in processor pool in 12.2(8)T4
Symptoms: A software-forced reload may occur on a Cisco 7200 series, and the console output may display memory corruption dumps. Conditions: This symptom is observed on a Cisco 7200 series that is running Cisco IOS Release 12.2(8)T4. Workaround: There is no workaround.
SegV exception at ip_feature_fastswitch
A Cisco router may crash with a SegV exception. There is no known workaround at this time.
NPE-G1 restarts by bus error while fast-switching
cisco 7206VXR NPE-G1 with c7200-is-mz.122-16.B.bin crashes with bus error There is no known workaround yet

C)Bug ID CSCec58210

Bus error @ ipnat_destroy_all_seq_delta_pairs.
Symptoms: A Cisco router that runs Cisco IOS Release 12.3(5a) may reload
because of a bus error. The output of the show version
command may show the following:

System returned to ROM by bus error at PC 0xXXXXXXXX, address 0xYYYYYYYY

Conditions: The symptom may be observed when IP NAT is configured.

Workaround: Enter the no ip nat service sip tcp port
5060 command and the no ip nat service sip udp port
5060 command.

The following link provides general information about bus errors:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800cdd51.shtml

Status
Fixed (Resolved)

Severity
2



Product
Cisco IOS software

Technology


1st Found-In
12.3(5b)M
Known Affected Versions This link will launch a new window.


Fixed-In
12.3(9.10)M
12.3(9.10)T
12.3(9b)M

Component(s)
nat

Regression
Y
Related Bugs
Bus error at ipnat_apply_seqdelta
Symptom: A router running IOS version 122-19 may reload wth a bus error. Conditions: Nat is configured. Workaround: There is no known workaround at this time.
NRP2 crashes at %ALIGN-1-FATAL:Illegal access to a low address
Symptom: A Cisco 6400 NRP2 , upgraded to c6400r2sp-g4p5-mz.123-9 , may experience a condition where the NRP raqndomly crashes and displays the following error message on the console output: %ALIGN-1-FATAL: Illegal access to a low address addr=0x38, pc=0x60DB8834, ra=0x60DAEA54, sp=0x629540D8 Unexpected exception, CPU signal 10, PC = 0x60DB8834 -Traceback= 60DB8834 60DAEA54 $0 : 00000000, AT : 61430000, v0 : 00000006, v1 : 00000006 a0 : 00000000, a1 : 0000FF00, a2 : 00000000, a3 : 01703D58 t0 : 00000038, t1 : 3400FF01, t2 : 00001782, t3 : FFFF00FF t4 : 00001700, t5 : 00000000, t6 : FFFFC0FF, t7 : 00000000 s0 : 62A12AB8, s1 : 00000000, s2 : 62DDB034, s3 : 00000001 s4 : 61520000, s5 : 629541C8, s6 : 629541D0, s7 : 61510000 t8 : 62367558, t9 : 000012E9, k0 : 30400801, k1 : 00300000 gp : 6143BF88, sp : 629540D8, s8 : 00000000, ra : 60DAEA54 EPC : 60DB8834, ErrorEPC : 602EC450, SREG : 3400FF03 MDLO : 00000000, MDHI : 00000000, BadVaddr : 00000038 Cause 0100000C (Code 0x3): TLB (store) exception NRP2-CP 00000202 / 00000000 System Bootstrap, Version 12.0(20000930:035957) [pgettner-r36 101], DEVELOPMENT SOFTWARE Copyright (c) 1994-2000 by cisco Systems, Inc. Composite Reset Reason = (0xa) Level 1 Watch Dog time out Level 2 Watch Dog time out Conditions: Occurs randomly after upgrade to this release. Workaround: None

D)Bug ID CSCee63116

Bus error @ ipnat_destroy_all_seq_delta_pairs.
Symptoms: A Cisco router that runs Cisco IOS Release 12.3(5a) may reload
because of a bus error. The output of the show version
command may show the following:

System returned to ROM by bus error at PC 0xXXXXXXXX, address 0xYYYYYYYY

Conditions: The symptom may be observed when IP NAT is configured.

Workaround: Enter the no ip nat service sip tcp port
5060 command and the no ip nat service sip udp port
5060 command.

The following link provides general information about bus errors:

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a00800cdd51.shtml

Status
Fixed (Resolved)

Severity
2



Product
Cisco IOS software

Technology


1st Found-In
12.3(5b)M
Known Affected Versions This link will launch a new window.


Fixed-In
12.3(9.10)M
12.3(9.10)T
12.3(9b)M

Component(s)
nat

Regression
Y
Related Bugs
Bus error at ipnat_apply_seqdelta
Symptom: A router running IOS version 122-19 may reload wth a bus error. Conditions: Nat is configured. Workaround: There is no known workaround at this time.
NRP2 crashes at %ALIGN-1-FATAL:Illegal access to a low address
Symptom: A Cisco 6400 NRP2 , upgraded to c6400r2sp-g4p5-mz.123-9 , may experience a condition where the NRP raqndomly crashes and displays the following error message on the console output: %ALIGN-1-FATAL: Illegal access to a low address addr=0x38, pc=0x60DB8834, ra=0x60DAEA54, sp=0x629540D8 Unexpected exception, CPU signal 10, PC = 0x60DB8834 -Traceback= 60DB8834 60DAEA54 $0 : 00000000, AT : 61430000, v0 : 00000006, v1 : 00000006 a0 : 00000000, a1 : 0000FF00, a2 : 00000000, a3 : 01703D58 t0 : 00000038, t1 : 3400FF01, t2 : 00001782, t3 : FFFF00FF t4 : 00001700, t5 : 00000000, t6 : FFFFC0FF, t7 : 00000000 s0 : 62A12AB8, s1 : 00000000, s2 : 62DDB034, s3 : 00000001 s4 : 61520000, s5 : 629541C8, s6 : 629541D0, s7 : 61510000 t8 : 62367558, t9 : 000012E9, k0 : 30400801, k1 : 00300000 gp : 6143BF88, sp : 629540D8, s8 : 00000000, ra : 60DAEA54 EPC : 60DB8834, ErrorEPC : 602EC450, SREG : 3400FF03 MDLO : 00000000, MDHI : 00000000, BadVaddr : 00000038 Cause 0100000C (Code 0x3): TLB (store) exception NRP2-CP 00000202 / 00000000 System Bootstrap, Version 12.0(20000930:035957) [pgettner-r36 101], DEVELOPMENT SOFTWARE Copyright (c) 1994-2000 by cisco Systems, Inc. Composite Reset Reason = (0xa) Level 1 Watch Dog time out Level 2 Watch Dog time out Conditions: Occurs randomly after upgrade to this release. Workaround: None


Burt

 

[burtsbees]

One more thing---looks like the fix is either upgrade the IOS or put in the commands no ip nat service sip tcp port 5060 and no ip nat service sip udp port 5060...

Burt

 

[americanmcneil]

Thanks tons for the breakdown burtsbees, I just put those commands in and will see how it goes. Ill update soon!



Scott "Thrown to the Wolves" McNeil

http://www.epproach.net

 

[americanmcneil]

Well, unfortunately the command line change did not fix the problem. I really hoped it would. That being said, if I am understanding things correctly, this means that it is a hardware problem. So my next question is this, if the error is that the router is trying to pull information from a memory address that no longer exists, is this a hardware problem with the router it's self or is it the memory card (it think it is 64mb, i am at home so am not in front of the router)? So is replacing the entire router in order (yikes) or replacing the memory card and reconfiguring the router?

Scott "Thrown to the Wolves" McNeil

http://www.epproach.net

 

[burtsbees]

Well, it just looks like it's a buggy IOS...I would say to upgrade the IOS first...

Burt

 

[americanmcneil]

That is what I was thinking, either that or possibly a hardware memory error and possible replacing the memory. I am going to look into both. I am still open to other suggestions as well, and thanks again burt, much appreciated.

Scott "Thrown to the Wolves" McNeil

http://www.epproach.net