smittydotbat
Technical User
RH 8.0 (Psyche) 2.4.20-28.2
System keeps hanging intermittently after kernel panics. Requires physical reboot of the box (box is remotely pingable, but Telnet/SSH do not work).
Any ideas?
---------------------------
Information from KSYMOOPS
---------------------------
ksymoops 2.4.5 on i686 2.4.20-28.8. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.20-28.8/ (default)
-m /boot/System.map-2.4.20-28.8 (default)
Warning: You did not tell me where to find symbol information. I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc. ksymoops -h explains the options.
Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/3w-xxxx.o) for 3w-xxxx
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
Warning (map_ksym_to_module): cannot match loaded module ext3 to a unique module
object. Trace may not be reliable.
May 31 04:37:44 lt068_iss1 kernel: <1>Unable to handle kernel paging request at
virtual address 2fc1c0d4
May 31 04:37:44 lt068_iss1 kernel: c01406b0
May 31 04:37:44 lt068_iss1 kernel: *pde = 00000000
May 31 04:37:44 lt068_iss1 kernel: Oops: 0000
May 31 04:37:44 lt068_iss1 kernel: CPU: 0
May 31 04:37:44 lt068_iss1 kernel: EIP: 0010:[<c01406b0>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
May 31 04:37:44 lt068_iss1 kernel: EFLAGS: 00010202
May 31 04:37:44 lt068_iss1 kernel: eax: c1a20100 ebx: c1a2011c ecx: c030ac80
edx: c1a20100
May 31 04:37:44 lt068_iss1 kernel: esi: 0000001e edi: 2fc1c058 ebp: 00000000
esp: c3491f10
May 31 04:37:44 lt068_iss1 kernel: ds: 0018 es: 0018 ss: 0018
May 31 04:37:44 lt068_iss1 kernel: Process kswapd (pid: 4, stackpage=c3491000)
May 31 04:37:44 lt068_iss1 kernel: Stack: f60cc000 00000000 00000000 c3491f5c c1
a20100 c1a2011c 00001ce2 c1a20100
May 31 04:37:44 lt068_iss1 kernel: 00001ce2 c0138aab c1879818 00000000 c0
30ae34 c030adb4 0000003f ffffffff
May 31 04:37:44 lt068_iss1 kernel: 00000009 0000013b 00026827 00000000 00
006697 c030ac80 00000000 00000100
May 31 04:37:44 lt068_iss1 kernel: Call Trace: [<c0138aab>] refill_inactive_zo
ne [kernel] 0x5eb (0xc3491f34))
May 31 04:37:44 lt068_iss1 kernel: [<c0139453>] rebalance_inactive_zone [kernel]
0xd3 (0xc3491f70))
May 31 04:37:44 lt068_iss1 kernel: [<c01394b8>] rebalance_inactive [kernel] 0x48
(0xc3491f90))
May 31 04:37:44 lt068_iss1 kernel: [<c0139601>] do_try_to_free_pages_kswapd [ker
nel] 0x51 (0xc3491fb4))
May 31 04:37:44 lt068_iss1 kernel: [<c0139893>] kswapd [kernel] 0x83 (0xc3491fd4
))
May 31 04:37:44 lt068_iss1 kernel: [<c0105000>] stext [kernel] 0x0 (0xc3491fec))
May 31 04:37:44 lt068_iss1 kernel: [<c010741e>] arch_kernel_thread [kernel] 0x2e
(0xc3491ff0))
May 31 04:37:44 lt068_iss1 kernel: [<c0139810>] kswapd [kernel] 0x0 (0xc3491ff8)
)
May 31 04:37:44 lt068_iss1 kernel: Code: 8b 44 b7 04 85 c0 0f 84 b9 00 00 00 89
c2 8b 0d b0 91 37 c0
>>EIP; c01406b0 <page_referenced+170/2c0> <=====
>>eax; c1a20100 <_end+1668d00/38455c60>
>>ebx; c1a2011c <_end+1668d1c/38455c60>
>>ecx; c030ac80 <contig_page_data+300/a80>
>>edx; c1a20100 <_end+1668d00/38455c60>
>>edi; 2fc1c058 Before first symbol
>>esp; c3491f10 <_end+30dab10/38455c60>
Trace; c0138aab <refill_inactive_zone+5eb/650>
Trace; c0139453 <rebalance_inactive_zone+d3/f0>
Trace; c01394b8 <rebalance_inactive+48/90>
Trace; c0139601 <do_try_to_free_pages_kswapd+51/1d0>
Trace; c0139893 <kswapd+83/1d0>
Trace; c0105000 <_stext+0/0>
Trace; c010741e <arch_kernel_thread+2e/40>
Trace; c0139810 <kswapd+0/1d0>
Code; c01406b0 <page_referenced+170/2c0>
00000000 <_EIP>:
Code; c01406b0 <page_referenced+170/2c0> <=====
0: 8b 44 b7 04 mov 0x4(%edi,%esi,4),%eax <=====
Code; c01406b4 <page_referenced+174/2c0>
4: 85 c0 test %eax,%eax
Code; c01406b6 <page_referenced+176/2c0>
6: 0f 84 b9 00 00 00 je c5 <_EIP+0xc5>
Code; c01406bc <page_referenced+17c/2c0>
c: 89 c2 mov %eax,%edx
Code; c01406be <page_referenced+17e/2c0>
e: 8b 0d b0 91 37 c0 mov 0xc03791b0,%ecx
May 31 11:45:02 lt068_iss1 kernel: 8139too Fast Ethernet driver 0.9.26
Jun 1 06:17:21 lt068_iss1 kernel: Unable to handle kernel paging request at vir
tual address fb303c2c
Jun 1 06:17:21 lt068_iss1 kernel: c022e50e
Jun 1 06:17:21 lt068_iss1 kernel: *pde = 00000000
Jun 1 06:17:21 lt068_iss1 kernel: Oops: 0002
Jun 1 06:17:21 lt068_iss1 kernel: CPU: 0
Jun 1 06:17:21 lt068_iss1 kernel: EIP: 0010:[<c022e50e>] Not tainted
Jun 1 06:17:21 lt068_iss1 kernel: EFLAGS: 00010246
Jun 1 06:17:21 lt068_iss1 kernel: eax: 00000000 ebx: f0f04580 ecx: f4a93c24
edx: 00000000
Jun 1 06:17:21 lt068_iss1 kernel: esi: f4a93c14 edi: f4a93b00 ebp: c3515180
esp: db0adf4c
Jun 1 06:17:21 lt068_iss1 kernel: ds: 0018 es: 0018 ss: 0018
Jun 1 06:17:21 lt068_iss1 kernel: Process savsecls.exe (pid: 15845, stackpage=d
b0ad000)
Jun 1 06:17:21 lt068_iss1 kernel: Stack: f0f04580 c022ef12 f4a93c14 c36901c0 c0
1eb29c f4a93c14 f4a93b00 c01eb879
Jun 1 06:17:21 lt068_iss1 kernel: f4a93c14 f0dadb00 00000000 f0dadb00 c0
144662 f4a93b00 f0dadb00 f0dadb00
Jun 1 06:17:21 lt068_iss1 kernel: ed9ceb80 00000000 bfff7568 c0142c6d f0
dadb00 ed9ceb80 f0dadb00 0808f130
Jun 1 06:17:21 lt068_iss1 kernel: Call Trace: [<c022ef12>] inet_shutdown [ker
nel] 0xc2 (0xdb0adf50))
Jun 1 06:17:21 lt068_iss1 kernel: [<c01eb29c>] sock_release [kernel] 0x5c (0xdb
0adf5c))
Jun 1 06:17:21 lt068_iss1 kernel: [<c01eb879>] sock_close [kernel] 0x39 (0xdb0a
df68))
Jun 1 06:17:21 lt068_iss1 kernel: [<c0144662>] fput [kernel] 0x102 (0xdb0adf7c)
)
Jun 1 06:17:21 lt068_iss1 kernel: [<c0142c6d>] filp_close [kernel] 0x4d (0xdb0a
df98))
Jun 1 06:17:21 lt068_iss1 kernel: [<c0142cee>] sys_close [kernel] 0x4e (0xdb0ad
fb0))
Jun 1 06:17:21 lt068_iss1 kernel: [<c010905f>] system_call [kernel] 0x33 (0xdb0
adfc0))
Jun 1 06:17:21 lt068_iss1 kernel: Code: c7 46 18 00 00 00 00 8b 83 b8 00 00 00
89 54 24 04 89 1c 24
>>EIP; c022e50e <inet_release+3e/70> <=====
>>ebx; f0f04580 <_end+30b4d180/38455c60>
>>ecx; f4a93c24 <_end+346dc824/38455c60>
>>esi; f4a93c14 <_end+346dc814/38455c60>
>>edi; f4a93b00 <_end+346dc700/38455c60>
>>ebp; c3515180 <_end+315dd80/38455c60>
>>esp; db0adf4c <_end+1acf6b4c/38455c60>
Trace; c022ef12 <inet_shutdown+c2/1a0>
Trace; c01eb29c <sock_release+5c/60>
Trace; c01eb879 <sock_close+39/60>
Trace; c0144662 <fput+102/130>
Trace; c0142c6d <filp_close+4d/80>
Trace; c0142cee <sys_close+4e/60>
Trace; c010905f <system_call+33/38>
Code; c022e50e <inet_release+3e/70>
00000000 <_EIP>:
Code; c022e50e <inet_release+3e/70> <=====
0: c7 46 18 00 00 00 00 movl $0x0,0x18(%esi) <=====
Code; c022e515 <inet_release+45/70>
7: 8b 83 b8 00 00 00 mov 0xb8(%ebx),%eax
Code; c022e51b <inet_release+4b/70>
d: 89 54 24 04 mov %edx,0x4(%esp,1)
Code; c022e51f <inet_release+4f/70>
11: 89 1c 24 mov %ebx,(%esp,1)
2 warnings and 5 errors issued. Results may not be reliable.
System keeps hanging intermittently after kernel panics. Requires physical reboot of the box (box is remotely pingable, but Telnet/SSH do not work).
Any ideas?
---------------------------
Information from KSYMOOPS
---------------------------
ksymoops 2.4.5 on i686 2.4.20-28.8. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.20-28.8/ (default)
-m /boot/System.map-2.4.20-28.8 (default)
Warning: You did not tell me where to find symbol information. I will
assume that the log matches the kernel and modules that are running
right now and I'll use the default options above for symbol resolution.
If the current kernel and/or modules do not match the log, you can get
more accurate output by telling me the kernel version and where to find
map, modules, ksyms etc. ksymoops -h explains the options.
Error (expand_objects): cannot stat(/lib/ext3.o) for ext3
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/jbd.o) for jbd
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/3w-xxxx.o) for 3w-xxxx
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
Warning (map_ksym_to_module): cannot match loaded module ext3 to a unique module
object. Trace may not be reliable.
May 31 04:37:44 lt068_iss1 kernel: <1>Unable to handle kernel paging request at
virtual address 2fc1c0d4
May 31 04:37:44 lt068_iss1 kernel: c01406b0
May 31 04:37:44 lt068_iss1 kernel: *pde = 00000000
May 31 04:37:44 lt068_iss1 kernel: Oops: 0000
May 31 04:37:44 lt068_iss1 kernel: CPU: 0
May 31 04:37:44 lt068_iss1 kernel: EIP: 0010:[<c01406b0>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
May 31 04:37:44 lt068_iss1 kernel: EFLAGS: 00010202
May 31 04:37:44 lt068_iss1 kernel: eax: c1a20100 ebx: c1a2011c ecx: c030ac80
edx: c1a20100
May 31 04:37:44 lt068_iss1 kernel: esi: 0000001e edi: 2fc1c058 ebp: 00000000
esp: c3491f10
May 31 04:37:44 lt068_iss1 kernel: ds: 0018 es: 0018 ss: 0018
May 31 04:37:44 lt068_iss1 kernel: Process kswapd (pid: 4, stackpage=c3491000)
May 31 04:37:44 lt068_iss1 kernel: Stack: f60cc000 00000000 00000000 c3491f5c c1
a20100 c1a2011c 00001ce2 c1a20100
May 31 04:37:44 lt068_iss1 kernel: 00001ce2 c0138aab c1879818 00000000 c0
30ae34 c030adb4 0000003f ffffffff
May 31 04:37:44 lt068_iss1 kernel: 00000009 0000013b 00026827 00000000 00
006697 c030ac80 00000000 00000100
May 31 04:37:44 lt068_iss1 kernel: Call Trace: [<c0138aab>] refill_inactive_zo
ne [kernel] 0x5eb (0xc3491f34))
May 31 04:37:44 lt068_iss1 kernel: [<c0139453>] rebalance_inactive_zone [kernel]
0xd3 (0xc3491f70))
May 31 04:37:44 lt068_iss1 kernel: [<c01394b8>] rebalance_inactive [kernel] 0x48
(0xc3491f90))
May 31 04:37:44 lt068_iss1 kernel: [<c0139601>] do_try_to_free_pages_kswapd [ker
nel] 0x51 (0xc3491fb4))
May 31 04:37:44 lt068_iss1 kernel: [<c0139893>] kswapd [kernel] 0x83 (0xc3491fd4
))
May 31 04:37:44 lt068_iss1 kernel: [<c0105000>] stext [kernel] 0x0 (0xc3491fec))
May 31 04:37:44 lt068_iss1 kernel: [<c010741e>] arch_kernel_thread [kernel] 0x2e
(0xc3491ff0))
May 31 04:37:44 lt068_iss1 kernel: [<c0139810>] kswapd [kernel] 0x0 (0xc3491ff8)
)
May 31 04:37:44 lt068_iss1 kernel: Code: 8b 44 b7 04 85 c0 0f 84 b9 00 00 00 89
c2 8b 0d b0 91 37 c0
>>EIP; c01406b0 <page_referenced+170/2c0> <=====
>>eax; c1a20100 <_end+1668d00/38455c60>
>>ebx; c1a2011c <_end+1668d1c/38455c60>
>>ecx; c030ac80 <contig_page_data+300/a80>
>>edx; c1a20100 <_end+1668d00/38455c60>
>>edi; 2fc1c058 Before first symbol
>>esp; c3491f10 <_end+30dab10/38455c60>
Trace; c0138aab <refill_inactive_zone+5eb/650>
Trace; c0139453 <rebalance_inactive_zone+d3/f0>
Trace; c01394b8 <rebalance_inactive+48/90>
Trace; c0139601 <do_try_to_free_pages_kswapd+51/1d0>
Trace; c0139893 <kswapd+83/1d0>
Trace; c0105000 <_stext+0/0>
Trace; c010741e <arch_kernel_thread+2e/40>
Trace; c0139810 <kswapd+0/1d0>
Code; c01406b0 <page_referenced+170/2c0>
00000000 <_EIP>:
Code; c01406b0 <page_referenced+170/2c0> <=====
0: 8b 44 b7 04 mov 0x4(%edi,%esi,4),%eax <=====
Code; c01406b4 <page_referenced+174/2c0>
4: 85 c0 test %eax,%eax
Code; c01406b6 <page_referenced+176/2c0>
6: 0f 84 b9 00 00 00 je c5 <_EIP+0xc5>
Code; c01406bc <page_referenced+17c/2c0>
c: 89 c2 mov %eax,%edx
Code; c01406be <page_referenced+17e/2c0>
e: 8b 0d b0 91 37 c0 mov 0xc03791b0,%ecx
May 31 11:45:02 lt068_iss1 kernel: 8139too Fast Ethernet driver 0.9.26
Jun 1 06:17:21 lt068_iss1 kernel: Unable to handle kernel paging request at vir
tual address fb303c2c
Jun 1 06:17:21 lt068_iss1 kernel: c022e50e
Jun 1 06:17:21 lt068_iss1 kernel: *pde = 00000000
Jun 1 06:17:21 lt068_iss1 kernel: Oops: 0002
Jun 1 06:17:21 lt068_iss1 kernel: CPU: 0
Jun 1 06:17:21 lt068_iss1 kernel: EIP: 0010:[<c022e50e>] Not tainted
Jun 1 06:17:21 lt068_iss1 kernel: EFLAGS: 00010246
Jun 1 06:17:21 lt068_iss1 kernel: eax: 00000000 ebx: f0f04580 ecx: f4a93c24
edx: 00000000
Jun 1 06:17:21 lt068_iss1 kernel: esi: f4a93c14 edi: f4a93b00 ebp: c3515180
esp: db0adf4c
Jun 1 06:17:21 lt068_iss1 kernel: ds: 0018 es: 0018 ss: 0018
Jun 1 06:17:21 lt068_iss1 kernel: Process savsecls.exe (pid: 15845, stackpage=d
b0ad000)
Jun 1 06:17:21 lt068_iss1 kernel: Stack: f0f04580 c022ef12 f4a93c14 c36901c0 c0
1eb29c f4a93c14 f4a93b00 c01eb879
Jun 1 06:17:21 lt068_iss1 kernel: f4a93c14 f0dadb00 00000000 f0dadb00 c0
144662 f4a93b00 f0dadb00 f0dadb00
Jun 1 06:17:21 lt068_iss1 kernel: ed9ceb80 00000000 bfff7568 c0142c6d f0
dadb00 ed9ceb80 f0dadb00 0808f130
Jun 1 06:17:21 lt068_iss1 kernel: Call Trace: [<c022ef12>] inet_shutdown [ker
nel] 0xc2 (0xdb0adf50))
Jun 1 06:17:21 lt068_iss1 kernel: [<c01eb29c>] sock_release [kernel] 0x5c (0xdb
0adf5c))
Jun 1 06:17:21 lt068_iss1 kernel: [<c01eb879>] sock_close [kernel] 0x39 (0xdb0a
df68))
Jun 1 06:17:21 lt068_iss1 kernel: [<c0144662>] fput [kernel] 0x102 (0xdb0adf7c)
)
Jun 1 06:17:21 lt068_iss1 kernel: [<c0142c6d>] filp_close [kernel] 0x4d (0xdb0a
df98))
Jun 1 06:17:21 lt068_iss1 kernel: [<c0142cee>] sys_close [kernel] 0x4e (0xdb0ad
fb0))
Jun 1 06:17:21 lt068_iss1 kernel: [<c010905f>] system_call [kernel] 0x33 (0xdb0
adfc0))
Jun 1 06:17:21 lt068_iss1 kernel: Code: c7 46 18 00 00 00 00 8b 83 b8 00 00 00
89 54 24 04 89 1c 24
>>EIP; c022e50e <inet_release+3e/70> <=====
>>ebx; f0f04580 <_end+30b4d180/38455c60>
>>ecx; f4a93c24 <_end+346dc824/38455c60>
>>esi; f4a93c14 <_end+346dc814/38455c60>
>>edi; f4a93b00 <_end+346dc700/38455c60>
>>ebp; c3515180 <_end+315dd80/38455c60>
>>esp; db0adf4c <_end+1acf6b4c/38455c60>
Trace; c022ef12 <inet_shutdown+c2/1a0>
Trace; c01eb29c <sock_release+5c/60>
Trace; c01eb879 <sock_close+39/60>
Trace; c0144662 <fput+102/130>
Trace; c0142c6d <filp_close+4d/80>
Trace; c0142cee <sys_close+4e/60>
Trace; c010905f <system_call+33/38>
Code; c022e50e <inet_release+3e/70>
00000000 <_EIP>:
Code; c022e50e <inet_release+3e/70> <=====
0: c7 46 18 00 00 00 00 movl $0x0,0x18(%esi) <=====
Code; c022e515 <inet_release+45/70>
7: 8b 83 b8 00 00 00 mov 0xb8(%ebx),%eax
Code; c022e51b <inet_release+4b/70>
d: 89 54 24 04 mov %edx,0x4(%esp,1)
Code; c022e51f <inet_release+4f/70>
11: 89 1c 24 mov %ebx,(%esp,1)
2 warnings and 5 errors issued. Results may not be reliable.