Database locked up and crashed during a pretty light load. I restarted into quiescent mode and then brought it back online a few minutes later and it crashed again almost immediately. Its back up now, though I cant really take any credit for doing anything to fix it, which frankly gives me the screaming heebie-jeebies.
I have included the salient portion of the log below. Anyone have any thoughts about what happened here? IBM's apar database lists LOTS of symptoms that look similar but nothing totally dead on is jumping out at me:
# first failure
18:31:32 Assert Failed: No Exception Handler
18:31:32 Informix Dynamic Server 2000 Version 9.21.FC7X3
18:31:32 Who: Session(3605, informix@schulerk1, 2272, 969740904)
Thread(122312, sqlexec, 70000003b998740, 4)
File: mtex.c Line: 422
18:31:32 Results: Exception Caught. Type: MT_EX_OS, Context: mem
18:31:32 Action: Please notify Informix Technical Support.
18:31:32 stack trace for pid 14972 written to /dbwork/informix_dumpdir/af.e1b02
e54
18:31:32 See Also: /dbwork/informix_dumpdir/af.e1b02e54, shmem.e1b02e54.0
18:35:05 mtex.c, line 422, thread 122312, proc id 14972, No Exception Handler.
18:35:06 Fatal error in ADM VP at mt.c:11498
18:35:06 Unexpected virtual processor termination, pid = 14972, exit = 0x100
18:35:06 PANIC: Attempting to bring system down
19:09:23 Informix Dynamic Server 2000 Started.
# after start with onint -ky and then brought online failed again
19:11:00 Begin recreating indexes deferred during recovery.
19:11:00 On-Line Mode
19:11:00 Assert Failed: No Exception Handler
19:11:00 Informix Dynamic Server 2000 Version 9.21.FC7X3
19:11:00 Who: Session(1, informix@scout, 0, 807153704)
Thread(7, main_loop(), 700000030183028, 4)
File: mtex.c Line: 422
19:11:00 Results: Exception Caught. Type: MT_EX_OS, Context: mem
19:11:00 Action: Please notify Informix Technical Support.
19:11:00 stack trace for pid 97090 written to /dbwork/informix_dumpdir/af.3ef37
93
19:11:00 See Also: /dbwork/informix_dumpdir/af.3ef3793, shmem.3ef3793.0
19:11:00 Recreating index: 'tool:"informix".subroload-create_files'
19:11:25 Booting Language <spl> from module <>
19:11:25 Loading Module <SPLNULL>
19:11:56 Completed recreating indexes.
19:12:16 mtex.c, line 422, thread 7, proc id 97090, No Exception Handler.
19:12:16 Fatal error in ADM VP at mt.c:11498
19:12:16 Unexpected virtual processor termination, pid = 97090, exit = 0x100
19:12:16 PANIC: Attempting to bring system down
I have included the salient portion of the log below. Anyone have any thoughts about what happened here? IBM's apar database lists LOTS of symptoms that look similar but nothing totally dead on is jumping out at me:
# first failure
18:31:32 Assert Failed: No Exception Handler
18:31:32 Informix Dynamic Server 2000 Version 9.21.FC7X3
18:31:32 Who: Session(3605, informix@schulerk1, 2272, 969740904)
Thread(122312, sqlexec, 70000003b998740, 4)
File: mtex.c Line: 422
18:31:32 Results: Exception Caught. Type: MT_EX_OS, Context: mem
18:31:32 Action: Please notify Informix Technical Support.
18:31:32 stack trace for pid 14972 written to /dbwork/informix_dumpdir/af.e1b02
e54
18:31:32 See Also: /dbwork/informix_dumpdir/af.e1b02e54, shmem.e1b02e54.0
18:35:05 mtex.c, line 422, thread 122312, proc id 14972, No Exception Handler.
18:35:06 Fatal error in ADM VP at mt.c:11498
18:35:06 Unexpected virtual processor termination, pid = 14972, exit = 0x100
18:35:06 PANIC: Attempting to bring system down
19:09:23 Informix Dynamic Server 2000 Started.
# after start with onint -ky and then brought online failed again
19:11:00 Begin recreating indexes deferred during recovery.
19:11:00 On-Line Mode
19:11:00 Assert Failed: No Exception Handler
19:11:00 Informix Dynamic Server 2000 Version 9.21.FC7X3
19:11:00 Who: Session(1, informix@scout, 0, 807153704)
Thread(7, main_loop(), 700000030183028, 4)
File: mtex.c Line: 422
19:11:00 Results: Exception Caught. Type: MT_EX_OS, Context: mem
19:11:00 Action: Please notify Informix Technical Support.
19:11:00 stack trace for pid 97090 written to /dbwork/informix_dumpdir/af.3ef37
93
19:11:00 See Also: /dbwork/informix_dumpdir/af.3ef3793, shmem.3ef3793.0
19:11:00 Recreating index: 'tool:"informix".subroload-create_files'
19:11:25 Booting Language <spl> from module <>
19:11:25 Loading Module <SPLNULL>
19:11:56 Completed recreating indexes.
19:12:16 mtex.c, line 422, thread 7, proc id 97090, No Exception Handler.
19:12:16 Fatal error in ADM VP at mt.c:11498
19:12:16 Unexpected virtual processor termination, pid = 97090, exit = 0x100
19:12:16 PANIC: Attempting to bring system down