IDS 7.31 crashing

[irishmc]

Environment:

IBM 7026-6M1
2-way
8GB

Informix 7.31.UC2X3 - Read Only - 4MB
Installed 05/02

Also on system:

Oracle 8.1.7 database - 6 instances running
Oracle 9I Apps


System was running fine (since 05/02) with Informix and Oracle until 2 weeks ago. Informix now crashes on it's own.

Brought system online with no db/apps running. Start Informix engine. Fails between 30 secs and 30 minutes.

Recent modification to Oracle possibly touched JRE

Log Info:

Mon Nov 25 15:52:48 2002

15:52:48 Event alarms enabled. ALARMPROG = '/usr/informix7.2/etc/no_log.sh'
15:52:52 DR: DRAUTO is 0 (Off)
15:52:52 AIX MP latch code enabled
15:52:52 Requested shared memory segment size rounded from 732KB to 736KB
15:52:52 Informix Dynamic Server Version 7.31.UC2X3 15:52:52 Informix Dynamic Server Initialized -- Shared Memory Initialized.
15:52:52 Physical Recovery Started.
15:52:52 Physical Recovery Complete: 0 Pages Restored.
15:52:52 Logical Recovery Started.
15:52:55 Logical Recovery Complete.
0 Committed, 0 Rolled Back, 0 Open, 0 Bad Locks

15:52:57 Dataskip is now OFF for all dbspaces
15:52:57 On-Line Mode
15:52:57 Checkpoint Completed: duration was 1 seconds.
16:00:00 Assert Failed: shmpoll.c, line 231, thread 7, errno=36, semop error.
16:00:00 Informix Dynamic Server Version 7.31.UC2X3
16:00:00 Who: Session(2, informix@, 0, 0)
Thread(7, sm_poll, 0, 1)
File: neterrb.c Line: 658
16:00:00 Assert Failed: semop: errno = 36

16:00:00 Informix Dynamic Server Version 7.31.UC2X3
16:00:00 Who: Session(0, @, 0, 0)
Thread(0, idle, 0, 3)
File: mt.c Line: 1223
16:00:00 Assert Failed: semop: errno = 36

16:00:00 Informix Dynamic Server Version 7.31.UC2X3
16:00:00 Who: Session(0, @, 0, 0)
Thread(0, idle, 0, 4)
File: mt.c Line: 1223
16:00:00 Assert Failed: semop: errno = 36

16:00:00 Informix Dynamic Server Version 7.31.UC2X3
16:00:00 Who: Session(0, @, 0, 0)
Thread(0, idle, 0, 6)
File: mt.c Line: 1223
16:00:00 Assert Failed: semop: errno = 36

16:00:00 Informix Dynamic Server Version 7.31.UC2X3
16:00:00 Who: Session(0, @, 0, 0)
Thread(0, idle, 0, 5)
File: mt.c Line: 1223
16:00:01 Assert Failed: shmpoll.c, line 353, thread 0, errno=22, semop error.
16:00:01 Informix Dynamic Server Version 7.31.UC2X3
16:00:01 Who: Session(0, @, 0, 0)
Thread(0, noname, 0, 0)
File: neterrb.c Line: 658
16:00:05 See Also: /tmp/af.3ab70
16:00:05 mt.c, line 1223, thread 0, proc id 131618, semop: errno = 36
16:00:05 PANIC: Attempting to bring system down

Informix Tech Support has all af.* files and log info. No idea yet.

Any help is greatly appreciated!!

 

[vpshriyan]

Hi,

The error log, in all probabilities, points to an out of shared memory and semaphores resources case. In this context I would like to draw your attention to the machine specific release notes related to the Informix database server located at $INFORMIXDIR/release/en_us/0333/IDS_7.3. You take note of the parameters suggested under the title "Informix Dynamic Server Shared Memory Parameters and Kernel Parameters" and may compare the actuals set at your server.

We have HP hardware and HP-UX 10.2 OS configured according to the following semaphore and shared memory paramerters for IDS 7.3. I am not an OS expert so I may not be in position to suggest the equivalent parameters for IBM-AIX. You can refer to your OS Admin for more information related to this field.

Semaphore Related Parameters
maximum value for semaphores(semaem)= 16384
Semaphore map(semmap)= 522
number of semaphore identifiers(semmni) = 4096
total number of semaphores in the system(semmns) = 4096
number of semaphore undo structures(semmnu) = 2048 to 4096
semaphore undo enteries per process(semume) = 4096
semaphore maximum value(semvmx) = 32767

Shared Memory Related Parameters
maximum shared memory segment size in bytes(shmmax) = 805306367 (could be 1GB)
minimum shared memory segment size in bytes(shmmin) = 1
maximum shared memory segments in system (shmmni) = 512
maximum shared memory segments per process(shmseg) = 256

Regards
Shriyan

 

[irishmc]

Shriyan

Thank you for your prompt response.

AIX manages memory different other UNIX vendors. The options you refer to are not tuneable within AIX. Below is the info from the IDS_7.3 file:

INFORMIX Dynamic Server Shared Memory and Kernel Parameters:
============================================================

SHMMAX: 268435456 (256 MB)
SHMMIN: 1
SHMALL: Not Applicable
SHMMNI: 4096
SHMSEG: Not Applicable
SHMBRK: Not Applicable
MAXMEM: Not Applicable
SEMMNI: Not Applicable
SEMMSL: Not Applicable
NFILES: Not Applicable

The Semaphore options don't exist.

I'm hoping someone has run across a similar scenario with Oracle and Informix and can shed some light.

Thanks again,
Patrick

 

[FCVV]

Hi,

semop: errno = 36

File /usr/include/errno.h
#define EIDRM 36 /* Identifier removed */

Try (work arounds):

1- Avoid Dynamic SHM allocation.
2- Re-brand all Informix Files (re-install it), just run the installserver script. Oracle Upgrade could touched several Shared Libs.
3- Make it Resident (firts check if it's supported).
4- Upgrade to 7.31 UC6.

Best Regards,
Francisco Virgilio.

 

[irishmc]

Francisco

Thanks for the reply.

After some discussion, we've decided to re-install the Informix files. I'll let you know how it goes.

Thanks
Patrick