Yet another performance problem

[sazzct]

Hi,
I am running IDS 9.3 on linux, and am having performance issues. I am hoping someone will be able to help me out.
As I read through all the other threads about tuning, and was about to do some of the suggestions, I thought it might be better to first see if that is truly what I need.
Here is the common outputs that most people ask for:

onstat -p
Informix Dynamic Server Version 9.30.UC1 -- On-Line -- Up 29 days 01:27:15 -- 692340 Kbytes

Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
33808421 34192118 3100599698 98.91 4451893 35656079 9790974 54.53

isamtot open start read write rewrite delete commit rollbk
794683080 8588492 455756957 3650092182 269499156 2121826 45214 2959645 179

gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0

ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 327552.95 4513.70 7531 16768

bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
704258 0 4133475411 0 0 839 279393 491110

ixda-RA idx-RA da-RA RA-pgsused lchwaits
236892 33553 32993422 33263570 28


*********************************************************


onstat -u

Informix Dynamic Server Version 9.30.UC1 -- On-Line -- Up 29 days 01:27:38 -- 692340 Kbytes

Userthreads
address flags sessid user tty wait tout locks nreads nwrites
16b4e018 ---P--D 1 informix - 0 0 0 12045 81209
16b4e618 ---P--F 0 informix - 0 0 0 0 565762
16b4ec18 ---P--F 0 informix - 0 0 0 0 507116
16b4f218 ---P--F 0 informix - 0 0 0 0 0
16b4f818 ---P--F 0 informix - 0 0 0 0 0
16b4fe18 ---P--F 0 informix - 0 0 0 0 0
16b50418 ---P--F 0 informix - 0 0 0 0 0
16b50a18 ---P--F 0 informix - 0 0 0 0 0
16b51018 ---P--F 0 informix - 0 0 0 0 0
16b51618 ---P--F 0 informix - 0 0 0 0 0
16b51c18 ---P--F 0 informix - 0 0 0 0 0
16b52218 ---P--F 0 informix - 0 0 0 0 0
16b52818 ---P--F 0 informix - 0 0 0 0 0
16b52e18 ---P--F 0 informix - 0 0 0 0 0
16b53418 ---P--F 0 informix - 0 0 0 0 0
16b53a18 ---P--F 0 informix - 0 0 0 0 0
16b54018 ---P--F 0 informix - 0 0 0 0 0
16b54618 ---P--F 0 informix - 0 0 0 0 0
16b54c18 ---P--F 0 informix - 0 0 0 0 0
16b55218 ---P--F 0 informix - 0 0 0 0 0
16b55818 ---P--F 0 informix - 0 0 0 0 0
16b55e18 ---P--F 0 informix - 0 0 0 0 0
16b56418 ---P--F 0 informix - 0 0 0 0 0
16b56a18 ---P--F 0 informix - 0 0 0 0 0
16b57018 ---P--F 0 informix - 0 0 0 0 0
16b57618 ---P--F 0 informix - 0 0 0 0 0
16b57c18 ---P--F 0 informix - 0 0 0 0 0
16b58218 ---P--F 0 informix - 0 0 0 0 0
16b58818 ---P--F 0 informix - 0 0 0 0 0
16b58e18 ---P--F 0 informix - 0 0 0 0 0
16b59418 ---P--F 0 informix - 0 0 0 0 0
16b59a18 ---P--F 0 informix - 0 0 0 0 0
16b5a018 ---P--F 0 informix - 0 0 0 0 0
16b5a618 ---P--F 0 informix - 0 0 0 0 0
16b5ac18 ---P--F 0 informix - 0 0 0 0 0
16b5b218 ---P--F 0 informix - 0 0 0 0 0
16b5b818 ---P--F 0 informix - 0 0 0 0 0
16b5be18 ---P--F 0 informix - 0 0 0 0 0
16b5c418 ---P--F 0 informix - 0 0 0 0 0
16b5ca18 ---P--F 0 informix - 0 0 0 0 0
16b5d018 ---P--F 0 informix - 0 0 0 0 0
16b5d618 ---P--F 0 informix - 0 0 0 0 0
16b5dc18 ---P--F 0 informix - 0 0 0 0 0
16b5e218 ---P--F 0 informix - 0 0 0 0 0
16b5e818 ---P--F 0 informix - 0 0 0 0 0
16b5ee18 ---P--F 0 informix - 0 0 0 0 0
16b5f418 ---P--F 0 informix - 0 0 0 0 0
16b5fa18 ---P--F 0 informix - 0 0 0 0 0
16b60018 ---P--F 0 informix - 0 0 0 0 0
16b60618 ---P--F 0 informix - 0 0 0 0 0
16b60c18 ---P--F 0 informix - 0 0 0 0 0
16b61218 ---P--F 0 informix - 0 0 0 0 0
16b61818 ---P--F 0 informix - 0 0 0 0 0
16b61e18 ---P--F 0 informix - 0 0 0 0 0
16b62418 ---P--F 0 informix - 0 0 0 0 0
16b62a18 ---P--F 0 informix - 0 0 0 0 0
16b63018 ---P--F 0 informix - 0 0 0 0 0
16b63618 ---P--F 0 informix - 0 0 0 0 0
16b63c18 ---P--F 0 informix - 0 0 0 0 0
16b64218 ---P--F 0 informix - 0 0 0 0 0
16b64818 ---P--F 0 informix - 0 0 0 0 0
16b64e18 ---P--F 0 informix - 0 0 0 0 0
16b65418 ---P--F 0 informix - 0 0 0 0 0
16b65a18 ---P--F 0 informix - 0 0 0 0 0
16b66018 ---P--F 0 informix - 0 0 0 0 0
16b66618 ---P--- 5 informix - 0 0 0 0 45
16b66c18 ---P--B 6 informix - 0 0 0 134 1020
16b67e18 ---P--D 10 informix - 0 0 0 0 0
16b69c18 Y--P--- 26361 stsdevl - 2f9f11b0 0 1 0 65
16b6f018 Y--P--- 26363 stsdevl - 2e5e2838 0 1 0 24
16b71a18 Y--P--- 26360 stsdevl - 2ef55018 0 1 0 69
16b73218 Y--P--- 26353 stsdevl - 248671b0 0 1 0 43
16b74a18 Y--P--- 26357 stsdevl - 2e699cd8 0 1 2 435
16b78018 Y--P--- 26358 stsdevl - 2eacd1d0 0 1 0 50
16b7b618 Y--P--- 26356 stsdevl - 2e699a08 0 1 1 111
16b7bc18 Y--P--- 26473 stsdevl - 2ed64a88 0 1 0 1
24765418 Y--P--- 26354 stsdevl - 2546d7b8 0 1 0 266
24769c18 Y--P--- 26362 stsdevl - 1ffaecc0 0 1 0 27
2476c018 Y--P--- 26355 stsdevl - 2cee0268 0 1 0 20
2476d218 Y--P--- 26302 stsdevl 4 2cee0918 0 1 0 0
80 active, 256 total, 200 maximum concurrent


*******************************************************



I will be greatly appreciative of any help I can recieve... and I am sure I will repay others in the future.

Thank you

 

[sazzct]

I do have lots extents.... could that be a problem?


And finally, my onconfig file
#**************************************************************************
#
# INFORMIX SOFTWARE, INC.
#
# Title: onconfig.std
# Description: Informix Dynamic Server Configuration Parameters
#
#**************************************************************************

# Root Dbspace Configuration

ROOTNAME rootdbs # Root dbspace name
ROOTPATH /dev/db-chunk1 # Path for device containing root dbspace
ROOTOFFSET 512 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 2048000 # Size of root dbspace (Kbytes)

# Disk Mirroring Configuration Parameters

MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH # Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)

# Physical Log Configuration

PHYSDBS rootdbs # Location (dbspace) of physical log
PHYSFILE 6000 # Physical log file size (Kbytes)

# Logical Log Configuration

LOGFILES 41 # Number of logical log files
LOGSIZE 2000 # Logical log size (Kbytes)

# Diagnostics

MSGPATH /informix/logs/online.log # System message log file path
CONSOLE /informix/logs/console.msg # System console message path
ALARMPROGRAM /informix/etc/log_full.sh # Alarm program path
TBLSPACE_STATS 1 # Maintain tblspace statistics

# System Archive Tape Device

TAPEDEV /dev/tape # Tape device path
TAPEBLK 16 # Tape block size (Kbytes)
TAPESIZE 100000000 # Maximum amount of data to put on tape (Kbytes)

# Log Archive Tape Device

LTAPEDEV /dev/null # Log tape device path
LTAPEBLK 16 # Log tape block size (Kbytes)
LTAPESIZE 10240 # Max amount of data to put on log tape (Kbytes)

# Optical

STAGEBLOB # Informix Dynamic Server staging area

# System Configuration

SERVERNUM 1 # Unique id corresponding to a OnLine instance
DBSERVERNAME actionplus_db # Name of default database server
DBSERVERALIASES actionplus_db # List of alternate dbservernames
DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env.
RESIDENT 0 # Forced residency flag (Yes = 1, No = 0)

MULTIPROCESSOR 0 # 0 for single-processor, 1 for multi-processor
NUMCPUVPS 1 # Number of user (cpu) vps
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one

NOAGE 0 # Process aging
AFF_SPROC 0 # Affinity start processor
AFF_NPROCS 0 # Affinity number of processors

# Shared Memory Parameters

LOCKS 20000 # Maximum number of locks
BUFFERS 50000 # Maximum number of shared buffers
NUMAIOVPS 1 # Number of IO vps
PHYSBUFF 32 # Physical log buffer size (Kbytes)
LOGBUFF 32 # Logical log buffer size (Kbytes)
CLEANERS 64 # Number of buffer cleaner processes
SHMBASE 0x10000000 # Shared memory base address
SHMVIRTSIZE 256000 # initial virtual shared memory segment size
SHMADD 32768 # Size of new shared memory segments (Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 300 # Check point interval (in sec)
LRUS 8 # Number of LRU queues
LRU_MAX_DIRTY 60 # LRU percent dirty begin cleaning limit
LRU_MIN_DIRTY 50 # LRU percent dirty end cleaning limit
TXTIMEOUT 0x12c # Transaction timeout (in sec)
STACKSIZE 32 # Stack size (Kbytes)

# System Page Size
# BUFFSIZE - OnLine no longer supports this configuration parameter.
# To determine the page size used by OnLine on your platform
# see the last line of output from the command, 'onstat -b'.


# Recovery Variables
# OFF_RECVRY_THREADS:
# Number of parallel worker threads during fast recovery or an offline restore.
# ON_RECVRY_THREADS:
# Number of parallel worker threads during an online restore.

OFF_RECVRY_THREADS 10 # Default number of offline worker threads
ON_RECVRY_THREADS 1 # Default number of online worker threads

# Data Replication Variables
DRINTERVAL 30 # DR max time between DR buffer flushes (in sec)
DRTIMEOUT 30 # DR network timeout (in sec)
DRLOSTFOUND /informix/etc/dr.lostfound # DR lost+found file path

# CDR Variables
CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional)
CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds)
CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kbytes)
CDR_NIFCOMPRESS 0 # Link level compression (-1 never, 0 none, 9 max)
CDR_SERIAL 0,0 # Serial Column Sequence
CDR_DBSPACE # dbspace for syscdr database
CDR_QHDR_DBSPACE # CDR queue dbspace (default same as catalog)
CDR_QDATA_SBSPACE # CDR queue smart blob space
CDR_QDATA_SBFLAGS 0 # Log/no-log (default no log)


# Backup/Restore variables
BAR_ACT_LOG /informix/logs/bar_act.log
# ON-Bar Log file - not in /tmp please
BAR_DEBUG_LOG /informix/logs/bar_dbug.log
# ON-Bar Debug Log - not in /tmp please
BAR_MAX_BACKUP 0
BAR_RETRY 1
BAR_NB_XPORT_COUNT 10
BAR_XFER_BUF_SIZE 31
RESTARTABLE_RESTORE on
BAR_PROGRESS_FREQ 0

# Informix Storage Manager variables
ISM_DATA_POOL ISMData
ISM_LOG_POOL ISMLogs

# Read Ahead Variables
RA_PAGES 64 # Number of pages to attempt to read ahead
RA_THRESHOLD 8 # Number of pages left before next group

# DBSPACETEMP:
# OnLine equivalent of DBTEMP for SE. This is the list of dbspaces
# that the OnLine SQL Engine will use to create temp tables etc.
# If specified it must be a colon separated list of dbspaces that exist
# when the OnLine system is brought online. If not specified, or if
# all dbspaces specified are invalid, various ad hoc queries will create
# temporary files in /tmp instead.

DBSPACETEMP temp1db # Default temp dbspaces

# DUMP*:
# The following parameters control the type of diagnostics information which
# is preserved when an unanticipated error condition (assertion failure) occurs
# during OnLine operations.
# For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No.

DUMPDIR /tmp # Preserve diagnostics in this directory
DUMPSHMEM 1 # Dump a copy of shared memory
DUMPGCORE 0 # Dump a core image using 'gcore'
DUMPCORE 0 # Dump a core image (Warning:this aborts OnLine)
DUMPCNT 1 # Number of shared memory or gcore dumps for
# a single user's session

FILLFACTOR 90 # Fill factor for building indexes

# method for OnLine to use when determining current time
USEOSTIME 0 # 0: use internal time(fast), 1: get time from OS(slow)

# Parallel Database Queries (pdq)
MAX_PDQPRIORITY 100 # Maximum allowed pdqpriority
DS_MAX_QUERIES 10 # Maximum number of decision support queries
DS_TOTAL_MEMORY 50000 # Decision support memory (Kbytes)
DS_MAX_SCANS 100 # Maximum number of decision support scans
DATASKIP off # List of dbspaces to skip

# OPTCOMPIND
# 0 => Nested loop joins will be preferred (where
# possible) over sortmerge joins and hash joins.
# 1 => If the transaction isolation mode is not
# "repeatable read", optimizer behaves as in (2)
# below. Otherwise it behaves as in (0) above.
# 2 => Use costs regardless of the transaction isolation
# mode. Nested loop joins are not necessarily
# preferred. Optimizer bases its decision purely
# on costs.
OPTCOMPIND 2 # To hint the optimizer

DIRECTIVES 1 # Optimizer DIRECTIVES ON (1/Default) or OFF (0)

ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 = ABORT, 2 = WAIT
OPCACHEMAX 0 # Maximum optical cache size (Kbytes)

# HETERO_COMMIT (Gateway participation in distributed transactions)
# 1 => Heterogeneous Commit is enabled
# 0 (or any other value) => Heterogeneous Commit is disabled
HETERO_COMMIT 0

SBSPACENAME sblobdb # Default smartblob space name - this is where blobs
# go if no sbspace is specified when the smartblob is
# created. It is also used by some datablades as
# the location to put their smartblobs.
SYSSBSPACENAME sblobdb # Default smartblob space for use by the Informix
# Server. This is used primarily for Informix Server
# system statistics collection.

BLOCKTIMEOUT 3600 # Default timeout for system block
SYSALARMPROGRAM /informix/etc/evidence.sh # System Alarm program path

# Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS
OPT_GOAL -1

ALLOW_NEWLINE 0 # embedded newlines(Yes = 1, No = 0 or anything but 1)

#
# The following are default settings for enabling Java in the database.
# Replace all occurrences of /usr/informix with the value of $INFORMIXDIR.

#VPCLASS jvp,num=1 # Number of JVPs to start with

JVPJAVAHOME /informix/extend/krakatoa/jre # JRE installation root directory
JVPHOME /informix/extend/krakatoa # Krakatoa installation directory

JVPPROPFILE /informix/extend/krakatoa/.jvpprops # JVP property file

JDKVERSION 1.3 # JDK version supported by this server

# The path to the JRE libraries relative to JVPJAVAHOME
JVPJAVALIB /lib/i386/

# The JRE libraries to use for the Java VM

JVPJAVAVM hpi:server:verify:java:net:zip:jpeg

# use JVPARGS to change Java VM configuration
#To display jni call
#JVPARGS -verbose:jni

# Classpath to use upon Java VM start-up (use _g version for debugging)

#JVPCLASSPATH /informix/extend/krakatoa/krakatoa_g.jar:/usr/informix/extend/krakatoa/jdbc_g.jar
JVPCLASSPATH /informix/extend/krakatoa/krakatoa.jar:/usr/informix/extend/krakatoa/jdbc.jar

********************************************************

 

[mberni]

lots of extents could really be a point. afaik informix recommends not more than 2 (two) extents per table. so you should maybe read about how to use storage parameters with "create table" to reduce the amount of extents. you could unload these tables, drop and recreate them (with better storage params) and reload the data again to optimize your extents.

maybe it could be useful to place some hot tables (i.e. tables with a lot of traffic) on separate disks, or place the indices on different disks than the data. to do this create some tablespaces on different disks (or mirrorsets) and specify the appropriate tablespace when creating the table/index.

some hints concerning your onconfig-file:

you have lots of logical logs (41) of small size (2000). Better increase the size of your logs so your logs will not become full too fast. i bet you are having a lot of full logs/hour. have a look at $INFORMIXHOME/online.log maybe the number of checkpoints (i.e. flushing the logs to disk) reduces your performance. i would increase the size to 16384 at a test. hope your logical logs reside on a separate dataspace (and disk), if not, move them.

how much physical memory do you have? your shmvirtsize is 256k but your shmadd only 32k. when you really allocate more than 256k of shared mem, increase your shmadd to lets say 128k, so you get fewer memory segments (you should have enough physmem to do this )

cleaners is set to 64. i bet you should go with 8 when you increase your logical-logsize.

you have "only" 50000 buffers, but your onstat tells of lots of bufwaits. this can be a result of too few buffers or of flushing them too slow (checkpoints and logsize) or both. try to increase them in steps of 10000 but always have a look at your free physmem (system command "top&quot.

IMPORTANT : do not make too many changes at the same time. always change one setting and then monitor the effect before taking the next step.

Last thing: most of performance tuning happens in the application not in the database-server. your stats tells of 491110 sequential scans, i.e. amount of reads against a full table without the use of an index. this kind of behaviour results in poor performance and can only be significantly improved by tuning the approriate statements.

hope this will get you some work for a while )

 

[sazzct]

Thank you so much... first thing I was going to try was the extents.... but now another problem.
The dbexport utility gives me this error:
Cannot make export directory,
and Yes, I am running as user informix and have tried all the switches.... it creates the <dbname>.exp directory fine, then just dies....
any more ideas?

 

[mberni]

maybe a silly question ...
but is there enough discspace available?

 

[olded]

sazzct:

Concerning your dbexport problem. If the export directory already exists when you execute dbexport you will get the &quot;export directory exists&quot; error and the dbexport fails.

Try removing the export directory first, and then run dbexport.

Regards,

Ed