dbdump taking very long time after fiber upgrade

[Guest_imported]

I have a strange issue. I am running informix Dynamic Server Version 7.30.UC5 on a solaris 7, E3500 box with 4 processors and 2 gig memory. Each night I perform a dbdump and it used to take about 3 hours. The box was connected to a Hitachi subsystem via a jni 1 gig fiber card. (direct) We have added a San solution with brocade 3500 switches. I changed the 1 gig jni card to a 2 gig jni card. I also installed Hitachi's Dynamic Link Manager 2.0. (I now have 2 links to the san)

My Problem is that the dbdump now takes about 7 hours. (up from 3!) It looks like it is taking all the extra time on just one of the tables. I did a dump/restore to make sure the tables were clean..ect.. no change. I upgraded the jni drives to 5.1.1, no change. Any ideas????

Here is the onstat -p, onstat -u and onconfig ----

onstat - p
Informix Dynamic Server Version 7.30.UC5 -- On-Line -- Up 3 days 17:40:43 -- 586752 Kbytes

Profile
dskreads pagreads bufreads %cached dskwrits pagwrits bufwrits %cached
11418698 1300816 45264310 74.77 120549 66715 183881 34.44

isamtot open start read write rewrite delete commit rollbk
59274109 1895995 4369954 11482426 36419 63 322 12622 0

gp_read gp_write gp_rewrt gp_del gp_alloc gp_free gp_curs
0 0 0 0 0 0 0

ovlock ovuserthread ovbuff usercpu syscpu numckpts flushes
0 0 0 6177.05 2488.40 241 2156

bufwaits lokwaits lockreqs deadlks dltouts ckpwaits compress seqscans
12650 1 56746802 0 0 11 2944 60105

ixda-RA idx-RA da-RA RA-pgsused lchwaits
16863 19 1216193 1232800 392

onstat -u
Informix Dynamic Server Version 7.30.UC5 -- On-Line -- Up 3 days 17:42:00 -- 586752 Kbytes

Userthreads
address flags sessid user tty wait tout locks nreads nwrites
26040018 ---P--D 1 informix - 0 0 0 215 1811
260404cc ---P--F 0 informix - 0 0 0 0 17712
26040980 ---P--F 0 informix - 0 0 0 0 8156
26040e34 ---P--F 0 informix - 0 0 0 0 3658
260412e8 ---P--F 0 informix - 0 0 0 0 64
2604179c ---P--- 9 informix - 0 0 0 0 1
26041c50 ---P--B 10 informix - 0 0 0 0 0
26042a6c ---P--D 13 informix - 0 0 0 66268 0
26042f20 Y--P--- 791 pasdbusr - 2631c680 0 1 0 0
260454c0 Y--P--- 796 pasdbusr - 2614a6c8 0 1 0 0
26045e28 Y--P--- 803 pasdbusr - 26196208 0 1 8 0
26046c44 Y--P--- 797 pasdbusr - 262086c8 0 1 0 0
260470f8 Y--P--- 794 pasdbusr - 261aef18 0 2 777 65
260475ac Y--P--- 801 pasdbusr - 261e9150 0 1 0 0
26047f14 Y--P--- 789 pasdbusr - 267f5b38 0 2 912 65
260483c8 Y--P--- 802 pasdbusr - 2614e208 0 1 678 0
26048d30 Y--P--- 792 pasdbusr - 2621b198 0 1 338 0
2604a000 Y--P--- 799 pasdbusr - 261cf420 0 2 898 65
18 active, 128 total, 38 maximum concurrent

onconfig -
#**************************************************************************
#
# INFORMIX SOFTWARE, INC.
#
# Title: onconfig.std
# Description: Informix Dynamic Server Configuration Parameters
#
#**************************************************************************

# Root Dbspace Configuration

ROOTNAME rootdbs # Root dbspace name
ROOTPATH /dev/online_rootdbs # Path for device containing root dbspace
ROOTOFFSET 1536 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 2095614 # Size of root dbspace (Kbytes)

# Disk Mirroring Configuration Parameters

MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH # Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)

# Physical Log Configuration

PHYSDBS rootdbs # Location (dbspace) of physical log
PHYSFILE 5000 # Physical log file size (Kbytes)

# Logical Log Configuration

LOGFILES 100 # Number of logical log files
LOGSIZE 4000 # Logical log size (Kbytes)

# Diagnostics

MSGPATH /pas/principia/tmp/online.log # System message log file path
CONSOLE /dev/console # System console message path
ALARMPROGRAM /pas/principia/informix-7.30.uc5/etc/log_full.sh
# Alarm program path
SYSALARMPROGRAM /pas/principia/informix-7.30.uc5/etc/evidence.sh
# System Alarm program path
TBLSPACE_STATS 1

# System Archive Tape Device
TAPEDEV /dev/null # Tape device path
TAPEBLK 16 # Tape block size (Kbytes)

# Log Archive Tape Device

LTAPEDEV /dev/null # Log tape device path
LTAPEBLK 16 # Log tape block size (Kbytes)
LTAPESIZE 10240 # Max amount of data to put on log tape (Kbytes)
# Optical

STAGEBLOB # Informix Dynamic Server/Optical staging area

# System Configuration

SERVERNUM 0 # Unique id corresponding to a Dynamic Server instance
DBSERVERNAME newsun # Name of default database server
DBSERVERALIASES # List of alternate dbservernames
DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env.
RESIDENT 1 # Forced residency flag (Yes = 1, No = 0)

MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor
NUMCPUVPS 3 # Number of user (cpu) vps
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one

NOAGE 0 # Process aging
AFF_SPROC 1 # Affinity start processor
AFF_NPROCS 3 # Affinity number of processors

# Shared Memory Parameters

LOCKS 500000 # Maximum number of locks
BUFFERS 200000 # Maximum number of shared buffers
NUMAIOVPS 2 # Number of IO vps
PHYSBUFF 32 # Physical log buffer size (Kbytes)
LOGBUFF 32 # Logical log buffer size (Kbytes)
LOGSMAX 300 # Maximum number of logical log files
CLEANERS 4 # Number of buffer cleaner processes
SHMBASE 0xa000000 # Shared memory base address
SHMVIRTSIZE 128000 # initial virtual shared memory segment size
SHMADD 8192 # Size of new shared memory segments (Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 300 # Check point interval (in sec)
LRUS 8 # Number of LRU queues
LRU_MAX_DIRTY 40 # LRU percent dirty begin cleaning limit
LRU_MIN_DIRTY 30 # LRU percent dirty end cleaning limit
LTXHWM 50 # Long transaction high water mark percentage
LTXEHWM 60 # Long transaction high water mark (exclusive)
TXTIMEOUT 0x12c # Transaction timeout (in sec)
STACKSIZE 32 # Stack size (Kbytes)

# System Page Size
# BUFFSIZE - Dynamic Server no longer supports this configuration parameter.
# To determine the page size used by Dynamic Server on your platform
# see the last line of output from the command, 'onstat -b'.


# Recovery Variables
# OFF_RECVRY_THREADS:
# Number of parallel worker threads during fast recovery or an offline restore.
# ON_RECVRY_THREADS:
# Number of parallel worker threads during an online restore.

OFF_RECVRY_THREADS 10 # Default number of offline worker threads
ON_RECVRY_THREADS 1 # Default number of online worker threads

# Data Replication Variables
# DRAUTO: 0 manual, 1 retain type, 2 reverse type
DRAUTO 0 # DR automatic switchover
DRINTERVAL 30 # DR max time between DR buffer flushes (in sec)DRTIMEOUT 30 # DR network timeout (in sec)
DRLOSTFOUND /pas/principia/informix-7.30.uc5/etc/dr.lostfound
# DR lost+found file path

# CDR Variables
CDR_LOGBUFFERS 2048 # size of log reading buffer pool (Kbytes)
CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional)
CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds)
CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kbytes)

# Backup/Restore variables
BAR_ACT_LOG /tmp/bar_act.log
BAR_MAX_BACKUP 0
BAR_RETRY 1
BAR_NB_XPORT_COUNT 10
BAR_XFER_BUF_SIZE 31

# Informix Storage Manager variables
ISM_DATA_POOL ISMData # If the data pool name is changed, be sure to
# update $INFORMIXDIR/bin/onbar. Change to
# ism_catalog -create_bootstrap -pool <new name>ISM_LOG_POOL ISMLogs
# Read Ahead Variables
RA_PAGES 10 # Number of pages to attempt to read ahead
RA_THRESHOLD 6 # Number of pages left before next group

# DBSPACETEMP:
# Dynamic Server equivalent of DBTEMP for SE. This is the list of dbspaces
# that the Dynamic Server SQL Engine will use to create temp tables etc.
# If specified it must be a colon separated list of dbspaces that exist
# when the Dynamic Server system is brought online. If not specified, or if
# all dbspaces specified are invalid, various ad hoc queries will create
# temporary files in /tmp instead.

DBSPACETEMP # Default temp dbspaces

# DUMP*:
# The following parameters control the type of diagnostics information which
# is preserved when an unanticipated error condition (assertion failure) occurs
# during Dynamic Server operations.
# For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No.

DUMPDIR /tmp # Preserve diagnostics in this directory
DUMPSHMEM 1 # Dump a copy of shared memory
DUMPGCORE 0 # Dump a core image using 'gcore'
DUMPCORE 0 # Dump a core image (Warning:this aborts Dynamic Server)
DUMPCNT 1 # Number of shared memory or gcore dumps for
# a single user's session

FILLFACTOR 90 # Fill factor for building indexes

# method for Dynamic Server to use when determining current time
USEOSTIME 1 # 0: use internal time(fast), 1: get time from OS(slow)

# Parallel Database Queries (pdq)
MAX_PDQPRIORITY 100 # Maximum allowed pdqpriority
DS_MAX_QUERIES # Maximum number of decision support queries
DS_TOTAL_MEMORY # Decision support memory (Kbytes)
DS_MAX_SCANS 1048576 # Maximum number of decision support scans DATASKIP off # List of dbspaces to skip

# OPTCOMPIND
# 0 => Nested loop joins will be preferred (where
# possible) over sortmerge joins and hash joins.
# 1 => If the transaction isolation mode is not
# &quot;repeatable read&quot;, optimizer behaves as in (2)
# below. Otherwise it behaves as in (0) above.
# 2 => Use costs regardless of the transaction isolation
# mode. Nested loop joins are not necessarily
# preferred. Optimizer bases its decision purely
# on costs.
OPTCOMPIND 2 # To hint the optimizer

ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 = ABORT, 2 = WAIT
LBU_PRESERVE 0 # Preserve last log for log backup
OPCACHEMAX 0 # Maximum optical cache size (Kbytes)

# HETERO_COMMIT (Gateway participation in distributed transactions)
# 1 => Heterogeneous Commit is enabled
# 0 (or any other value) => Heterogeneous Commit is disabled
HETERO_COMMIT 0

# Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS
OPT_GOAL -1

# Optimizer DIRECTIVES ON (1/Default) or OFF (0)
DIRECTIVES 1

# Status of restartable restore
RESTARTABLE_RESTORE off
NETTYPE tlitcp,1,50,CPU # Configure poll thread(s) for nettype
NETTYPE ipcshm,1,10,NET # Configure poll thread(s) for nettype

 

[roelwe]

First of al: How do you do your dbdump?
You should check I/O queues. Is your kernel I/O on?
check onstat -g ioq and onstat -g iof. to see what thread is performing the i/o.

I checked your onconfig parameters:

Since you have 3 CPU VPs, you don't need a NETTYPE on a NET VP.
I would set it to
NETTYPE TLITCP,2,25,CPU
NETTYPE IPCSHM,1,10,CPU

You didn't set you DBSPACETEMP variable. The temporary dbspace is the heaviest dbspace. Set the default dbspace to improve your performance big time

For savety I would recommend you to set LBU_PRESERVE to 1.