Concurrent IO performance.

[jprabaker]

Hi,

I'm having performance issues after implementing CIO on my Oracle 10g database filesystems. After the initial switch over the users complained of jobs taking twice as long as they did before the change. We've tried increasing the SVA size (from 400M to 2G), pinning the SGA memory, using large memory pages all to little avail.

Is there something glaringly obvoius that we are missing?

 

[kHz]

CIO for redologs? archivelogs? datafiles? binaries?

Are you using Enhanced JFS or JFS?

If you are using Large Pages there are a couple of extra settings that need to be made.

What are you using for numfsbuffs? (vmstat -s) and (vmstat -v)

What is numclust?

What size is the agblksize for redologs?

Do you have any numbers to prove it is twice as slow?

 

[jprabaker]

Hi thanks for the reply.

We are using CIO for all but the oracle install filesystem. numfsbuffs, numclust & agblksize are all default. All filesystems are jfs2.

We have a daily forcast which runs on the box for over an hour, this is what was taking twice as long.

We've had to revert back to normal jfs2 filesystems at the request of the users. I'm in the process of building a new install on an isolated machine to try and get things sorted so any recommendations/suggests are welcome!


Thanks,
john

 

[kHz]

Have you ran an iostat or lvmstat at the time? Or better, run nmon for that hour and see what info you get.

I run agblksize of 512 for redologs and numclust=8. I also upped the numfsbuf due to the output of vmstat -v. I run CIO on JFS2 except for the filesystem with Oracle binaries. I haven't seen any perf degredation.

Again, run nmon during that hour and see what you get back. And run it with CIO mounted. nmon doesn't cause a perf hit like trace.

You might also want to look at vmstat -s to see if you are paging to paging space and/or to the filesystem.

 

[p5wizard]

kHz,

numfsbufs is only for JFS, for JFS2 it is the parameter j2_nBufferPerPagerDevice - this defaults to 512 which is probably sufficient.


HTH,

p5wizard

 

[jprabaker]

Hi p5, do you bother with j2_dynamicBufferPreallocation?

Thanks,
John

 

[p5wizard]

I don't think our JFS2 filesystems are pushed to that extreme that it would need tweaking j2_dynamicBufferPreallocation. But I'l keep it in memory.

j2_dynamicBufferPreallocation
Purpose:
Specifies the number of 16k slabs to preallocate when
the filesystem is running low of bufstructs.
Values:
Default: 16 (256k worth)
Range: 0 to 256
Type: Dynamic
Diagnosis:
N/A
Tuning:
The value is in 16k slabs, per filesystem. The filesystem
does not need remounting. The bufstructs for Enhanced JFS
are now dynamic; the number of buffers that start on the
paging device is controlled by j2_nBufferPerPagerDevice,
but buffers are allocated and destroyed dynamically past
this initial value. If the value of external pager
filesystem I/Os blocked with no fsbuf (from vmstat -v)
increases, the j2_dynamicBufferPreallocation should be
increased for that filesystem, as the I/O load on the
filesystem could be exceeding the speed of preallocation.
A value of 0 (zero) disables dynamic buffer allocation
completely.


HTH,

p5wizard

 

[kHz]

You actually need to tune them based on the output of vmstat. And some of the defaults are not sufficient. And I realize it is for JFS and JFS2 are preceded with j2. And I have increased because of blocked buffers on both JFS and JFS2 because they were not sufficient.

 

[kHz]

These are my AIX 5.3 for Oracle settings:
vmo -p -o lru_file_repage=0
vmo -p -o maxclient%=80
vmo -p -o maxperm%=80
vmo -p -o minperm%=5
vmo -p -o v_pinshm=1
ioo -p -o j2_nBufferPerPagerDevice=2048
ioo -p -o numfsbuffs=2048
ioo -p -o pv_min_pbuf=1024
ioo -p -o numclust=8
no -p -o rfc1323=1
no -p -o sb_max=1310720
no -p -o tcp_sendspace=262144
no -p -o tcp_recvspace=262144
no -p -o udp_sendspace=65536
no -p -o udp_recvspace=655360

 

[kHz]

Additionally -

if you are not on Oracle 10g then you should

export AIXTHREAD_SCOPE=S

And if you are pinning your SGA (recommended), then you should set pinned memory:
Pinned Memory=(SGA + 10%) = Value/RealMemory ## If this number is over the default
of 80% then

vmo -p -o maxpin%=<value>

otherwise, if it is less than the 80% default, then leave it at default.

If you are using Large Page Memory then:

chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGAE oracle

num_of_large_pages=INT((SGA-1)/16MB)+1

vmo -p -o lgpg_regions=num_of_large_pages -o lgpg_size=16777216

 

[jprabaker]

Hi,

Ive now setup a test environment with a 200G worth of 10g database on there.

Without CIO I'm getting approx 16 thousand disk writes per second, with CIO we saw this drop to 9 thousand. I've changed the redo log filesystem to use a 512byte agblksize and we saw the writes go up about 9 1/2 thousand per second.

I'm using nmon to gather stats and I'm looking at the "Disk total kb/s" graph.

We are using the following tuneables:

lru_file_repage=0
maxclient%=80
maxperm%=80
minperm%=5
v_pinshm=1
lgpg_regions=128
lgpg_size = 16777216
j2_nBufferPerPagerDevice=1024
j2_dynamicBufferPreallocation=64
numclust=8

My test jobs are now taking approx twice as long with CIO enabled! Please help!!

Thanks,
John

 

[kHz]

What are your 'vmo -L' and 'ioo -L' output, as well as 'vmstat -s' and 'vmstat -v'.

Are you using JFS2 or JFS? If using JFS, you don't want to use CIO, and you can't. Are you using SSA, EMC, Shark, Hitachi, or what for disk? Are you using RAID and what level?

 

[jprabaker]

Hi,

We were using DS8300 shark configured to use raid5. We had 4 * 50G luns, using one per filesystem. Unfortuantly I no longer have access to the systems so can't post output to commands but I'd still like to understand why we saw a degregation.