Problems with Memory on a LPAR AIX

[Mkoert]

Hi All

WE have found a problem with one of our applications.
This application runs perfect on a AIX stand alone machine. But when we run it on a LPAR AIX it seems not able to load its librarys in SHARED MEMORY. Because of this the amount of memory used by every process is 10 times more then we are used to. I did the following command on both machines and you can see the difference.

ps -Af -o pid,pcpu,pmem,vsz,user,comm

(LPAR)
53924 2.7 4.0 146652 fundt1 Approve#1.1.7.00_AIX5.2

(stand alone )
38842 0.0 0.0 13036 clsqa Approve#1.1.7.00_AIX5.2

Any ideas. Because we have ran out. ( The question is also outstanding by IBM but they have had it now for over a week and they have not given use any indication)

 

[khalidaaa]

For the AIX as an operating system it doesn't matter wether you run it on a stand alone server or an LPAR! it should act the same!

I believe something is not right with the application itself. Any way, Could you please get the output of the following commands:

(on LPAR)
svmon -P 53924

(on Stand alone)
svmon -P 38842

Regards,
Khalid

 

[mkoert2]

Sorry that it is a different handle it seems it did not take my first handle.

We have stopped and started the application already a few times so i do not have the processes anymore that where in the post.

So i took 2 different processes.

LPAR

-------------------------------------------------------------------------------
Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
90050 Entry#1.1.7.0 53416 4310 0 52518 N N N

Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
3386e7 2 work process private - 36231 2 0 36231
b8097 d work shared library text - 7987 0 0 7987
0 0 work kernel seg - 7034 4308 0 7034
120da4 3 work shared memory segment - 629 0 0 629
3586eb - work overflow heap - 424 0 0 424
248449 - clnt /dev/lvfund:8235 - 322 0 - -
1c8539 - clnt /dev/lvfund:8279 - 300 0 - -
5050a 1 clnt code,/dev/lvfund:4155 - 231 0 - -
90712 f work shared library data - 213 0 0 213
280450 - clnt /dev/lvtux:35657 - 16 0 - -
238747 - clnt /dev/lvfund:8542 - 14 0 - -
26044c - clnt /dev/lvtux:35195 - 11 0 - -
2f845f - clnt /dev/lvfund:4264 - 2 0 - -
2f045e - clnt /dev/lvfund:4265 - 2 0 - -

Stand alone

Pid Command Inuse Pin Pgsp Virtual 64-bit Mthrd LPage
49868 Entry#1.1.7.0 21655 2675 11900 55644 N N N

Vsid Esid Type Description LPage Inuse Pin Pgsp Virtual
224f3 2 work process private - 3 2 3405 3408
3c01e d work shared library text - 3687 0 3324 42956
0 0 work kernel seg - 3928 2673 2880 6386
1d1e8 f work shared library data - 0 0 1831 2325
1d3a8 - work - 380 0 460 569
1c48c - pers /dev/users:526624 - 0 0 - -
3497c - pers /dev/users:526611 - 13639 0 - -
10b69 - pers /dev/users:2451457 - 18 0 - -
3c0dc 1 pers code,/dev/users:2314289 - 0 0 - -
182ee - pers /dev/users:2314375 - 0 0 - -
242f0 - pers /dev/users:2314376 - 0 0 - -
2b80 - clnt - 0 0 - -
16b6a - clnt - 0 0 - -

 

[RMGBELGIUM]

could you please post the output of vmo -a of both standalone and lpar?

rgds,

R.

 

[mkoert2]

I can give you the LPAR and ill request the stand alone machine and give you that as soon as they wake up in the US.

LPAR
cpu_scale_memp = 8
data_stagger_interval = 161
defps = 1
force_relalias_lite = 0
framesets = 2
htabscale = n/a
kernel_heap_psize = n/a
large_page_heap_size = n/a
lgpg_regions = 0
lgpg_size = 0
low_ps_handling = 1
lru_file_repage = 1
lru_poll_interval = 0
lrubucket = 131072
maxclient% = 80
maxfree = 128
maxperm = 5306409
maxperm% = 80
maxpin = 5452596
maxpin% = 80
mbuf_heap_psize = n/a
memory_affinity = 1
memory_frames = 6815744
mempools = 1
minfree = 120
minperm = 1326602
minperm% = 20
nokilluid = 0
npskill = 2048
npswarn = 8192
num_spec_dataseg = n/a
numpsblks = 262144
pagecoloring = 0
pinnable_frames = 6607894
pta_balance_threshold = 50
relalias_percentage = 0
soft_min_lgpgs_vmpool = 0
spec_dataseg_int = n/a
strict_maxclient = 1
strict_maxperm = 0
v_pinshm = 0
vmm_fork_policy = 0

 

[mkoert2]

Stand alone machine is

memory_frames = 524288
pinnable_frames = 489875
maxfree = 128
minfree = 120
minperm% = 20
minperm = 99873
maxperm% = 80
maxperm = 399493
strict_maxperm = 0
maxpin% = 80
maxpin = 419431
maxclient% = 80
lrubucket = 131072
defps = 1
nokilluid = 0
numpsblks = 2621440
npskill = 20480
npswarn = 81920
v_pinshm = 0
pta_balance_threshold = 50
pagecoloring = 0
framesets = 2
mempools = 1
lgpg_size = 0
lgpg_regions = 0
num_spec_dataseg = n/a
spec_dataseg_int = n/a
memory_affinity = n/a
htabscale = -1
force_relalias_lite = 0
relalias_percentage = 0
data_stagger_interval = 161
large_page_heap_size = n/a
kernel_heap_psize = n/a
soft_min_lgpgs_vmpool = 0

 

[mkoert2]

Just some more info

oslevel 5200-01

vmstat
System Configuration: lcpu=2 mem=26624MB
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 1 3794328 2297999 0 0 0 0 0 0 361 2504 480 1 1 98 0

 

[hfaix]

Are both systems 5200-01? I noticed the vmo -a stats are slightly different. No lru_file_repage on the stand a lone, memory frames, pinnable frames,...

What kind of "Lpar" is this? Power5 lpars require 5.2 ML4+ or higher. It's possible that the ML level isn't reporting the usage correctly. I know if 5300-03 svmon -Pu doesn't always report the correct usage...an APAR was released to fix this.

 

[khalidaaa]

There are some tuning parameters that are different

 memory_affinity
 numpsblks
htabscale
minperm
maxperm

I don't think that these are the cause of the problem but why don't you go through the below links and read more about these parameters:

http://www.unet.univie.ac.at/aix/aixbman/prftungd/vmmpage.htm

http://www.circle4.com/jaqui/eserve...OOandVMO.pdf#search="AIX + minperm + maxperm"

I think its the you need to review the application Itself you are running (coding on that application!)

Regards,
Khalid

 

[mkoert2]

Both systems are on 5200.01
I know that is not the reporting. I have problems that some processes do not start because of memory problems. We now can run the application but have 26 Gb in the LPAR. The stand alone has 2Gb.

 

[hfaix]

Is it a power 5 or power 4 lpar?

 

[mkoert2]

/app/cls/Fundtech/tools>lsattr -El proc0
frequency 1656424000 Processor Speed False
state enable Processor state False
type PowerPC_POWER5 Processor type False

 

[hfaix]

Interesting. I believe the minimum ML level for 5.2 with power 5 is ML4+ or ML5. You should probably upgrade this. I doubt it will fix your problem though.

 

[mkoert2]

We are in conversation with IBM and the Vendor of the software but they both are not certain what is going on.
We are also making a second LPAR machine to see if the same thing is going to happen. On the first LPAR machine we had installed AIX 64 we are now going back to AIX 32. The first machine was switched back to 32 but we want to try it with a clean install to see if it makes any differnce.

If there are any idears out there let me know.