SAP Portal taking too much time to start on AIX 6.1 TL5 SP6

[MoreFeo]

Hi, we have some performance issues starting SAP Portal on AIX 6.1 TL5 SP6.
It takes more than 10 minutes to start, and there is a jlaunch process with high CPU and PgSp values in topas for more than 10 minutes.

It is a Power7 LPAR (0'39 CPU, 1 virtual cpu, 16GB RAM, SMT on). I've also tried with SMT on and off, with more and less virtual cpus, and results vary from 10 minutes to 15 minutes but they're still too high.

It all started when we were asked to upgrade Oracle from 11.2.0.2 to 11.2.0.3, so we had to update AIX from 6.1 TL4 SP2 to 6.1 TL5 SP6 (Oracle upgrade asked for fix IZ97457).

I'm trying to troubleshoot this, but there are several things I don't understand.
PgSp value for the jlaunch process keeps growing in topas, but there is no paging space used (topas shows 0% of 20448MB, and vmstat shows no pi or po:

Topas:
Name	PID	CPU%	PgSp	Owner
jlaunch	6422560	49.5	[b][u]291.2[/u][/b]	pppadm

# vmstat 3
kthr    memory              page              faults              cpu
----- ----------- ------------------------ ------------ -----------------------
 r  b   avm   fre  re  pi  po  fr   sr  cy  in   sy  cs us sy id wa    pc    ec
 4  0 983146 1844191   0   0   0   0    0   0  45 110361 4018 74 17  8  0  0.39  99.9
 3  0 984826 1842516   0   0   0   0    0   0  32 108305 3884 74 15 11  0  0.39  99.9
 4  0 985067 1842272   0   0   0   0    0   0  37 112300 4140 70 18 12  0  0.39 100.0
 3  0 985263 1842033   0   0   0   0    0   0  43 107183 4175 75 14 11  0  0.39  99.9
 6  0 986726 1840508   0   0   0   0    0   0  40 106498 3875 74 16 10  0  0.39  99.8
 4  0 986389 1840658   0   0   0   0    0   0  49 106811 4452 71 17 11  0  0.39 100.0
 3  0 986680 1840365   0   0   0   0    0   0  31 113041 3946 63 24 13  0  0.39  99.8

Also, topas show some faults and PageIn values in the PAGING section, but there is still memory available, and as I said vmstat shows no pi:

Topas:
		MEMORY
PAGING		Real,MB   16384
Faults	2377	% Comp     26
Steals	0	% Noncomp   7
PgspIn	0	% Client    7
PgspOut	0
PageIn	95
PageOut	0
Sios	19

Also, I see the jlaunch process consuming CPU in topas for more than 10 minutes, but in ps -ef it shows less than 2 minutes in the TIME column:

    UID      PID     PPID   C    STIME    TTY  TIME CMD
  pppadm  6422560  4849668   0 10:55:05      -  [b][u]1:55[/u][/b] /usr/sap/PPP/JC00/j2ee/os_libs/jlaunch pf=...

Wait IO is really low, and vmstat shows no blocked processes.

Can anyone explain this results?
What can I do to troubleshoot this?

Thanks

 

[MoreFeo]

Another thing that we are seeing is that the Entc column in topas is always showing 99.9% or even 100%, even when the cpu is almost idle (96.2% idle).

CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   
ALL    0.3    3.5    0.0   96.2   0.50   99.9

In other servers the Entc column remains low when there is no cpu usage:

CPU  User%  Kern%  Wait%  Idle%  Physc   Entc   
ALL    0.0    0.5    0.0   99.4   0.02    2.5

What can cause this?

 

[MoreFeo]

We've also seen that ec column in vmstat is always 99 or 100.

I've found a fix (IV01111 - WAITPROC IDLE LOOPING CONSUMES CPU. APPLIES TO AIX 6100-05) that should solve this:

https://www-304.ibm.com/support/entdocview.wss?uid=isg1IV01111

It is included in 6.1 TL5 SP8.