Solaris 10 memory issue

[terrywashington]

I recently upgraded one of our development V440s from Solaris 9 to Solaris 10 (update 9). The system has 4 X 1.6GHz CPUs, 8GB of RAM, an 8GB swap partition on the internal boot disk and a 20GB swap file on a SAN disk. One of the developers reported that a test that starts 1001 processes slows the system to a crawl and never finishes. This test reportedly ran successfully in a few minutes in Solaris 9. Unfortunately I just configured sar on this system so I have no performance history to compare from Solaris 9. I ran top, vmstat and prstat while the test ran and all seemed to indicate the system ran out of physical memory. I have included samples of the prstat, top and vmstat output from during the test. Is there a kernel setting that can be set to allow the system to perform as it reportedly did in during Solaris 9?

**prstat -Z output**

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
5895 deadm 23M 12M sleep 10 0 0:00:33 7.0% mcadbs/1
5846 root 3264K 1288K cpu3 59 0 0:00:04 1.1% prstat/1
5898 deadm 35M 19M sleep 60 0 0:00:02 0.4% fame/1
6769 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6766 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6772 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6755 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6770 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6767 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6753 deadm 29M 13M sleep 59 0 0:00:00 0.2% mcadbs/1
6765 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6750 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6754 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6756 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6762 deadm 29M 13M sleep 59 0 0:00:00 0.2% mcadbs/1
6752 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6758 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
233 root 197M 16M sleep 59 0 1:23:27 0.2% nscd/286
6757 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6760 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6764 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6768 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6731 deadm 29M 13M sleep 59 0 0:00:00 0.2% mcadbs/1
6740 deadm 29M 13M sleep 59 0 0:00:00 0.2% mcadbs/1
6771 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6751 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6736 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6749 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6759 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6738 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6729 deadm 29M 8752K sleep 59 0 0:00:00 0.2% mcadbs/1
6733 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
6739 deadm 29M 13M sleep 59 0 0:00:00 0.2% mcadbs/1
6746 deadm 29M 17M sleep 59 0 0:00:00 0.2% mcadbs/1
5791 richard. 2808K 688K sleep 59 0 0:00:05 0.1% top/1
6726 deadm 29M 8880K sleep 59 0 0:00:00 0.1% mcadbs/1
6727 deadm 29M 13M sleep 59 0 0:00:00 0.1% mcadbs/1
6743 deadm 29M 13M sleep 59 0 0:00:00 0.1% mcadbs/1
6748 deadm 29M 17M sleep 59 0 0:00:00 0.1% mcadbs/1
6761 deadm 29M 17M sleep 59 0 0:00:00 0.1% mcadbs/1
6730 deadm 29M 13M sleep 59 0 0:00:00 0.1% mcadbs/1
733 root 29M 3864K sleep 59 0 0:00:00 0.2% mcadbs/1
ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
0 838 16G 7641M 93% 50:07:14 34% global
Total: 838 processes, 1304 lwps, load averages: 3.11, 1.09, 0.43


**top output**

load averages: 3.10, 1.18, 0.47 soldev1 14:30:29
1082 processes:1071 sleeping, 6 running, 1 stopped, 4 on cpu
CPU states: 44.4% idle, 34.5% user, 21.1% kernel, 0.0% iowait, 0.0% swap
Memory: 8.0G real, 156M free, 20.2G swap in use, 44.6G swap free

PID USERNAME THR PR NCE SIZE RES STATE TIME FLTS CPU COMMAND
5895 deadm 1 53 0 23.1M 11.8M sleep 0:35 0 5.51% mcadbs
5846 root 1 60 0 3488K 1608K cpu01 0:04 45 0.81% prstat
5898 deadm 1 59 0 39.0M 23.4M sleep 0:02 32 0.33% fame
6839 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.25% mcadbs
6848 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.25% mcadbs
6842 deadm 1 59 0 29.2M 16.6M sleep 0:00 0 0.24% mcadbs
6844 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.24% mcadbs
6849 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.24% mcadbs
6846 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.23% mcadbs
6840 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.23% mcadbs
6852 deadm 1 59 0 29.2M 16.6M sleep 0:00 0 0.23% mcadbs
6841 deadm 1 59 0 29.2M 16.6M sleep 0:00 0 0.22% mcadbs
6853 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.22% mcadbs
233 root 286 59 0 197M 17.3M sleep 83:27 173 0.21% nscd
6847 deadm 1 59 0 29.2M 16.7M sleep 0:00 0 0.20% mcadbs

**vmstat output**

root@soldev1:/]# vmstat 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 5 33 18905016 6647272 85 228 328 161 162 0 51 0 13 5 0 442 3774 354 3 1 96
0 0 74 56034928 7245784 30 113 49 0 0 0 0 0 2 2 0 390 1054 287 4 0 95
0 0 74 56035808 7246520 6 52 54 0 0 0 0 0 9 3 0 397 21887 291 3 2 94
0 0 74 56035808 7246488 0 8 41 0 0 0 0 0 26 1 0 455 21831 329 3 3 94
0 0 74 56028816 7234224 846 414 6702 0 0 0 0 0 117 2 0 737 22840 1105 10 3 87
0 0 74 56002032 7204936 345 1499 2387 2 2 0 0 0 184 4 25 618 21238 745 5 4 91
0 1 74 55800696 7062536 331 2533 1184 5 5 0 0 0 51 119 0 878 15787 2460 4 12 84
0 4 74 53907784 5763064 0 3124 1745 0 0 0 0 0 126 297 0 1038 21632 4554 4 28 69
0 1 74 51602344 4180568 113 4157 1451 0 0 0 0 0 71 272 0 1463 28515 4805 11 36 53
3 0 74 49457456 2707352 217 3736 602 0 0 0 0 0 24 166 0 3325 22810 4630 61 33 6
7 1 74 48195408 1837136 158 2833 597 0 0 0 0 0 97 123 7 3844 18825 4811 69 27 3
4 1 74 46946048 976048 203 3922 804 0 0 0 0 0 37 157 0 3168 23962 4156 56 40 4
6 1 74 45780264 195344 117 2681 881 195 158 0 23750 0 42 99 0 3049 14804 4070 55 31 14
3 2 74 45225640 119184 68 1375 560 83933 85554 0 25008 0 261 299 0 2316 10306 2403 26 29 45
0 3 74 44799112 112456 67 1173 707 54368 59309 0 25375 0 199 199 0 1805 10475 2371 18 25 57
0 4 74 44297920 96832 60 917 220 62251 67962 0 35145 0 207 206 0 1493 6808 1760 12 23 65
1 5 74 43906520 136472 75 1197 442 61409 61355 0 17943 0 185 244 1 1652 8005 1884 14 23 63
1 3 74 43335592 113008 87 1495 227 83296 83211 0 25368 0 252 292 0 2191 10156 2278 25 31 45

 

[cndcadams]

you really need to look into memory leak. There are many differences in solaris 10 compared to solaris 9.

See

https://blogs.oracle.com/pnayak/entry/finding_memory_leaks_within_solaris

 

[Annihilannic]

Each of those processes has a resident set size of 16.7MB. He's starting 1000 of them... that's 16.7GB, in a system with 8GB of memory? I can't imagine how it *didn't* run out of memory on Solaris 9...

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

 

[terrywashington]

Thanks to both of you for your input ;-)