Performance Problem

[ponetguy2]

Hello again everyone. I have three Solaris 9 servers (
SunOS hostname 5.9 Generic_118558-27 sun4u sparc SUNW,Sun-Fire-480R). All three are currently non-production boxes. I installed the latest and greatest Solaris 9 patch cluster on all three at roughly thesame time in single user mode. During this install process, I noticed that there was a significant difference on how long it took to completely install the patch cluster on box A. Box C and B took alot longer to finish (roughly 5 to 10 minutes) vs. box A.

I ran some test and found very little CPU load on all three when installing patches. However, there was a slight higher iowait load on box B and C vs. A. Despite of the iowait difference, I don't think it should cause much of a difference in performance.

Should I run format>analyze>test on all three?

Any suggestions on how to attack this problem would be greatly appreciated.

 

[Annihilannic]

Were the systems in single-user mode at the time you installed the patches?

Annihilannic.

 

[ponetguy2]

hello annihilannic. i'm surprised you are in on a weekend. thank you for the reply.

yep. single usermode. should i run a format>analyze>test?

 

[ponetguy2]

After looking a little more than I did. I do see a difference with box A.

One thing I forgot to mention is that all three boxes are mirrored. Each box has two disk. I looked at the disk type and revision and there is a difference on revision number and ssd assignment. I'm not sure if this would make a difference, but this info scratches my old belief that they are identical. Can anyone tell me if having a different revision number and ssd assignement will make a difference in performance?

Please see below:

Box A:

# netstat -k ssd1,err
ssd1,err:
Soft Errors 0 Hard Errors 0 Transport Errors 0 Vendor SEAGATE
Product ST336605FSUN36G Revision Revision 0638 Serial No 0233P1YRX9 Size 36418595328 Media Error 0 Device Not Ready 0
No Device 0 Recoverable 0 Illegal Request 0 Predictive Failure Analysis 0

# netstat -k ssd2,err
ssd2,err:
Soft Errors 0 Hard Errors 0 Transport Errors 0 Vendor SEAGATE
Product ST336605FSUN36G Revision Revision 0438 Serial No 0211P1BSF1 Size 36418595328 Media Error 0 Device Not Ready 0
No Device 0 Recoverable 0 Illegal Request 0 Predictive Failure Analysis 0


Box B & C:

# netstat -k ssd0,err
ssd0,err:
Soft Errors 0 Hard Errors 0 Transport Errors 0 Vendor SEAGATE
Product ST336607FSUN36G Revision Revision 0207 Serial No 0315A1FELG Size 36418595328 Media Error 0 Device Not Ready 0
No Device 0 Recoverable 0 Illegal Request 0 Predictive Failure Analysis 0

# netstat -k ssd1,err
ssd1,err:
Soft Errors 0 Hard Errors 0 Transport Errors 0 Vendor SEAGATE
Product ST336605FSUN36G Revision Revision 0638 Serial No 0230P1WYDP Size 36418595328 Media Error 0 Device Not Ready 0
No Device 0 Recoverable 0 Illegal Request 0 Predictive Failure Analysis 0

 

[ponetguy2]

I did a test by unzippin solaris 9 patch cluster and watching the output from top.

I did see a little difference in performance between the three. I was watching the cpu load and iowait. iowait on box B and box A was about 20 to 25% and box A was around 10 to 15%. CPU load was about thesame on all three.

ssd assignment or patch revision might be the root of this problem. Please advise.

 

[dandan123]

netstat or iostat ?

 

[Annihilannic]

-k is an undocumented feature in netstat for reading kernel variables.

I don't know why it's in netstat either.

ponetguy, personally I wouldn't worry about this too much, but you could try checking for any updated firmware for the disks in question. Does the application run from these disks or just the operating system? Have you had any performance problems or noticeable differences between the three systems from the application perspective?

Annihilannic.

 

[marrow]

Just out of interest does each box have the same amount of memory, same number of cpu's and same power?

prtconf |head
prtdiag|grep MHz

 

[ponetguy2]

I also ran some performance test between all three servers. I did see a significant difference
in performance between the three. I was watching the cpu load and iowait. iowait on server B and
Server C was at 20 to 25% and Server A was at around 10 to 15%.
CPU load on all three servers were about thesame. All applications are running locally.

There is a new st, sd and ssd drivers patch (113277-45) for SunOS 5.9.


All three have thesame CPU and Memory.

 

[Annihilannic]

I wasn't referring to driver patches so much as disk firmware, such as the following (this is only an example, there may be newer firmware):

http://sunsolve.sun.com/search/document.do?assetkey=1-21-113671

Annihilannic.

 

[ponetguy2]

Forgive me. I'm not a hardware expert.

I tried to look for the disk firmware (ST336605FSUN36G), but I was unsuccessful. I will keep googling for the firmware. Hopefully I find something.

 

[ponetguy2]

109962-14
Hardware, FC-AL Disks: Download program and FC-AL Disk Drive firmware

CRAP!!! I found this firmware for the disk, but it requires me to unmirror
Solaris Disksuite prior to upgrading the firmware. I'm not sure if I
want to do that

 

[kHz]

unmirroring disksuite is simple but it requires a reboot for the root slice. if the root slice isn't on this disk though it would be quick to do and then mirror again.

 

[Annihilannic]

Unmirror disksuite?? What on earth for? If it was me I would ignore that (follow my advice at your own risk)... as long as there is zero traffic going to those disks I believe you would be fine (i.e. filesystems unmounted, etc.) I can't see what reason they would have to recommend that other than to ensure that the disks were idle.

Annihilannic.

 

[kHz]

I wasn't the one who mentioned breaking the mirrors using disksuite. I was just stating that it is a simple procedure.

And FYI I have read the instructions for firmware upgrades that advise any mirrors be broken before applying.

So I would actually follow the instructions that come with the firmware. If the instructions read to break a mirror then break the mirror, regardless what you have been told here. If they read don't break the mirror then don't break the mirror.

 

[Annihilannic]

I know, kHz, I wasn't responding to your post in particular.

Annihilannic.

 

[marrow]

To check your f/w is the same or not, run this command:-

/usr/platform/sun4u/sbin/prtdiag -v | grep OBP

 

[Annihilannic]

We're talking about disk drive firmware here marrow? Not OBP?

Annihilannic.

 

[ponetguy2]

thank you guys. i'm afraid i will need to unmirror and apply the firmware.

i also noticed that box b and c have two different model numbers:

box A:
========

c1t0d0
Vendor: SEAGATE Product: ST336605FSUN36G Revision: 0638

c1t1d0
Vendor: SEAGATE Product: ST336605FSUN36G Revision: 0438
box B:
=========

c1t0d0
Vendor: SEAGATE Product: ST336607FSUN36G Revision: 0207

c1t1d0
Vendor: SEAGATE Product: ST336605FSUN36G Revision: 0638

box c:
=========

c1t1d0
Vendor: SEAGATE Product: ST336607FSUN36G Revision: 0207

c1t0d0
Vendor: SEAGATE Product: ST336605FSUN36G Revision: 0638

i'll swap the disk between box b and c to match disk model numbers. i know it should'nt
matter since the specs are thesame, but who knows

i'll keep you guys posted.

 

[ponetguy2]

The statement below for patch 113671-03 is unclear to me. Tell me how you guys interpret the statement below.
It seems to refer to storege systems only. i have the disk on a V240 and mirrored via SDS.

This patch release is for the 6120 ARRAY and SE3510 RAID Storage systems
and also for NON RAID Storage systems that use Sun disk drives ONLY.
This PATCH DOES NOT APPLY TO the T3 OR T3B Array.
If you are looking to update with the latest patch release for the T3
or T3B Array, please download their respective patches elsewhere.