AIX and DS4300 performance issues

[chrisw09]

I have a P570 with 2 FC adapters, each connected to their own McData 4500 switch which is in turn connected to a controller.

HBA0 --> switch 1, port 0 --> controller A
HBA1 --> switch 2, port 0 --> controller B

I have a couple of luns, hdisk3 and hdisk4 both about 10gig.

When I assign (ownership/preferred path) the logical volumes to controller A, my system performance tanks. When you do a 'iostat 2', you will consistently see %tm_acct near or at 100% with no disk activity. When you assign the lv's to controller B, no disk issues.

I looked at the switch error logs and there is nothing about errors on any of the ports, nor is there any errors on the SAN or in AIX. With that said, i have to feel that something is definitely wrong with the path to Controller A.

Here's my list of physical things that can be wrong:
1. Bad HBA
2. Bad Cable between server and switch
3. Bad SFP for server in switch
4. Bad SFP for SAN in switch
5. Bad cable between switch and SAN
6. Bad SFP in SAN
7. Bad controller in SAN
8. Bad switch

So far, i have replaced 2,3 and 5 with no change, things are bad on controller A path and good on controller B path.

This morning I moved SAN controller A connection to a different port in the switch. I re-ran cfgmgr -v and waited (a very, very long time). The lcd screen on the P570 displayed '0538' and cfgmgr was stuck at '/usr/lib/methods/fdarcfgrule'.

After 10 minutes, it finished and i checked the error logs and found:
B8FBD189 0228065708 T S fscsi0 SOFTWARE PROGRAM ERROR
825849BF 0228065708 T H fcs0 ADAPTER ERROR
D9770360 0228065708 P H dac0 ARRAY OPERATION ERROR
B8FBD189 0228065708 T S fscsi0 SOFTWARE PROGRAM ERROR
825849BF 0228065708 T H fcs0 ADAPTER ERROR
B8FBD189 0228065708 T S fscsi0 SOFTWARE PROGRAM ERROR
825849BF 0228065708 T H fcs0 ADAPTER ERROR
B8FBD189 0228065608 T S fscsi0 SOFTWARE PROGRAM ERROR
825849BF 0228065608 T H fcs0 ADAPTER ERROR
B8FBD189 0228065608 T S fscsi0 SOFTWARE PROGRAM ERROR
825849BF 0228065608 T H fcs0 ADAPTER ERROR
B8FBD189 0228065508 T S fscsi0 SOFTWARE PROGRAM ERROR
825849BF 0228065508 T H fcs0 ADAPTER ERROR
825849BF 0228065508 T H fcs0 ADAPTER ERROR

So, thinking i had a bad SFP, i swapped it with another one. Same result. I moved the SAN controller connection back to it's original port and re-ran 'cfgmgr -v'; finished in 3 seconds and no errors in aix.

Can you not swap SFP's like that? I did for the one that the P570 attaches to and things worked fine but swapping either the SFP in the SAN controller or the SFP in the switch, i get errors and AIX (or any other host that uses controller A) cannot see their disks.

Thanks for any help!

 

[RMGBELGIUM]

if you have a bad sfp, it will show in the logs of the switch normally.I would suspect there's sth fishy with your controller of the DS4300

 

[chrisw09]

I opened and issue with IBM and sent support data for them to analyze. They came back and said that they don't see anything wrong. Said there is a little noise coming from a switch but other than that, things are clean.

 

[baddos]

I'm no expert on SAN switches, but I know that my switch there is a special port used for storage controllers and other ports used for hosts. Maybe that could explain why when you swapped ports, you started with all those errors.

One thing I would do is check for errors in your HBA driver on your host, and the port counters on your switch. Errors there might mean a bad cable or sfp/gbic, etc.

Also, is this a new install or an existing one? If existing has anything changed like new drivers, firmware, etc.?

 

[RMGBELGIUM]

@baddos,your can plugin a storage controller and a host nicely besides each other,worst case scenario you implement zoning to separate traffic to hosts that do not need a connection to this storage,but there are by default no "reserved ports" on a switch for a storage controller.Your san switch will have one or multiple zones, and 1 of them is a zone for your storage controller,and the network guy propably told you this is a reserved zone for your storage.

rgds,

R.

 

[xmsre]

I'll take a stab at translating the error log from the eunuchese:

An array operation failed (D9770360) beacuse an adapter call failed (825849BF) because a protocol command wasn't supported (B8FBD189-Communication error has been detected at the FC I/O controller protocol device.).

Try updating the firmware on the HBA and switch. This recommendation assumes the protocol command that wasn't supported originated from the SFP.

 

[RMGBELGIUM]

beware as well when updating the firmware of your HBA and switch, that you fall within the support matrix of your constructor / product.Most of the times, array controllers require you to have specific HBA firmware versions in combination with a specific switch OS.