I/O Error on AIX Using ESS Shark

[KPKIND]

Hi *,

I have got a 7028-6C4 system which uses Shark storage. Yesterday we have received a lot of I/O errors continuously for a couple of minutes. During this time the oracle service hosted on this storage area has also gone down and indeed reported the error unable to write to log file etc...

I have checked from OS level, I can't see any obvious problems, so went to our SAN team who has also confirmed that there are no errors reported on the SAN for this server.

One thing I have observed is that there was one error saying "Unable to communicate with the device" just before this error flooding. I am enclosing the output of the errpt with this.

Any clues would be of great help. Thanks in Advance.

Cheers
Kumar


D7B7FF7E 0908161705 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161705 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161705 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161705 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161705 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161705 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161705 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161705 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161705 I O SYSJ2 USER DATA I/O ERROR

More like this

D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
C1348779 0908161505 I O SYSJ2 LOG I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
D7B7FF7E 0908161505 I O SYSJ2 USER DATA I/O ERROR
613E5F38 0908161505 P H LVDD I/O ERROR DETECTED BY LVM
F4D25312 0908161505 P H vpath6 UNABLE TO COMMUNICATE WITH DEVICE

 

[RMGBELGIUM]

are you fiber attached? Trough San switches?
could be your switch or fiber has gone bad

regrds,

R.

 

[mickeli]

Hi,

whats about the Hostadapters of the Shark. Did you check then Problemlog of the Shark?

 

[KPKIND]

Hi RMBELGIUM/MICKELI,

Thanks for the response. My scenario is a bit tricy one, b'coz it is not directly my team which maintains the other side (I mean the SAN and it's accessories). But the SAN team who manage everything on this front say that there were no errors thrown.

To safegaurd myself, I wanted to make sure if I can determine anything from the OS level or not. So pl let me know how do I rule out that this issue is not in my area(OS Level).

Anyway I would try to know the details you have asked me.

TIA
Kumar