Can't mount filesystem

[Yrrk]

I have a p615 running AIX 5L that I just booted after it being turned of for some time. Now that it's up, I notice the /tsmdblog filesystem didn't successfully mount. I also see one of the hdisks is defined, not available. How do I proceed in troubleshooting this based on what i've learned so far? I'm pretty rusty and appreciate any help

root@PS-BS-P615-02:/usr/tivoli/tsm/server/bin$ df -k
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           262144    227116   14%     1626     2% /
/dev/hd2          1310720     44844   97%    24933     8% /usr
/dev/hd9var        262144    244456    7%      321     1% /var
/dev/hd3           262144    175304   34%       79     1% /tmp
/dev/hd1           262144    253748    4%       18     1% /home
/proc                   -         -    -         -     -  /proc
/dev/hd10opt       262144    232892   12%      464     1% /opt
root@PS-BS-P615-02:/usr/tivoli/tsm/server/bin$ lsvg -l rootvg
rootvg:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
hd5                 boot       1     2     2    closed/stale  N/A
hd6                 paging     2     4     2    open/syncd    N/A
hd8                 jfslog     1     2     2    open/stale    N/A
hd4                 jfs        1     2     2    open/stale    /
hd2                 jfs        5     10    2    open/stale    /usr
hd9var              jfs        1     2     2    open/stale    /var
hd3                 jfs        1     2     2    open/stale    /tmp
hd1                 jfs        1     2     2    open/stale    /home
hd10opt             jfs        1     2     2    open/stale    /opt
tsmdbloglv          jfs2       1060  1060  2    closed/syncd  /tsmdblog
loglv00             jfs2log    1     1     1    closed/syncd  N/A
root@PS-BS-P615-02:/usr/tivoli/tsm/server/bin$ lsdev -Cc disk
hdisk0 Defined   1S-08-00-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive root@PS-BS-P615-02:/usr/tivoli/tsm/server/bin$
root@PS-BS-P615-02:/usr/tivoli/tsm/server/bin$ errpt IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
C1348779   0427135705 I O SYSJ2          LOG I/O ERROR
A6DF45AA   0427134405 I O RMCdaemon      The daemon is started.
BE0A03E5   0427134405 P H sysplanar0     ENVIRONMENTAL PROBLEM
C1348779   0427134305 I O SYSJ2          LOG I/O ERROR
078BA0EB   0427134305 I S LIBLVM         Forced activation of a volume group
0BA49C99   0427134305 T H sisscsia1      SCSI BUS ERROR
BE0A03E5   0427134205 P H sysplanar0     ENVIRONMENTAL PROBLEM
9DBCFDEE   0427134305 T O errdemon       ERROR LOGGING TURNED ON
A6DF45AA   0427122705 I O RMCdaemon      The daemon is started.
C1348779   0427122605 I O SYSJ2          LOG I/O ERROR
EAA3D429   0427122605 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122605 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
078BA0EB   0427122505 I S LIBLVM         Forced activation of a volume group
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
EAA3D429   0427122505 U S LVDD           PHYSICAL PARTITION MARKED STALE
9DBCFDEE   0427122605 T O errdemon       ERROR LOGGING TURNED ON

 

[plamb]

One can asuume that rootvg was mirrored at thr vg level some added tsmdbloglv not mirrored then hdisk0 had problems so the system is running from hdisk1 unmirrored and since tsmdbloglv was made on hdisk0 which is bad it will not mount.

 

[ggauthier]

I agree with plamb.
The LVs in the rootvg show to be mirrored, except for tsmdbloglv. And it also appears that the jfs2log loglv00 is used for /tsmdblog.

What is a bit weird is that hd6 shows being on 2 PVs and is synced. Perhaps my understanding of a mirrored page device is rusty, but I would expect it to be stale like the other mirrored devices.

Also, hd7 is missing.

Beyond all that, you should contact your hardware support vendor (e.g., IBM) to determine if the drive needs to be replaced based upon the details of the error messages you received.

Oh. .. you should probably reboot to see if the drive comes on-line. Almost forgot that.

Regards,
gg

 

[Yrrk]

Well it turns out hdisk0 wasn't completely seated. It was seated enough that i was getting the power LED to come on but after powering down, pulling both drives, blowing them out with a can of air, then reseating them, the system is up and fine with both hdisks as available and the partition mounted..

problem solved. Thanks for the advice otherwise

 

[p5wizard]

Re hd6 not being stale

LVM has not yet found that out...
hd6 would become stale with enough write activity on it, LVM would then find out that it cannot write to the second mirror of its LPs and mark the LPs in question and the LV as stale.

 

[rs6000er]

try this:
1,errpt to check the error log and make sure if the hdisk0 is damaged or not.
2,if so, unmirror the rootvg and reduce the rootvg to get rid of t he bad drive.
3,if not, cfgmgr can refresh the devices datebase and then see if the hdisk0 is available.

 

[jie]

Try ruuning fsck to the filesystem ...