disk has 'REMOVED' state after installing tape drive

[zyrag]

HI,

Machine Config:
type: 7013, model: j40 oslevel: aix 4.2.0

A supplier technician is physically installing an IBM 4mm4gb tape drive to replace a defective one already installed. Upon booting, the system hangs at 556. After waiting a considerable amount of time, the technician decided to shutdown the system, physically removed the tape drive and boot in normal mode. The system was able to complete the boot but there were file systems not mounted. I began to troubleshoot with the following results..

--------------------------------------------------------

# mount /inv
Unable to read superblock (TERMINATED)
Cannot mount /dev/lvinv on /inv: There is an output or input error

# errpt
ID        T  C  RESOURCE  DESCRIPTION
1A2E7186  U  S  LIBLVM    PHYSICAL VOLUME DEFINEd AS MISSING

# errpt -aj 1A2E7186
LABEL: LVM_MISSPVADDED
PROBABLE CAUSE:
      POWER, DRIVE, ADAPTER OR CABLE Failure
DETAIL DATE:
      MAJOR/MINOR DEVICE NUMBER
      000C 0001

# lsdev -Cc disk
  hdisk0 Available
  hdisk1 Available
  hdisk2 Available
  hdisk3 Available

# lslv lvinv
  VG STATE: Active/Complete     VG: vgAR
  TYPE: Jfs                     LV STATE: closed/syncd
  COPIES: 1
  STALE PPs: 0

# lsvg vgAR
  VG_STATE: Active
  TOTAL PVs: [COLOR=red]2[/color]
  STALE PVs: 0
  ACTIVE PVs: [COLOR=red]1[/color]
  QUOROM : 2
  VG DESCRIPTORS: 2
  STALE PPs: 0
  AUTO-CONCURRENT: DISABLED

# lsvg -p vgAR
  PV_NAME   PV_STATE  TOTAL_PPs  FREE_PPs
  hdisk0    [COLOR=red]removed[/color]   537          1
  hdisk3    active    537          15

# lspv hdisk0
  PV STATE: [COLOR=red]removed[/color]
  FREE PPs: 1 (8 megabytes)

----------------------------------------------------

I have in mind some steps to fix the problem but, i'm not quite sure which should come first. i need a step by step procedure to solve this problem - just to make sure..
I hope you could share your experiences on this.

Thanks in advance,

 

[bonsky]

Did the guy tried to check the SCSI ID of the existing tape and compare it with the new one. Im not so sure, wether due to wrong scsi id might be the caused of this, but hardly and i havent face such.. luckily..
anyways... here are the steps ( from problem determination redbook, try to check that one. ) Also, what i often do before replacing or do some upgrades, always make it a practise to have a full reboot of the machine to make sure that its pretty working before touching it.

LED 556
A LED code 556 during a standard disk-based boot indicates that a failure occurred during the varyon of the rootvg volume group. This can be the cause of:
>>>> A damaged file system
>>>> A damaged journaled file system (JFS) log device
>>>> A bad IPL-device record or bad IPL-device magic number (the magic number
indicates the device type)
>>>> A damaged copy of the Object Data Manager (ODM) database on the boot logical volume
>>>> A hard disk in the inactive state in the root volume group
>>>> A damaged superblock

A corrupted ODM in the BLV is also a possible cause for these LED codes. To create a usable one, run the following commands that remove the system's
configuration and save it to a backup directory:
# /usr/sbin/mount /dev/hd4 /mnt
# /usr/sbin/mount /dev/hd2 /usr
# /usr/bin/mkdir /mnt/etc/objrepos/bak
# /usr/bin/cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
# /usr/bin/cp /etc/objrepos/Cu* /mnt/etc/objrepos
# /usr/sbin/umount all
# exit

After this, you must copy this new version of the ODM in the RAMFS to the BLV. This is done with the savebase command. Before that, make sure you place it on the disk used for normal boot by executing:
# lslv -m hd5
Save the cleanODM database to the boot logical volume. For example:
# savebase -d /dev/hdisk0
Finally, recreate the BLV and reboot the system. For example:
# bosboot -ad /dev/hdisk0
# shutdown -Fr

Another possible reason for these error codes is a corrupted superblock. If you boot in maintenance mode and receive error messages such as Not an AIX file
system or Not a recognized file system type, it is probably due to a corrupted superblock in the file system.
Each file system has two super blocks: One in logical block 1 and a copy in logical block 31. To copy the superblock from block 31 to block 1 for the root file system, issue the following command (before you use this command, check the product documentation for the AIX release you are using to make sure all of the parameters shown are correct):

# dd count=1 bs=4k skip=31 seek=1 if=/dev/hd4 of=/dev/hd4

Worst thing.. you might need to reinstall or restore from your recent backup. Good luck dudes!

 

[zyrag]

# /usr/sbin/mount /dev/hd4 /mnt
# /usr/sbin/mount /dev/hd2 /usr
# /usr/bin/mkdir /mnt/etc/objrepos/bak
# /usr/bin/cp /mnt/etc/objrepos/Cu* /mnt/etc/objrepos/bak
# /usr/bin/cp /etc/objrepos/Cu* /mnt/etc/objrepos
# /usr/sbin/umount all
# exit
After this, you must copy this new version of the ODM in the RAMFS to the BLV. This is done with the savebase command. Before that, make sure you place it on the disk used for normal boot by executing:
# lslv -m hd5
Save the cleanODM database to the boot logical volume. For example:
# savebase -d /dev/hdisk0
Finally, recreate the BLV and reboot the system. For example:
# bosboot -ad /dev/hdisk0
# shutdown -Fr

Thanks bonsky.. Do these require that these series of commands be done in maintenance mode?

Anyway, i'll try it out but in case it still fail, I'm figuring out of doing these series of actions as the solution. Please verify if i'm doing the right course of action.

1. migrate LVs from hdisk0 to hdisk3 (migratepv), but dtermine first for a space in hdisk3
2. remove hdisk0 from vgAR (reducevg)
3. remove definition from ODM (rmdev -dl)
4. verify the disk
5. if disk still usable, add back to the system (cfgmgr)
6. extend the disk back to vgAR (extendvg)
6. migrate back the LVs from hdisk3 to the disk

If you know that i'm in the wrong way, pls advise.. OR, if there should be something im not aware of..

Still need your advises..