replacing rootvg/hd

[Guest_imported]

i've tried things but i want simpler and secure way of doing it right:

i have a bad hdisk0 (rootvg), it's making sound. as per my initial observation, it seems to me that the /tmp & /var FS are corrupted.

i tried using "mksysb" to backup the system so that i can restore to a new HD - but it failed. it gives me error: "backup completed with errors"

i successfully backup hdisk2 (datavg)using "savevg", but when i tried it for hdisk0 (rootvg), i got the same error message.

i also tried backing up the entire system using tar but when it reaches the corrupted portion, it display something like &quot;tar <file/directories> error&quot;, but continue to backup other files/directories.

i can re-boot the system and so far - it's up and running. the only thing i want for this system is to replace the hdisk0 or possibly hdisk1 as well. i don't want to wait all my data suffer because of defective hard disks.

i would appreciate any idea to solve this particular concern.

 

[vickersdc]

If you have another disk, extend the rootvg to include it, then use the migratepv command to copy the data across.

Unfortunately, it's not quite as simple as that because you also have to recreate the paging space and install the blv as well.

I can get you the details if you need them, but plenty of resources on the web (and elsewhere) detail the steps to take.

Hope it helps..?
Cheers,
Dave Vickers.

 

[Guest_imported]

Dave Vickers,

i'm desperate... can you give me the details on how to do it?

thank you.

 

[bi]

johny,

check what the errors are on the mksysb. If you have a file opened during the mksysb process and then close it, mksysb will tell you it completed with errors because it couldn't find the temporary file that was open during the mksysb process.

Try making a mksysb again and watch the errors that come out. If it is a temp file that is caused by someone vi'ing a file, you probably aren't going to need that file to bring the system back.

Have you brought the system down to service mode and tried fsck'ing /tmp and /var? If you haven't, I'd try that first, because it seems to me if the filesystems are corrupted, migrating the pv (or mirroring the volume group) is just going to migrate or mirror the corruption.

Have you reconstructed the /tmp or /var filesystems recently? If you have, check the permissions on them. Because these are created when you install a system, you rarely pay attention to the permissions on them because you don't have to build them. /tmp is owned by bin:bin and is wide open so everyone has read/write/execute permissions. /var is owned by root:system and its permissions are 755.

Here are the steps I have in my AIX Sys Admin course book. I would recommend having the mksysb before you do this anyway. Following these steps, I've listed what I've done to mirror rootvg, which is another way you can accomplish this.

1. Connect the new disk to the system.
2. Add the new disk (hdiskY) to rootvg.
3. Does the bad disk contain hd5? use these commands: migratepv -l hd5 hdiskX hdiskY
bosboot -ad /dev/hdiskY
mkboot -cd /dev/hdiskX (This clobbers the blv on the bad disk. This worries me, personnally.)
bootlist -m normal hdiskY hdiskX cd0 ...(etc)
[Does the bad disk contain the primary dump device? If so type the following:
sysdumpdev -p /dev/sysdumpnull]
migratepv hdiskX hdiskY
[Did the old disk contain the primary dumpdevice? If so, type the following:
sysdumpdev -v /dev/hdiskX]
4. remove the old disk from the volume group.
5. Remove the old disk from the system (with the rmdev command).

Here is how I'd do it with mirroring. This allows you to test it before you get rid of the old disk. This also assumes that all of rootvg is on one disk only. I used this on a 4.3.3 system.

1. Connect the new disk to system.
2, Add the new disk to rootvg. (extendvg -f rootvg hdiskX)
3. Mirror the vg: mirrorvg rootvg
4. Execute bosboot: bosboot -a
5. Update bootlist as follows: bootlist -m normal hdiskX hdiskY cd0 (...etc.)
bootlist -m service hdiskX hdiskY cd0 (...etc.)
6. Boot the system (shtudown -r). Watch as the system comes up to make sure it is booting off the new disk.
7. When the system comes back up and everything checks out, you can unmirror rootvg as follows:

unmirrorvg -c 1 rootvg hdiskY (the disk you will be getting rid of)

8. Follow the instructions on the screen regarding the hd5 lv and type the following: chpv -c hdiskY

9. Redo bosboot so only the new disk is on record: bosboot -a.

10. Change your bootlist: bootlist -m normal hdiskX cd0 (...etc)
and
bootlist -m service hdisk0 cd0 (...etc)
11. Remove the disk from rootvg and remove the disk from the system (rmdev).

Good luck and let us know what happens.

 

[bi]

CORRECTION JOHNY:

In step 3 of the migrate process, DON'T include hdiskY in your bootlist for either normal or service mode, as you clobbered blv in the previous step! (My error, not my Sys admin course book!)

 

[vickersdc]

I'd just like to add a couple of points here...

1. If you are considering mirroring the rootvg you need to ensure that the correct APAR's are in place. Not knowing which version of AIX you are using, please consider the following...

- in order for mksysb to work correctly when restoring a mirrored volume, you need fix IX56564, which IS built in to AIX4.2 and 4.3, but needs to be installed in 4.1 - check with:

instfix -ik IX56564

- AIX 4.2 requires fix IX68483 in order to boot correctly (it's not needed for 4.1 and is built into 4.3).

- you may require IX61186 for bosboot to work properly for 4.1, or IX62417 for 4.2. Nothing is needed for 4.3.

- there are also fixes for device driver mirroring: 4.1 IX60521; 4.2 IX70884; 4.3 none required.

- fixes for mirrorvg / unmirrorvg: 4.1 not required; 4.2 IX72058; 4.3 IX72550

Remember you can check for all these using 'instfix -ik IXnnnnn'

Hope it helps.
Dave V.

 

[Guest_imported]

bi & dave v,

actually, it is related to thread52-137341. kindly check.

the system has:

* LVs from lv00 -> lv08
* aside from default Filesystems (/,/usr,/tmp,/var,/home,etc)

i tried:

1. shutdown -m
2. df -k (shows /,/usr,/var only)
3. umount /var (can't umount / & /usr)
4. fsck -y /dev/hd4 & /dev/hd2 -> shows same below messages:

** Checking /dev/hd4 (/) MOUNTED FILE SYSTEM; WRITING SUPPRESSED;
Checking a mounted filesystem does not produce dependable results.
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Inode Map
** Phase 6 - Check Block Map
1006 files 52208 blocks 5136 free

5. fsck -y /dev/hd9var
6. fsck -y /dev/hd3
7. fsck -y /dev/hd1
8. fsck -y /dev/lvXX -> from 00 to 08 (do i need to fsck all FSs?)
9. type &quot;exit&quot;
10. confirm run level (opted for 2 = multiuser)
* system re-mounted FSs
11. sync;sync;sync;shutdown -Fr
12. after successful re-boot, i logged in as root
13. mksysb -i /dev/rmt0
14. i got this error:

Creating info file....
Creating tape boot image...
Creating list of files to back up...
Backing up 32801 files...
176 of 32801 files (05)....../usr/bin/mksysb[801]:23894 Bus error(coredump)

0512-005 mksysb:Backup completed
The backup command completed with errors.
The messages displayed on Standard Error contained additional information.

----end----

will this be a good enough or can i use this mksysb backup?

 

[bi]

After you got the bus error, did you continue to see mksysb backing up files? Did the number of files progress to include all of the files?

 

[bi]

And also, re your question #8: You probably don't have to fsck all the lvs, but I would. And did you check hd4 (/usr)?

 

[Guest_imported]

bi,

i think you're the only one with me now. thank you for standing by.

re question/queries:

#1. are you refering to smit.log? if you are - YES! i checked, it's the same error message. also, errpt -a shows Label : DISK_ERR1 (see Thread52-137341)

#2. no - the number of files didn't progress a bit [xxx of xxxxx files (0%).....] and it's looping back (i waited for an hour and it just keep on cycling).

#3. i did - fsck'd with -y option all FSs, just to make sure.

 

[vickersdc]

I'm still here - just catching up with what's going on!!



Dave V.

 

[vickersdc]

Can I just check that the SCSI bus is terminated properly..?

Are you still wanting to / have you already mirrored the rootvg..?

Cheers,
Dave V.

 

[Guest_imported]

i would like to mirror the rootvg if it's going to be the last resort but for now i want to try things that will fix a corrupted filesystems as simple as possible.

unfortunately, i got a new error message:


Creating information file (/image.data) for rootvg...

Creating tape boot image........

Creating list of files to back up.....
Backing up 32805 files..............................
176 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)..............................
177 of 32805 files (0%)...............^C.

*** i have to press control-C to force to stop it as it loops back...nonstop for more than an hour***

any idea?

can u possibly umount / & /usr under maint mode? i tried and i failed.

desperately seeking help - johny2001

 

[vickersdc]

You can only unmount / & /usr if you boot into maintenance mode either from CD or tape. If you try to boot into maint mode from the hdisk it is automatically busy, and therefore canot be unmounted.

 

[bi]

johny,

I don't think you are going to get a good mksysb tape, whether because of the disk or your tape drive.

I would recommend mirroring now and then breaking the mirror. Don't forget to do your bosboot command and change your bootlist to use the new drive rather than the old one.