Disaster Recovery 1

[dpsmith]

Hello, please don't laugh, AIX fool here. I do daily full backups and once a week system backups on our RS6000 F50 AIX server and it occurs to me that I don't know what do do in the event of a system crash.
No one in the company has ever done a fresh install!
It was setup 5 years ago and except for the recent OS upgrade we haven't touched it.
I use SMIT to back up to tape
The OS is on a mirrored drive and the data is on a RAID5.
I feel comfortable with the data , due to help I've recieved here!
But don't have a clue about the OS.
We have a software/hardware contract with IBM but I'd like to at least have it on paper what I'm supposed to do.
*But* I need it "dummified" with AIX commands and all that.

Can anyone point me in the right way?
I tried the IBM site for white papers but had very little luck.

Thanks in advance.

Dsmith

 

[dblair]

man makesysb

 

[KenCunningham]

Or even man mksysb

There are a few threads in this forun dealing with system recovery, try a keyword search for mksysb. Hope this helps.

 

[renaissancetek]

I am actually in the middle of a bare metal RS/6000 recovery and you are right this is the time to be sure all is working well.
The mksysb is a great start, but the only way you personally will know if this works is if you practise a restore and document your results. When I worked at a hospital years ago, they practiced 2 times each year do a bare bones restore with a few applications to confirm all worked ok. We all learned a lot from this.
The point is to practice, because you never know what you will discover or how much longer the process will be before you try.
I would suggest try to buy or rent / borrow a lower end computer they are not that expensive. The RS/6000 should be a simlar line machine, like a low end p series for that type or a F-30 or something like that if you are on that machine type. The reason is that the similar comptuers will restore with the same microcode otherwise it is more difficult to move from one to the other. You might even try to team up with another company to share the expence.
Remeber hardware failure is not an if it will happen but when, virtually all equipment does break at some point! Drives are mechanical they will wear out eventually.
Good luck, rememeber that being prepared is more than half the battle!!

Tony

 

[jtrisc]

As mentioned above, to successfully perform a recovery onto alternate hardware you will need to ensure that hardware is as close as possible a match to your existing hardware on which the mksysb was cut. Even then a mksysb recovery can fail. It is essential that you have your installation cd's available at the OSlevel of the mksysb.

Before I launch into the method, PLEASE PLEASE PLEASE do NOT try this on any production system and if your systems are business critical, consider taking out a DR contract with the likes of IBM or Sungard, or one of the independents suxh as Indigo Stone (

http://www.indigostone.com)

Practice this procedure as often as you can preferably on the hardware you would recover onto and if you upgrade the firmware on your production server remember to do the same to your recovery server,

Well here we go - sorry about the formatting but you should be able to tell the difference between the screen dumps and the instruction.
________________________________________________________

Rebuilding from the mksysb tape
q Locate the correct AIX install cd (ie equivalent level to the mksysb image) and place in the drive of the RS/6000
q Boot the system from the AIX cd. If the system is currently has no operating system loaded or you know that the first boot device is the cd rom then the system can be left to boot until the Welcome to the Base Operating System Installation and Maintenance screen is displayed.
q If you are not sure about the bootlist or operating system status then it is safest to follow the procedures below to point the first boot device at the cdrom.
Setting the bootlist

RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/600/6000RS/6000RS/6000RS/StartingSoftwareRS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6PleaseWaitRS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000RS/6000R

Memory Keyboard Network SCSI SPEAKER

When this window appears, as soon as the Keyboard comes up on the screen, press 1 (F1 if a graphics screen) about 5 times and wait until it takes you into the System Management Services.
Select Multiboot
Select Select boot devices
Select Configure 1st Boot Device
Select the device number for the cdrom
Exit completely from the menus until the boot is continuing
After 2-3 minutes, a prompt will be displayed on any attached terminals requesting a key to be hit to select that terminal to work from
After 1-2 minutes, the system will ask you to choose the language that you will use. English is the default choice. Press [Enter] to accept the default value.
You will then see the BOS Installation and Maintenance menu:

Installing from a mksysb

Welcome to Base Operating System
Installation and Maintenance

Type the number of your choice and press Enter. Choice indicated by >>>

>>>>1 Start Install Now with Default Settings
2 Change/Show Install Settings and Install
3 Start Maintenance Mode for System Recovery



88 Help?
99 Previous Menu
>>>Choice [1] : 3



1. Select the Start Maintenance Mode for System Recovery option.
2. Select the Install from a System Backup option.
3. The System Backup Installation and Setting screen is displayed. Selec the disk(s) wher you want to install rootvg.
4. Select the drive containing the backup tape and insert the mksysb tape. The system reads the media and begins the installation.
5. You will be prompted again for the BOS installation language, and the Welcome screen should be displayed. Continue with the Prompted Installation, as cloning is not supported in non-prompted installations.
6. Wait while the system restores from tape. This will take between 45 minutes to 2 hours depending on the amount of data.
7. After the mksysb installation completes, the installation program automatically installs additional devices and the kernel (uniprocessor or microprocessor) on your system using the original product media you booted from. Information is saved in BOS installation log files. To view BOS installation log files, enter cd /var/adm/ras and view the devinst.log file in this directory.
8. If the system looks hung on nimserver requests after an install, hit enter to get a login.
IF the root password is not available follow the instructions to break the root password

Breaking the root password
Reboot the system with the cd still in the drive and enter the maintenance mode (by hitting 5 at keyboard banner during boot ).

RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000
RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000 RS/6000


memory keyboard network scsi

Select "Start Maintenance Mode for System Recovery" from the displayed menu. Select the option to import rootvg and to mount rootvg filesystems.

Maintenance

Type the number of your choice and press Enter.

>>> 1 Access a Root Volume Group
2 Copy a System Dump to Removable Media
3 Access Advanced Maintenance Functions
4 Install from a System Backup

Select "Access a Root Volume Group".

Warning:

If you choose to access a root volume group, you will not be able to return
to the Base Operating System Installation menus without rebooting.


Type the number of your choice and press Enter.

0 Continue

Hit 0 and enter to continue through this warning

Access a Root Volume Group

Type the number for a volume group to display the logical volume information
and press Enter.

1) Volume Group 0040536a7be2eb8c contains these disks:
hdisk0 8678 10-60-00-8,0
2) Volume Group 0040536a68cbc793 contains these disks:
hdisk1 8678 10-60-00-9,0

Assuming hdisk0 is available select that

Volume Group Information

------------------------------------------------------------------------------
Volume Group ID 0040536a7be2eb8c includes the following logical volumes:

hd5 hd6 paging00 paging01 hd8 hd4
hd2 hd9var hd3 hd1 tftplv lv03
lv00
------------------------------------------------------------------------------



Type the number of your choice and press Enter.

1) Access this Volume Group and start a shell
2) Access this Volume Group and start a shell before mounting filesystems


Select "Access the Volume Group and start a shell". When this completes you have normal access to all your system files, and you are executing as root.

Importing Volume Group...
rootvg
Checking the / filesystem.
log redo processing for /dev/rhd4
syncpt record at 6fc028
end of log 73b30c
syncpt record at 6fc028
syncpt address 6fc028
number of log records = 2577
number of do blocks = 159
number of nodo blocks = 1
/dev/rhd4 (/): ** Unmounted cleanly - Check suppressed
Checking the /usr filesystem.
/dev/rhd2 (/usr): ** Unmounted cleanly - Check suppressed
Saving special files and device configuration information.
Mounting the / filesystem.
Mounting the /usr filesystem.
Checking and mounting the /tmp filesystem.
/dev/rhd3 (/tmp): ** Unmounted cleanly - Check suppressed
Checking and mounting the /var filesystem.
/dev/rhd9var (/var): ** Unmounted cleanly - Check suppressed
Files systems mounted for maintenance work.

You will be at a shell command prompt similar to the following. See what is mounted and act accordingly:
#
# df
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/ram0 163840 29192 83% 4390 11% /
/dev/cd0
df: /SPOT: No such file or directory
/dev/hd4 163840 29192 83% 4390 11% /
/dev/hd2 2555904 121648 96% 55924 18% /usr
/dev/hd3 786432 759888 4% 232 1% /tmp
/dev/hd9var 786432 507624 36% 6593 7% /var
# passwd
Changing password for "root"
root's New password:
Re-enter root's new password:
#
# pg /etc/security/passwd

root:
password =
lastupdate = 989932521
flags =
# shutdown -Fr

Notes: You may need to set TERM=lft and EXPORT TERM in order to use your display in full-screen mode.
The system should come up in a maintenance mode without requiring a password to be entered. Find the path at which the hd4 logical volume is mounted (the root filesystem, (/) could be either mounted from RAM or hd4). If hd4 is mounted on the root filesystem (/) then run the passwd command resetting root’s password. If hd4 is mounted on another directory (such as /tmproot) then change to /tmproot/etc/security and edit the password file, removing the text for the password.
If this is a HA’d node it would be useful to edit the /etc/hosts file as below at this point to stop the system hanging on reboot.


Note that if a reboot is required on a hacmp machine it will take a long time to come up due to the hacmp stuff not being removed and hanging on services not available.

Good Luck

JT

 

[dpsmith]

Wow!
Thanks "jtrisc" and "renaissancetek".
Lots to digest and figure out.

I'll check out the makesysb too as others suggested.

Maybe convince the $$guys to buy a lowend server to test.
EBay here I come.

Quick follow up to renaissancetek, you said to make sure I have the OS installation CD's available when doing the restore.

We recently upgraded to 5 from 4.
I'm a Microsoft flunky and didn't do the update myself.
Is the AIX upgrade a total replacement or do I need the CD's from 4 and then 5?

Or rather, once updated can I toss the AIX 4 CD's in a box and forget about them?

Thanks again.

Dpsmith

 

[mrn]

Never mind about the MS stuff, we forgive you.

You can just install 5 straight away, no need for the 4 cd's

-
| Mike Nixon
| Unix Admin
-------------