Hi Folks, A NIGHTMARE has happen 1

[KOG]

Hi Folks,

A NIGHTMARE has happened to me .. and it is probably every DBA's nightmare ... walked in work this morning to find there is an error message regarding with disk corruption. when I took over the DBA post, I was very concerned with the fact that ALL datafiles for the production instance is held on ONE disk and THAT disk corrupted ... with the new server we purchased (F50) I plan to spread it across several disk.

An engineer is coming on site this morning.

As a 'beginner' AIX, I want to check if I am right in thinking ... should I have done mksysb of all the volume groups i.e. root, oracle etc ... as I have not got around to it yet. Will that cause probs?

I have done physical and logical oracle backups. They are on different disks.

Is there anything I should do while waiting for an engineer to come on site, it could be later this afternoon as he need to order the new disk first.

Running around like mad !

Regards

Katherine

 

[MoshiachNow]

Hi,

1.mksysb can be used ONLY to create a root VG backup.

2.Is it a system (rootvg) disk that crashed ?

3.Does your DB reside on JFS filesystem or on a plain logical volumes? "Long live king Moshiach !"

 

[KOG]

Hi Levw,

Many thanks for your immediate response, it is very much appreciated. I am panicking around here as no one seems to know about oracle and AIX and it is so hard trying to work out what to do before an engineer arrives (not known yet it all depends on the availability of the disk ordered).

Now to answer your questions;

1. The corrupted disk is not rootvg, it is another volume group for oracle stuffs only. So should I do the mksysb now? ? OR wait till an engineer come on site?

2. The datafiles for our db is JFS? Is that good?

3. As for backups, done physical backups yesterday but it is on different disk - SHOULD I copy it onto tape now? OR Again wait for an engineer.

Many thanks for your help.

Regards

Katherine

 

[MoshiachNow]

HI,

1.mksysb will not help - as I said,it backs up rootvg only.

2.It's good .There are lots of things one can do to try and recover the corrupted filesystem.

Tried running "fsck -y /dev/lvx" on the problematic one ?

3.It's always prefereble to have a tape backup as well,I would make one.

See some notes on FS recovery:

7.2 Can disk be accessed ?
If the disk fails to import or refuses to be defined as a new disk - first try to access it on a physical level.

# dd if=/dev/hsdisk4 count=5 bs=32 of=/dev/null

This will verify that the disk is accessible phisically, and not totally dead.

# dd if=/dev/hdisk4 count=1 |od -x | grep 0000200

This will attempt to read the disk PVID = physical volume ID

7.3 Recovering a logical volume
Sometimes you will get messages regarding problems with logical volumes.
The following procedure rebuilds the bad LVCB - logical volume control block in ODM.

# getlvcb -AT slv4.3 ;views LV info
# putlvcb -N slv4.3 ;rebuilds LVCB
# synclvodm -v rootvg slv4.3 ;resyncs ODM

7.4 Copy a superblock
If you have a dirty superblock error message during mount or fsck -you might try to do “fsck”. If this does not work try the following (This procedure does not promise 100% success).
USE this procedure ONLY as a last resort if fsck fails !!!!!
(The following example relates to a bad filesystem in slv4.0)
Copy the original Superblock into a file (called sd0 in /tmp - places can be changed):

dd if=/dev/rslv4.0 of=/tmp/sb0 bs=4k count=1 skip=1

Note: if=Input File, of=Output file, bs=Block Size.
Copy the backup Superblock into a file (called sd1 in /tmp - places can be changed):

dd if=/dev/rslv4.0 of=/tmp/sb1 bs=4k count=1 skip=31

Copy the Backup Superblock file over the original Superblock:

dd if=/tmp/sb1 of=/dev/rslv4.0 bs=4k seek=1

Do “fsck” again on this filesystem
Note: If you want to restore the original Superblock, do:
dd if=/tmp/sb0 of=/dev/rslv4.0 bs=4k seek=1

Then we'll run fsck on this filesystem to ensure it's back.

7.5 Format jfslog
If the jfslog has been damaged for any reason, it will cause the filesystems to not be mounted.
Assume the filesystem /dataVolumes/proof7.2.0 fails to mount and fsck does not help.
Find out the name of the jfslog that belongs to this filesystem:

# lsvg -l svg2

svg2:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
slv2.0 jfs 1080 1080 2 open/syncd /dataVolumes/proof7.2.0
loglv02 jfslog 1 1 1 open/syncd N/A

we will reformat the jfslog. Enter:

# logform /dev/loglv02

This command will ask the horrible question "Do you wish to destroy the log?". Just say yes. This will get you a new, clean jfslog.
Then we'll run fsck on this filesystem to ensure it's back.

"Long live king Moshiach !"

 

[KOG]

That sounds like another language to me .. I think I will wait till an engineer comes on site and watch him ... many thanks for you help but I am a bit concerned about backing up to tape as thsi takes hours .. and he is due to arrive in about 30 mins .. so that would be too late.

Many thanks and Regards

Katherine

 

[mrn]

For future reference, use savevg to backup non rootvg volume groups, not a lot use now I know, but it's always good to learn from mistakes.

Mike --
| Mike Nixon
| Unix Admin
|

http://www.instantflashshop.com

----------------------------

 

[KOG]

Hello Mike,

Is it poss to use savevg command through smitty? If I wish to save the other three vgs onto rmt tape, can I do this through smitty?

Many thanks

Katherine

 

[pascaldy]

Just use 'smitty savevg'.