Help Can't boot V440 solaris 9 server!

[McQueen]

Hello all,

I've have a problem with my solaris9 V440 server, I originally couldn't login I finally got in and found the all the sym links to in the /dev directory where all corrupted, I also check the /var/adm/messages file and it was sending out the following messages :-

[ID 879645 kern.notice] NOTICE: /: unexpected free inode 60096, run fsck(1M)

My root disk is mirrored using Solaris volume manager so I did there following

1) Tried to detach the sub mirror but this didn't work
I assume because the sym links in the /dev directory where all corrupted

2) init 0
3) boot cdrom -s
4) fsck -y -o f /dev/rdsk/c1t0d0s0
5) At this point I attached a tape drive and did a ufsdump
of the root disk as I don't have any backup if this server!
5 init 0
6) boot

When I tried to boot the system it just comes up with the same error and the server just hangs!

Do I need to run an fsck on the mirror device as well?.


Any help Please I'm getting desperate

Regards

Steve

 

[coffeysm]

Well if your drives are mirrored odds are the same problems will exist on both drives. Did you try doing a boot -rs from the ok prompt?

 

[McQueen]

hi Coffeysm

Thanks for your reply I tried the boot -rs still the system
just come up with the same errors:-



Rebooting with command: boot -rs
Boot device: /pci@1f,700000/scsi@2/disk@0,0 File and args: -rs
SunOS Release 5.9 Version Generic_117171-07 64-bit
Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed
WARNING: forceload of misc/md_sp failed
Hardware watchdog enabled
NOTICE: /: unexpected free inode 1384, run fsck(1M)
NOTICE: /: unexpected free inode 1384, run fsck(1M)
NOTICE: /: unexpected free inode 202, run fsck(1M)
NOTICE: /: unexpected free inode 202, run fsck(1M)
NOTICE: /: unexpected free inode 197, run fsck(1M)
NOTICE: /: unexpected free inode 1384, run fsck(1M)
NOTICE: /: unexpected free inode 1384, run fsck(1M)
NOTICE: /: unexpected free inode 1384, run fsck(1M)
NOTICE: /: unexpected free inode 1378, run fsck(1M)
NOTICE: /: unexpected free inode 1384, run fsck(1M)
NOTICE: /: unexpected free inode 1384, run fsck(1M)

I'm going to shutdown and boot to cdrom again and run another fsck do I need to do it on the mirrored drive as well, Or is there a way to bring the server up without the mirror drive?

Thanks for your help still desperate!

Regards

STeve

 

[cndcadams]

McQueen;

If you have a sunsolve account check out doc

76407

At the bottom is a link to a ufs fsck doc.

It makes reference to a boot -b which If you do not have a sunsolve account post back and we will see what I can do for you.

Also try the boot -b see if it gets you to maintenance mode.

Thanks

CA

 

[marrow]

I have had this on one one of our small V210 mirrored systems, as far as I can remember I didn't break any mirrors but ran fsck 3>4 times to clear the problem. I don't think it mattered if we selected via md or rdsk

 

[coffeysm]

Well if you boot off the cd-rom drive it will not have the drivers loaded for SDS. So, you will probably have to fsck both drives, instead what I would do is disable SDS and try to boot off one drive instead. You can always re-setup SDS for disk mirroring. I would also setup UFS logging if you have not done so.

boot cdrom -s
mount /dev/dsk/c0t0d0s0 /mmt
vi /mnt/etc/system - Remove rootdev: entry
vi /mnt/etc/vfstab - Change all /dev/md entries to original
reboot

 

[McQueen]

Sorry to get back to you so late and thanks for all your help in the end I did the following:-

1) Boot cdrom -s
2) mount /dev/dsk/c1t0d0s0 /a (Mount the root disk)
3) vi /a/etc/vfstab - Changing all the /dev/md entries to original
4) vi /a/etc/system - Romoved the rootdev: and the forced load entries
5) Renamed the following file and directories:-

/a/etc/lvm to /a/etc/OLD_lvm
/a/etc/rc2.d/S95svm.sync to /a/etc/rc2.d/OLD_S95svm.sync
/a/etc/rcS.d/S35svm.init to /a/etc/rcS.d/OLD_S35svm.init
/a/etc/init.d/svm.sync to /a/etc/init.d/OLD_svm.sync
/a/etc/niit.d/svm.init to /a/etc/niit.d/OLD_svm.init

6) init 0
7) boot - The system booted with the following errors

SunOS Release 5.9 Version Generic_117171-07 64-bit
Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hardware watchdog enabled
Configuring ATM interfaces:
configuring IPv4 interfaces: ce0.
Hostname: eservicesdev
The system is coming up. Please wait.
ifconfig: socket: lo0: Protocol wrong type for socket
ifconfig: socket: lo0: Protocol wrong type for socket
checking ufs filesystems
/dev/rdsk/c1t0d0s4: is logging.
mount: warning: - not found in /etc/vfstab
open of /dev/tcp failed: No such file or directory
open of /dev/tcp failed: No such file or directory
starting rpc services:Sep 1 11:34:24 rpcbind: netid tcp: device /dev/tcp does not exist
Sep 1 11:34:24 rpcbind: netid ticlts: device /dev/ticlts does not exist
Sep 1 11:34:24 rpcbind: netid ticotsord: device /dev/ticotsord does not exist
Sep 1 11:34:24 rpcbind: netid ticots: device /dev/ticots does not exist
Sep 1 11:34:24 rpcbind: /usr/sbin/rpcbind: found 4 errors with network configuration files. Exiting.
/usr/sbin/rpcbind: found 4 errors with network configuration files. Exiting.
rpcbind done.
Setting netmask of ce0 to 255.255.0.0
Setting default IPv4 interface for multicast: add net 224.0/4: gateway eservicesdev
/export/home0/swapspace/swap0: No such file or directory
syslog service starting.
savecore: open("/dev/dump"): No such file or directory
WARNING: autofs: lookupname: 2
automount: mount /net: No such file or directory
WARNING: autofs: lookupname: 2
automount: mount /home: No such file or directory
WARNING: autofs: lookupname: 2
automount: mount /xfn: No such file or directory
syslogd: /dev/sysmsg: No such file or directory
savecore: open("/dev/dump"): No such file or directory
dumpadm: failed to open /dev/dump: No such file or directory
open of /dev/eri failed: No such device or address
open of /dev/eri failed: No such device or address
open of /dev/eri failed: No such device or address
open of /dev/eri failed: No such device or address
open of /dev/eri failed: No such device or address
setsockopt: Bad file number
open of /dev/tcp failed: No such file or directory
open of /dev/tcp failed: No such file or directory
volume management starting.
Fri Sep 1 11:34:27 2006 fatal: unable to t_open "/dev/ticlts"

8) devfsadm - When logged as root
9) Rebooted Everything came up fine


Thanks for all your help Coffesym, cndcadams and Marrow, I
Couldn't of done it without your help. If it happens again
I'll just run the fsck on both mirrors, I was panicing becouse I had no backup but all's well now thanks once again.

I've attached a link to a pdf I found with helped also:-

http://ftp.support.veritas.com/pub/support/products/Bare_Metal_Restore_for_NBU/267856.pdf.

 

[KenCunningham]

Presumably you now have a backup too? Well done all, anyway!!

All I ask of you
Is make my wildest dreams come true

 

[coffeysm]

Glad to hear you got it up and running. You should have ran reboot -- -r, when you were off the cd-rom though. That would have avoided all those errors. There also looks like there might be a typo in your /etc/vfstab file. You might not want to use disk mirroring since you said you did not have a backup. The problem with mirroring is if you lose a file or have a misconfiguration then it exists on both drives. You might want to look into the dd command (takes awhile) or using the ufsdump/ufsrestore method for copying hard drives.

mount: warning: - not found in /etc/vfstab

 

[McQueen]

I did fix the typo in the /etc/vfstab file and I have done a few backups now. I will probabley go back to the disk mirroring after I've got a few weeks good backups. Unless you guys think it's more touble than it worth?.

 

[coffeysm]

Well if you have some good backups now and plan on backing up the server, then I would say stick with the mirroring. Less of a hassle then having to do the other two methods...

 

[tuka]

The V440 internal controller support hardware raid1 (mirror).
See the V440 Server Admin. guide for setup info. It is very easy to setup and manage.

The raidctl utility is used to setup the mirror.