ESX server crashed

[amberlynn]

HELP!

We had some kind of power hiccup today. My virtual server (running ESX 3.0.2) was frozen. I rebooted it, and it displays several errors (e.g. 0:00:01:26:25 CPU1:1038 LVM:1358: One or more devices not found (file system [storage1, bunch of numbers]).

I can access it through the web interface, but only 2 of my virtual machines are showing - the rest say unknown1, unknown2 etc.
The two that are named correctly won't start - the error on their console is:
PXE-E53 No boot filename received
PXE-MOF Exiting Intel PXE ROM
Operating system not found.

How can I get this server back up and running ASAP?

Thanks,
A

 

[itsp1965]

Not knowing the full extent as to what happened to your server, it sounds like at minimum you may have a experienced a disk failure where your VMs reside. You might also be having issues with one of your CPUs (from your error messages). Have you checked the server itself for any issues (ie hardware diagnostics)?

 

[SimonDavies]

LVM stands for Logical Volume Manager, I would check that your Datastores are back up.

Simon

The real world is not about exam scores, it's about ability.

 

[amberlynn]

I ran the diagnostics (accessed by F10 on boot) - which ran diagnostics on CPUs and memory - test results were fine.

Are there other diagnostics I should run?

How do I check if the datastores are back up?

Thanks,
A

 

[amberlynn]

I've exported my logs - but can't really make any sense of these on my own.
I should mention - I do have backups - I just don't know how to go about getting them on this server!

 

[SimonDavies]

On the vcenter console (or if you are just connecting directly to the server) load up the page for the ESX Host, next you want to check the Datastores (they will all be in that area, I don't have it handy at the moment so I can't tell you for sure).

Simon

The real world is not about exam scores, it's about ability.

 

[amberlynn]

There is one datastore - and it contains 'folders' of each of the 2 visible virtual machines - but none for the other missing ones.
I'm not sure if these are good or not - but I do see them...

 

[ArizonaGeek]

Is your storage direct attached or are you on a SAN or NAS? Do any of your physical drives have amber lights to indicate they were bad? Have you tried to SSH into your host to see if you can see all the disks or to see if there are any errors? You should be able to SSH into the host, SU - for Root access then cd to /vmfs/volumes then do an ls to see what volumes are listed. You can CD to each of your volumes and do an ls to see what VMs are there.

What type of backups do you have? More importantly what are you using to back up your VM's? Or are you doing guest level back ups, like back ups of the virtual servers themselves?

Good luck ... hopefully I can be more help with a bit more info.

Cheers
Rob

The answer is always "PEBKAC!