Dell PowerEdge 2650 REBOOTS AT WILL 3

[oakuniv]

Hi - I recently purchased 2 Dell PowerEdge 2650 servers.
They have 2 dual 2.4 GHZ processors, 2 GB of RAM, 5 36 GB 15K RPM Harddrives in a RAID 5 configuration.

It is running as an Apache Webserver - running an application called WebCT. We also use a software product called DoubleTake to mirror our install of WebCT to another identical server. I use the Gigabit network card with a cross over cable to send the double take data to the other server.

Anyway - here is the problem. The server reboots itself at random times. 2 to 3 times per week. The last software I installed before the reboot problem appeared was Windows Service Pack 3. I have since uninstalled service pack 3 but the reboot problem continues.

Using Dell Openmanage software I have run diagnostics and found no problems. I have also taken the server down and re-seeded all the components. But it still re-boots.

The only error that I get once the server has rebooted is that the server restarted unexpectedly. I haven't used the Failover capabilities of the DoubleTake software yet - because I believe the other server may have the same problem.

The first time these servers rebooted - one of our other server admins tried to access each of the machines with terminal services. When he did that each of the machines rebooted as he tried to connect to them. This was several days after we had already gone into production with these machines.

At one point I was rebooting the server myself and I did get a blue screen but I wasn't able to read it fast enough before the system re-booted. I know there is an option to not allow the system to re-boot so that it just hangs on the blue screen. . .and that might be the last thing we try. . .I can't afford to have the systems down for long and it is so hard to know when they might re-boot.

Any help would be appreciated. Thanks in advance.

 

[galileo3000]

If the server is rebooting randomly, sounds like a hardware problem. Nothing in your software combination seems out of the ordinary. I would swap parts with the other 2650 to isolate the problem. The 2650 has hardly any wires so swapping parts should be pretty easy. Because of "ChipKill" it is unlikely that the memory is at fault. And a faulty processor would not allow you build anything. I am betting on the integrated video or nic. Hope this helps.

 

[dusme]

What's your backup software? We recently narrowed intermittent spontaneous reboots down to our backup software on poweredge 2550.

 

[oakuniv]

We are using Veritas Backup Exec. . .we have been using it for a while. . .on 2550's too. .

 

[dusme]

See if unchecking 'write checksums to media' in Veritas will resolve the spontaneous reboots and if it does; it's an issue of how checksums are handled and you may need to replace your motherboard.

 

[JasR]

Just a way out possibility - we had NT4 and Win2K boxes rebooting when shares on them were accessed from Macintosh OSX machines. Mac Xs can (and do) create files with null characters in the filename. Win2K doesn't know what to do with these files and - yes - reboots.

 

[rgpjr]

I had the same problem- My Dell Poweredge would reboot almost everyday if the backup was still running. I too used Veritas 8.5 and believe the configuration had something to do with it. I just reconfigured and it's been fine ever since.

 

[oakuniv]

Since my system is in production I have not been able to take it down long enough to rebuild it. . .however I found a short term fix to my problem. By leaving the system logged in with a user account it has not re-booted. But if I lock the system or log out. . .it will reboot at some point in time.

I am going to be rebuilding this system at the end of the sememster. . .I think that Dells Servers are really sensitive to how they are set up. . .I didn't use the Dell Setup disks when I set these systems up. . .I just installed off the windows disk. . .I will use the Dell Setup disks when I rebuild it.

Thanks.

 

[umlguy]

I too am having random reboots of my dual processor PowerEdge 2200. This only seems to happen when I boot with both processors active (i.e. Linux-smb, or Windows Server 2000). If I boot with only a single CPU active in Linux, I don't seem to have problems. I have swapped the CPUs with the same results.

This is a new machine for me, a hand me down if you will. I am not running any backup processes, not much else either since I am still trying to get it going right to start with.

Any help here would be great.

Current config:
128MB
2 PII-333 CPUs
17GB disk
using integrated video
3Com Etherlink III
Mandrake Linux 9.1

 

[jkupski]

>By leaving the system logged in with a user account
>it has not re-booted. But if I lock the system
>or log out. . .it will reboot at some point in time.

Are you in a Domain environment running Software Update Services?

 

[oakuniv]

In September of 2002 I was not using SUS. However the server does now have SUS configured. . .but is not set to reboot.

So I have returned to this page. . .after a year and several months. . .last time I was here I had a reboot problem for a 9 month period and it was eventually resolved by reinstalling the entire server and installing the newest BIOS and FIRMWARE updates. . .

Then I purchased a 3rd POWEREDGE 2650. . .and I made sure it was totally up to date with the latest of all updates. Now it seems to be doing the same thing as was going on with my other two servers last year.

It has rebooted several times when I logged on with Terminal Services and then serveral times just in the middle of the night when no one was logged in. There is one Bios Update that I see is available to be applied, but there isn't anything else to update it with

 

[PineGar]

I was having this problem with a PE 2600 system. Working with Dell no hardware problems were found. I am using Veritas 9.0 Build 4454. When I contacted Veritas they stated that they didn't support tape drives installed on a RAID controller (PERC 4/di installed on my server). They suggested that I installed a non-RAID SCSI card and put the tape drive on that. I did one better and moved the drive to a different machine. So far no reboots, but it's only been out of the server for 2 days. My reboots were happening about ever 5-6 days. I'll post again in about a week and let you all know if the problem went away.

 

[PineGar]

It's been 9 days since I removed the tape drive from the PERC 4/di and I haven't had a reboot since. Has anyone else attempted this and solved their issue?

 

[brs154]

I have been having this problem with a PE 2550 for about 1 year now. It is RAID 5 running Win2K SP4 and Oracle 9i. (and a Veritas Backup Exec Agent, no tape drive directly attached) I have upgraded and updated pretty much everything possible. When I went to SP4 it seemed to be fine for 3 months, now it is back at it again. There used to be a thread on Dell's web that had approx. 500 posts of people complaining about the same issue. I can no longer find it. The only abnormality I've noticed is that I get a message to condition the PERC3/Di Controller's battery much more often than I think would be necessary.
I hope someone can resolve this issue.

 

[technome]

Try shutting down Veritas' leaking drivers once a day.
(The leak can be seen in task manager over a >24 hour period, just watch the drivers slowly grow. This does not affect all machines.). This problems been around for at least 3 years.

From the CD, copy the \util directory to your machine, there are two files, bestop.bat and bestart.bat you need.
In the "Services" console, change all the Veritas services to Manual, schedule bestop and bestart to shutdown and restart the services once a day.

Also even if you are not using the remote agent, place the the "priority" to the lowest setting.

Don't waste your money calling Veritas on this problem, they want your money but their support really sucks. Basically they read through the same info you can get at the site, and charge you, with attitude. There are some excellent techs there, but they are in the upper tier tech area. Last time I needed help I told them I would do a charge back ( breach of contract) if I was not bumped up the tech heirarchy; with a bit of shock and background discussion I was bumped up.

 

[MJewell]

One other suggextion would to Right Click on My Computer, select Properties, Select the Advanced Tab and click on Startup & Recovery, uncheck 'Automatically restart'

However when the server crashes it will not restart, but when you come in in the morning, you will be able to see the error message....

-Mike

 

[xmsre]

gflags and poolmon otter help you track down which device driver is leaking. If you prefer a gui, try pooltag

http://www.osronline.com/article.cfm?id=98

 

[Jrak]

Could it be that your UPS battery is low enough to cause your computer to reboot when ever there are any power surges?
This happened to our server, it would be rebooted when we came in and there was nothing unusual in the event viewer that indicated any problem.
Our UPS battery light was showing green even though there wasn't enough power to protect it from power spikes and surges.
Just a thought.

 

[technome]

Jrak, curious, what make of UPS were you having this happen with. This might be more prevalent than most techs realize.

I have been using APC units and had some odd shutdowns during the grid power outages last summer (even with full charge, as shown by the LEDs).

 

[brs154]

"One other suggextion would to Right Click on My Computer, select Properties, Select the Advanced Tab and click on Startup & Recovery, uncheck 'Automatically restart"

I've tried this, and when our machine reboots it comes back up to logon.