All PCs mysteriously restarted in our environment...

[XRJoe]

Yesterday at 2:20pm about 90% of our PCs (along with several servers) mysteriously restarted with no messages or errors and nothing listed in Even Viewer. I know this is a shot in the dark but has anyone else ever experienced something like this?

A little about our environment:
- Over 4,000 PCs w/ Several hundred servers. Mostly Windows environment.
- Large Citrix Farm (presentation server 3.0)
- Most clients managed with Altiris
- Cisco network componants

Any suggestions would be appreciated. Thanks!

 

[LawnBoy]

Windows Update? It happened to Skype last patch tuesday.


"We must fall back upon the old axiom that when all other contingencies fail, whatever remains, however improbable, must be the truth." - Sherlock Holmes

 

[2ffat]

Were they on UPSes? Could it be a power glitch?


James P. Cottingham
-----------------------------------------
^{I'm number 1,229!
I'm number 1,229!}

 

[XRJoe]

It was a couple thousand PCs across several remote locations around the state so I do not believe it was a power glitch.
Users that experienced it described as if they were to shutdown their PC with all their applications open. The applications slowly closed with some prompting to save changes, and then they rebooted. There's nothing in the event logs that point to a cause.

 

[2ffat]

Were they running the same app, maybe some company owned program?



James P. Cottingham
-----------------------------------------
^{I'm number 1,229!
I'm number 1,229!}

 

[XRJoe]

I'm sure many of them were running the same app but not all. There were a couple servers that also rebooted, below is the event log from the restart on a Windows 2003 server:

Date: 08/22/07 Source: User32
Time: 2:20:41PM Category: None
Type Information EventID: 1074
User: NT AUTHORITY\SYSTEM
COMPUTER: <NAME>

The process winlogon.exe has initiated the restart of computer <NAME> on behalf of user NT AUTHORITY\SYSTEM for the following reason: No title for this reason could be found
Reason Code: 0x80030004
Shutdown Type: restart
Comment:

For more information, see Help and Support Center at

http://go.microsoft.com/fwlink/events.asp.

 

[2ffat]

Well, we're back to what LawnBoy mentioned. Is their MS update program set to download and install patches at the same time? If they are, maybe they all updated and rebooted at the same time.


James P. Cottingham
-----------------------------------------
^{I'm number 1,229!
I'm number 1,229!}

 

[XRJoe]

I think if it were any sort of update you would see something in Event Viewer. There are no other System logs between 1:17:53 PM and 2:20:41 PM when the Server, and the PCs, restarted.

We are treating this as an isolated incident as there have been no other issues since this. The environment is fairly large and complex so it could be something as simple as an application update that broadcast a shutdown, or it could be some sort of attack, or possibly an accident. Only concern is we haven't been able to figure out what caused the issue.

 

[lhuegele]

It could have been a disgruntled employee who knows how to issue a domain-wide command called "shutdown". You can use it to shutdown and reboot machines on a network. Looks like if that was the case, that they also knew how to hide their tracks in the event logs.

Have you had any lay-offs lately? Any disgruntled IT employees leave recently?

 

[rn4it]

Do you have an application like SMS or Aliris? these are application where you can schedule application or patch installs and of shutdown or reboots PC's remotely. If you do I'd check there as well.

 

[baddos]

Date: 08/22/07 Source: User32
Time: 2:20:41PM Category: None
Type Information EventID: 1074
User: NT AUTHORITY\SYSTEM
COMPUTER: <NAME>

The process winlogon.exe has initiated the restart of computer <NAME> on behalf of user NT AUTHORITY\SYSTEM for the following reason: No title for this reason could be found
Reason Code: 0x80030004
Shutdown Type: restart
Comment:

This is usually followed from a Windows Update or Automatic update install.

 

[LawnBoy]

<cough>


"We must fall back upon the old axiom that when all other contingencies fail, whatever remains, however improbable, must be the truth." - Sherlock Holmes

 

[crochetinggeek]

Have you checked your Altiris logs? I know Altiris is capable of something like this. Has it been doing auto discovery and installs of clients on machines? (I ask because you say most of the clients - if this had happened on all clients, the auto-discovery would explain why there are more of them)

Do you have a new person - or a person who has just been granted access rights to Altiris, and they've been playing around?

 

[normntwrk]

If it happens again you might look at this aspect:

If you have ever experienced your Windows Server 2003, or even a Windows 2000 or Windows XP computer rebooting automatically, or if you have received a "serious error" message or a blue screen of death, your computer may be infected with a Spyware.Service.MiscrosoftUpdate (Trojan)


The root cause of all these problems is typically a kernel driver that's installed by a couple of known rootkit spyware programs: msupd5.exe and reloadmedude.exe.

good article here :

http://mcpmag.com/columns/article.asp?EditorialsID=1329

Norm

 

[drew1701d]

What antivirus solution do you have? I know our enterprise McAfee product generates the messages you stated when it does a reboot to complete an antivirus engine upgrade. If you have the option not checked to prevent a reboot or prompt the user then that sounds like a possible culprit.

Also, is folder redirection in use on the workstations? If so did a server take a hit the hosts a dependent process that caused a cascade reboot to resync?

Did your active directory events reveal anything else?

"I'm certifiable, not certified. It just means my answers are from experience...not a book

 

[electronicsfreak]

For finding rootkits in case you have any.

http://www.microsoft.com/technet/sysinternals/utilities/RootkitRevealer.mspx

There is a point in wisdom and knowledge that when you reach it, you exceed what is considered possible - Jason Schoon

 

[2ffat]

See if this applies: thread779-1420859



James P. Cottingham
-----------------------------------------
^{I'm number 1,229!
I'm number 1,229!}