Empty WINNT folder on ePO Server 1

[simonjcook]

We have had one ePO server go down completely and another exhibiting the same problems.
We were unable to log on to the first, rebooted it and received an NTOSKRNL.EXE missing or corrupt error.
We are now unable to log on to a second ePO server.
On investigation the WINNT folder is largely empty.

Can anyone throw any light on this please.


Regards

Simon J Cook

< Keyboard Error - Press F1 to continue >

 

[simonjcook]

OK, Spoke to NAI and found the cause...

The temp and tmp system variables are missing.
Reason currently unknown.

Windows defaults the temp folder to C:\Winnt.
The repository pull task downloads to the temp folder...
with me so far...
After the pull task it cleans up by deleting the contents of the temp folder.

Pop... windows folder empty... I can feel an ePO 3 patch coming on...

Regards

Simon J Cook

< Keyboard Error - Press F1 to continue >

 

[FlipFlop1974]

Hello!

We expirienced the same problem but we don't use ePO. We're only using McAfee AutoUpdate Architect and McAfee VirusScan 7.1.
The behavior seems to be similar to yours, except the TEMP and TMP variables. There are correct and both are relating to C:\TEMP. Even so I can find from time to time NAI relating temp folders in C:\WINNT named naixxx.tmp (where xxx - a random number is be used by the software).
And at a time the systems performance is getting worse and worse and the C:\WINNT folder shranks down to about 50 MB. A reboot results in the message "NTOSKRNL.EXE NOT FOUND".

Is this also a NAI Problem? I can't find any stuff in their KB. Also I was not able to find a setting for the path to a temp folder.

Any ideas?

regards, mike

 

[simonjcook]

Hi there,

Yes, after investigation you are right the TEMP and TMP variables were present.

NAI's ascertion they were missing was wrong.
Those variables are stored in the user section of the registry and not HKLM.

I have not had any further feedback from NAI regarding this issue.

Regards

Simon J Cook

< Keyboard Error - Press F1 to continue >

 

[simonjcook]

Hi again,

Mike, McAfee AutoUpdate Architect has been integrated into ePO 3.

As this behaviour occurs under ePO 3 I am pretty certain the problem relates back to MAA also.

The files left in the windows directory were those that were locked at the time of deletion.


Regards

Simon J Cook

< Keyboard Error - Press F1 to continue >

 

[FlipFlop1974]

Yes, yor're right. Only locked files are still remaining in the winnt directory.
Is there any workaround? Is there any registry key or config file such SiteList.mgr where the temp path can be configured? It's quite funny because we rolled out the MAA on 70 Serevrs located over the world. They're running many services such Mail, SQL and so on... an than there comes MAA and crashes down the machine from time to time?

However, many thank for your reply!

regards, mike

 

[simonjcook]

Hi there,

This issue started occuring on one out of three ePO servers after upgrading to ePO 3 SP2a Patch 4.

As it was quite serious we rolled back to SP2a without any patches.

Unfortunately vanilla SP2a causes high CPU usage for Naimserv.exe on our two main ePO servers.

Stuck between a rock and a hard place we upgraded the two main servers the Patch 3.

The CPU usage dropped to reasonable levels.
Unfortunately now Naimserv.exe is crashing periodically on those two servers which has been logged with NAI as our top priority.

Because I am not prepared to upgrade the remaining server to patch 3 where I am sure it will start deleting the contents of the WINNT folder again, NAI have closed the call.

Patch 3 and 4 are very similar with Patch 4 including an additional security fix. The core is the same.

Sorry it took a long time to get to the point...

Until NAI address our Naimserv.exe stability issue with our two main servers we will not be putting the third server in a situation where the deletion issue can re-occur.

So unfortunately no solution at present...

Regards

Simon J Cook

< Keyboard Error - Press F1 to continue >

 

[FlipFlop1974]

Seems to be not that easy...
I'm currently trying to catch some technician at NAI. I've been put on hold since 1 hour (does someone have troubles too ;-)
If I can get further information I'll post it here!

Thanks so far!

regards,
mike

 

[simonjcook]

OK further information on this one,

We have had another ePO server go down with exactly the same issue overnight.

After another conversation with NAI...

This apparently *is* related to the TEMP and TMP path variables.
It *is* confirmed as an issue... no patch yet.

But we do now have a solution... Yippee.
The TEMP and TMP variables are being lost somehow, reason still unkown.
In our experience, on operational systems the TMP and TEMP variables are present in the User environment variables.
The suggestion from NAI is to add TEMP and TMP to the System Variables.
In addition to this NAI have stated that the "everyone" group needs "Full Control" to the TEMP directory.

I am not a big fan of the everyone group nor full control for non admin accounts
After thrashing this one out, the everyone group needs read, write, execute, delete and owner.

Quite why the everyone group needs to be used I am not sure.
I would hazard a guess that another group would be more suitable.

However for a quick fix I have implemented their suggestion.

We'll see...

Regards

Simon J Cook

< Keyboard Error - Press F1 to continue >

 

[JimmyZ1]

That seems kind of crazy, but what isn't with EPO.... good luck and thanks for the post..



Doh!!

 

[FlipFlop1974]

Hi there again and good day to all especially to Simon J Cook ;-) !

After waiting about 1,5 hours on the support hotline from NAI I was able to get a technician to the line. I told him exactly the issue about the deleted files, the produced temp files in C:\winnt\ and the behavior about the whole cleaning of the suggested "temp" folder from Nai products. He had no satisfied answer for me at this time, but hours later I got a surprising mail from NAI, and guess.. I got the same solution except the everyone permission.

So fact is that if you use ePO, MAA, or any other NAI products be aware that you can get in big troubles if you don't add TEMP and TMP to your system variables ;-)

What I found out on our Test server, maybe it's useful for someone:
I turned on auditing with the delete option on the c:\winnt folder and could get this information: The delete task was started from the system account. Our MAA is scheduled at 01:00:00AM and now guess when the delete process has started????
Tadaaaa, exactly at 01:05:24AM !

Small workaround if your server hast lost it's system files:
Copy all *.exe files from C:\winnt and C:\winnt\system32 from a similar server to their original destination via network. Now you should be able to log on again. After that open the task manager start your Backup software and try to restore the entire C:\winnt folder --> reboot and pray

regards, mike

 

[n1koolkat]

I have'nt yet experienced this problem although i have experience every other problem that can be imagined epolicy including the registry settings for network associates disappearing, anyway i am trying to preempt this particular problem.

My system variables have set C:\winnt\Tmp and C:\winnt\Tmp as the temporary folders. Are you suggesting that i change the temprary folders from the above to c:\Temp or are the current settings sufficient.

Thanks

 

[FlipFlop1974]

The suggestion from NAI is to put both variables into the system variables.
Don't ask me why, it seems that the program searches for both entries. If neither of them exist the C:\winnt path is used to store the temp files.
All in all we'll set both variables on every server to go sure. So we have the TEMP pointing to C:\winnt\temp and the TMP also pointing to C:\winnt\temp. So the program can clean up the whole folder every minute if it feels funny about it ;-)

regards mike

 

[simonjcook]

Hi there,

The key is not to rely on the variables TEMP and TMP being in the User Environment section.
They should be applied to the System Variables (Global for all users).
The actual path is not significant.
The everyone group requires full access to the folder.

Since we implemented these suggestions no WINNT folders have emptied themselves to date...

Under NT4 it is not immediately obvious how to apply variables to the System Variables pane.

Right click on the My Computer Icon on the desktop.
Choose properties from the popup menu.
The System Properties form will be displayed.
Select the Environment tab.
Click any line in the System Variables list view.
Scroll down until you see a blank line.
Click on the blank line.
Enter the Variable name TEMP.
Enter the required path in the Value text box.
Click on the Set button to apply the variable.
Repeat for the TMP variable.

It is recommended that the Everyone security group has full control to the temp folder.
I am pretty sure this can be cut down to Change and Owner permissions.

Regards

Simon J Cook

< Keyboard Error - Press F1 to continue >

 

[FlipFlop1974]

Hi there again!

Here's my last conclusion about this topic. Since this issue is now few weeks ago, I can definitly say that the creation of the system variables temp and tmp are the solution. Since we have implemented those variables, all our servers (around 70) are still up and running.

I think now it's time to spend some weeks in greece!
Have a nice day!

so long, mike