Urgent: Cannot logon to server console 2

[MarkLappin]

I'm grasping at straws at this point

2k3 SBS server w/ exchange, not SQL --- local and term serv console sessions froze up but server would still serv the users in all manners. Attempted remote shut down, failed, powered off manually, restarted, cannot logon to console or term serv in; can telnet in. Made sure workstation service was started, attempted restart, Symantec AV system wouldn't stop, booted to safemode and turned auto startup off for all the anti-virus stuff. On restart no change, turned services back on --- reboot, console takes about 30-40 minutes for logon screen and term serv login freezes at applying user settings, services are starting, no errors in event logs, exchange operating, av operating, shares, authentication, dns, all that stuff is working but we can't logon to the server itself


Any ideas here welcome, my AIM screen names are TNGPicard and TNGData --- if you have any ideas please put them here or grab me on AIM.

Mark L.

 

[NetIntruder]

what error are you getting trying to log in?

~Intruder~
CEH, MCSA/MCSE 2000/2003

"The Less You Do, The Less Can Go Wrong"

 

[MarkLappin]

There is no error, at the console it hangs at "applying computer settings" and no logon prompt, through terminal services (admin mode only) it hangs at "applying personal settings" --- does not error, just hangs.

 

[NetIntruder]

were there any recent policy changes? If safe mode works,you may want to try disabling the GPO from the OU this server is in and try to reboot again.

~Intruder~
CEH, MCSA/MCSE 2000/2003

"The Less You Do, The Less Can Go Wrong"

 

[MarkLappin]

No, no recent policy changes which have been reported to me -- i'm 2500 miles away right now talking to the person there and well, thats not going so well.


Since the last post I've gotten some new info;
--- there are no services hung in the starting state
--- The person on site CAN logon to the console, console logon takes about 2 hours though, most of the time it is hung/sitting at "applying personal settings" AFTER a very long hang during startup at "applying computer settings" -- users are still able to logon and are noticing no problems.

The other server (2003 non SBS running SQL 2000 sp3a) is not experiencing the same logon problems, the SQL agent process and retrospect service are hanging on startup; these affect our backups but we're going to reboto that server during lunch, we can force the backups and auto tasks to run so that is less critical.

It bothers us very much about the SBS/Exchange server however.

Mark

 

[porkchopexpress]

Can you remotly view the event logs using the event viewer on another station?

 

[captaincrunch00]

Did you just install Symantec Antivirus 10.0.359 or 10.0.100?
I had the exact same problem with both of those versions.

You'll have to get the SCSCleanwipe utility from Symantecs website to do a full uninstall of the program.


Check the services running on that computer remotely by using the MMC Console. You will probably find a service that is in the "starting" phase and not "started". You wont be able to stop it or disable it remotely (I wasn't), so reboot the server into safe mode and then change the service to Manual instead of Automatic. Then reboot and you should be able to log in.

 

[captaincrunch00]

Hit post too soon:

Then use the SCScleanwipe thing from the symantec site, wipe the program from your server, reboot it again. It should be fine and dandy to run now.

I have reinstalled version 9.0 of Symantec and it works much better.

Also with Version 10 it has made 2 of my servers bluescreen at random times during the night 4 nights in a row. No reason for it, nothing is running, no backups, no one is on the network, absolutely nothing. Its real fun to come into a crashed server monday morning.

 

[MarkLappin]

porkchopexpress: yes, nothing erroneous, just a lot of POP3 connector alerts from connection issues and host/rsctaxbusters.local errors (not sure what those are about -- been there for months).

captaincrunch00: No, using symantec 9.


I talked to two or three IT admins at other law firms who were having some problems very similar, they said their problems were due to a DDOS attack on servers with their terminal services ports directly exposed to the internet. We've blocked all but SSL for OWA and restarted the servers, waiting on them to let us login now to the console or terminal services now but at 20 minutes and waiting.......

Mark

 

[MarkLappin]

Howdy folks --- after a long day we've gotten this resolved! yay! A very long day; the below has been posted to a specialized mailing list for the law office system we use but the story and solution, symptoms and such are there.

mark



Evening/Afternoon,

Last night I got a call from the law firm I worked at and still contract with regarding Prolaw and their server systems. The problem seemed simple: could not logon to the server at the console or via Terminal Services. Several hours of troubleshooting last night with me 2,700 miles away and the person in the office having similar results.

I VPN'd into the office network and RDP'd to a user's workstation with the 2003 Admin tools installed (didn't need them just the normal XP ones would work), no event log errors, nothing really out of shape in services, I went ahead and disabled the Prolaw Exchange agent becuase there were a few hundred PID entries within about 1 minute and there were a number of items in the application log regarding the event sync and transactions or something so that seemed like a place to start; no go.

Next said ok, lets disable the AV systems and see, so hadthe person at the office boot the server to safe mode, disable the av on the server and reboot, still no go. AT this point I e-mailed a real expert (read true IT professional with MCSE style credentials and such over at Unif of Pacific where I wnet to school) to see if they'd go look this weekend on site becuase I was stumped. Left the servers attempting to boot but users could login.

When I say attempting to boot, the servers (a SBS 2k3 w/ Exchange and a 2003 Standard with MS-SQL SP3a) hung at "Applying computer settings", attempted login through TS/RDP and hung at "Applying personal settings". We could telnet into both servers and execute commands at the command line and had limited RPC connectivity from XP workstations connected to another computer. The servers would still "serv", users could login, access files and run prolaw. The SQL Server agent was stuck in a start state as well as the Retrospect Helper services (system backup software -

http://www.dantz.com).

This morning users logged in OK, although we still had the prolaw agent disabled (we have had a lot of problems with it). Still out of ideas and googling and searching on Tek-Tips.com (btw, it apears that people are having issues with Symanted AV corp v. 10 so be advised) I posted to a yahoo group dealing with Prolaw about if anybody else was having trouble and gave them my AOL screennames and MSN messenger account asing them to contact me, a few minutes later I got an e-mail back on on the way out the door I ahd the person call my cell phone while I drove to meet sombody for lunch. I spoke with somebody from: reinischmackenzielaw.com in Portland who was having the same problems.

His inital research indicated this may be a DDOS attack on port 3389 (RDP/TS) and to close the port from direct internet connectivity. Did that, restarted servers during lunch, same problem.

About an hour later, begin doing more Troubleshooting and discover that the APC agent and APC server on both servers is hung on start up. E-mailed the person I spoke with and he was on the phone with APC who was aware of the problem (though they still charged him $250 for the call) and he had been on the phone with Microsoft. The solution he had was: boot your server to safe mode, disable the APC services, reboot to normal, uninstall the APC systems and install the latest version (

http://www.apcc.com/tools/download/software_comp.cfm?sw_sku=SFPCBE704&id=125&TSK=).

At CLG we have 2 servers which take a very long time to shut down (especially with exchange) and with the state they were in we couldn't do a normal power down procedure, we had to hold power button in for 10 seconds (the person there) which is not the ideal way to do things at all!

As an alternative to that since I could get into the servers via Telnet and have access to the command line, we used the following two commands:

taskkill /IM apcserver.exe /f
taskkill /IM apcagent.exe /f

Immediately on doing these the pending logins for TS logged in, and the consoles presented thier logon screens. We promptly set the APC services in services manager to disabled and uninstalled the APC agents. We are waiting until tomorrow morning and going to reboot again just to ensure that this was the problem before we install the newer ones.

A possible alternative to this method and if you have control over your group policies would be to set the policy to disable the APC services if you can and then reboot and hope your servers goto update their policy -- I have nto tested this scenario, it just makes sense if you have a lot of servers and are using GPO's to help control your services states.

These problems started 48-72 hours ago after a "spontaneous reboot" as it was described to me by the person in the office; I suspect that the servers are set (not a setting I maintain on servers usually just for this reason) to auto update themselves from windows update and something required a restart which caused thei APC disruption.



Mark Lappin
Contractor to Calone Law Group, LLp in Stockton, CA

 

[toddman33]

Hi MarkLappin,

Thank you for your post, you helped me out. I also have Windows 2003 Server with APC and Symantec Antivirus 9.0.

I actually started experincing the slow logon right after I installed the critical updates from Windows. It would literally take 2 hours to display the log on screen but right before that it would display an Application error with IcePack.exe. IcePack.exe is the core component for Symantec Antivirus on the quaritine side, so I thought for sure it was Norton causing the slow logon. But after taking a chance and following your tip(booting to safe-mode and disabling the APC service) I was able to boot right into Windows with no lag time.

Since APC is the culprit in this problem, I was only able to find the website stating a critical update, but no descriptions on possible problems. It's looks as if once the java time is expired, your server will take forever to load windows if the APC service is still enabled. What a huge problem.

Also APC on their website states:
CRITICAL UPDATE REQUIRED PowerChute Business Edition - Customers Using 6.x Must Upgrade to 7.x due to Java Runtime Environment expiration

http://nam-en.apc.com/cgi-bin/nam_e...dp.php?ISOCountryCode=us&p_faqid=7202&p_olh=1

This is a longshot, but I think the expiration may be the whole issue and that's conflicting with Norton because now on Norton's site they are stating problems:

http://service1.symantec.com/SUPPOR...tec AntiVirus Corporate Edition&ver=9.0&tpre=

That article was published two days ago, so it's still a farely new problem. If this is causing Windows 2003 to take an extremely long time to start, I'd expect more activity in the forums.

Todd