Citrix Server randomly unresponsive/extremely slow

[ascarpella]

Hi -

I've got two Citrix servers in a farm, running PS 4.0, updated as of 9/1/07. Both servers run CA eTrust Antivirus. Both servers are identical and load balanced, users mainly run the Desktop, but some do run published office apps. We have a mix of thin terminals and PCs. One of the servers is randomly becoming unresponsive and the users' Citrix desktops will "lock up". If I shake the mouse on the server, it will come out of screen-saver after 2-3 minutes. It'll then take another 3-4 minutes to log in. Hit Ctrl+Alt+Del, another 2-3 minutes. I haven't been able to leave it sit long enough to bring up the task manager, I usually kill it after 15 minutes or so as I need to get users back in. I've tried leaving it (Task Manager) open and sorted by processor time, but it's minimized when I get logged back in and it just won't come up. Users are eventually disconnected, at some point during this slowdown, so I just hard boot the server to get it back up. There are no errors in the Event Viewer. I assumed I had some bad RAM, but it's all been replaced one stick at a time and the issue still occurs.

If someone could point me in the right direction to look it'd be much appreciated.

Thanks!

 

[enigma99]

ascarpella,
It sounds like you have a run away process thats taking up all of your CPU cycles.
Do you have RM configured in your farm?

If so, I would start looking at the summary reports around the time it started to go down.
It will show you the users logged in and give you a list of processes running at the time and CPU %.

I would look at some CPU clamping software that will prevent runaway processes on your Citrix server.

Hope that helps.

 

[ascarpella]

enigma99 -

Thanks for the advice. I don't have RM configured because we're running PS 4.0 Advanced. Should have mentioned that in the original post.

We had a trial of EdgeSight a little while back and I'm sure that would've proven beneficial now, but man, it's expensive.

Using your suggestion for CPU clamping software, I'm looking at installing ThreadMaster (

http://threadmaster.tripod.com/).

This looks like a good, free package to do the job.

Anyone have any experience with this? I'll hit the search button...

 

[Dublin73]

how many users are logged in when the server freezes? what sort of hardware is on the servers.... CPU, RAM etc.?

running task manager and looking at the CPU, before the server freezes, should give you some good information on the problem

 

[ascarpella]

There are 25-35 users on the server at any given time throughout the day.

The servers are matching HP Proliant ML350's with Server 2003, 2x Intel Xeon 3.06 GHz processors, 4GB RAM, and 3 pairs of 36.4 GB SCSI drives running in RAID 1+0.

This issue used to happen at least once a week, but after updating PS to the same rollups on both machines, running the Proliant driver/software update package, and updating to 2003 SP 2 from SP 1, it cut down to once a month.

Happened twice in a 24 hour period this week so I'm a little concerned.

I've tried leaving the Task Manager up so I can catch the offending process, but due to the random nature I haven't been able to catch it. I'm going to disable the screensaver and hopefully I'll be able to see it the next time it happens.

I'm also looking into setting up the user-initiated memory dump on the server, so hopefully I can get a dump while it's slow.

Good times...

 

[bran2235]

I've had this happen to me, and as Enigma says- probably a runaway process. Are you running the HP Foundation Agents? I ran those for a while and finally realized we didn't even use them (didn't need to)- Try stopping these services to see if you get any performance gain. I would also disable any virus software to check and see if you get any better performance-

My two-cents,
Brandon

 

[Dublin73]

when you've got a full load on the server, 35 users, how much RAM and CPU are being utilized? slowness like this is usually caused when there's no more RAM or CPU resources available.

Does this always happen when there's 35 users on the server, or can it happen with only around 20 users or so?

you don't need to wait for the server to crash to start investigating CPU and RAM usage through task manager

 

[ascarpella]

It happens with around 30-35 users. I'm basing that number on the fact taht that's how many users are in the other server and they're load balanced. CPU runs around 60% and RAM usage is at 3GB of the 4GB installed.