It is known that some device drivers can cause a terminal server to crash. Check that you have the latest drivers for all of your server's hardware, and that all your printer drivers are fully compatible.
Voyager1's suggestion of examining the crash dump files is the best way to find the root cause. The last few, in the list of drivers given, usually indicate where the problem lies.
You will not get a crash dump file if your pagefile is either not partially on the system partition, or at least equal to the amount of RAM in your system + 12Mb, ie if your pagefile is split over multiple partitions, the portion on your system partition must be equal to that formula.
I hope this information is helpful.