I agree with sleipnir214, X on a server is only asking for trouble, staying with a simple box is the best way to go, install only what you need for the box to serve, you will see better performance and have less security concerns.
If, like said before, you are intending this question for a workstation machine, I have to disagree with the KDE suggestion, I happen to like the look/feel of GNOME 2.2.x much better than KDE, I think it is much more polished and UNIX-like than the KDE, however KDE is much better if you intend to build it yourself as you only need three source packages and it builds rather quickly, still giving you a full blown desktop. Both perform well though in my opinion and neither give a faster desktop on my system.
-Bryan