DNS could cause the delay if it isn't set up correctly and/or someone has made a change in networking and hasn't told you about it. To test if it is DNS, try telneting to the IP address rather than the name. If you don't get the delay using the IP, then you know it is DNS. You can either try to correct the problem or use dbase77's solution.
However, if you just have
hosts: files
in your /etc/nsswitch.conf file, be sure to add the host you are telnetting to to your /etc/hosts file, or you won't be able to use the server name. I would recommend
hosts: files dns
so that if the system can't find the name in /etc/hosts, it will try DNS.
I also recommend trying to find out what the networking config problem is rather than ignoring it.