Seems as if your problem is a bit vague, but it sounds familiar. Actually I would start with any DNS information that pertains to your website, such as your MX records. Inside of each DNS name table is an entry for a Mail exchanger (MX), if this machine resides with your ISP maybe have them look into it there, otherwise check on your DNS tables and make an entry similar to this:
IN MX 10 mailserver
The number 10 is your priority level, where mailserver would be the actual name of your machine. When mail tries to come to your domain it bounces around looking for your MX information, which is what takes it so long to arrive. Hope this helps.
--bp