Why is there a RST flag from receiving mail server?

[Heathereen]

I'm using NA's Sniffer v 4.50.05 to track down a mail tx issue. My company (A) can't send mail to company (B). We get "mail delayed" receipts and eventually "mail delivery failed." We can mail everywhere else OK and receive just fine. Company B's IT dept can't determine what the problem is, even after whitelisting our SMTP IP.

In the Sniffer capture, I see the 2 servers talking away happily, transmitting, receiving, acknowledging... it all looks OK until company B's server sends back a RST flag. There are no FIN or OK commands. So there is a pause... any my server starts over. Does anyone know what might cause a receiving mail server to send a RST command like this?

Thanks much!! -H

 

[PalmTest]

You say that the servers are talking happily.
I assume this is on TCP/IP level.
A possible reason why you receive a RST is because the Service on Server B hasn't heard anything from Side-A for a distict period (timeout) and assumes the other side is unreachable (at Service level, not at TCP/IP level). The underlying TCP/IP connection gets terminated with a RST.
(Forum readers, please correct me if you think I am wrong)

I once had a similar case at hand when the TCP/IP conections runs perfectly, but the upperlevel applications could not communicate. The solution was to apply the latest ServicePack once again on Side-A.
The Server guy guaranteed me that the service pack was already applied, so he did not believe that it would help.

Technically I found out that Side-A transmitted an empty message to Side-B, but there was no "Contentlength=0" field included in the message which Side-A transmitted.
Side-B received the message and waited for the rest of the message, Side-A had nothing to send and waited for a response (ACK) from Side-B.
At this stage both sides were waiting on each other.
Both servers were PINGable all the time!
The timeout on Side-B was set to 45 seconds. After 45 seconds Side-B ended the communication over the TCP/IP session with a RST.
When the latest service pack was applied to Side-A the "Contentlength=0" field was present and everything went smoothly.

Later on I discovered that Side-A was installed, patched with the latest ServicePack and configured correctly.
Then a third party application was installed.
The latest ServicePack was not aplied after the installation of the third party application.
After this case was closed the Server guy and I went along very well !!

Also look at the "IP Identification" value.
In the case described above, I saw a few packages with the "IP Identification" value of 0 (zero), which puzzled me.
These packages were generated by a Firewall with SYN-Defense active.

HTH