Exchange server floods the network - help!

[esmithbda]

As of 9:10am this morning, our Exchange server will flood the network if it is running. We have a 128K frame relay connection and the Exchange server will overflow that sending data out.

If I stop the SMTP process, then the issue is resolved.

I looked in the System manager for Exchange at the Queues and there are about 20-25, all that show no messages, and one that shows 1 message of a size of 3K that has been trying to send since yesterday.

I haven't changed anything on this machine - the only change to the network was that there was a new fully patched WinXP Pro machine added last night - it is just a user machine on the network.

Can anyone please offer some ideas - our network is dead if the Exchange server is up and we can't do anything at all - if I take the Exchange server down, then we can have the network, but no e-mail which is just as bad for most of the users.

 

[esmithbda]

I've scanned every single machine with two virus scanners from different companies.

The problem remains, and no viruses are found.

If the Exchange SMTP service is running, it floods the network. If it is not running, then we have net access.

I have setup performance monitor to monitor what computers are sending out data and I'm graphing it - there is little to no traffic at all.
Then when I start up the SMTP service, it goes up dramatically for the server that houses that of course, and then drops off when the SMTP service is shut down.

There are a few servers that blip up a bit - but that is likely due to them trying to connect to send mail that they have built up throughout the day.

Nothing on the server has changed in a long time. I went in today because of this stuff and backed off on the number of connections to servers that it makes, as well as put even more restrictions on the message sizes.

But even then, nothing shows up in the queues.

Talking to the people that we lease out frame relay from, when our SMTP server is on, they see the line flooded with data coming out from us.

I'm trying to install packet sniffing tools now - but even as I do so, I don't think that this machine is on the same switch as the SMTP server - so I'm not sure it will even see the proper traffic.
Our firewall is a Sonicwall SOHO 2 and its logging features are none too impressive.

I can tell from it that we are getting a lot of SMTP data sent - but it doesn't tell me from where, or which way - it just goes up by about 7 megs each time I turn on/off the SMTP server for under a minute or so to test new theories.

I have a network consultant coming in later today.

I think we are likely going to have to resort to shutting down the entire network and then bringing it back up, one machine at a time to try to resolve what/who is causing it.

This is not going to be pretty.

 

[esmithbda]

It seems to be resolved now - and I'm not entirely sure why.

To add to the oddity of this problem, we had out network consultant come in to look at it. He pretty much reporduced all that I had done, so there was a lot of double tracking.

All of the virus checking was redone.

We then downloaded Ethereal and installed that. We installed some extra logging on the firewall while we were in there - but I don't think that solved the issue.

We noticed that the connection looked slow (even slower than the pitiful 128K that we have allotted for us) and so we blocked everyone on the network from getting out to the net and only allowed the machine that we were on (the mail server with the SMTP process turned off) to get web access in and out through the firewall.

We tested the connection and saw that it was close enough to the speed that we should have (108 instead of 128).

We disabled Norton AntiVirus for Exchange on the server in case that was the issue.

Over our testing period, we started and stopped the SMTP process probably 50 times as we tried new things and tested to see how/why/when it caused the connection to die.

If we were on the machine itself and doing things, then we could actually get out (after opening some ports for it). We could telnet and ping out of it with the SMTP on - but not an initial connection - we would have to connect to the service with SMTP off, and then we could turn SMTP on and the connection would slow down, but exist.
But if we used terminal services from another machine into the mail server, then we couldn't get any connections at all no matter what if the SMTP service was running.

There were fewer and fewer listings in the queue while each time that we started and stopped the SMTP process - the whole time there were only 3 messages waiting to go out, the largest was about 3K in size.

Then we saw that we could go out to webmail that we had on a server outside of our network and mail into our admin account here with the SMTP service going - something that wouldn't happen before.

So we reverted to our normal firewall settings and the mail server and network seemed to be working together happily.
The admin account got a message showing that it couldn't deliver something to an address.
Our network consultant saw that and was convinced that we were an open relay. So I then showed him what I have already checked what seems to be a million times before - we weren't an open relay - can't telnet into port 25 from outside, proper settings, etc etc - proper MS KB was followed.

So that leaves three theories as to why this happened - and one of them seems to be ruled out.

1) We were hit by a ton of span to users that don't exist on our domain. The Exchange server then sends a response back that says "sorry, that user doesn't exist here" to the e-mail, but that e-mail account that it is replying to usually either never existed, or has since been shut down for spamming.
As a result, there is a lot of traffic there.
This one sounds reasonable - except that the tech that mangages our actual connection said that all he saw was a massive amount of data streaming out of our connection and nothing going into us at all since it was full.
That would make it hard for the spam to get in and keep the process going.

2) A user computer here (or a server) has a virus/worm/trojan/spyware that is sending out mail through our Exchange server. The Exchange server is setup not to allow anyone to send out through it unless they can authenticate - their machine would be allowed to authenticate and therefore get out.
This seems improbable because we didn't see the messages in the queue, and we have anti-spyware programs running on all of the machines (two different programs), and we have anti-virus programs running, and I did a walk through and ran a different companies virus scan on all of the machines just to make sure there wasn't a miss.

3) Norton AntiVirus for Exchange was turned off - it didn't immediately solve the problem - but it was after that when we saw a resolution. So perhaps it somehow went bad and was e-mailing out. I have no idea how/why it would do that, but perhaps.
I am about to start it up again to see if that is going to kill us again or not.

Now that I have Ethereal, I can at least sniff the line if this happens again and get an idea of what is getting sent out and hopefully get a better idea of what is going on.


A problem that is gone, but unfortunately not really resolved.

 

[ymeq123]

Sounds like fun. Ugggh. I had a somewhat similar issue with mine. Someone sent out an email to many recipients with about 7 meg of attachments. At 200k upstream max I was having some of the same issues as you. Especially since some of the addresses were incorrect or would not accept that large of an attachment. Pain in the but to fix. I tried to enumerate the message and then delete it but it would not stop one of them. I had my retries set kinda high so I lowered the retries and then I went into my DNS and added the domain the last message was sending to and directed it nowhere. The retries dies out fast at that point.

 

[esmithbda]

Our message limits and connections etc were all fairly low to start with. I lowered them some more in hopes of helping and/or preventing the issue again.

I still don't think it was that though in this case - the queues only showed 3 messages ever waiting to be sent.

It was as if the SMTP server (the service itself, not just that machine and some other service) were sending out tons of data, but no mail... if that is even possible.

 

[esmithbda]

This happened again with no warning and/or explanation recently. We were going to upgrade to Win2k3 and Exchange 2k3 anyway, so we took the unexpected severe downtime as an opportunity to more quickly make the move (fortunately it is a very easy move, especially if the mailbox sizes aren't huge - we have one user that refuses to ever delete anything... for over 5 years).

I still have no idea what causes it. The problem could be solved by either going into the Exchange manager and turning off the SMTP virtual server and/or going into the services and turning off the SMTP service.
But turning them back on would flood the connection again.

If you looked at the queue, there were only 3 messages trying to get out, and they were all small (less than 1K in size).

I am totally baffled as to what caused it.

In the end, a consultant said that it was probably because the machine was so heavily overloaded (it was the PDC, it had Exchange 2K on it, it had IIS running for OWA, it has SQL server on it, it has Great Plains and Pervasive on it, it was a file server, and two Access programs are run over the network off of it - and it scanned the Exchange mail for viruses and had its own anti-virus client as well - it only had 512MB RAM and it was a PIII 1Ghz) - not sure how that would do it, but I hope it doesn't happen again.
The last time we had it, it just fixed itself over time (a day) while we repeatedly stopped and started the services (and tried restarts of the whole machine as well).

I would love to know why it did that and what exactly was happening.

I tried to run a packet sniffer on it to see what was going on, but it was hard to find anything unusual in what was going on (I am more familiar with them on a Unix system than in Windows).

 

[windows00xp]

I had a similar problem it was the blaster virus

 

[esmithbda]

I immediately thought that it was a virus each time and couldn't find a single machine on the network that had it. Ran virus scans on all of them and none of them turned up with it - all of the machines are fully patched as well.

At least if it were that and I could determine what machine it was, I would know how to resolve it in the future if it does turn up again (I sure hope not).

 

[de1458]

if your exchange is directly process incoming email, a 128k link might be the problem, one thing about smtp is it might use all your available bandwidth this is true for both incoming and outgoing, that is someone flood your exchange with spam will potentially bring you down for all the net service, if you have a router that is able to config for traffic sharing/shaping, you might have a better luck to assign the priority for net traffic. but this just one possibility might not explain exactly what happen in your situation.

 

[esmithbda]

We changed the outgoing bandwidth on the firewall and the Exchange server to limit it - but there is little we can do against the incoming.
We are changing the service which we get our connection to and finally today one of the higher-ups finally said that we need a faster connection as well.