How to enable SMTP filtering on ISA Server

Microsoft has some articles on this issue as has Thomas Shinder (www.isaserver.org).

However, as mistakes are easy made I deceided to create this FAQ.

This FAQ assumes you are using Windows 2000 Server and ISA Server 2000 (english versions) and that the filtering is to be done on the ISA Server itself. Futhermore, I'll assume the network looks like this:
[tt]
+-------------------------------+
| Internet |
+-------------------------------+
|
| Internet
|
+--------------+----------------+
| ISA Ext. 1.2.3.4/27 acme.com |
| ISA Int. 192.168.1.254/24 |
+-------------------------------+
|
| Intranet (internal.lan)
|
+--------------+----------------+
| Exchange 2K 192.168.1.1/24 |
+-------------------------------+
[/tt]

First off all, it's important that you install the SMTP Service BEFORE you install the ISA Server. Also, have any ServicePack and Security Fixes installed before your start the installation of ISA Server.

After you have installed the SMTP Service, the next step is to configure this service. It is important that socket pooling for the SMTP Service is being deactivated. There are numerous ways to do this. One way would be to use the ADSUTIL.VBS script:
[tt] cscript adsutil.vbs set /smtpsvc/1/DisableSocketPooling 1[/tt]
or use MetaEdit from Microsoft.

NOTE Socket pooling allows a service to listen on all IP addresses and all NICs. You can verify this by using the netstat command (when the virtual server is started):
[tt] netstat -na[/tt]

it should return something like this:
[tt]Proto Local Address Foreign Address State
TCP 0.0.0.0:25 0.0.0.0:0 LISTENING
[/tt]
Since the incoming messages are to be filtered on the ISA Server and then be forwarded to the internal Exchange server, the properties of the "Default SMTP Virtual Server properties" must be set accordingly. In this case, the IP address to listen on is the internal IP address of the ISA Server when using server publishin rules. If you are using Packet Filters then the external address is to be used.

Since we use Server Publishing rules, set the IP to the internal IP of 192.168.1.254

Also, a very important step, is to change the relay restrictions. Click on "Relay" on the "Access" page of "Default SMTP Virtual Server Properties" and select "Only the list below" and deselect "Allow all computers which successfully authenticate to relay, regardless of the list above.". Make sure the list of computers is empty.

You can make other options on the Messages page in order to limit message size etc. At this point there is no need to define outbound security. Anonymous Security is sufficient enough for the SMTP relay server to communicate with the internal Exchange server.

The SMTP Server relay no messages at this point. All incoming messages are dropped. It's time to define a remote domain for which incomming messages are accepted.

Open Internet Services Manager, expand Default SMTP Virtual Server and right click on the "Domains" node and select "New" then "Domain". The "New SMTP Domain Wizard" will ask for the kind of domain type. In this case it's "Remote" domain. The domain name is:
[tt] *.acme.com[/tt]

If your internal Exchange Server accepts/host other domains too, these need to be defined here as well.

NOTE The internet domain name does not necessarily need to be the same as your active directory domain name. You might for instance name your Active Directory "internal.net" but host e-mail for the domain acme.com. Which domains your internal Exchange Server accepts are done with the Recipient Policy on the internal Exchange Server.

After the domain is created, right click on it and select "Properties" and select "Allow incoming mail to be relayed to this domain" and select either "Use DNS to route to this domain" or "Forward all mail to smart host".

In this case, we'll allow incoming mail to be relayed and we're using a smart host,
[tt] [192.168.1.1][/tt]

If you do not put brackets around the IP address, the SMTP relay server will try to resolve the IP address to an IP address.

After completing this step, Stop and Start the "Default SMTP Virtual Server".

To verify that socket pooling is disabled and that the server is listening on correct interface, trigger another netstat -na. The output should read like this:
[tt]Proto Local Address Foreign Address State
TCP 192.168.1.1:25 0.0.0.0:0 LISTENING
[/tt]

When You install the ISA Server, make sure that if you want to filter on keywords, that the "Message Screener" component is installed. It's found under "Add-in Services" when you install the ISA Server. Also, do not forget to install the ISA SP1 and ISA Feature Pack.

To enable SMTP Filter, open ISA Management, expand your server name, click on application filters end then right click "SMTP Filter" and select enable if not already done.

Define any rules you would like in this place, like filtering on keywords, mail attachments etc.

NOTE If you define Users/Domains to be filtered, these objects are not store on the ISA Server, they are simply dropped. Other rules, like Keywords and Attachments can be either deleted, without notice to anyone, hold in the BADMAIL folder on the ISA Server or forwarded to someone's attention.

The last step, after creating SMTP Filter rules is creating the Server Publishing rule. IP Address of internal server is the IP address of the ISA Server, in this case 192.168.1.254. External IP address is the one which SMTP requests are expected. In this case 1.2.3.4. This rule should be applied to the "SMTP Server" protocol and to requests from "Any Request".

To test if this scenario works, use telnet on the external interface of the ISA Server, port 25.

If you have changed SMTP Filter Properties, it might be a good idea to stop and start the "Default SMTP Virtual Service" as well as Firewall and Proxy Service on the ISA Server. Else, it will most probalby not be working correctly.

[color #ff0000]NOTE I am not in any way responsible for any damage that may occur to you due to mistakes or errors in this FAQ. This FAQ is written only on behalf of personal experience, if anything is wrong in this FAQ please let me know. Always try out first, before you implement it in a live environment.[/color]