OWA HTTPS traffic SOHO firewall 5.2.8a

[mleggatt]

I have a Outlook Web Access server on our internal network which is accessed from the internet via HTTPS (port 443) through a Watchguard SOHO firewall box using NAT. Initially I was running 5.2.7 of the firmware but this seemed to drop HTTPS traffic from external hosts, on looking in the log files it appeared the HTTPS traffic from even one host created lots of connection to the firewall from increaing numbers of ephemeral ports on the host (client). Eventually the firewall would refuse the incoming HTTPS connections with the following error:

2003-05-01 08:59:40 Local0.Warning 192.168.xxx.xxx IP: Packet discarded from 212.140.xxx.xxx port 3680 to 213.2.66.74 port 443 (TCP)(incorrect state)

This showed as the external OWA users being unable to access the OWA pages.

I believe this is because the firewall was mistakenly identifying the HTTPS traffic as a SYN flood attack and blocking the connections. This however had no effect on outgoing traffic. The only way to resolve this was to reboot the SOHO box.

I then upgraded to 5.2.8a firmware which is supposed to resolve the webserver HTTPS issue (see release notes). On installing this everything seemed o.k with the following showing in log:

05-14-2003 17:07:07 Local0.Info 192.168.x.xxx IP: allowed from 167.247.xxx.xxx port 38637 to 213.2.xxx.xxx port 443 TCP SYN (HTTPS)

but then the firewall started dropping outgoing connections with the following error:

2003-05-14 12:09:48 Local0.Error 192.168.xxx.xxx NAT: DYNAMIC Translation pool exhausted

This suggest to me that the 5.2.8a upgrade now allows the OWA traffic through instead of blocking it as a SYN attack but the large number of ephemeral orginating ports from the client (web browers) means that the firewall runs out of NAT ports and so drops both incoming and outgoing connections.

Watchguard have assured me that lots of users use OWA through SOHO boxes but I cannot see how they can be if it has this kind of issue. I have some questions:

1. Is this large number of originating ports normal for HTTPS traffic? It seems like it is creating one connection for every object on the OWA pages?

2. Is there any method for resolving this issue?

3. Does anyone else run OWA through a SOHO box with success or is this firewall not up to the job?

4. Had anyone had a similar issue and resolved it?

Any assistance or hints would be appreciated.

Thanks
Martin

 

[Ntr0P]

Dynamic Translation pool exhausted messages are caused by outgoing traffic, not incoming traffic.

For each outgoing packet from a specific IP addr to a specific IP addr & port number, the SOHO sets up a session, and assigns an available high port number from the Dynamic Translation pool to this session.
Sessions have default timeout values - so if the session does not seem to get terminated nicely, eventually the session and used port info will be deleted. For TCP packets, the default timeout value is fairly long (hours), while the UDP timeout is very short (seconds).

If a PC runs an application which makes many outbound sessions (from the SOHO perspective), then it can use up the entire available Dynamic Translation pool. This can happen if there's some kind of trojan on one of
the PCs or if one or more of your users is using p2p file sharing software.

You might want to go around to the PCs and make sure no one is running
anything like Kazaa, morpheus, winmx, gnutella clients (bearshare, limewire,
etc..).

 

[mleggatt]

NtrOP thanks for your reply it seems to be what Watchguard have told me the same in the past. I know for fact that at least one user here uses Kazaa, and lots use MSN messenger e.t.c The trouble is until now I have had no concrete reason to ban this type of software. I know it is a security risk but peers here seem to have the attitude that it has not caused problems in the past so why now?

I do have several questions, you say the "Dynamic Translation pool exhausted messages are caused by outgoing traffic, not incoming traffic" but from past experience with Linux IPTABLES type firewalls there is DESTINATION NAT and SOURCE NAT (SNAT & DNAT) surely there is either one NAT pool shared by the outgoing and incoming connections or one pool for each outgoing and incoming traffic. If the NAT pool is shared by both the incoming and outgoing connections then the OWA HTTPS access, which I know for a fact utilises lots of ephemeral ports even for one user could exhaust the NAT pool and cause outgoing connections to be dropped.

I get the impression that on version 5.2.7 firmware the HTTPS traffic created by OWA was incorrectly identified as a SYN flood attack hence the "(TCP)(incorrect state)" errors and the dropped packets. The 5.2.8a patch resolved this issue but now meant that instead of being dropped the OWA HTTPS traffic exhausted the NAT pool and hence the problems with outgoing traffic. I have noticed that one user using OWA HTTPS can create 20-30 port connections in one transaction. I think that using HTTPS, each object on the page is given a separate connection on increasing ephemeral ports on the client and an OWA page has 20-30 objects (gifs e.t.c).

You also mention a TCP default timeout value of hours, wouldn't it be possible to alter this to a shorter value or alter it so that HTTPS TCP ports timeout sooner. I am not sure if this is possible as its probably defined in an RFC somewhere. I know on IPTABLES firewalls you an configure lots of these timeout values e.t.c.

The 5.2.8a patch did not fix the OWA issue as instead of the "Incorrect state errors" I know get after a short time with only one user using OWA the following error:

Local0.Warning 192.168.xxx.xxx IP: discard from 212.xxx.xxx.xxx port 1077 to 213.xxx.xxx.xx port 443 TCP SYN (maximum number of inbound sessions)

Which is pretty self explanatory, it looks to me as though the SOHO box is now confusing the large number of TCP ports connecting inbound with lots of user. When in fact all of these ports come from one host. The only way I can see of resolving this is either altering the TCP HTTPS port timeout or increasing the size of the NAT pool on the SOHO box (more memory??? or a higher spec box??)


Thanks again for the info, any pointers on the above would be great.

Thanks
Martin