Trouble with NAT/PAT to inside resources from internet

[fieryhail]

I'm having a very difficult time establishing NAT connectivity to resources on the PIX inside interface from the internet. At one point, had things working fine, but had a bunch of Layer-2 switches connected to separate interfaces on the PIX. Now the separate networks are on a Layer 3 switch, and connected to the PIX via the "inside" interface. When logged into the PIX, I can fully access all resources connected to the layer 3 switch. I've tried everything I can think of, but must be missing something, probably a NAT misconfiguration. I'll post a "clean" version of PIX config here. A symptom is, when I try to access a PATted resource from the internet, the connection times out. I have a media server that is accessed using custom port 420 that translates to private port 80. Packet-tracer allows traffic fine to the outside routable IP, but I get dropped on phase 6 "rpf-check" when I try to go to the private inside address. Any suggestions? At this point I'm just trying to get NAT setup correctly then will tighten the PIX down further. I'm going to be ommitting the VPN parts of the config as I don't believe them to be relevant to this, if I am mistaken I'll post them also, just trying to make the post a little smaller. Thanks in advance for any suggestions.



PIX Version 8.0(4)
!
hostname PIX0
domain-name rcserveny.com
enable password 7U6F27URx059dL3Q encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface Ethernet0
nameif OUTSIDE
security-level 0
ip address 96.xx.xx.174 xx.xx.xx.xx
!
interface Ethernet1
speed 100
duplex full
nameif INSIDE
security-level 100
ip address 192.168.0.1 255.255.255.0
!
interface Ethernet2
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet3
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet4
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet5
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet6
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet7
shutdown
no nameif
security-level 100
no ip address
!
interface Ethernet8
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet9
shutdown
no nameif
no security-level
no ip address
!
ftp mode passive
dns server-group DefaultDNS
domain-name rcserveny.com
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
access-list inside_access_in extended permit ip any any
access-list inside_access_in extended permit icmp any any
access-list outside_access_in extended permit icmp any any
access-list OUTSIDE_access_in extended permit ip any any
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.172 eq smtp
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.172 eq 2080
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.172 eq https
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.172 eq lotusnotes
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.171 eq www
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.171 eq lotusnotes
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.171 eq https
access-list OUTSIDE_access_in extended permit object-group TCPUDP any eq domain host 96.xx.xx.173 eq domain
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.173 eq www
access-list OUTSIDE_access_in extended permit tcp any host 96.xx.xx.173 eq ftp
access-list INSIDE_access_in extended permit ip 192.168.0.0 255.255.0.0 any
access-list INSIDE_access_in extended permit ip 172.16.0.0 255.255.255.0 any
access-list rcsinternal_splitTunnelAcl standard permit 192.168.110.0 255.255.255.0
access-list rcsinternal_splitTunnelAcl standard permit 192.168.10.0 255.255.255.0
access-list rcsinternal_splitTunnelAcl standard permit 192.168.20.0 255.255.255.0
access-list rcsinternal_splitTunnelAcl standard permit 172.16.0.0 255.255.255.0
access-list INSIDE_nat0_outbound extended permit ip 192.168.110.0 255.255.255.0 192.168.110.192 255.255.255.192
access-list INSIDE_nat0_outbound extended permit ip 192.168.10.0 255.255.255.0 192.168.110.192 255.255.255.192
access-list INSIDE_nat0_outbound extended permit ip 192.168.20.0 255.255.255.0 192.168.110.192 255.255.255.192
access-list INSIDE_nat0_outbound extended permit ip 172.16.0.0 255.255.255.0 192.168.110.192 255.255.255.192
access-list OUTSIDE_IN extended permit ip any any
pager lines 24
logging enable
logging asdm warnings
mtu OUTSIDE 1500
mtu INSIDE 1500
ip local pool RCSVOICE 192.168.110.221-192.168.110.230 mask 255.255.255.0
no failover
icmp unreachable rate-limit 1 burst-size 1
asdm image flash:/asdm-61551.bin
no asdm history enable
arp timeout 14400
global (OUTSIDE) 101 interface
nat (INSIDE) 0 access-list INSIDE_nat0_outbound
nat (INSIDE) 101 0.0.0.0 0.0.0.0
static (INSIDE,OUTSIDE) udp 96.xx.xx.173 domain 192.168.20.5 domain netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.173

http://www 192.168.20.5

http://www netmask

255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.173 ftp 192.168.20.5 ftp netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 2080 192.168.10.2

http://www netmask

255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 smtp 192.168.10.2 smtp netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.171

http://www 192.168.10.4

http://www netmask

255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 lotusnotes 192.168.10.2 lotusnotes netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.171 https 192.168.10.4 https netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 7080 192.168.10.6 7080 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 7443 192.168.10.6 7443 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 7090 192.168.10.6 7090 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 7444 192.168.10.6 7444 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 18180 192.168.10.6 18180 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 18443 192.168.10.6 18443 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 11100 192.168.10.6 11100 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp 96.xx.xx.172 11099 192.168.10.6 11099 netmask 255.255.255.255
static (INSIDE,OUTSIDE) tcp interface

http://www 192.168.10.41

420 netmask 255.255.255.255
access-group OUTSIDE_access_in in interface OUTSIDE
access-group INSIDE_access_in in interface INSIDE
!

!
route OUTSIDE 0.0.0.0 0.0.0.0 96.xx.xx.169 1
route INSIDE 192.168.10.0 255.255.255.0 192.168.0.2 1
route INSIDE 192.168.20.0 255.255.255.0 192.168.0.2 1
route INSIDE 192.168.110.0 255.255.255.0 192.168.0.2 1

aaa authentication ssh console LOCAL

!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect netbios
inspect rsh
inspect rtsp
inspect skinny
inspect esmtp
inspect sqlnet
inspect sunrpc
inspect tftp
inspect sip
inspect xdmcp
!
service-policy global_policy global
prompt hostname context
Cryptochecksum:13b90c05b926368a274eb4e8ed8de4f3
: end

 

[Viconsul]

Your translation should be

static (INSIDE,OUTSIDE) tcp interface 420 192.168.10.41 [URL unfurl="true"]www netmask[/URL] 255.255.255.255

 

[fieryhail]

I appreciate the syntax correction, I modified it, however, when I try to access an internal resource, I still get a connection timeout. Also, when I run packet-tracer, I still get the packet dropped at phase 6 nat. The packet-trace output is:


Phase: 6
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
static (INSIDE,OUTSIDE) tcp interface 420 192.168.10.41

http://www netmask

255.255.255.255
match tcp INSIDE host 192.168.10.41 eq 80 OUTSIDE any
static translation to xx.xx.xx.174/420
translate_hits = 0, untranslate_hits = 0
Additional Information:
Forward Flow based lookup yields rule:
out id=0x79a7de0, priority=5, domain=nat-reverse, deny=false
hits=1, user_data=0x79a7d40, cs_id=0x0, flags=0x0, protocol=6
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=192.168.10.41, mask=255.255.255.255, port=80, dscp=0x0

Thanks again in advance for any suggestions. They're much appreciated.

 

[Viconsul]

You need to permit port 80

http://www on

interface outside.

access-list OUTSIDE_access_in extended permit tcp any interface outside eq www

 

[NetRx]

Viconsul, that will not help with his current config. He is already permitting any traffic to the outside interface:

access-list OUTSIDE_access_in extended permit ip any any

Fieryhail, that statement is a huge security risk and effectively disables your firewall. I would recommend removing it and only granting needed access. You already have specific statements defined on your outside ACL--you should use those.

Regarding your issue though, could you please specifically state what the source IP address, destination and needed ports are? For example

Source: 4.2.2.2
Destination: 192.168.1.x
Ports: TCP 80

Of course blank out any addresses that shouldn't be seen like you have been doing. Thanks.

 

[fieryhail]

I realize that it is in effect disabloing the firewall, please note, that the cleaned config was taken with that statement enabled for testing only, just to verify that there wasn't an ACL in effect that was stopping the traffic, normally that statement is NOT enabled lol. Purely testing, otherwise it's selectively enabled for traffic.

Regarding the specific issue, I have connected to a remote system and am testing from there coming inside to my network.

Source: any internet: 420
Destination: Outside IP:xx.xx.xx.173, internal: 192.168.10.41:80
Ports TCP 420 on outside translating to 80 on inside.


Thanks again for the assistance as well as for the heads up on the "ip any any" statement.

 

[NetRx]

I might be overlooking something but the config looks good based on the IPs you provided. Did you clear xlate and clear local-host after you changed the static? Note that running those commands without any arguments will sever any existing connections.

Try these

clear local-host 192.168.10.41
clear xlate local 192.168.10.41

Those will only terminate any connections associated to that local IP address.

 

[fieryhail]

I tried that, and still no success. I appreciate the pointer however. What I am wondering, is if there is another nat statement that is needed somehow? What seems to be happening according to the packet-trace output on phase 6 is apparently a NAT issue. As referenced like such:

static (INSIDE,OUTSIDE) tcp interface 420 192.168.10.41

http://www netmask

255.255.255.255
match tcp INSIDE host 192.168.10.41 eq 80 OUTSIDE any
static translation to xx.xx.xx.174/420
translate_hits = 0, untranslate_hits = 0

Another thing, I have 5 static IPs on the outside interface, so am I incorrect in thinking that the static statement would have to be the routable IP that DNS is programmed with, i.e, xx.xx.xx.173 instead of using the "interface" in the translation statement? I thought that using the syntax "interface" in the translation statement would perform NAT using the IP assigned to the interface. Maybe I am wrong in this? Any suggestions as always, much appreciated. Thanks again.

 

[NetRx]

Packet tracer can be unreliable at times, fyi. Your static statement is good and is binding to the outside IP address as you stated. All you need to get a firewall working is usually:

1. A NAT/static (can be waived if nat-control is disabled, which it is by default on an ASA)
2. ACL
3. Route

You've met all three criteria. Can you setup logging?

logging buffered 7
logging buffer-size 8192
logging timestamp
--Generate traffic--
show log

Also you could simply remove the PAT and replace it with a NAT for troubleshooting

no static (INSIDE,OUTSIDE) tcp interface 420 192.168.10.41 [URL unfurl="true"]www netmask[/URL] 255.255.255.255
static (INSIDE,OUTSIDE) interface 192.168.10.41 netmask 255.255.255.255
clear local-host 192.168.10.41
clear xlate local 192.168.10.41

Another thing, I have 5 static IPs on the outside interface, so am I incorrect in thinking that the static statement would have to be the routable IP that DNS is programmed with, i.e, xx.xx.xx.173 instead of using the "interface" in the translation statement?

I'm assuming the .173 was a typo and should be .174? Both addresses are routable to the outside world?

 

[fieryhail]

Yes, I know that PT can be unreliable at times lol. However, I have also tested from a web browser from a remote site and get nowhere. Site timed out. Yes, I know the site locally is working because I have full access to the resources from hosts inside the network, just wanted to let you know that it's not an error with the host resource(s) itself. No, the .173 is not a type, and yes, both IPs are routable. I can not remove the PAT entirely, as I have other services that also need PAT. Unfortunately, I have only 5 routable IPs currently, and at this time, no option for more without changing WAN provider (commercial cable) but many more services that need to be internet accessable than 5 routable IPs can enable. A /27 subnet would do wonders! lol.

After viewing the log, there is only 1 line that references the IP 192.168.10.41, I've removed the public IP but here is the line:

JPIX-6-302014: Teardown TCP connection 160877 for OUTSIDE:64.xx.xx.85/2575 to INSIDE:192.168.10.41/80 duration 0:00:30 bytes 0 SYN Timeout

Obviousely a SYN timeout, and while I know what SYN packets are, I admit I don't understand what is causing this issue. Thanks again.

 

[NetRx]

Typo on the destination on the static? Sorry I gotta ask, that's what the logs tell me is happening!

Can you telnet to 192.168.10.41 over port 80 from another host on the 192.168.0.1/24 network?

 

[fieryhail]

the only hosts on the 192.168.0.0 network are the PIX 0.1, and the Catalyst 3550 switch, 0.2. Possibly I have a misconfiguration, so I'll explain the setup a little more and exactly how I am testing:

On the 3550 I have multiple networks each with an SVI. Each host in those networks uses the SVI as their gateway. So 192.168.10.0/24 has the SVI as 192.168.10.1 and hosts in that network use 192.168.10.1 as their gateway. Same goes for the other networks, 192.168.110.0/24, 172.16.0.0/24, and 192.168.20.0/24. The PIX routes data destined for hosts in these networks with the "route 192.168.10.0 255.255.255.0 192.168.0.2" statement. There is also an SVI on the 3550 with the IP 192.168.0.2, this is how the PIX sends traffic to the switch. There is a static route configured on the 3550, "ip route 0.0.0.0 0.0.0.0 192.168.0.1" Perhaps this will help some as far as explanation.

To test aside from using packet-tracer, I RDC into a remote system 64.xx.xx.85 and open a web browser on that machine, and then try to access:

http://site.domainname.com:420/page

The connection invariably times out. From any host on any network in the 3550, yes, I can telnet to the host on port 80. Port 420 is only used when connecting from an outside network, the internet. I'm still very new in regards to PIXes so I realize it's very possible I setup the config wrong. Basically, what I want to do is to have the 3550 switch handle the heavy intervlan routing instead of the PIX, at one time I had each network connected directly to an interface on the PIX, there are plenty of interfaces lol, and there was no inter-vlan routing on the 3550, which is a waste since it is a layer 3 switch, and also a bottleneck. Especially noticeable when there was lots of traffic going from hosts on one network to another, multi-gigabyte etc. The PIX would slow performance quite a bit. Also, I work with a fairly high-speed WAN link, 100x15 (down/up) so really need to have the PIX primarily for WAN security.

I hope this helped clear up the whole setup some, thanks for all your help and I look forward to hearing back. Thanks again.

 

[NetRx]

Thanks for the description, that was helpful. Let's see if we can rule the firewall out. Ideally we are not looking for a return SYN-ACK packet back to the inside interface of the firewall. My apologies if you've tried some of these steps already.

/**Generate Traffic from outside**/

pixfirewall# show conn det
   -You should see flags, aABS

/**Push these commands**/

pixfirewall(config)#access-list 150 extended permit ip host 64.xx.xx.85 interface outside
pixfirewall(config)#access-list 200 extended permit ip host 192.168.10.41 host 64.xx.xx.85
pixfirewall(config)#capture WEB_CAP_IN access-list 150 circular-buffer interface outside
pixfirewall(config)#capture WEB_CAP_RETURN access-list 200 circular-buffer interface inside

/**Generate Traffic from outside**/

show capture WEB_CAP_IN det
show capture WEB_CAP_RETURN det

If you do not get any return data from 'show capture WEB_CAP_RETURN' then you will need to look at routing on the 3550 or server. Make sure the server has the right gateway, try a route print, ping that RDP machine, etc...Ultimately this may involve setting up a SPAN session on the switch and monitoring the traffic.

 

[fieryhail]

Firstly, thank you. No I have not tried these as yet, still new to PIX, the tips are greatly appreciated however, while issue not solved yet, definitely is giving me more insight to the PIX in general. Anyway, in response to the first, I am not getting the aABS, but instead just aB as shown below:

TCP OUTSIDE:64.xx.xx.85/2839 INSIDE:192.168.10.41/80,
flags aB, idle 0s, uptime 3s, timeout 30s, bytes 0

The strange thing (or maybe no so strange since it is an RDC connection) is the output from the WEB_CAP_IN only displays the ip 64.xx.xx.85 on port 3389 going to the IP of my workstation, 192.168.110.101 (not a typo, different network) but nothing on port 420 going to 192.168.10.41:80. The capture from WEB_CAP_RETURN however shows this:


1: 08:51:31.706689 0011.bb5b.1e00 0017.59ef.7092 0x0800 62: 192.168.10.41.80 > 64.xx.xx.85.2848: S [tcp sum ok] 85813780:85813780(0) ack 3394636966 win 5840 <mss 1460,nop,nop,sackOK> (DF) (ttl 63, id 0)

Perhaps this is helpful? It appears that something is trying to get back, but communication is breaking down somewhere. Thanks again.

 

[NetRx]

The traffic over 3389 is normal from WEB_CAP_IN. When you generate inside traffic to an outside host, in this case RDP traffic, your internal machine gets PAT'd to the external IP on the outside interface. WEB_CAP_IN will include the RDP traffic since it binds the the outside interface.

Note that you should also see an attempt from WEB_CAP_IN like 64.xx.xx.85.2848 > 96.xx.xx.174.420: S. That is the traffic we are looking for.

aB means that 2/3 of the TCP handshake completed. The inside host sent a SYN-ACK packet back to the RDC machine as the capture from WEB_CAP_RETURN shows. The 'a' character usually means that the return 'syn' was successfully sent out the outside interface and back to the outside host but timed out somewhere. I'm thinking maybe something is up with the return path upstream. Can you run these and make sure the PAT is right:

pixfirewall#show xlate interface outside det
pixfirewall#show xlate local 192.168.10.41 det
pixfirewall#clear asp drop
/**Generate outside traffic**/
pixfirewall#show asp drop
 -This command will tell us if a flow is dropping the packet. Similar to packet-tracer

What was the statement you ran for packet-tracer? It should be similar to this:

packet-tracer input outside tcp 64.xx.xx.85 1025 96.xx.xx.174 420 det

Have you tried accessing the site by IP address only from the RDC machine? You mentioned DNS earlier. What happens when you do an nslookup site.domainname.com on the RDC box?

Rich
Network Engineer - CCNA