Pix506e TCP SYN Negotiation Fails

[DameSlap]

Hello All,

I'm completely stumped by a problem I'm having with an "edge" connection between us ourselves and a customer and wonder if anyone can see why.

Here's the gist:

We have a Pix 506e at the edge of our network, plugged virtually back-to-back with our customers firewall.
Our Firewall Outside Int = 192.168.27.1
Cust Firewall Outside Int = 192.168.27.2

They have an AS400 inside their WAN that connects to an IPDS print server inside ours.
AS400 appears to us as translated address 192.168.27.22
They connect to translated address 192.168.28.22

A week ago, printing stopped working. Both parties claim not to have changed config on their respective firewalls.

I've done some packet capture and it looks like the AS400 and the print server start, but never finish their TCP/IP negotiation. It looks like this:

1. The AS400 sends a TCP SYN to 192.168.28.22
2. Firewall translates correctly and I see the packet traverse our WAN
3. The print controller sees the TCP SYN and generates a SYN ACK for 192.168.27.22
4. I can see this packet reach the inside interface of our Pix506e
5. It doesn't seem to traverse the firewall. Our customer swears blind he's not seeing the returning traffic and eventually both sides give up and issue TCP RST. But only AFTER I see the SYN ACK hit the inside interface of our firewall.

Here's the config:

FWXXXXXXXXXX(config)# sh conf
: Saved
: Written by enable_15 at 15:39:59.989 UTC Wed Sep 21 2005
PIX Version 6.3(3)
interface ethernet0 10full
interface ethernet1 100basetx
nameif ethernet0 outside security0
nameif ethernet1 inside security100
enable password aOZCVVhxvp5Yy4dG encrypted
passwd aOZCVVhxvp5Yy4dG encrypted
hostname FWXXXXXXXXXX
domain-name XXXXXXXXXXX
fixup protocol dns maximum-length 512
fixup protocol ftp 21
fixup protocol h323 h225 1720
fixup protocol h323 ras 1718-1719
fixup protocol http 80
fixup protocol rsh 514
fixup protocol rtsp 554
fixup protocol sip 5060
fixup protocol sip udp 5060
fixup protocol skinny 2000
fixup protocol smtp 25
fixup protocol sqlnet 1521
fixup protocol tftp 69
names
access-list acl_out permit icmp any any
access-list acl_out permit ip host 192.168.29.1 host 192.168.28.22
access-list acl_out permit ip host 192.168.27.2 host 192.168.28.22
access-list acl_out permit ip host 192.168.27.22 host 192.168.28.22
access-list acl_out permit tcp host 192.168.27.22 host 192.168.28.22
pager lines 24
logging on
logging trap debugging
logging host inside 10.10.1.31
logging host inside 10.10.3.123
logging host inside 10.10.1.90
mtu outside 1500
mtu inside 1500
ip address outside 192.168.27.1 255.255.255.0
ip address inside 10.10.10.18 255.255.255.0
ip audit info action alarm
ip audit attack action alarm
pdm history enable
arp timeout 14400
global (outside) 1 192.168.28.1-192.168.28.200
nat (inside) 1 10.10.39.20 255.255.255.255 0 0
nat (inside) 1 10.10.1.0 255.255.255.0 0 0
nat (inside) 1 10.10.3.0 255.255.255.0 0 0
nat (inside) 1 10.10.7.0 255.255.255.0 0 0
nat (inside) 1 10.10.10.0 255.255.255.0 0 0
nat (inside) 1 10.10.22.0 255.255.255.0 0 0
nat (inside) 1 10.10.39.0 255.255.255.0 0 0
nat (inside) 1 10.10.0.0 255.255.0.0 0 0
static (inside,outside) 192.168.28.10 10.10.10.19 netmask 255.255.255.255 0 0
static (inside,outside) 192.168.28.22 10.10.39.20 netmask 255.255.255.255 0 0
access-group acl_out in interface outside
route outside 0.0.0.0 0.0.0.0 192.168.27.2 1
route inside 10.10.0.0 255.255.0.0 10.10.10.19 1
route outside 192.168.0.0 255.255.0.0 192.168.27.2 1
route outside 192.168.29.0 255.255.255.0 192.168.27.2 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 rpc 0:10:00 h225 1:00:00
timeout h323 0:05:00 mgcp 0:05:00 sip 0:30:00 sip_media 0:02:00
timeout uauth 0:05:00 absolute
aaa-server TACACS+ protocol tacacs+
aaa-server RADIUS protocol radius
aaa-server LOCAL protocol local
no snmp-server location
no snmp-server contact
snmp-server community public
no snmp-server enable traps
floodguard enable
telnet 10.10.0.0 255.255.0.0 inside
telnet timeout 20
ssh timeout 5
console timeout 0
terminal width 80
Cryptochecksum:471d9b655ccc19adc65d099cb7c23622
FWXXXXXXXXXX(config)#

I know there are some overlapping rules, very lazy I know, but I've reached the end of my knowledge here and have been trying anything and everything to get it working.

Any help very much appreciated.

Thanks.

 

[chicocouk]

What's the other firewall? It would be interesting to find out whether it's a brand that requires a manual proxy arp entry for the 192.168.27.22 address or not, because your pix will see that address as being part of it's outside subnet, rather than a remote network, so it won't route it to the default gateway (which you have configured as 192.168.27.2)

In other words, when the packet leaves your pix, it won't get routed to 192.168.27.2, the pix will just send an arp request asking who has 192.168.27.22, and if nothing replies, the tcp connection will eventually timeout.

So if the other firewall is a Nokia box, or anything *nix based, or a netscreen or any of a few other brands, get their firewall tech to confirm they're proxy arp-ing for the 192.168.27.22 address. If they're not, there's your problem.

Alternatively, you could add a specific host route, and see if that sorts the issue out, along the following lines;

route outside 192.168.27.22 255.255.0.0 192.168.27.2 1

I don't have a pix in front of me to test, but I think the above may work, if that's the problem.

I'm afraid I also dislike your nat and global statements, because they overlap your statics. I wonder if something else on your network is natting through as the static address you've supposedly reserved for this print server? Statics should take precedence over nat/global combinations, but in any case your config shouldn't be set up like that.

A quick test would be to do a clear xlate on the pix, and then try the connection. Better still, change the global pool to

global (outside) 1 192.168.28.23-192.168.28.200

and then clear xlate and test.

Let us know how you get on, this is an interesting one. Cheers

CCNA, CCSA, MCSE, Cisco Firewall specialist, VPN specialist, wannabe CCSP

 

[DameSlap]

Thanks Chicocouk,

Some really good looking possibilities here.

I've already shaved the bottom end off the global pool to exclude the static ranges, though actually the Pix seems to start using addresses from the middle of the range rather than the bottom (all the dynamically assigned xlate addresses were 192.168.28.100 upwards) so I don't think it was interfering with the the print server static, but worth doing anyway.

Also, though I didn't question it at the time, our customer raised the issue of ARPing while he was watching the debug on his firewall for inbound traffic. My PIX was indeed ARPing like crazy for 192.168.27.22 so (at his request) I did add a static ARP entry for the MAC of his firewall's external interface against 192.168.27.22. ARPing stopped, but didn't seem to affect the overall problem.

I'll try a host specific route now and see if that makes any difference (will update you).

Thank you so much again, really very much appreciated.

 

[chicocouk]

Based on what you're saying, my feeling is that the problem is with the other firewall rather than your pix. If the pix is arp-ing for 192.168.27.22, then it's trying to send traffic to it, and needs to know what mac address to send it to. At which point you basically know your access lists etc are allowing the traffic correctly (which we know anyway, as you don't have an outbound access-list applied, so all traffic will leave as it should)

If this previously worked, but now doesn't, and your partner can see the pix making a lot of arp requests, that means his firewall must at some point have been set up to answer those arp requests, and now isn't. Or else traffic for 192.168.27.22 would never have routed to his firewall.

Check your translation table using "show xlate" to prove that you're natting the local device out properly, if so I'd be inclined to ask the other party to check their proxy arp settings and do a capture to libpcap format from their external interface, so you can examine what they're seeing in ethereal.

You can do the same on the pix, using the capture command, you need to create an acl to do so, like below

access-list ethereal permit ip any host 192.168.27.22
capture ethereal access-list ethereal packet-length 74 interface outside

And then browse to

https://10.10.10.18/capture/ethereal/pcap

to pick up the capture file. You should be able to prove the packets are leaving your pix that way.

Capture command docco is here:

http://www.cisco.com/en/US/products...erence_chapter09186a00801727a6.html#wp1053548

Cheers

CCNA, CCSA, MCSE, Cisco Firewall specialist, VPN specialist, wannabe CCSP