3560 as router connected to FW

[DallasBPF]

Ok, so I have been banging my head against the wall and need a fresh set of eyes...

Network set up is as follows;

ISP -> 1841 Router -> Endian Mini Firewall -> 3560 layer 3 switch

Connected to the 3560 is the Firewall, 2960 Switch for server connections and a 1131AG Wifi AP.

I will post the config below, but to explain the problem... There are 4 networks (management, server, users, wifi) and they can ALL talk to one another, but the ONLY way for them to go out the firewall is if you ping the firewall first and get a connection going... So long as you leave the ping going you can access the internet, but if you stop the ping it loses its connection in approx 30 seconds.

Ok, here is the config to the 3560;

Building configuration...

Current configuration : 5909 bytes
!
! Last configuration change at 13:47:38 UTC Mon Apr 20 2009
! NVRAM config last updated at 13:26:52 UTC Mon Apr 20 2009
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log datetime
service password-encryption
!
hostname 3560Gateway
!
enable secret 5
!
username brent privilege 15 password 7
no aaa new-model
clock timezone UTC -6
system mtu routing 1500
ip subnet-zero
ip routing
ip domain-name
!
!
!
!
no file verify auto
!
spanning-tree mode pvst
spanning-tree extend system-id
spanning-tree vlan 1-2,5,10,20 priority 24576
!
vlan internal allocation policy ascending
!
interface Port-channel1
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
!
interface FastEthernet0/1
description Copier
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/2
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/3
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/4
description Chris Lafield Office
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/5
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/6
description Danny Thompson Office
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/7
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/8
description Jack Lafield Office
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/9
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/10
description Susan Samore Office
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/11
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/12
description Rick Moncrief Office
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/13
description Boardroom
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/14
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/15
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/16
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/17
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/18
description Receptionist
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/19
spanning-tree portfast
!
interface FastEthernet0/20
spanning-tree portfast
!
interface FastEthernet0/21
spanning-tree portfast
!
interface FastEthernet0/22
spanning-tree portfast
!
interface FastEthernet0/23
spanning-tree portfast
!
interface FastEthernet0/24
spanning-tree portfast
!
interface FastEthernet0/25
spanning-tree portfast
!
interface FastEthernet0/26
spanning-tree portfast
!
interface FastEthernet0/27
spanning-tree portfast
!
interface FastEthernet0/28
spanning-tree portfast
!
interface FastEthernet0/29
spanning-tree portfast
!
interface FastEthernet0/30
spanning-tree portfast
!
interface FastEthernet0/31
spanning-tree portfast
!
interface FastEthernet0/32
spanning-tree portfast
!
interface FastEthernet0/33
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface FastEthernet0/34
spanning-tree portfast
!
interface FastEthernet0/35
spanning-tree portfast
!
interface FastEthernet0/36
spanning-tree portfast
!
interface FastEthernet0/37
spanning-tree portfast
!
interface FastEthernet0/38
spanning-tree portfast
!
interface FastEthernet0/39
spanning-tree portfast
!
interface FastEthernet0/40
spanning-tree portfast
!
interface FastEthernet0/41
spanning-tree portfast
!
interface FastEthernet0/42
spanning-tree portfast
!
interface FastEthernet0/43
spanning-tree portfast
!
interface FastEthernet0/44
spanning-tree portfast
!
interface FastEthernet0/45
switchport access vlan 99
spanning-tree portfast
!
interface FastEthernet0/46
spanning-tree portfast
!
interface FastEthernet0/47
description Cisco1131AG WiFi
switchport trunk encapsulation dot1q
switchport trunk native vlan 20
switchport mode trunk
!
interface FastEthernet0/48
description Endian Firewall
switchport access vlan 5
switchport mode access
no cdp enable
spanning-tree portfast
!
interface GigabitEthernet0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
channel-group 1 mode on
!
interface GigabitEthernet0/2
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport mode trunk
channel-group 1 mode on
!
interface GigabitEthernet0/3
!
interface GigabitEthernet0/4
!
interface Vlan1
no ip address
shutdown
!
interface Vlan2
description Management VLAN
ip address 192.168.0.1 255.255.255.0
!
interface Vlan5
description Server VLAN
ip address 192.168.1.1 255.255.255.0
!
interface Vlan10
description User VLAN
ip address 192.168.2.1 255.255.255.0
ip helper-address 192.168.1.2
!
interface Vlan20
description Wireless VLAN
ip address 192.168.3.1 255.255.255.0
ip helper-address 192.168.1.2
!
ip classless
ip route 0.0.0.0 0.0.0.0 192.168.1.100
ip http server
!
!
control-plane
!
!
line con 0

line vty 0 4
privilege level 15

login
length 0
line vty 5 15
privilege level 15

login
!
ntp clock-period 36028547
ntp server 192.168.1.2
end

3560Gateway#

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[DallasBPF]

I ended up changing the interface fa0/48 from a switchport to a L3 port and assigning it an IP address if 192.168.100.2/30 and giving the firewall the same ip network aswell as keeping the other networks on it... If I have something streaming out to the web I keep my connection like internet radio, but if you are just browsing websites you loose connection to the web...

changes made to the interface;

3560Gateway#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 192.168.100.1 to network 0.0.0.0

C 192.168.0.0/24 is directly connected, Vlan2
C 192.168.1.0/24 is directly connected, Vlan5
C 192.168.2.0/24 is directly connected, Vlan10
192.168.100.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.100.0/30 is directly connected, FastEthernet0/48
D 192.168.100.0/24 is a summary, 00:30:47, Null0
C 192.168.3.0/24 is directly connected, Vlan20
S* 0.0.0.0/0 [1/0] via 192.168.100.1
3560Gateway#sh run int fa0/48
Building configuration...

Current configuration : 157 bytes
!
interface FastEthernet0/48
description Endian Firewall
no switchport
ip address 192.168.100.2 255.255.255.252
keepalive 1
spanning-tree portfast
end

3560Gateway#

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[DallasBPF]

3560Gateway uptime is 1 hour, 1 minute
System returned to ROM by power-on
System restarted at 14:12:53 UTC Mon Apr 20 2009
System image file is "flash:c3560-ipservicesk9-mz.122-50.SE1.bin"



------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[unclerico]

i would like to know if the Endian's MAC address is in the MAC address table when you lose connectivity.

I hate all Uppercase... I don't want my groups to seem angry at me all the time! =)
- ColdFlame (vbscript forum)

 

[DallasBPF]

Nope, not showing up in the mac address table of the switch at all.... which is wierd...

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[DallasBPF]

Just added it statically and testing that.

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[DallasBPF]

still to early to tell, but no drops in pings in the last 20 mins since I added the static mac address into the table.

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[DallasBPF]

Turns out to be the Firewall and not the Cisco equipment. For some reason even though you can add multiple networks to the interface on the firewall, it will not automatically route them w/o opening a connection to the firewall first and maintaining it.

very strange... guess it's time to open a ticket with Endian to find out whats going on...

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[VinceWhirlwind]

Traffic coming in from the internet hits the firewall - if the firewall has an interface in the same subnet as the destination IP on the packet, then the firewall DOES NOT ROUTE THE TRAFFIC ON TO YOUR L3 SWITCH. It attempts to switch the traffic directly to the recipient. Which means it needs a valid ARP entry for that recipient.

As a basic good design premise, you should not have the same VLAN trunked to multiple L2 devices and NEVER trunked to multiple L3 devices.

Remove all "internal" networks from your firewall and configure an exclusive point-to-point subnet for the purpose of routing traffic back-and-forth between L3 switch and firewall. Configure appropriate "internal" routes on the firewall.

 

[DallasBPF]

What you recommended was done today, did not work. What is working for the time being so that the users can access the internet is by putting any device that needs access to the internet into the same VLAN that sends untagged packets to the firewall. And yes, I have tried putting the switchport into trunking mode and it does no difference. There is an option on the firewall for VLAN configuration, and everytime I do this I lose my connection to the firewall and I have to put it back to factory defaults and reconfigure it. I have already opened a support ticket to find out what I am missing on the firewall.



------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[VinceWhirlwind]

So is this how it is currently setup?
"I ended up changing the interface fa0/48 from a switchport to a L3 port and assigning it an IP address if 192.168.100.2/30 and giving the firewall the same ip network as well as keeping the other networks on it"

I'm not perfectly clear on what you mean by "keeping the other networks on it" - but it needs to be said that you should not have any of your "internal" subnets on the firewall, you should have routes pointing those subnets at the link to the 3560.

 

[DallasBPF]

No that is not how it is currently set up... it is currently set up where VLAN5 is pretty much what everyone is set to on the switchports (also the VLAN the firewall main IP is attached to).

As for your question about 'keeping the other networks on it'. You have 4 subnets; 2, 5, 10, 20 as in the above config shows. Since it is connected to VLAN 5 (192.168.1.100) a user on VLAN 10 can not ping the outside world (its not hitting the default route and going to 192.168.1.100) unless you have a continuous ping going to 192.168.1.100. Then it allows the connections to the internet.

How did I prove this? Well, I opened 2 command prompts, first one pinging 4.2.2.2 -t (fails) then on the 2nd one I started to ping 192.168.1.100 -t and I get the replys, immediately the 4.2.2.2 starts to recieve replys. When I stop the ping to 192.168.1.100, 4.2.2.2 pings start to fail within 30 seconds.

Now, out of curiousity... why would you not have the internal networks attached to the trusted interface? I am confused by that, since that is what the firewall is suppose to do... Differentiate between the Trusted and Untrusted network. Because when I did not have the internal networks on the firewall, they would never go outside even in the above scenario.

------------------------------------
Dallas, Texas
Telecommunications Tech
CCVP, CCNA, Net+

CCNP in the works

 

[VinceWhirlwind]

So,
"VLAN5 is pretty much what everyone is set to on the switchports (also the VLAN the firewall main IP is attached to)"

...means that you have:
- client devices on the VLAN 5 subnet
- their default GW is on your L3 3560
- another VLAN 5 subnet IP interface on the firewall.

The reason this setup is wrong is this: return traffic will NOT be routed from the firewall to the VLAN 5 subnet router address, the FW will attempt to SWITCH it (as it has an interface in the same subnet).

You need to consider that the boundary between L3 devices is a clear demarcation point which absolutely separates subnets.

Of course if your FW could make successful ARP queries or if you configured static ARP entries on it for every device within your network it would work, but it would still be an odd way to setup a network.