Upgrade server farm advice 2

[crazyitguy]

We would like to upgrade our network that supports our server farm(about 25 servers made). Mostly web and SQL databases.

Currently we have 2 2811 routers running BGP. Each is connected to a different ISP; one via T1, the other via FastEthernet 100Mb.

The routers are then connected to an unmanaged switch(100Mb). The unmanaged switch is then connected to a Checkpoint firewall.

The firewall is then connected to a second unmanaged switch. The servers are connected to this switch.

As you can see there is no redundency besides the edge routers.

Most importantly we would like to add redundency, increase the speeds, use vlans to seperate the servers.

The 6500 route is a little to expensive. We were thinking a pair of 3760 switches that will connect directly to the firewalls. Something like this:

Advice? Suggestions?

Thanks

 

[crazyitguy]

>If Router1 goes down and router2 becomes the virtual primary >router, how will firewall1 send traffic to that router, if >the routers are on different subnets?

That was going to be my next question....

If ISP1 or Router1 went down, would the primary ASA(remember this is Active/Standby, even if we stayed with checkpoint) be able to detect this and failover to the secondary ASA?

If there is no way and we have to throw in switches between the routers and firewalls then how would they physically be connected(making sure there is no single point of failure)? What models?

Thanks

 

[UnaBomber]

If ISP1 or Router1 went down, would the primary ASA(remember this is Active/Standby, even if we stayed with checkpoint) be able to detect this and failover to the secondary ASA?

Why would you want it to? The primary FW can continue to act as the primary, as long as you have layer2 connectivity between the 2 routers.

I suppose if you dont want to do that, you could create a tunnel between the adjacent router and firewall and have VRRP on the Firewall track that tunnel, if it goes down then make the other firewall the primary. I dont have a proof of concept for that however, but I believe you could do it. Although NAT inbound may be a headache to configure.

As for cabling: Firewalls to both switches and Routers to both switches, switches to each other. Be careful of loops...

UnaBomber
ccnp mcse2k

 

[crazyitguy]

1)That does sound like a headache. If I go with layer-2 switches then HSRP would be back on the table, correct?

2) I also would not need the direct connection between the 2 edge routers anymore, correct?

3)Can you elaborate on this statement: "..and with a multilayer router you have more options for securing your server farm/s"?

How does a layer-3 switch have more options? Wouldn't a full fledged firewall give me more options/protection than a simple Layer-3 switch ACL(DOS attacks, spoofs, etc)?

4)I want to throw one more thing into the mix. We are actually connected to one of our ISPs via fiber. We have a media converter on our premises that converts the fiber to ethernet before the link hits our router:

Would it be better to get rid of that 2811 and replace it with a layer-3 switch with /EMI? I would imaging cost would prohibitive though ....

 

[UnaBomber]

2) I also would not need the direct connection between the 2 edge routers anymore, correct?

No you dont but its probably better you do, in the case as remote as it may sound, one ISP goes down, and the other routers FA interface also goes down

3)Can you elaborate on this statement: "..and with a multilayer router you have more options for securing your server farm/s"?

WRT Vlans, you can chroot systems off better, and route at wirespeed between them.

Would it be better to get rid of that 2811 and replace it with a layer-3 switch with /EMI? I would imaging cost would prohibitive though ....

I wouldnt put a core layer switch at your edge, its bad design, I am not even sure how it would handle BGP... Never thought of it, but I am sure its possible, whether it is acceptable/good practise remains to be seen.

UnaBomber
ccnp mcse2k

 

[crazyitguy]

Thanks for all you help. Final diagram.....

Anymore comments or suggestions?

 

[globalchicken]

The final diagram looks good. A router can not have 2 interfaces in the same subnet though. This is where i was trying to relay to you about the L2 switch. the HSRP standby address should be in the same subnet as the devices pointing to it. Have you thought about load balancing between your 2 ISPs? right now without load balancing, the active router will be the one forwarding traffic to the WAN and the secondary router will just be for redundancy.

Also the link between your 2 L3 switches could be a L2 link and HSRP for all subnets could be run between the 2 switches. Subnet C is being used as a L3 link between the 2 switches but how is HSRP being utilized? The servers default gateway should be the standby address not the actual address used as your VLAN IP for HSRP to function.

 

[crazyitguy]

> A router can not have 2 interfaces in the same subnet though.

Are you positive about this? Can anyone concur? If this is the case then wouldn't the same logic apply to the firewalls. The firewalls act as routers but they will have 2 interfaces in the same subnet......

If that is true than here is a new diagram along with facts and addresses:

-BGP running on edge routers(Router1 and Router2) We would perfer all traffic to go out through ISP1 as it is faster and cheaper. The other one is there for backup. As a result we set BGP's "local preference" higher on Router1.

-HSRP is running on edge routers(Router1 and Router2); Router1 being the active router. For now we do not care about load-balancing.

-Firewall1 is the active firewall and Firewall2 is only on standby. The Layer-3 Switches use Firewall1's IP address as the default gateway.

-Layer-3 Switches will run HSRP for all the internal VLAN gateway addresses. If a server is connected to both L3Switch1(set as gefaut gateway) and L3Switch2, and L3Switch1 goes down, L3Switch2 will become the default gateway.

Subnet A: 205.55.5.0/24
Subnet B: 206.66.6.0/29
Subnet C: 10.0.1.0/28
Subnet D: 10.0.2.0/30
VLAN 100: 10.0.3.0/24

HSRP Virtual IP address: 205.55.5.1

Router1
Fa0/0: 205.55.5.2
Fa0/1: 206.66.6.2

Router2
Fa0/0: 205.55.5.3
Fa0/1: 206.66.6.3

Firewall1
G0/0: 205.55.5.4
G0/1: 205.55.5.5
Default gateway: 205.55.5.1

Firewall2
Fa0/0: 205.55.5.6
Fa0/1: 205.55.5.7
Default gateway: 205.55.5.1

Thanks

 

[globalchicken]

You can not overlap ip addresses, that is putting interfaces in the same network, really defeats the purpose of routers. Your routers segment networks.

Ok, so no load balancing. You could set router 1 with a higher HSRP priority on both subnets A and B so that it will become your active standby address so that your firewalls always forward traffic to it. Dont forget to set preempt in the case that ISP 1 goes down and then comes back up.

Your firewalls 2 external interfaces: 1 should be in subnet A and the other in subnet B. This will give you connectivity to both routers through both switches. Prefer the route through subnet A and B will be your alternate. Once again route to your HSRP standby addresses no the actual physical ip on the interface.

Scenario: if the L2 switch 1 were to fail then in your config, router B would take over even though ISP 1 is still active. Have router 1 be the HSRP active for both subnets A and B so that after the switch fails subnet A link on the firewall would go down, resorting to the second external interface that should be in subnet B. If it is in subnet B, and router 1 is active in your HSRP scheme then your second path through your primary firewall, L2 switch 2, and then on to Router 1, which is also primary for HSRP subnet B.

I dont know if i am making sense, but to summarize: Put firewall external interfaces in subnet A and B, connect to each L2 switch, then make router 1 primary HSRP active address for both subnet A and B and router 2 secondary for A and B.

HSRP has to be run for every subnet that is available.

Have an issue will comment on your L3 switch issues later tonight..

 

[globalchicken]

Here is a little change that you might think about.

The interfaces that will be connecting to the firewalls can be in their own vlan. I would make those ports L2 and configure them to be a member of vlan. Then allow your L3 switch to route among vlans pointing to your firewalls.

 

[crazyitguy]

You can not overlap ip addresses, that is putting interfaces in the same network, really defeats the purpose of routers. Your routers segment networks.

my bad...

...if the L2 switch 1 were to fail then in your config, router B would take over even though ISP 1 is still active......

If I put a direct connection between Router1 and Router2, and L2Switch1 fails, would traffic still go out through ISP2. We set BGP's "local prefernce" higher on Router1, the BGP tables on router2 will still be showing the best route for most destinations would be through router1. Wouldn't traffic flow like this: firewall1 -> L2switch2 -> Router2 -> Router1 ->ISP1 (as long as it the best BGP route). This seems it would be optimal as ISP1 would still be used even if L2switch1 goes down.

Prefer the route through subnet A and B will be your alternate.

How would I tell the firewall to use the second interface if the primary interface fails? Alternate static default gateway?

-----------------------

I don't mean to go around in circles, but.....

Couldn't I apply the logic you just posted to a configuration without the layer 2 switches:

Subnet A: 205.55.5.0/29
Subnet B: 206.66.6.0/29
Subnet E: 207.77.7.0/30
Subnet C: 10.0.1.0/28
Subnet D: 10.0.2.0/30
VLAN 100: 10.0.3.0/29

Subnet A HSRP Virtual IP: 205.55.5.1

Subnet B HSRP Virtual IP: 206.66.6.1

Firewall1
Default Gateway: 205.55.5.1
Alt Default Gateway: 206.66.6.1

Firewall2
Default Gateway: 205.55.5.1
Alt Default Gateway: 206.66.6.1


thanks

 

[globalchicken]

...if the L2 switch 1 were to fail then in your config, router B would take over even though ISP 1 is still active......


With a direct connection between Router 1 and Router 2, then yes with your BGP weights, the preferable route would be through Router 1. In that aspect, a direct connection would work.\\\

How would I tell the firewall to use the second interface if the primary interface fails? Alternate static default gateway?

Well if your link to the L2 switch dropped, then that interface would go down, and no data would be communicated. I dont have experience with that firewall vendor so as how to configure that, I am not sure, but would be surprised if there was not any way to do that.

Taking the L2 switch out of the picture would not enable HSRP to function. The routers need to be able to exchange hellos on the same segment.

This scenario would work if you can figure out how to configure those firewalls to roll over to the functioning interface if you router was to go down. Now your points of failure are your routers and firewalls butt all have full redundancy. This last topology looks like the way to go! Nice job!

 

[globalchicken]

You would not be using HSRP, but you would not need to. IF your primary firewall went down then all traffic would route through your router 1, if router 1 went down then all traffic would go through router 2.

 

[crazyitguy]

That makes sense...

I am going to ask the firewall people what they think.

What about the Layer 3 device configs. Was there something wrong there? I am guessing that the internal interfaces on the firewalls need to be in different subnets as well, correct?

Should subnets C or D be VLANs or Layer-3 connections (I do not remember the correct terminology).

Thanks

 

[UnaBomber]

A router can not have 2 interfaces in the same subnet thoug

Yes this is true... I setup VLANs in this case...

I think we have managed to "overdesigned" this by the way.

I dont think I helped with my physical topology description:

IMHO there is no need to use 2 subnets between your firewalls and border routers. Even Cisco doesnt recommend this, your network is fully redundant without this. You should crossover your firewalls and border routers however.



UnaBomber
ccnp mcse2k

 

[crazyitguy]

IMHO there is no need to use 2 subnets between your firewalls and border routers. Even Cisco doesnt recommend this, your network is fully redundant without this. You should crossover your firewalls and border routers however.

So what are you saying, to use 2 VLANs instead of 2 subnets between the routers and firewalls?

What does Cisco recommend?

 

[UnaBomber]

A vlan is a subnet, or should be.

No I am saying that you should have:

2 border routers
2 layer2 switches
2 Firewalls

Like you have. But you dont really need to add extra subnets:

Here is a physical diagram, this is very very high level, you are going to have to do the ground work to ensure you have everything setup correctly. I included hsrp on your border routers, but it isnt necessary, I thought about this, and perhaps however it would reduce convergence when u have a physical problem to an isp, not a black hole..

I did a physical drawing for you this time, because the logical drawing doesnt change...

UnaBomber
ccnp mcse2k

 

[UnaBomber]

For some reason it didnt want to render it, although it did in the preview section

here it is again

UnaBomber
ccnp mcse2k

 

[crazyitguy]

1)If Firewall1 is active and Firewall2 is standby and switch1 goes down, how does traffic get from Firewall1 to the edge routers? I am not sure how ASA failover works(I am assumming it works like PIX) but the connection between the 2 firewalls is a "failover cable". It is for heartbeat type communications.

2)What devices is OSPF configured on?

3)Is VRRP configured so internal hosts can get to the firewalls or is it so the firewalls can get to the edge routers?

 

[crazyitguy]

If the L2 switch ( which is connected to the active firewall) goes down, will the firewalls recognize this and force the standby firewall to become active? According to this article:

http://www.informit.com/articles/article.asp?p=24686&rl=1

A failover occurs when one of the following situations takes place:

*The standby active command is issued on the Primary PIX.

*The failover active command is issued on the Secondary PIX.

*Block memory exhaustion occurs for 15 consecutive seconds or more on the active PIX Firewall

*Network Interface Card (NIC) status. If the Link Status of a NIC is down, the unit will fail. "Down" means that the NIC is not plugged into an operation port. If a NIC has been configured as "down," it does not fail this test.

*Failover Network communications. The two units send "hello" packets to each other over all network interfaces. If no "hello" messages are received for two failover poll intervals, the non-responding interface is put in testing mode to determine who is at fault.

*Failover cable communication. The two units send "hello" messages to each other over the failover cable. If the standby doesn't hear from the active within two failover poll intervals, and the cable status is OK, the standby takes over as active.

*Cable errors. The failover cable is wired so that each unit can distinguish between:
o A power failure other unit.
o A cable unplugged this unit.
o A cable unplugged other unit.

*If the standby detects that the active is powered off (or reload/reset), it takes active control. If the failover cable is unplugged, a syslog is generated but no switching will occur.

If this is true then will the same thing happen if the internal L3 switch dies?

 

[UnaBomber]

It doesnt have to be failover cable, you can set them up over ethernet now... This is irrelievent

Both Firewalls are on the same LAN, so if there is no route to the border router, it will converge on the other firewall, Dynamic Routing.

OSPF on Firewalls, Border Routers, though out your whole organisation, you need a routing protocol, right?

On firewalls, internal hosts... But there are other options for fail-over on firewalls.


UnaBomber
ccnp mcse2k