Multiple interface failover... How to set up?

[jonheese]

Okay, I'm not sure if this question belongs here, so let me know if there's a better place to ask. Here goes:

I've got a trio of Win2k3 servers here in the office, along with a handful (~15) of workstations. Our T-1 connection to the office has been a little flaky recently, so we're going to get a DSL connection, to be used as a backup to the T-1, for the times when it's down.

So, each of the servers (and the ISA server for the WSs) will have 3 NICs, as follows:

NIC 1: Local subnet (i.e. 192.168.3.x)
NIC 2: Public IP from T-1 router
NIC 3: Public IP from DSL router

Now, what I want is automatic failover: Use the T-1 for outbound connections all the time, unless it goes down, at which point it should switch to the DSL connections, and switch back again when the T-1 comes back online.

Now, I know the theory of setting the metrics on each NIC appropriately so it will use the T-1 NIC by default, and will switch to the DSL NIC if the T-1 NIC goes down. The problem is that in this failure mode, the T-1 NIC link is still "up". That is, it still has physical link with the router (next hop), but the second hop is down...

Is there a way to do this?

I've tested this on one of the servers already, and it works perfectly if I unplug the T-1 NIC, or pull the power on the T-1 router, but if I just pull the T-1 line out of the T-1 router (killing the next hop, obviously), the server just keeps dumping packets into the abyss...

Any and all suggestions welcome.

Regards,
Jon Heese

 

[cajuntank]

One public facing security appliance.
Natting public to private for each server. Allowing specific open ports to those servers. Not quite sure how you are using your ISA (if just a proxy or what), but that's it.

I have done this many times with Cisco ASA and Sonicwall appliances with companies connecting to T1 for primary and cable/dsl/wireless/T1(other carrier) for backup.

You just need the one appliance. If your using software based firewall on your servers, your natting anyway.

I just used the Linksys as a last resort. I would recommend at least a Sonicwall that will give you a more hardened OS as well as some threat management choices like IPS and Anti-X at the firewall level. A TZ170 from Sonicwall is a great choice. You can purchase the options ala cart or you can get the unlimited user total security bundle which gives you 1 year of all the threat management services along with the enhanced OS for failover. That one will probably cost a little over a 1K but well worth the money for all the protection you get. You can get a lower end model, the WAN failover option, and licensed for say 25 or so users for less money. You'll need to talk to a vendor for pricing and options.

 

[jonheese]

So you mean a single firewall device with 8 public IPs (2 for each server)?

Regards,
Jon Heese

 

[jonheese]

Is the SonicWall TZ170 capable of being assigned multiple public IPs for each WAN link?

eg: Say the T-1 IPs are 1.2.3.[1-4] and the DSL IPs are 4.3.2.[1-4], and the 2 WAN interfaces on the SonicWall are eth0 and eth1. Can I do this:

eth0:1 - 1.2.3.1 (NAT to server1)
eth0:2 - 1.2.3.2 (NAT to server2)
eth0:3 - 1.2.3.3 (NAT to server3)
eth0:4 - 1.2.3.4 (NAT to ISAServer)

eth1:1 - 4.3.2.1 (NAT to server1)
eth1:2 - 4.3.2.2 (NAT to server2)
eth1:3 - 4.3.2.3 (NAT to server3)
eth1:4 - 4.3.2.4 (NAT to ISAServer)

Does this make sense? This is what I'd need to happen in order for something like this to work... I'm assuming from Pat's post above that the Cisco devices will do this sort of thing, but I'm curious if the Sonicwalls will...

Regards,
Jon Heese

 

[cajuntank]

I don't see a post I replied to you earlier today about this. I now understand want you want. You are wanting failover both ways, in and out. From talking to my ISP buddies a while back, you can do this but you have to get the ISP(s) involved to coordinate with each other in their BGP routes. The WAN failover for outbound is easy since its on your side, but the inbound does not involve you and is kind of tricky since both ISP(s) have to have failover routes to the each other for your subnets since that failover has to happen before it ever gets to you. Think of it like this, if your web server was

http://www.abc.com

and resolved to 1.2.3.1, even if you had a firewall that could have two IP address natting to the same box, the internet will never know about that other address since by default there is no route being broadcast from your 1st ISP to route to another ISP should that first route fail.

If you can get that addressed with your ISP(s), I think you can do what your wanting to do. You might need to touch base with Sonicwall or Cisco, or whomever to get the right appliance to do such a enterprise level procedure. My fist guess would be the smaller appliances might not be capable and you might have to scale up to say the Pro series in the Sonicwall.

Good luck and let me know how it turns out. My first guess to you however is since you don't have two enterprise level connections, I think you going to have trouble getting your ISP(s) to play nice nice with each other, but who knows till you try.

 

[jonheese]

Actually, I was just thinking about round-robin DNS with some SRV entries to give a higher preference to the T-1 IPs... Does that make sense?

Regards,
Jon Heese

 

[cajuntank]

Sounds like it should work... I think you'll still need to touch base with Sonicwall or whoever to get the right appliance model for those dual IP address requirements.

Let me know.

 

[cajuntank]

Found out that with round robin, because it can't tell if a route is down or not, you'll run into the potential issue that you'll have a route down and to the end user if it happens to be that path's turn, it will fail to resolve. So the only guaranteed method of having this work is the BGP scenario. The problem you'll run into with this, is that you are wanting a enterprise level solution on a small business price tag. You don't tend to find many ISP's that will both do this unless your paying big monthly dollars. But who knows... it doesn't hurt to ask.

 

[jonheese]

Yeah, I already shot down the round robin idea because it isn't "smart" enough...

Instead, I think I'm going to go with low DNS TTLs and a script that swaps out the A records on the DNS server as the links go up and down.

Thanks!

Regards,
Jon Heese

 

[Knackster]

Jon,

let us know how this works out for you.

But, for WAN failover and multiple servers, I use a Watchguard Firebox x750e with 6 public ip addresses set up for auto WAN failover. Once the NATing is set up, all you need to worry about is the WAN failover setup which is built in to the device.

Chris
IT Manager
Houston, Texas

 

[cajuntank]

Chris, the issue he is trying to deal with is not from traffic originating in to out, but from traffic originating out to in perspective.

 

[jonheese]

Well, actually, it's both. But the latter is definitely posing a bit more of a challenge.

Regards,
Jon Heese

 

[cajuntank]

Well the traffic from in to out is easy... It's handled like we've been discussing from a firewall appliance that has WAN failover port(s) for your ISP(s). But like I mentioned, the traffic originating from the outside coming in is going to be contingent on your service provider(s) telling people on the outside how to get to you (normally handled through BGP to establish multiple routes to different subnets (multiple ISP(s)).

 

[jonheese]

Nah, that's WAY more than our ISPs will do. I'm going to "keep it simple, stupid" and just go with a script that switches the DNS records for the servers in question based on the WAN link that's active at that time. If I keep the DNS TTL values low (i.e. 5 minutes or so), we should be able to balance the switchover downtime and the increased load from more DNS requests.

I'm just hoping we don't have a problem with DNS caching servers that don't enforce TTL values properly...

Until we actually get the hardware (we're going with a Cisco 1841, BTW), I won't know exactly how I'm going to trigger the DNS updates, but I have confidence that we'll be able to figure something out.

Thanks for all your help, cajuntank, and others!

Regards,
Jon Heese

 

[wdoellefeld]

jonheese,

If you have any luck getting two internet providers to cooperate together routing traffic on your public IPs, let me know.

(yay! shameless advertising. my side business)

http://www.frcp.net

 

[jonheese]

To tell you the truth, if our ISPs were that willing to cooperate and fill our needs, we could probably just get the T-1 to be stable and we wouldn't have to spend $3000 on Cisco hardware to make this work.

As it is, they know that they are the "only game in town" and we are stuck with them. They are totally happy sitting on their hands taking our money, so I don't see why they'd take the time and effort to go that far beyond the call of duty...

Regards,
Jon Heese

 

[hairstraightback]

Well, here is a very low budget idea, not totally fleshed out, you could look at. Use a service like dyndns, have all of your inbound traffic connecting through the url you choose as your hostname, and run the dyndns client software on each of your servers. I'm pretty sure you can set up your client software to cycle the address as frequently as you want, set it up on a short cycle, and it should work. This would not offer "failover" to connected clients, if a link went down while they were connected, they would have to reconnect, but it would be very easy to implement and try.

 

[jonheese]

Yes, that is a viable answer to the inbound DNS failover problem. Thanks for the suggestion. I know that No-IP.com has something like a 5 minutes TTL, so that should be fine.

Of course, we will still need the hardware to do the outbound failover and manage the two WAN links, and that purchase is already in the works anyway.

Regards,
Jon Heese