San Francisco T-1 flooded out, questions on fail over possibilities

[BigShyBear]

I am the network administrator with a company that has offices in Los Angeles, Las Vegas, San Francisco and San Diego. Since the rains the T-1 to our San Francisco office is down with no quote yet on when it will be up.
I have been told to present options on modifying the network so there is a resilency in the network in the future. I know that a dial up backup won't provide enough bandwidth it so I am looking for a better solution.
Strangely enough, the phone service to our San Francisco office never went down, just the T-1.

Current configuration:
5 T-1 lines total, Los Angeles has 4 of them, and 1 runs from San Francisco to Las Vegas.
Los Angeles - 4 T-1's, 1 each to San Diego, Las Vegas and San Francisco, 1 for internet. 3 routers, Cisco 1720 for the internet, outside our firewall, 2 Cisco 2620's, one with 2 T-1 WIC cards - for San Francisco and Las Vegas, and 1 2620 with a single T-1 WIC for San Diego. San Francisco and Las Vegas have a single 2620 with 2 T-1 WIC's, one for the run to Las Angeles, one for the run to each other.
San Diego has a 2620 with a single T-1 WIC, for the run to Los Angeles.
Does anybody have any suggestions for me to look at that won't cost us another $5000 a month. (That's what we are paying for our current T-1 lines)

 

[artesian2000]

Are these point to point lines? I have an idea or two you might want to kick around.

 

[BigShyBear]

Yes they are.

 

[artesian2000]

Have you consider moving you network onto an MPLS based solution? With that the sites would be automatically messed with eachother within the network. No more need to to manage multiple PL's going to different locations. LA sounds like your hub location where your apps/databases reside. So you would need to set up setup one of the othe remotes as a possibly secondary hub location. Not maybe all your apps but ones critical to your business. With that said in the event of your hub location going down you can still conduct your business day to day with the impact of losing everthing. Just a thought here more than anything. Without a full understanding on how your network traffic flows building disaster recover can be a little hard to judge. MPLS may be able to save you some $$ verses a PL network and could help in reducing the complexity of managing your network.

 

[JOAMON]

If the phone service stayed up then good chance a DSL circuit would also. DSL relatively low cost and could work as a backup should you lose your T1. T1 probably went down because the CO flooded. If DSL terminates to the same CO then likely it would not fair any better than the T1. For it to be redundent failover it must also reside in a differnet CO to eliminate both failing if the CO goes down.

 

[BigShyBear]

I just got word from Broadwing that "technicians are at the manhole that is flooded and they are pumping it out.' yeesh.
How does MPLS work? Not the internal tech details, I've read the docs on MPLS, but the hardware and software pieces at the router end I would need. Do I need a separate T-1 from each location from a T-1 WIC in the Cisco 2620's? and do I use Cisco Express forwarding in the routers between the two lines?
I was also looking at the ADSL cards and seeing if the 2620's could use CEF between the T-1 and an adsl line, and ipsec encryption on the adsl line.
My goal is to get a parallel link that won't be affected by backhoes or squirrels or rain flooding manholes.

 

[BigShyBear]

A side note, I just found out we flew one of our secretaries up to San Franscisco this morning with all the paychecks for all the people working out of the San Francisco office. $300 for a round trip ticket just to drop off a box of paychecks. All because the link is down and the payroll dept could not print checks up there like they are used to.

 

[lgarner]

I'd consider a VPN over a wireless Internet connection. Any land-based connection is going to be subject to the same problem, though it can be mitigated by using two local carriers, two points of entry into your building, etc.

You could use CEF, but I'd use the "lesser" connection as a backup route.

 

[BigShyBear]

We looked at wireless, but we can't find anybody that services the physical location where our San Francisco office is located. Technically it is near Port Chicago which is north and east of Oakland on the east side of the bay behind some mountains.

 

[JOAMON]

DSL or Cable internet for backup.

 

[plshlpme]

you have to look to your telco and see if they can offer you some diversity..
you have to be specific though.. because there are many knids of diversity...

CO diversity.. Manhole diversity etc..
they may be able to bring you a redundant pair in that is on a totally seperate cable system entering the building from a different location and of course routing back to the CO via a different path.

of course the more diverse.. the more you pay.
are you looking to add this diversity to all locations though or just the sanfran office?

also with the MPLS solution.. the telco will in most cases provide a CE router to you that you can connect your router too.. this will create a mesh network. You would then just need to make sure you have a backup at your main hub site where all your applications and db's are served from.