Emergency Recovery Plan 1

[Sofestibeest]

Hello,

I am working on an Emergency Recovery Plan for my companies network infrastructure in case of a disaster.

This is the first time I have had to create such a plan. Have anyone of you created something like this before and if yes, could you send me a copy.

Our network has 25 servers. among these servers we have SQL 7, Exchange 5.5, 2 Windows 2000 DC, DNS servers, IIS servers, Proxy server, VPS servers and printservers.

Any help would be useful! Thank you!

Robert

 

[Shift838]

There are various types of Disaster Recovery plans. an example would be say you have a server completely crash that has tons of user data on it, running windows 2000. You have been backing up nightly with Differential except for Saturdays and that is a full backup, the server crashed on Tuesday before the Tuesday Differential backup was performed. You need to get the server back up with current data.

1. Notify helpdesk of problem and Estimated Time of resolution in order for the helpdesk to answer questions from the calling users.
2. Perform your Diagnostics to determine if hardware is the issue.
a. If so then replace hardware
b. If not and the OS is corrupt the best solutions a reload of OS/Apps and Data.
3. If new hardware then setup with your configuration utilities in accordance to your load procedures. If existing hardware you should be able to just erase the sytem.
4. Load Windows 2000 in a temporary directory say "w2k" and and backup software.
5. Request a complete restore of all data to its original location from the Saturday backup.
6. Request a differential restore of the differential backup from Sunday and Monday to their original locations.
7. Modify the boot.ini file to point back to the original boot directory (winnt)
8. Notify help desk issue is resolved
9. Notify all users affected that the data is only good up through Monday and all data saved on tuesday will have to be redone.

Hope this helps..

 

[Ticedoff]

Step 1 - Create an Up to Date Resume.
Step 2 - Write a Bogus DR Plan.
Step 3 - When disaster strikes, look for another job.

No kidding.

DR Planning is NOT "How do I do a bare metel restore on a server if it crashes". That is just day-to-day Server Admin stuff.

"Notify the Helpdesk"??? All of the folks at the HelpDesk are DEAD. The HelpDesk is on FIRE, or under WATER, or the building collapsed on it.
Now what?

If you are tasked to come up with a real "Disaster Recovery Plan" (DR Plan), and you have never done one before, then it says there is no real commitment to this plan - starting at the top of your company - The CEO / CTO / CIO needs to be commited to this.
My guess is they have been told by some analyst that their company needs a "Disaster Recovery Plan" to meet some auditors requirment - a check box.
If they aren't fully on board with this, then you are wasting your time. You can't tell them (or your boss) that, of course. You need to come up with a viable looking document, that allows you to keep your job until the Disaster hits.
If they are truly on board, then the first thing is to request training with well quilified and recognized sources. Do a search on Google for Disaster Recovery to find a few. It will cost $$$$ - if they are committed to this, it won't matter - within reason.
You will have the satisfaction of knowing that you did a great job on the DR Plan, and the training will look good on your Resume for a later job.

The basic steps for disaster recovery start with the PEOPLE not the hardware / software. How many of your key IT personnel were KILLED (you??) when the airplane / earthquake / fire / flood / tornado hit your building. That means your documentation has to account for every little detail of the hardware and assume nothing.
Then consider the hardware. How will you replace that old piece of S**T SunOS 4.1 SPARCStation 2 that was your DNS server for the last 10 years. You have to find hardware replacements for EVERYTHING.
Then consider the OS's for everything. Look at your routers for example - How many subnets? How many WAN links? Have you saved the ACLs for these routers? What about firewalls?
Then consider Application software. Is there an old HP3000 runing MPE/IX with MANMAN on it? The hardware is easy to find. You might even be able to find the OS. The MANMAN application though may not be available anymore.
THEN consider the data:
Mission Critical (available in minutes)
Business Critical (Available in Hours)
Business Neccessary (Available in days)

Your CEO / CTO has to sign off on this sorting of the data; otherwise, you end up (assuming you survived the disaster) in the position of deciding which Position / Hardware / Application / Data gets restored first. Do you need Email first? Do you need order entry first? Do you need inventory control first? What about phones? The list is endless but has to be managed.

Staples decided the most important thing to have up and runnig after a disaster was Inventory Control & Distribution. Why? The local Staples stores can sell things using pencils & paper - they can make change using calculators, but they can't sell something that isn't on the shelf - Inventory Control & Distribution.

Then PRACTICE your DR Plan.

My suggestion? Don't ask for a Disaster Recovery Document from a users group. You won't get anything worth while. Develope the skills & knowlege to write your own that is custom tailored to your company.
If the managment of your company will not allow the time and money to for you to be trained in this and write a real DR Plan, then write a bogus plan and update your resume. Then keep an eye open for low flying planes, dark clouds and stay close to the exits.

 

[Sofestibeest]

Hi,

Thanks for the input. It gave me ideas and point of views I did not acknowledge before.

Keep the input coming please!

any suggestions or tips you want te send me: feestbeest@enschede.com

Thanks
Robert

 

[adsc]

Hi,

We had to write a DR plan when the last gulf war (Iraq) happened.

I can’t say it was easy to do it, but it was a really hill.

I have Sun, NT, W2K, Firewall, Exchange 5.5, Routers, & Oracle servers, etc…. about 8 servers. Plus 2 remote sites.

I’ll give you some hits about what you need to do.
First I totally agree with “Ticedoff” said, and would like to add the following:

What type of DR Plan you need?
Do you need to he operating & continuing your work even after the disaster happened directly, with full/partial functionality? (Here you should consider the cost)
Or only you need to be able to restore everything as it was before the disaster happened?

Start with documentation. You have to document all your current setup, this what actually many IT’s ignore it and don’t by attention to it.
And most of the person I have seen they don’t like to document because they are afrade that they could be fired any time if the management have all his work secrets and configuration, which I think it’s wrong thought.

Document all your current applications, databases, setup, server by server, you need to get also some information from the IT key persons.

Then put this plan in a testing procedure (review it), make sure you and your teem didn’t miss any thing.
Whey I am saying documentation? Because when you want to re-setup any of your current systems this will be easy, you have all the old WORKING configuration in front of you.

Then consider to have a “full set” off backup out-side your building, in case of a disaster on the building. You can rent a box save in the bank and store them there, also keep updating this outside backup copy frequently.

HTH

 

[SteveBurkett]

I'll add to ADSC's point on documentation by saying you probably should have a folder prepared for each machine, with all the documentation on its config, plus a copy of all the software installed on it.

If things really kick off then the last thing you want is to have to search round for who last borrowed your CD of Windows 2000 Advanced Server or what have you!

Course I haven't actually got round to making my disaster folders yet... Erk!