Business Continuity Project

[mayurkirti]

This is a really innovative project for me. We are a mid size IT company with around 350 employees. As a part of my support engineer job I am working on a business continuity and disaster recovery project.

Here's where I need help: Our IT infrastructure is a combination of physical and virtual windows servers. We mostly use Windows server 2003, VMWare, Cisco and XP pro in our environment. One problem that an on-call engineer faces in the event of disaster is to successfully shutdown the critical services before UPS runs out of power. Gracfull shutdown can prevent the hardware from powering down abruptly and other evident damage. I am trying to automate this process considering interdependencies between devices (clients/servers/networking stuff/ virtualization) and designing the correct sequence of powering down the services.

One challenge is the dynamic environment where severs change every day, new services are added and removed. I also have to factor in the business needs so that if necessary, I should be able to shutdown X non-critical services and still keep the Y number of business-critical services running.

I need some ideas and implementation suggestions for this project. If anyone has done this in the past or if this interests you, please drop some notes. I will appreciate it. Let me know if you need any other details.

Thanks,
Mayur

 

[mayurkirti]

anyone and ideas?

 

[A0C61ZZ]

Mayurkirti,
I have worked on Application BCP (Disaster Recovery) for a reputed Wall Street financial institution for the last 3 years, this being my 4th year.

1. Typically we start with creating BCP runbook detailing the applications involved.
2. This includes providing business process flows and application setup flows. You're already working on the correct sequence of powering down the services. However these devices/applications would have a different flow/dependencies of their own compared the business flow.
3. You will need to start with both the business and device flows separately.
4. After this step you will need to evolve this and intermingle both, so as to achieve a common flow which caters to both the business and h/w dependencies.
5. Break down each task (be it business or hardware/application) at the smallest level and work your way upwards.
6. Bottom-up approach helps for this compared to top-down approach.
7. Each task, should have an entry criteria that has to be satisfied before it can kick-in.
8. Each task should have an exit criteria that needs to be set for the next task to fire.
9. Assign priorities/criticality to each task
10. Ask your stakeholders, how realistic they want this BCP to be. If they're ok with a simulation that guarantees basic readiness then you may NOT a detailed approach as say compared to real production failover scenario. Get this clarified first.

Most important FREEZE the SCOPE of the BCP - Identify and PINDOWN the exact (and critical) number of business processes and devices you plan to failover in this test. You can improvise and build more on this in the subsequent years.

Let me know if this works.

- A0C61ZZ.

 

[mayurkirti]

Hi A0C61ZZ,

I was meaning to reply to your post for a while but I waited to see what I can get out of your suggestions. Your reply actually helped me a lot. Right now I have all business related services mapped out as they relate to our hardware infrastructure. As you said, once this has been done, I will have to find a sequence to power down the services (or h/w) when required. This also gives me the picture of what can be shutdown and when, so that certain business critical services can be prioritized when there are limited resources (UPS/generator power/Air conditioning/space).

After I nail down this first task, I need to find out a way to automate this process. I dont have any resources to develop an application in-house. I have no idea what my options are and how to go about transitioning this logic into an application.

Let me know if you have any ideas or suggestions.

Thanks,
Mayur