How to Recover Exchange 5.5 - and troubleshoot errors.

Exchange 5.5 Disaster Recovery White Paper
By: Al Rozon û System Administrator, MCSE, MCP+Internet

Introduction:

This case study is to outline the steps needed to the recovery of a Microsoft Exchange Server 5.5 server w/ SP4, which resides upon a Windows NT 4.0 SP6 server and a Blackberry Enterprise Server.

Materials needed during this entire installation process, and things you should know before going further.
- Know the hardware in the machine that is being used for Exchange; have all the necessary drivers.
- In this case, we needed 3 machines.
o Exchange Restore Machine w/ Backup Exec.
o PDC (Primary Domain Controller) w/ mass storage (80gigs)
o BlackBerry Server on NT 4.0
- Have the Microsoft Exchange 5.5 Installation Disk and all Exchange Service Pack Disks.
- Have all appropriate device drivers.
- Have Veritas Backup Exec 8.0 Disks.
- Have Sybari Antigen Disks.

Disaster Recovery Strategy:

Planning is essential to a good recovery. The strategy listed here is one of many different strategies one can take. This is the one I have chosen for our corporation. It has been tested and has been proven successful.

When recovering an Exchange 5.5 server, it is absolutely imperative that it has access to the ORIGINAL SAM from the PDC of the original Domain. That said, there are three ways one can proceed to ensure that the SAM is available.

1. Use an NT Service aware Backup Program like Veritas Backup Exec to backup the PDCÆs Registry and SAM.
2. Install a temporary BDC and use Symantec Ghost to get an image of the installation.
3. Install a permanent BDC on the original domain and store it offsite.

I have chosen plan 3. Ideally, make an installation of your new BDC on a laptop. Laptops are portable enough so that if there are any major corporate changes to the Account Database, the laptop can be easily brought back into the domain and synchronized with the PDC; obtaining the newest possible Account Information. IN the event of a catastrophic failure (IE: Sprinklers go off and pooch your entire domain), you have a working Domain Controller that can be promoted to a PDC, giving you an active Account SAM Database without even having to restore a thing.

Next, you must install your New Exchange Server.
The first thing that you must do, is delete the current Exchange Server Account from Server Manager in your RECOVERY ENVIRONMENT ONLY.
It is very important to note that you must install the new server into the same state as your original server. Document all the paths to the MTA, Log Files, and to the Database files (PUB.EDB, PRIV.EDB). And make sure that the Site Name, Organizational Unit names and the Server name are typed identically to your production server. These values are Case-Sensitive (we found this out the hard way). Should you miss a capital letter anywhere in these values, the restores will not work.

The Backup type we use is an ONLINE Backup (Which is to say that services are live as the backups run).
I have Circular Logging off. This give us the best possible chances for a full and successful recovery. With Veritas Backup Exec, once logs are backed up on the tapes, the logs are trimmed down and kept to a manageable size û so circular logging is not needed in our case.

If the backup strategy is one of Offline Backups, Circular Logging is acceptable because when the services are turned of, all changes made during that day, are committed to the databases. Therefore, the log files would not be needed to recover the databases. To turn Circular logging off and on, you would go into the properties of the server:

Site\OU\Servers\Server-name
FILE | Properties | Advanced Tab.
NOTE: It is here, also, where you can find the paths to all of your Database Files and logs to document from.

Ensure that all the proper connectors and add-ins are installed. IF you have a Blackberry Enterprise Server (BES), then you must install another server to be the new BES for your new server. Name the Server the same as your active BES. Ensure that the Mailbox and NT accounts used in your new environment are identical to that of the production environment.

We then installed our Anti-virus software, Antigen by Sybari. Of course, this integrates directly into the server, and Mailbox and NT accounts have to be properly associated as they are in your Production Environment.

At this stage, collect the appropriate backup tapes to recover. Inventory and Catalog all the tapes for your restores first before you begin with the restoration. Depending on the size of your Information Store and the speed of your backup tape drive, the catalogs can take a couple of hours.

Restoring Procedures:

After all the appropriate Cataloging has been done, you may wish to shut down all the Exchange services, and copy the PRIV.EDB and PUB.EDB to another location. This is a good practice so that if your first attempt at restoring your server is unsuccessful, you simply copy the PRIV and PUB back to their original location, delete all log files and checkpoint files, and start the services into a pre-restoration state.

1. First, you must stop all services except the System Attendant.
2. Restore the Information store, along with all of its associated Incremental backups (if there are any). With Veritas Backup Exec, use a NO LOSS recovery.
3. Delete everything within the DSADATA folder û this will effectively delete your current DS. This is a MUST. Without this step, upon restoration, the DS will fail to start.
4. Then Restore a Directory Store that is OLDER than the Information Store youÆve just restored. Two weeks or two months, it does not matter how old it is. It simply should not be as up to date as your IS. When starting the IS later on in these steps, the IS will play out the logs and bring the DS up to date.
This completed, delete the DSÆs checkpoint file: c:\exchsrvr\dsadata\edb.chk. (Page 5 explains Checkpoint Files).
Then start the Directory Store Service.
With the DS started, start the Information Store û KNOWING that it may fail. Monitor the Event Logs in NT to ensure that the Logs are being replayed by pushing F5 over 5 second intervals. It will/should fail with an Error 1180 û Database is inconsistent. This is expected. If it starts, you may feel free to start all other Exchange Service and skip the Troubleshooting Steps below.
Start the services in the following order once the IS has started:
Event Service
IMS
MTA

Troubleshooting Steps:

In the event that your IS did not start as mentioned above, the following steps must be taken to make your databases consistent and ported to the proper Global Unique Identifier.

The first pre-emptive step is to delete a registry key that will hinder the running of the ISINTEG utility. This key is:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\SERVICES\MSEXCHANGEIS\RestoreInProgress
You must delete the RestoreInProgress key. With the logs having been already played, this key is not needed and will only hinder progress later on, as the registry believes the restore failed. When it tries update the GUIDÆs later on, it finds that the Information Store is more up to date than the logs showed in the RestoreInProgress Key, and therefore the ISINTEG fails.

WARNING: Messing around, or deleting the wrong key in the registry COULD make your server useless. Be comfortable with what you are doing.
To look this up, go to: http://www.microsoft.com/technet and do a search for RestoreInProgress. There are many kbÆs on this topic. Follow steps from the Microsoft Knowledge Bases as I have to be sure you are doing the right thing. This document is, but, a guide, not the bible, on restoring an Exchange Server.

ESEUTIL.EXE / EDBUTIL.EXE
To make the, now, restored databases consistent, you must run the ESEUTIL.EXE command. Copy the c:\exchsrvr\bin directory to the largest partition on your server (usually this is where the PUB.EDB and PRIV.EDB files are).
When running this utility with Exchange 5.5 or greater, it swaps information in and out of a temp file that it creates. It will take up a minimum of 40% of the size of your Information store, and so enough space is need to carry out this function.
If you are using Exchange 5.0 or earlier, you must use the EDBUTIL and its temp file can grow in excess of the size of your IS.

This document assumes that you are using Exchange 5.5 or greater.
Open a Command Prompt.
Browse to the correct drive where you copied the exchsrvr\bin directory
Go into the BIN directory
Type this command to make the Private Store Consistent:

Eseutil.exe ûP /ISPRIV
Depending on the size of your Private Store, this can take up to an hour or more.
You may monitor the progress with the progress meter this utility provides you.
Once this is complete, you must make your Public Store consistent.
Type this command:

Eseutil.exe ûP /ISPUB
Usually, this takes considerably less time. 15 to 30 seconds, depending on how many Public Folders you have. IF you run a list server û it may take a lot longer.

ISINTEG.EXE

ISINTEG.EXE, when restoring an Exchange Server an a physically-different machine than your production server is a æmust-runÆ utility.
When restoring to a different machine, the SIDÆs are different, no matter what. Each mailbox and every single container within the Exchange server has a unique identifier called a GUID, or Global Unique Identifier.
As it stands right now, at this stage of the game, the data within the Databases contain the GUIDÆs of your production serverÆs Directory Store and they do not ôjiveÆ with that of your current server. The ISINTEG utility will reach into your new registry, find out THIS serverÆs GUID and apply it to all the containers and mailboxes.

ISINTEG.EXE is located in the exchsrvr\bin directory. Your Command Prompt should still be open.
Type this command:

Isinteg.exe ûpatch
It will take a little while û but will conclude with a message that your databases have been successfully updated.

Checkpoint files:

Checkpoint files keep track of which logfiles are current, and which ones need to be replayed in the event of an unscheduled stoppage of the services (like you accidentally pushed the reset button). Before starting the services, the checkpoint files need to be deleted, because these files contain information based on your primary installation before you restored. If you do not delete these files, services will fail to start.
Rest assured that the files are automatically regenerated upon restarting the Directory and Information Store services.

Usually, these checkpoint files are located at C:\exchsrvr\dsadata (for the directory store checkpoint file) and c:\exchsrvr\mdbdata (for the Information store). They are both called EDB.CHK. Delete these files.


CONCLUSION:
Once all of these steps have been done, you are now free to start all of your services.

DS should be started.
System Attendant should be started.
Next is the Information Store.
Next is the Event Service.
Next is the Internet Mail Service
Next and finally will be the Message Transfer Agent Services.

If you have any questions, or confused .. or feel that it's not written clearly enough - i'm always open to suggestions to making it better. I have used this countless times.. and it works..

If you have Q's or comments and feedback.. email me, pls, Alshrim@rogers.com