Commvaultdude
Technical User
Just thouhgt I would post this in case any one every needed it.
Background
Exchange administrator is using Perfect Disk to defrag the DB's on a Sunday evening during our maintenance window, Perfect Disk deletes the DB that he was defragging, I am not kidding here it just disappeared, he then calls me to restore the DB. Problem is the last backup was a Friday night so we are going to lose email on Saturday and Sunday, we have the logs from those days but they are not in the backup, I know I know, replay the logs, this is where it got interesting.
According to Microsoft you just need to restore the DB, delete the check file, mount the DB and presto it should read all the logs, BS, does not happen this way.
After a lot of research I discovered that if a DB has been deleted then when you restore it is called a Victimized database. You have 2 options here (neither of which we were aware of Sunday night when we were restoring the DB). Option 1: when restoring the DB locate the folder that Commvault is using to store the temporary logs to(stores them in commmvault systems, galaxy, etc......) Copy all the logs after the last backup up until the issue occurs into the temporary folder, when the restore finishes, it will replay the logs, you can mount the DB and you are done. Option 2, this is what we had to do since we had already recovered the DB that was deleted. Create an RSG in Exchange 2003, add the DB you want to restore and then start the restore in commvault same thing as above, copy all the logs into the Commvault temp location and then when the restore complete it will replay the logs and voila all email is back, from here you can go into the RSG and use Exchange tasks to push the email back to the DB. In the 2nd option the restore may throw some error message about failing at around 94% but you can ignore it, the restored DB will still mount and you can still locate and merge the email back to production.
Basically what we did is below:
1. We created a recovery storage group on our production server and configured it for that database in question.
2. We had Commvault run its restore job to that RSG.
3. While Commvault was restoring the db to that RSG we located the temporary folder Commvault was going to use to replay the log files that it had backed up on Friday. I found what the last log file it had backed up and copy the next one in sequence including all the other log files from over the weekend to that temporary folder that Commvault was going to use to replay it logs.
4. When the restore of the database finished, Commvault put its logs into that temporary directory it created to replay its logs and we figured that Commvault wasn’t smart enough to realize that additional logs files were there and we were correct. It replayed all the log files in that folder including the ones from over the weekend .
5. We mounted the DB in the RSG. Then we selected all the mailboxes (in the RSG), right clicked and chose Exchange Tasks.
6. That gave us the option to Merge any missing emails back up to the production database.
7. Success.
I hope this helps someone out in the future, took me about 3 hours on Monday morning to locate this and get it working so not too bad, people were missing some email for about 3 hours when they got into the office, we were lucky and there were no execs in this DB but I will take 3 hours any day of the week.
Background
Exchange administrator is using Perfect Disk to defrag the DB's on a Sunday evening during our maintenance window, Perfect Disk deletes the DB that he was defragging, I am not kidding here it just disappeared, he then calls me to restore the DB. Problem is the last backup was a Friday night so we are going to lose email on Saturday and Sunday, we have the logs from those days but they are not in the backup, I know I know, replay the logs, this is where it got interesting.
According to Microsoft you just need to restore the DB, delete the check file, mount the DB and presto it should read all the logs, BS, does not happen this way.
After a lot of research I discovered that if a DB has been deleted then when you restore it is called a Victimized database. You have 2 options here (neither of which we were aware of Sunday night when we were restoring the DB). Option 1: when restoring the DB locate the folder that Commvault is using to store the temporary logs to(stores them in commmvault systems, galaxy, etc......) Copy all the logs after the last backup up until the issue occurs into the temporary folder, when the restore finishes, it will replay the logs, you can mount the DB and you are done. Option 2, this is what we had to do since we had already recovered the DB that was deleted. Create an RSG in Exchange 2003, add the DB you want to restore and then start the restore in commvault same thing as above, copy all the logs into the Commvault temp location and then when the restore complete it will replay the logs and voila all email is back, from here you can go into the RSG and use Exchange tasks to push the email back to the DB. In the 2nd option the restore may throw some error message about failing at around 94% but you can ignore it, the restored DB will still mount and you can still locate and merge the email back to production.
Basically what we did is below:
1. We created a recovery storage group on our production server and configured it for that database in question.
2. We had Commvault run its restore job to that RSG.
3. While Commvault was restoring the db to that RSG we located the temporary folder Commvault was going to use to replay the log files that it had backed up on Friday. I found what the last log file it had backed up and copy the next one in sequence including all the other log files from over the weekend to that temporary folder that Commvault was going to use to replay it logs.
4. When the restore of the database finished, Commvault put its logs into that temporary directory it created to replay its logs and we figured that Commvault wasn’t smart enough to realize that additional logs files were there and we were correct. It replayed all the log files in that folder including the ones from over the weekend .
5. We mounted the DB in the RSG. Then we selected all the mailboxes (in the RSG), right clicked and chose Exchange Tasks.
6. That gave us the option to Merge any missing emails back up to the production database.
7. Success.
I hope this helps someone out in the future, took me about 3 hours on Monday morning to locate this and get it working so not too bad, people were missing some email for about 3 hours when they got into the office, we were lucky and there were no execs in this DB but I will take 3 hours any day of the week.