Unable to mount DB after power down of AD/Exch in Primary site 2

[davy2k]

Dear All,
We have 2 AD/Exchange 2010 sites (each with its own AD). During a user acceptance test where it is required to power down both AD/Exchange at the primary site.

I get the message:

Error:
Couldnt't mount the database that you specified. Specified database: TKY-DB: Error Code: An active manager operation failed. Error: The MS Exchange Replication service may not be running on the "Exchange on Primary Site". Specific Error message Error 0x6ba (The RPC server is unavalable) from cli_GetPrimaryActiveManager..

Error: 0x6ba (The RPC server is unavalable) from cli_GetPrimaryActiveManager..

Also, the cluster manager is empty.

I have also configured the preferred server on the secondary site to be the AD on that site using the: "Set-ADServerSettings command. This time the database shows unknown in the Mounted column in the EMC. and the error below:


[PS] C:\Windows\system32>Test-ReplicationHealth | fl Error


Error :

Error :

Error : Active Manager is in an unknown state on server 'LRGFUK-EXCH'. Basic da
tabase administrative operations such as mounting or dismounting and th
e ability to have failovers or switchovers is not available. Review the
event logs for more details. Error: The server is not yet a fully oper
ational member of the database availability group.


Error :

Error :

Error : Failures:
Server 'lrg-exch' is a member of database availability group 'LR
GDAG-DR' and is not up.
Server 'fuk-exch' is a member of database availability group 'LR
GDAG-DR' and is not up.




[PS] C:\Windows\system32>Test-ReplicationHealth -Identity lrgfuk-exch

Server Check Result Error
------ ----- ------ -----
FUK-EXCH ClusterService Passed
FUK-EXCH ReplayService Passed
FUK-EXCH ActiveManager *FAILED* Active Manager is in a...
FUK-EXCH TasksRpcListener Passed
FUK-EXCH TcpListener Passed
FUK-EXCH DagMembersUp *FAILED* Failures:...

Your help will be highly appreciated.

Thank you

 

[ShackDaddy]

Is the file share witness server online? If it is not, any database that is a member of a DAG may not be able to come online.

Dave Shackelford
ThirdTier.net
TrainSignal.com

 

[davy2k]

Hi Shackdaddy,
Thank you for your response. I realised after my post and more troubleshooting that the FSW server was offline that was why the DB refused to mount.

My questions are:
- since the FSW can only be on a single site at a time so if the site hosting the FSW goes down then the DB on that site will be unavailable, is there a work around for this? Since there are only 2 sites.
- what is the purpose of the Alternate witness if the servers cannot failover to this alternate FSW

Also, the failover cluster GUI is empty. I tried to switch to the FSW on the site that is up and running (using command line) I got some errors about cluster service/RPC not running on the server on the failed site. I tried from the properties of the DAG, this was greyed out as well.

Thank you

 

[ShackDaddy]

I would check out this article:

http://port25.wordpress.com/2009/12...er-recovery-scenario-in-exchange-2010-part-3/

To bring up a database that's in a different site than the FSW, you are going to need to use the stop-databaseavailabilitygroup command to "turn off" the servers that are down in the site that had the FSW. Once you've done that, you should be able to mount the database at the other site. The process is always going to be manual.

After you've done that manual failover to a new site that has more than one database server, then an Alternate Witness would be useful, to play witness in that new site. But it only comes into play once you've already done what I listed earlier. All this "no-worky" is trying to avoid a split-brain situation in which the connection between the sites simply fails and both sites think that they are the production site.

Dave Shackelford
ThirdTier.net
TrainSignal.com

 

[davy2k]

Hi ShackDaddy,
Thank you for your invaluable advice.

I worked on this in a test environment and it worked nicely. here are the steps I took in resolving this issue.

SiteB is the host for file share witness. I powered down both exchange/AD in SiteB.

- I removed the "Default-First-Site-Name" and created separate sites, SiteA and SiteB
- used the Get-DatabaseAvailabilityGroup | select Name,*server* to view all started/stopped mailbox servers
- used the Stop-DatabaseAvailabilityGroup –Identity DAG1 – ActiveDirectorySite SiteB -ConfigurationOnly (ignored the error message regarding the non availability of server in SiteB)
- ran the Get-DatabaseAvailabilityGroup | select Name,*server* command again to view all started/stopped mailbox servers
- ensure that the cluster service is stopped
- ran the Restore-DatabaseAvailabilityGroup –Identity DAG1 –ActiveDirectorySite SiteA (this restored the alternate file share witness to SiteA that is up and running and then mounted the database...)

However on the outlook 2003 client I get the 0X80004005 Microsoft Exchange offline address book synchronization error. (I use the haproxy load balancer as the client access server). Any idea why this is occurring. I am about to test using outlook 2007 client.

Thank you for your assistance.

 

[58sniper]

Giving ShackDaddy a purple start for his excellent answer. Feel free to do the same.

Do you have your Tek-Tips.com Swag? I've got mine!

Stop by the new Tek-Tips group at LinkedIn.