Second guessing USN role back

[cantgetright]

I am working on a broken domain in a server 2003 single domain, single site environment configured as the following.

DC1
Server 2003 Enterprise SP2
FSMO: Schema owner, Domain role owner, PDC, RID pool owner, infrastructure owner
Primary DNS Server
Exchange server

DC2
Server 2003 Enterprise SP2
BDC
Secondary DNS server
DHCP Server
Application server
Terminal server
Print Server

We have recently taken over this network from someone else. I know that they recently recovered DC2 with Ghost. I also know the the DC1 was recovered with Symantec Back up Exec System Recovery. We will be redesigning this but I would like to clean up the domain before introducing a new server.

They are experiencing netlogon failures, default profiles load intermittently, Exchange authentication fails when using the domain name rather than the exchange server name. Replication is failing and DHCP no longer authenticate. I originally suspected a USN role back on DC2. After further investigation I am second guessing that because I can not find any evidence of this. The event log has been long over written. We have a script restarting the netlogon service just to keep it alive. We are not getting events 1113, 1115 or 2103 when the service is restarted. Repadmin is reporting the following.

On DC1
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxf8903 @ USN 5101644 @ Time 2008-10-13
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx401bd @ USN 1011169 @ Time 2009-11-06
Default-First-Site-Name\DC1 @ USN 5714752 @ Time 2010-10-09
Default-First-Site-Name\DC1 <retired> @ USN 5664826 @ Time 2010-08-15
Default-First-Site-Name\DC2 @ USN 616743 @ Time 2010-8-19


On DC2
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxf8903 @ USN 5101644 @ Time 2008-10-13
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxx401bd @ USN 1011169 @ Time 2009-11-06
Default-First-Site-Name\DC1 @ USN 5612746 @ Time 2010-06-29
Default-First-Site-Name\DC2 @ USN 696431 @ Time 2010-10-09

Both are reporting them selves to have the higher USN. I do not know why DC1 is listed twice, once as retired on DC1. I also am suspecting that there were a couple other DC's not removed properly. There is a non existent server listed as a domain controller in AD on both servers.

At this point I thought I would attempt a metadata cleanup to remove the orphans before going any further. ntdsutil list servers in site only shows DC1 and DC2 with no orphans listed.

Any help in further trouble shooting this would be greatly appreciated. I am losing what little hair I have left. I would really not like to start from scratch because of the amount configuration that would be involved.

Also, Our redesign includes a new server "DC3" running windows server 2008. This will take over most domain functionality. I would like to see DC2 become a member and demote DC1 to BDC.

Thank you in advance. Please let me know if I can offer any more information to help.

 

[58sniper]

No such thing as a BDC or PDC. Hacing Exchange on a DC is going to be problematic with these issues. You'd be better off moving Exchange off of a DC.

Having Terminal Services running on a DC is also a security concern.

A better (even if temporary) solution might be to spin up some VMs, migrate apps and data, flatten the original boxes and reload, and move stuff back.

It's best practice to have no applications running on DCs other than DNS and DHCP.

Pat Richard MVP
Plan for performance, and capacity takes care of itself. Plan for capacity, and suffer poor performance.

http://www.ucblogs.net/blogs/exchange/

 

[pagy]

The symptoms suggest a USN rollback but your output from what I assume was a repadmin /shoutdvec does not seem to support that.

Are the servers on at least 2003 SP1? If not you will not get any 2095, 1113,1115 or 2103 in the event logs.

Run through some DNS checks. run dcdiag /test:dns
dcdiag /test:replications

I would consider DC1 to be the more reliable here as it was at least recovered with a proper back system, although I don't know whether backup exec sys recovery works for a DC. DC2 being restored from a Ghost image is bad mojo.

I think the thing to do here would be to;

grab a ntbackup of dc2
if dc2 has the fsmo roles move, or seize, if you have to to dc1
demote dc2 out. with a /forceremoval if you have to
clean up dc2 metadata
clean up dns srv records of dc2
then see how things run with dc1

Post back and let us know



Paul
VCP4

https://www.mcpvirtualbusinesscard.com/VBCServer/PaulP/profile

RFC 2795 - The Infinite Monkey Protocol Suite (IMPS)

http://www.faqs.org/rfcs/rfc2795.html

Difficult takes a day, impossible takes a week

 

[pagy]

repadmin /shoutdvec

Obviously I meant repadmin /showutdvec ;-)

Paul
VCP4

https://www.mcpvirtualbusinesscard.com/VBCServer/PaulP/profile

RFC 2795 - The Infinite Monkey Protocol Suite (IMPS)

http://www.faqs.org/rfcs/rfc2795.html

Difficult takes a day, impossible takes a week