!!Extremely messed up domain-wide network problems!!

[aziel11]

I am relativly new to the IT world, about two years ago I was thrust into the position of running a computer network of about 200 computers, 4 servers and 500 users. The network was grown up over time and everything was a mess when I took over. Everything seemed to run fine untill about four months ago when something happened that caused many of the computers to loose their connection to the domain controller. We were running all windows server 2000 with one gateway server handling DNS, DHCP, WINS. One server running exchange and the primary DC. One server doing file sharing and running some server/client software and a backup DC.
I added another server as a DC with a clean install of 2003. Since then I have taken the gateway server offline and replaced it with a clean install of 2003. The current setwup is this:
One server (2003) running DNS, DCHP, WINS, the internet gateway, VPN, and its a DC. One 2003 server just doing backup DC. One server (2000) doing file sharing and software. One server (2000) doing Exchange 2000 and DC as well as secondary DNS. Right now I can't get a VPN connection to the gateway server, it says the username password is invaild on the domain. If I terminal server into that server and restart Routeing and Remote Access service then the VPN will connect, but I have to restart that service everytime I want to VPN in. Also occasionally I can't terminal server into that server, I get a RPC server is unavailible error. If I physically log in at the console and restart the RPC locator service then I can log in remotely again. The exchange server is off site and there is a VPN link through the routeing and remote access controll panel, that link is allways functional and seems unaffected by the above mentioned problems.
Also all the windows NT machines are unable to loginto the domain and must either use local accounts or cached accounts to log in. Every computer on the local network can ping every other computer includeing the DC's its just that some of them don't find the DC on login.

Does anyone have any idea of where I should begin fixing this?
Also I have talked to our finance manager about getting some money to buy three new servers to replace the four two-year-old servers we have now. Has anyone tried replacing all of the servers on a domain? If so what is the easiest way to do it? Please help me?
M

 

[rperissi]

Have you run "dcdiag /a" from the cmd line on a dc to diagnose any issues?

Can you ping machines by FQDN as well as netbios name?

 

[aziel11]

Yes I can ping by name most of the time, I have run dcdiag, here is what it outputs from the DC/gateway/DNS/DHCP server:


Domain Controller Diagnosis

Performing initial setup:
Done gathering initial info.

Doing initial required tests

Testing server: Default-First-Site-Name\SHIVA
Starting test: Connectivity
......................... SHIVA passed test Connectivity

Testing server: Default-First-Site-Name\DOC
Starting test: Connectivity
......................... DOC passed test Connectivity

Testing server: Default-First-Site-Name\BICH
Starting test: Connectivity
......................... BICH passed test Connectivity

Doing primary tests

Testing server: Default-First-Site-Name\SHIVA
Starting test: Replications
[Replications Check,SHIVA] A recent replication attempt failed:
From DOC to SHIVA
Naming Context: CN=Schema,CN=Configuration,DC=carehawaii,DC=com
The replication generated an error (8524):
The DSA operation is unable to proceed because of a DNS lookup failure.
The failure occurred at 2005-04-13 11:56:01.
The last success occurred at 2005-03-25 10:56:40.
77 failures have occurred since the last success.
The guid-based DNS name 954f8a22-7e03-4782-ae49-28748fbadc43._msdcs.carehawaii.com
is not registered on one or more DNS servers.
[Replications Check,SHIVA] A recent replication attempt failed:
From DOC to SHIVA
Naming Context: CN=Configuration,DC=carehawaii,DC=com
The replication generated an error (8524):
The DSA operation is unable to proceed because of a DNS lookup failure.
The failure occurred at 2005-04-13 12:43:57.
The last success occurred at 2005-03-25 10:56:40.
675 failures have occurred since the last success.
The guid-based DNS name 954f8a22-7e03-4782-ae49-28748fbadc43._msdcs.carehawaii.com
is not registered on one or more DNS servers.
[Replications Check,SHIVA] A recent replication attempt failed:
From DOC to SHIVA
Naming Context: DC=carehawaii,DC=com
The replication generated an error (8524):
The DSA operation is unable to proceed because of a DNS lookup failure.
The failure occurred at 2005-04-13 12:47:15.
The last success occurred at 2005-03-25 11:07:39.
1620 failures have occurred since the last success.
The guid-based DNS name 954f8a22-7e03-4782-ae49-28748fbadc43._msdcs.carehawaii.com
is not registered on one or more DNS servers.
......................... SHIVA passed test Replications
Starting test: NCSecDesc
......................... SHIVA passed test NCSecDesc
Starting test: NetLogons
......................... SHIVA passed test NetLogons
Starting test: Advertising
Fatal ErrorsGetDcName (SHIVA) call failed, error 1355
The Locator could not find the server.
......................... SHIVA failed test Advertising
Starting test: KnowsOfRoleHolders
......................... SHIVA passed test KnowsOfRoleHolders
Starting test: RidManager
......................... SHIVA passed test RidManager
Starting test: MachineAccount
......................... SHIVA passed test MachineAccount
Starting test: Services
......................... SHIVA passed test Services
Starting test: ObjectsReplicated
......................... SHIVA passed test ObjectsReplicated
Starting test: frssysvol
......................... SHIVA passed test frssysvol
Starting test: frsevent
There are warning or error events within the last 24 hours after the

SYSVOL has been shared. Failing SYSVOL replication problems may cause

Group Policy problems.
......................... SHIVA failed test frsevent
Starting test: kccevent
An Warning Event occured. EventID: 0x800004F1
Time Generated: 04/13/2005 12:40:47
Event String: An attempt by the Knowledge Consistency Checker

An Warning Event occured. EventID: 0x800004F1
Time Generated: 04/13/2005 12:41:04
Event String: An attempt by the Knowledge Consistency Checker

An Warning Event occured. EventID: 0x800004F1
Time Generated: 04/13/2005 12:41:21
Event String: An attempt by the Knowledge Consistency Checker

......................... SHIVA failed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0xC0001F60
Time Generated: 04/13/2005 12:37:45
Event String: The browser service has failed to retrieve the

......................... SHIVA failed test systemlog
Starting test: VerifyReferences
......................... SHIVA passed test VerifyReferences

Testing server: Default-First-Site-Name\DOC
Starting test: Replications
......................... DOC passed test Replications
Starting test: NCSecDesc
......................... DOC passed test NCSecDesc
Starting test: NetLogons
......................... DOC passed test NetLogons
Starting test: Advertising
......................... DOC passed test Advertising
Starting test: KnowsOfRoleHolders
......................... DOC passed test KnowsOfRoleHolders
Starting test: RidManager
......................... DOC passed test RidManager
Starting test: MachineAccount
......................... DOC passed test MachineAccount
Starting test: Services
......................... DOC passed test Services
Starting test: ObjectsReplicated
Authoritative attribute options on BICH (writeable)
usnLocalChange = 45347
LastOriginatingDsa = DOC
usnOriginatingChange = 3725167
timeLastOriginatingChange = 2005-03-28 14:03:50
VersionLastOriginatingChange = 3
Out-of-date attribute options on SHIVA (writeable)
usnLocalChange = 8515
LastOriginatingDsa = DOC
usnOriginatingChange = 1363
timeLastOriginatingChange = 2003-01-10 17:17:19
VersionLastOriginatingChange = 1
......................... DOC failed test ObjectsReplicated
Starting test: frssysvol
......................... DOC passed test frssysvol
Starting test: frsevent
......................... DOC passed test frsevent
Starting test: kccevent
......................... DOC passed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 12:15:15
Event String: The connection was aborted by the remote WINS.

An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 12:45:15
Event String: The connection was aborted by the remote WINS.

......................... DOC failed test systemlog
Starting test: VerifyReferences
......................... DOC passed test VerifyReferences

Testing server: Default-First-Site-Name\BICH
Starting test: Replications
......................... BICH passed test Replications
Starting test: NCSecDesc
......................... BICH passed test NCSecDesc
Starting test: NetLogons
......................... BICH passed test NetLogons
Starting test: Advertising
Warning: DsGetDcName returned information for \\doc.carehawaii.com, when we were trying to reach BICH.
Server is not responding or is not considered suitable.
......................... BICH failed test Advertising
Starting test: KnowsOfRoleHolders
......................... BICH passed test KnowsOfRoleHolders
Starting test: RidManager
......................... BICH passed test RidManager
Starting test: MachineAccount
......................... BICH passed test MachineAccount
Starting test: Services
......................... BICH passed test Services
Starting test: ObjectsReplicated
Authoritative attribute servicePrincipalName on DOC (writeable)
usnLocalChange = 3712410
LastOriginatingDsa = BICH
usnOriginatingChange = 37006
timeLastOriginatingChange = 2005-03-26 14:04:14
VersionLastOriginatingChange = 8
Out-of-date attribute servicePrincipalName on SHIVA (writeable)
usnLocalChange = 79943
LastOriginatingDsa = BICH
usnOriginatingChange = 24848
timeLastOriginatingChange = 2005-03-24 14:56:47
VersionLastOriginatingChange = 5
Authoritative attribute msDS-HasInstantiatedNCs on DOC (writeable)
usnLocalChange = 3706595
LastOriginatingDsa = BICH
usnOriginatingChange = 32821
timeLastOriginatingChange = 2005-03-25 16:12:27
VersionLastOriginatingChange = 5
Out-of-date attribute msDS-HasInstantiatedNCs on SHIVA (writeable)
usnLocalChange = 79748
LastOriginatingDsa = BICH
usnOriginatingChange = 24607
timeLastOriginatingChange = 2005-03-24 11:59:27
VersionLastOriginatingChange = 3
Authoritative attribute msDS-hasMasterNCs on DOC (writeable)
usnLocalChange = 3706595
LastOriginatingDsa = BICH
usnOriginatingChange = 32821
timeLastOriginatingChange = 2005-03-25 16:12:27
VersionLastOriginatingChange = 7
Out-of-date attribute msDS-hasMasterNCs on SHIVA (writeable)
usnLocalChange = 79748
LastOriginatingDsa = BICH
usnOriginatingChange = 22370
timeLastOriginatingChange = 2005-03-24 11:42:27
VersionLastOriginatingChange = 5
......................... BICH failed test ObjectsReplicated
Starting test: frssysvol
......................... BICH passed test frssysvol
Starting test: frsevent
There are warning or error events within the last 24 hours after the

SYSVOL has been shared. Failing SYSVOL replication problems may cause

Group Policy problems.
......................... BICH failed test frsevent
Starting test: kccevent
......................... BICH passed test kccevent
Starting test: systemlog
An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 11:58:55
Event String: The connection was aborted by the remote WINS.

An Error Event occured. EventID: 0x40011006
Time Generated: 04/13/2005 12:28:55
Event String: The connection was aborted by the remote WINS.

......................... BICH failed test systemlog
Starting test: VerifyReferences
......................... BICH passed test VerifyReferences

Running partition tests on : Schema
Starting test: CrossRefValidation
......................... Schema passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Schema passed test CheckSDRefDom

Running partition tests on : Configuration
Starting test: CrossRefValidation
......................... Configuration passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... Configuration passed test CheckSDRefDom

Running partition tests on : carehawaii
Starting test: CrossRefValidation
......................... carehawaii passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... carehawaii passed test CheckSDRefDom

Running partition tests on : DomainDnsZones
Starting test: CrossRefValidation
......................... DomainDnsZones passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... DomainDnsZones passed test CheckSDRefDom

Running partition tests on : ForestDnsZones
Starting test: CrossRefValidation
......................... ForestDnsZones passed test CrossRefValidation
Starting test: CheckSDRefDom
......................... ForestDnsZones passed test CheckSDRefDom

Running enterprise tests on : carehawaii.com
Starting test: Intersite
......................... carehawaii.com passed test Intersite
Starting test: FsmoCheck
......................... carehawaii.com passed test FsmoCheck

Is that helpful?
M

 

[aziel11]

Here is an update on another symptom:
We currently have three DC's two in the main office and one which is the exchange server in an offsite location linked through routing and remote access. I want to demote the offsite DC and upgrade it to server 2003 (its running 2000) and upgrade exchange from 2000 to 2003. All monitoring utilities tell me that AD replication is functioning fine but when I run DCPROMO on that werver it allways returns an error that it can' find any DC's to trasfer it's roles to. I've already trasfered all five master roles to another DC as well as the global catalog, but I don't think those changes have taken effect. Anyone got any ideas?

 

[aziel11]

As relating to the RPC server error on remote login, here is the event associated with the error:

Event ID:1219
Source: Winlogon
Description: Logon rejected for DOMAIN\Administrator. Unable to obtain Terminal Server User Configuration. Error: The RPC server is unavailable.

Hope that helps.