Kick the crashed AD controller out

Sometimes bad things happen, sometimes ugly things happen. One of the very bad things that can happen is when an active domain controller crashes and there is no backup available. The case becomes really ugly if nobody cares about the crashed controller for about 60 days (forest that was created on a domain controller running Windows Server 2003 and earlier) or 180 days (forest that was created on a domain controller running Windows Server 2003 sp1 and later). On this depends the default tombstone lifetime of directory objects. Later we will investigate on the tombstone. 

 

A good place to fix this whole bunch of problems is by verifying the backup strategy and ensuring that all system-states are saved on all domain controllers. The second step is verifying that DNS are fine and syncing the proper way. Now we are ready to move the FSMO roles. For everyone that is not familiar with the five FSMO Friends, here is a small overview from Wikipedia 

Flexible Single Master of Operation (FSMO, F is sometimes floating ; pronounced Fiz-mo), or just single master operation or operations master, is a feature of Microsoft’s Active Directory (AD). As of 2005, the term FSMO has been deprecated in favor of operations masters. 

FSMOs are specialized domain controller (DC) tasks, used where standard data transfer and update methods are inadequate. AD normally relies on multiple peer DCs, each with a copy of the AD database, being synchronized by multi-master replication. The tasks which are not suited to multi-master replication, and are viable only with a single-master database, are the FSMOs. 

Domain-wide FSMO Roles: 

Every domain in an Active Directory forest must contain one of each of the following FSMO roles:
The Relative ID Master allocates security RIDs to DCs to assign to new AD security principals (users, groups or computer objects). It also manages objects moving between domains.
The Infrastructure Master maintains security identifiers, GUIDs, and DNS for objects referenced across domains. Most commonly it updates user and group links. This is another domain-specific role and its purpose is to ensure that cross-domain object references are correctly handled. For example, if you add a user from one domain to a security group from a different domain, the Infrastructure Master makes sure this is done properly. As you can guess however, if your Active Directory deployment has only a single domain, then the Infrastructure Master role does no work at all, and even in a multi-domain environment it is rarely used except when complex user administration tasks are performed, so the machine holding this role doesn’t need to have much horsepower at all.
The PDC Emulator operations master role processes all password changes in the domain. Failed authentication attempts due to a bad password at other domain controllers are forwarded to the PDC Emulator before rejection. This ensures that a user can immediately login following a password change from any domain controller, without having to wait several minutes for the change to be replicated. The PDC Emulator Operations Master role must be carefully sited in a location to best handle all password reset and failed-authentication forwarding traffic for the domain. 

Forest-wide FSMO Roles: 

Regardless of the number of domains in an Active Directory forest, the following FSMO roles exist only once:
The Schema Master maintains all modifications to the schema of the forest. The schema determines the types of objects permitted in the forest and the attributes of those objects.
The Domain Naming Master tracks the names of all domains in the forest and is required to add new domains to the forest or delete existing domains from the forest. It is also responsible for group membership. 

Normally it’s very easy to move these roles by right clicking the forest level and choose Move … in the Active Directory Schema snap-in, Active Directory Domains and Trusts snap-in and Active Directory Users and Computers snap-in. But it will fail to 99% with an obscure error. The reason for the error is one domain controller in the replica ring is missing and marked as Tombstone. Let’s get to the bigger guns and start “ntdsutil.exe”, open a command prompt and enter “ntdsutil.exe”. If the shell is bugging you that the exe is missing, you need to install the server support tools. They are located on the Windows CD in the support folder. Other ways you can download it from Microsoft using Google ☺. 

!! Remember at this point you can do very large harm to the directory so please be sure that you have properly working backups!! 

After “ntdsutil.exe” has successful started, type “roles” and press enter. Type “connections” and press enter. Now Type “connect to server xyz.planetgeek.ch”, where xyz.planetgeek.ch is the name of the server where you want to transfer the roles to. A message will appear: 

“Binding to xyz.planetgeek.ch …
Connected to servername using credentials of locally logged on user.” 

Tipe “quit” to leave the selection menu. Now appears: “fsmo maintenance:” now enter: 

“Seize schema master” if you want move the schema master.
“Seize domain naming master” if you want move the naming master.
“Seize PDC” if you want move the PDC.
“Seize RID master” if you want move the Relative ID master.
“Seize infrastructure master” if you want move the infrastructure master. 

Next thing to do is kicking the metadata out of the directory. To do this I know two possible ways. The first is use a VB script written by Clay Perrine from Microsoft. The second way is to use ntdsutil.exe. I prefer the VB script. It works on the most common Windows Operating systems (2k, XP, 03, Vista and 08). The script is below ore you can obtain it directly from Microsoft (http://go.microsoft.com/fwlink/?LinkID=123599). 

REM ==========================================================
REM GUI Metadata Cleanup Utility
REM Written By Clay Perrine
REM Version 2.5
REM ==========================================================
REM This tool is furnished "AS IS". NO warranty is expressed or Implied. 

on error resume next
dim objRoot,oDC,sPath,outval,oDCSelect,objConfiguration,objContainer,errval,ODCPath,ckdcPath,myObj,comparename 

rem =======This gets the name of the computer that the script is run on ====== 

Set sh = CreateObject("WScript.Shell")
key= "HKEY_LOCAL_MACHINE"
computerName = sh.RegRead(key & "\SYSTEM\CurrentControlSet\Control\ComputerName\ComputerName\ComputerName") 

rem === Get the default naming context of the domain==== 

set objRoot=GetObject("LDAP://RootDSE")
sPath = "LDAP://OU=Domain Controllers," & objRoot.Get("defaultNamingContext") 

rem === Get the list of domain controllers==== 

Set objConfiguration = GetObject(sPath)
For Each objContainer in objConfiguration
outval = outval & vbtab & objContainer.Name & VBCRLF
Next
outval = Replace(outval, "CN=", "") 

rem ==Retrieve the name of the broken DC from the user and verify it's not this DC.=== 

oDCSelect= InputBox (outval," Enter the computer name to be removed","")
comparename = UCase(oDCSelect) 

if comparename = computerName then
msgbox "The Domain Controller you entered is the machine that is running this script." & vbcrlf & _
"You cannot clean up the metadata for the machine that is running the script!",,"Metadata Cleanup Utility Error."
wscript.quit
End If 

sPath = "LDAP://OU=Domain Controllers," & objRoot.Get("defaultNamingContext")
Set objConfiguration = GetObject(sPath) 

For Each objContainer in objConfiguration
Err.Clear
ckdcPath = "LDAP://" & "CN=" & oDCSelect & ",OU=Domain Controllers," & objRoot.Get("defaultNamingContext")
set myObj=GetObject(ckdcPath)
If err.number <>0 Then
errval= 1
End If
Next 

If errval = 1 then
msgbox "The Domain Controller you entered was not found in the Active Directory",,"Metadata Cleanup Utility Error."
wscript.quit
End If 

abort = msgbox ("You are about to remove all metadata for the server " & oDCSelect & "! Are you sure?",4404,"WARNING!!")
if abort <> 6 then
msgbox "Metadata Cleanup Aborted.",,"Metadata Cleanup Utility Error."
wscript.quit
end if 

oDCSelect = "CN=" & oDCSelect
ODCPath ="LDAP://" & oDCselect & ",OU=Domain Controllers," & objRoot.Get("defaultNamingContext")
sSitelist = "LDAP://CN=Sites,CN=Configuration," & objRoot.Get("defaultNamingContext")
Set objConfiguration = GetObject(sSitelist)
For Each objContainer in objConfiguration
Err.Clear
sitePath = "LDAP://" & oDCSelect & ",CN=Servers," & objContainer.Name & ",CN=Sites,CN=Configuration," & _
objRoot.Get("defaultNamingContext")
set myObj=GetObject(sitePath)
If err.number = 0 Then
siteval = sitePath
End If
Next 

sFRSSysvolList = "LDAP://CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System," & _
objRoot.Get("defaultNamingContext")
Set objConfiguration = GetObject(sFRSSysvolList) 

For Each objContainer in objConfiguration
Err.Clear
SYSVOLPath = "LDAP://" & oDCSelect & ",CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System," & _
objRoot.Get("defaultNamingContext")
set myObj=GetObject(SYSVOLPath)
If err.number = 0 Then
SYSVOLval = SYSVOLPath
End If
Next 

SiteList = Replace(sSitelist, "LDAP://", "")
VarSitelist = "LDAP://CN=Sites,CN=Configuration," & objRoot.Get("defaultNamingContext")
Set SiteConfiguration = GetObject(VarSitelist) 

For Each SiteContainer in SiteConfiguration
Sitevar = SiteContainer.Name
VarPath ="LDAP://OU=Domain Controllers," & objRoot.Get("defaultNamingContext")
Set DCConfiguration = GetObject(VarPath)
For Each DomContainer in DCConfiguration
DCVar = DomContainer.Name
strFromServer = ""
NTDSPATH = DCVar & ",CN=Servers," & SiteVar & "," & SiteList
GuidPath = "LDAP://CN=NTDS Settings,"& NTDSPATH
Set objCheck = GetObject(NTDSPATH)
For Each CheckContainer in objCheck
rem ====check for valid site paths =======================
ldapntdspath = "LDAP://" & NTDSPATH
Err.Clear
set exists=GetObject(ldapntdspath)
If err.number = 0 Then
Set oGuidGet = GetObject(GuidPath)
For Each objContainer in oGuidGet
oGuid = objContainer.Name
oGuidPath = "LDAP://" & oGuid & ",CN=NTDS Settings," & NTDSPATH
Set objSitelink = GetObject(oGuidPath)
objSiteLink.GetInfo
strFromServer = objSiteLink.Get("fromServer")
ispresent = Instr(1,strFromServer,oDCSelect,1) 

if ispresent <> 0 then
Set objReplLinkVal = GetObject(oGuidPath)
objReplLinkVal.DeleteObject(0)
end if
next 

sitedelval = "CN=" & comparename & ",CN=Servers," & SiteVar & "," & SiteList
if sitedelval = ntdspath then
Set objguidpath = GetObject(guidpath)
objguidpath.DeleteObject(0)
Set objntdspath = GetObject(ldapntdspath)
objntdspath.DeleteObject(0)
end if
End If
next
next
next
Set AccountObject = GetObject(ckdcPath)
temp=Accountobject.Get ("userAccountControl")
AccountObject.Put "userAccountControl", "4096"
AccountObject.SetInfo
Set objFRSSysvol = GetObject(SYSVOLval)
objFRSSysvol.DeleteObject(0)
Set objComputer = GetObject(ckdcPath)
objComputer.DeleteObject(0)
Set objConfig = GetObject(siteval)
objConfig.DeleteObject(0)
oDCSelect = Replace(oDCSelect, "CN=", "")
msgval = "Metadata Cleanup Completed for " & oDCSelect
msgbox msgval,,"Notice."
wscript.quit 


An easy to use description of the ntdsutil.exe way you find under http://technet.microsoft.com/en-us/library/cc736378.aspx

Next thing that will drive you crazy are the millions of ntfrs errors in the Eventlog. Ntfrs is the “New Technology File replication Service” from Windows. It is used for the replication of the sysvol/ netlogon. Remember Since Windows 2003 R2 nftrs is replaced trough DFS. First of all we are saving the eventlog to a file then clean it and boot every Domain Controller in the domain and wait a few minutes. On my experience this will fix half of the problems, like swiss admins tend to say “ein boot tut immer gut” ; -).