Working with replicated Active Directory

General presentation

Active Directory provides a set of authentication servers that replicate to allow:

  • Redundancy (protection against failure).

  • Better load management (several ADs to distribute requests).

  • Or to provide a local authentication server for remote sites.

Note

Bandwidth consumption for Active Directory replication is generally negligible on modern networks. Indeed, the Active Directory architecture was imagined in the 90’s when an inter-site ISDN 64kbps connection was already an envied luxury.

Replication is one of the most complicated and least understood topics in Active Directory operation. Even if it is not necessary to understand these concepts (and most sysadmin, even certified ones, don’t really master them), it is still important to understand the basics to facilitate diagnosis in case of problems.

Putting DCs on remote sites is rarely a choice related to a bandwidth issue, but rather a desire to ensure continuity of service even if the WAN network has fallen down.

Note

Samba-AD is a mature product that can manage domains with several dozen domain controllers in replication without any problem. The largest domain that Tranquil IT co-administers has more than 140 domain controllers using replication.

How replication works

The operation of Active Directory replication is very different from the replication mode of OpenLDAP Syncrepl or other replication systems:

  • Active Directory replication works in Pull mode (the server pulls modifications from other servers) and not in Push mode (the server sends its modified data).

  • The state of the replications is contained in the AD tree itself, there is no log file that records all the modifications.

  • Although AD servers replicate together, they are not exactly identical. Indeed there are objects and attributes that are not replicated, such as the state of the replications. If we have to compare two ADs we must therefore exclude certain attributes to avoid unnecessary noise, and thus facilitate a post-incident replication search for example.

Notification vs. polling

Active Directory servers on the same Active Directory site send each other notifications when changes are made.

Example: when srvads1 has a change, it sends a notification to srvads2, srvads2 receives the notification from srvads1 and connects to srvads1 to ask for changes that have taken place since its last replication.

This ensures that changes are spread quickly. Indeed, since user workstations and servers can connect indifferently to the domain controllers of the same site, it is important that consistency between ADs to be kept.

On the other hand if the AD servers are on different sites, the remote server srvads3 will regularly ask its replication partner srvads1 if there have been any changes since the last replication. This mode of operation is designed to save bandwidth.

USN Operation

USN is a monotonous counter (which only increases) which is specific to each AD server (it is not replicated between ADs in a domain). The USN counter is incremented by 1 each time a change is made to an object, and the value is assigned to the uSNChanged field of the changed object.

Thus the object with the highest USN is the last object changed in the AD. Each AD object has a unique USN on the same AD. Since USNs are not replicated, the same object has a different uSNChanged value on each Active Directory. Since there is no relationship between the uSNChanged attributes between two DCs, the same value can be set on two DCs (but never on the same domain controller).

When an AD replicates with another AD, it stores the USN max value of its last replication with the remote AD. Each AD stores a value of the USN max of the replicating ADs at the root of the main partition (DC=mydomain,DC=lan) in the repFrom and repTo attributes (these are binary data not directly readable) which are used to display the result of the samba-tool drs showrepl command.

whenChanged attribute

The whenChanged attribute is not replicated from one AD to another. The value of this attribute corresponds to the moment when the object was modified, either by an external call (LDAP modification made by an administrator, a user station, an application, etc.) or by a replication.

That is to say that on the AD where the modification was made, the attribute whenChanged will have the timestamp of the moment of the modification, while the other ADs will have the timestamp corresponding to the moment when it was replicated.

Deleting tombstoneLifetime objects and attributes

As mentioned above, the replication status is stored in the AD tree itself. The case of deleted objects then arises. If we definitively delete an object from the LDAP tree, it will have no more USN and therefore we will not be able to replicate the deletion.

To solve this problem the AD developers have used the concept of a trash can where objects are moved when they are deleted: the container CN=Deleted Obects,DC=mydomain,DC=lan. When an LDAP entry is deleted several things happen:

  • The attribute ISDELETED is added with the value 1.

  • Attributes that are not necessary for deletion replication are removed (members, memberOf, address, mail, etc).

  • The object is moved into the CN=Deleted Objects container.

  • The DN of the object, and thus its CN for users and groups, is renamed by suffixing it with 0ADEL: and its objectGUID. For example, when deleting the object CN=dcardon would have as new DN:

    DN: CN=dcardon\0ADEL:<object_guid>,CN=Deleted Objects,DC=mydomain,DC=lan

Note

With Active Directory 2003 Microsoft has introduced a new concept to help restore a deleted object: the Active Directory Recycle Bin. The recycle bin allows to recover a deleted object and to put it back in the normal tree structure.

This feature is not yet supported in Samba-AD.

Managing replication conflicts

When the same object has been modified on two domain controllers during replication, this will cause reconciliation problems. Active Directory has different methods for reconciling conflicting entries. For example, if a mail attribute was changed on two ADs at the same time, the entry with the most recent timestamp will be kept.

Note

For replications, it is not the timestamps that are used to make comparisons between the different DAs but the USNs. The timestamps are only used here for reconciliation when changes are in conflict during replications.

If the DC is unable or unwilling to make a decision on which value to keep, e.g. the same account created on two DCs at the same time, the entry with the highest timestamp will be renamed according to the following standard:

CN=dcardon\0ACNF:<object_guid>,OU=company_users,DC=testing,DC=lan

This prevents DN from conflicting and allows to finish replication (you can’t have two objects with the same DN in an AD tree). On the other hand it does not guarantee the uniqueness of the samAccountName. Indeed to manage this case, Microsoft engineers did not put a unique index on the attribute samAccountName.

Renaming samAccountName to avoid duplicates is the role of another recurring task that validates the uniqueness of this attribute, and renames the conflicting attributes. This recurring task is not currently implemented in Samba-AD. However, this case occurs very rarely.

Conflicting objects can appear in different partitions of the AD, especially in the DNSDomainZones and ForestDNSZones partitions. Entries renamed OACNF in the DNS console can normally be deleted without any impact.

Replication topology

Managing replication connections, role of the KCC

In order for replications to go well in a domain, it is important that each controller is connected to every other controller through one or more hops. If a change is made to srvads1, it can be replicated to srvads2 which will replicate it to srvads3. Even if srvads1 and srvads3 do not replicate directly, the modification will be propagated by transitivity.

The role of interconnecting ADs is that of the KCC, the verification of the consistency of the knowledge. It is a name that reflects its role well. The KCC uses the information configured by the administrator in the Site and Services Active Directory console to choose its AD interconnection strategy. However, if the configuration set by the administrator does not allow the KCC to have a fully connected network, the KCC may make decisions that may be in disagreement with what the administrator had intended.

By default the KCC will make at least one replication connection with another AD at the same site, and with at least one AD at a remote site to which it must make replications.

In practice we advise to define star replication topologies with a replication site in the center and replication sites around it that replicate only with the replication site. This simple method will avoid loops (see paragraph on the subject).

The notion of bridgehead can be defined on a remote site for the KCC to choose this default bridgehead domain controller to manage replications with the remote site.

Depending on the number of controllers in the domain, it can be interesting to have a site dedicated to replication.

Replication loop

The replication topology can cause replication loops. That is to say, for example, that server A replicates with B and C, and that B and C replicate with each other. We can therefore have a modification on A which is propagated to B which is propagated again to C which propagates it again to server A.

In this case the replication protocol has a mechanism to recognize this duplicate replication information and to ignore it.

Replicating the SYSVOL directory

The SYSVOL directory of an Active Directory contains the GPO definition files and the NetLogon login scripts for the domain. The SYSVOL directory must be available on all AD servers of a domain with the same content.

Replication of the LDAP DRS (Directory Replication Service) tree is a completely different protocol from that used for replication of the SYSVOL directory.

To ensure the replication of the SYSVOL directory, Microsoft has implemented two protocols:

  • FRS: protocol integrated in AD from AD2k to AD2k16 (except last version October 2019).

  • DFS-R: protocol integrated in AD since AD2k8.

Currently Samba-AD does not support FRS or DFS-R. There is no official Samba methodology for replicating the SYSVOL directory. There are different methods. You can use the Tranquil IT method.

The DFS-R protocol used for the SYSVOL directory is the same protocol used to replicate file shares between Microsoft file servers. It is an extension of the DFS protocol. Samba supports the DFS protocol, but not the DFS-R protocol.

Case of read-only domain controllers

Starting with Active Directory 2008, it is possible to have read-only Active Directory servers RODC. As the name implies, these domain controllers do not allow local data modification. Changes are redirected to a normal domain controller (which will be called in this documentation by abuse of language RWDC).

However, as stated earlier in this documentation, the replication status data itself is stored in the AD. This implies that the AD is not entirely read-only. Indeed some attributes are still read/write. The instanceType attribute defines whether the attribute is read/write (value 4) or read-only (value 2).