Active Directory Permissions Go Missing

The other day I encountered a situation with an obscure Active Directory process called “AdminSDHolder” that, by design, is intended to keep things protected and secure but is not well known by Administrators and can cause a real headache.

The Situation

A user reported a permissions problem with an Active Directory group they manage. Normally, the user can control membership of the group but they’d be denied access and couldn’t perform this operation.

Looking at the Advanced Security Settings of the group, I found the setting to “Include inheritable permissions from this object’s parent” had been disabled and that all the Access Control Entries (ACEs) in the groups Discretionary Access Control List (DACL) where set to a generic list. Since the user was assigned permissions through inheritance their rights where effectively denied and removed resulting in the inability to perform any functions on the group.

The Workaround

The fix was to check the inheritance box and apply the permissions from the parent. Checking with the user all things where back to normal and they could do their work.

Or so I thought…

Later that day, that same user reported the same problem. Looking again at the DACLs on the Advanced Security Settings I found the same condition as before…Inheritance was denied and ACEs where set back to generic.

I set up an Audit Policy on the Domain Controllers to audit all Directory Service Access and applied an audit control setting to the group to monitor all access and changes made to the group. I then tested my settings by making modifications to the group and saw an audit entry in the security log showing where I had made the change and what change I had made.

A few hours later, I checked the object again and saw the same conditions as before; inheritance was unchecked and security entries where reset.

Looking through the security log there was no evidence of anything done…no entries showing changes or access to the group in question. If there are no entries in the security log for either users or services making changes to Active Directory then it’s a process operating outside of normal user-mode.

The Cause

As it turns out, Active Directory has an internal process called “AdminSDHolder” that runs every hour to maintain the security settings of protected groups and their nested groups (i.e., groups that are members of protected groups). The permissions settings used are defined by the security-descriptor of the AD object cn=AdminSdHolder,cn=System,dc=yourdomain,dc=com.

When this process runs it check all “protected groups” in Active Directory for an attribute known as adminCount. If the value of adminCount is greater than 0, it changes the permissions on this object and resets the flag to disable inheritance of parent objects. It does this on all protected groups and the groups nested within the membership of these groups.

While it makes sense to have an automated process to keep standard permission levels set on protected groups within Active Directory, it can become a challenge to Administrators when they begin to nest groups and users together to form chains of permissions on objects.

Active Directory protects the following built-in groups by default:

  • Enterprise Admins
  • Schema Admins
  • Domain Admins
  • Administrators
  • Account Operators
  • Server Operators
  • Print Operators
  • Backup Operators
  • Cert Publishers

Any groups affiliated with these protected groups through membership are automatically flagged as protected by AdminSDHolder.

So back to my situation with the user group losing permissions. It turns out that the group being managed was a member of a protected group and therefore was having its permissions reset every hour. By removing the group membership from the protected group it was no longer subject to the AdminSDHolder process.

More Information:

http://blogs.technet.com/b/askds/archive/2009/05/07/five-common-questions-about-adminsdholder-and-sdprop.aspx

http://support.microsoft.com/?id=817433

Virtual Infrastructure Management

What does a VMWare Virtual Infrastructure (VI) Administrator have to do to get their head around the dozens of host servers, subnets and virtual machines in their environment?  Stop…Take A Deep Breath…Move Forward.  VI sanity can be meet with a reasonable amount of planned activity equal in part to the regular due diligence Administrators apply to their physical infrastructure counter parts: host clean-up, log reviews, monitoring, etc.

Who Moved My Cheese: VI Change Management

First, get your bearings.  In a well-defined VI, a Virtual Machine (VM) resides on a dynamic cluster of host servers, those servers are part of a group and that group is composed of hardware (e.g., servers, SAN, networks, disks, etc.).  That being said, your VMs can jump from host to host and group to group at any point during the day or night.  As shared resources (CPU, RAM, etc.) are required and contention for those resources becomes an issue, VMs can automatically be evacuated from one host to another.  If your cluster is configured with Dynamic Resource Scheduling (DRS) or High Availability (HA), your VMs can reside on any host in the cluster at any time.  In this configuration, the VI dynamically makes room on a host if the intensive processing of one or more VMs be comes and issue and then re-balance those resources later on as things settle down.

Given the dynamic nature of this architecture, chasing these moving targets is often like the old adage…”hearding cats”.  So what should be considered an offical “Request For Change” (RFC) in a Virtual Infrastrucuture environment?  Here are some common changes that make sense to account for in your formal Change Control process:

  1. Manual vMotion or movement of a VM between hosts
  2. VM configuration changes – extending the permanent allocation of virtual hardware or resource share changes
  3. Deployment / introduction of new VMs into the environment
  4. Host configuration changes – maintenance changes
  5. Patches and updates to ESX hosts, host hardware maintenance, etc
  6. DRS – Automatic load leveling on hosts using vMotion, could occur daily or hourly
  7. Cluster change – Addition of LUNs, rescan of storage
  8. Cluster change – Removal of LUNs
  9. Cluster change –Host upgrade (Major change, VM downtime not always required)
  10. VMtools upgrades (after major host version upgrade, VM restarts after tools install)
  11. Addition of hosts to an existing cluster
  12. vCenter upgrades – no VM changes, but possible loss of access to VMs via vCenter
  13. Critical to performance and stability

Walk The Talk: The Practice of  VI maintenance

Like any administrative function, keeping an active eye on your hosts and VMs will give you a finger on the pulse of your environment.  Remember the three key concepts in VI management: shared storage, shared resources (CPU, RAM, Networks) and VM placement.  As things progress disks (SAN LUNs) will fill-up, VMs will need care and feeding and performance monitoring will tell yo how things are progressing.  Here are some best practices that most VI admins perform on a regular basis to keep their head in the game.  I’m sure your see many similarities to your regular duties when managing a physical environment.

Daily Tasks

  1. Gather Statistics and review previous day performance and utilization data
  2. Look for changes between current and previous day data on both VMs and hosts

Weekly Tasks

  1. Review host logs, vSphere logs document errors or issues to troubleshoot
  2. Review VMFS volume capacity; do not deploy VMs to LUNs with <20% available space
  3. Look for VMs with open snapshots; these can grow to big and cause performance issues or lock ups
  4. Monitor host drive space
  5. Decommission Test/Dev VMs to ensure to reclaim unused space

Monthly Tasks

  1. Create a capacity reports for IT management; there is a great tool for this from vKernel called Capacity Analyzer
  2. Update your VM templates with the latest hotfixes and patches approved for the environment