My Domain Controller Couldn't Find Itself for a Year

The short version

A routine pre-patch health check on my lab's only domain controller turned up something embarrassing: its AD-integrated DNS had never loaded. Not once. Not in the entire life of the domain.

The directory itself was healthy by every test I could run. The root cause was stranger than the symptom: the domain controller had lost its secure channel with itself. Its machine-account password no longer matched what Active Directory held for it, so every service running as SYSTEM was being refused by its own directory — while every admin tool I ran worked fine.

The fix was fifteen minutes and one Microsoft KB (2751452): stop the Kerberos Key Distribution Center, reset the machine-account password with netdom resetpwd pointed at the DC's own IP address, and reboot. The zones loaded for the first time ever.

This is how a domain ran broken for a year without anyone noticing — and what finally gave it away.

The environment

The lab is a 3-host vSAN ESA cluster running ESXi 9.1. Active Directory lives on a single domain controller, built early in the lab's life as a learning exercise. By design, AD is not the lab's DNS: two Pi-hole instances handle name resolution for the LAN, with hand-built static records for every lab host, plus conditional forwarding that sends queries for the AD domain to the domain controller itself.

That design is why the failure hid so well. At promotion time, the DC's DNS role was never really finished — the thinking was "Pi-hole will be the DNS." Everything the lab actually uses resolves through the Pi-holes' static records, so day to day, nothing appeared wrong. Domain joins were flaky, once upon a time, and the joined machines were eventually unjoined because "it just caused issues." In hindsight, that was the smoke. Nobody looked for the fire, because nothing anyone depended on was burning.

Even the nightly backup of the DC — application-aware, using a domain credential — worked the whole time, because it connects by IP and static name and never needed AD's own DNS. Nothing did. That was the problem.

How it surfaced

The incident started with the least dramatic sentence in systems administration: I ran a health check before patching.

dcdiag /q failed one test — Connectivity. The DC's own GUID record in _msdcs could not be resolved to an IP address. Resolving the domain name from the DC itself returned public internet addresses — which turned out to be a subplot of its own (see the end of this article). Probing the DC's DNS service directly returned NXDOMAIN: the service was alive, answering, and empty. DNS Manager showed a red-badged server that "has not been configured," with zero zones loaded.

Zero. Not corrupted. Never loaded.

The acquittal parade

What followed was several hours of methodically proving that everything was fine, one subsystem at a time. I'm keeping the dead ends in this write-up because they're the actual content — the fix took fifteen minutes; knowing what the fix wasn't took the afternoon.

Services: DNS, NTDS, Netlogon, DFSR, ADWS — all running, all healthy.
The directory: LDAP binds worked locally and over the NIC. All naming contexts present. isSynchronized and isGlobalCatalogReady both true. A full dcdiag /v failed only Connectivity — Advertising, Netlogons, FSMO checks, all passed.
The zone data: The registry said four AD-integrated zones were configured. The zone objects existed in the directory, with two dozen child records in the main zone. Both application directory partitions existed and were hosted on the DC. Cross-reference records pointed at exactly the right places; replica-location attributes matched the DC's current NTDS Settings object letter for letter.
Permissions: The zone ACLs were pristine. My admin token carried Domain, Enterprise, and Schema Admins, elevated — and yet Get-DnsServerZone returned access denied, zone creation returned access denied, and dnscmd eventually stopped answering altogether.
Replacement: Restarting services didn't help. A reboot didn't help. Uninstalling and reinstalling the DNS role didn't help. Wiping the registry's zone table didn't help — the service re-derives its zones from AD, and still failed with zero local configuration. Deleting the zone objects themselves (after LDIFDE exports, because you export before you excise) didn't help.

Two real faults did turn up along the way, found by eye rather than by tool: the DC's subnet mask was wrong for the LAN's actual prefix length, and its network profile was set to Public. Both were worth fixing. Neither was the cause. The failure didn't budge a millimeter.

The error that finally said something

Event 4000 — "the DNS server was unable to open Active Directory" — had been logging on a roughly ten-minute cycle, carrying sub-error 9005. For a year. The headline said "can't open AD." The useful information was in Event 4007's XML payload: an invalid directory-partition registry state for the _msdcs zone's forest partition. Something narrower, and stranger, than the headline.

That's what led to KB 2751452, whose symptom list reads like a transcript of my afternoon: DNS zones don't load, events 4000 and 4007, the DNS console reports access denied, dnscmd is refused, and the service itself runs. Its stated cause #1: the DC/DNS server has lost its secure channel with itself.

Honest grading, then and now: the match was symptom-level, not proof. My 4000 sub-error wasn't the credential-class code the article's scenario implies. It was a prime suspect, not a diagnosis — until the cure worked.

One irony deserves its own paragraph: the secure-channel theory had actually come up hours earlier, and I'd set it aside as untestable. Test-ComputerSecureChannel locates a domain controller through DNS. DNS was the casualty. The test for the broken thing needed the broken thing.

The fix

On the DC, elevated:

Stop-Service KDC -Force

netdom resetpwd /server:<DC's own IP> /userd:<domain>\Administrator /passwordd:*

Restart-Computer

Three details matter. First, the KDC has to be stopped, or the DC keeps trying to authenticate with the credential you're about to replace. Second, the server is specified by IP address — that's the whole trick, because name resolution is what's broken. Third, the password prompt rejected several attempts before accepting one — plausibly the broken channel refusing valid authentication until one attempt punched through. Even the fix had to knock twice.

After the reboot, DNS Manager opened fully alive for the first time all day — no red badge, a complete console tree. I recreated the zones as AD-integrated (they'd been deleted during the experiments), re-registered the DC's records, and restarted Netlogon so it could register itself.

Verification, in the present tense this time: dcdiag /q no longer fails Connectivity. gpupdate /force succeeds for both Computer and User policy. And the original design finally works end to end — a LAN client's query for the domain now travels through Pi-hole's conditional forwarding and lands on a zone that actually loads, and the LDAP SRV lookup for the domain answers for the first time in the domain's life.

The subplot: I was my own stranger

Early in the incident, the DC resolving its own domain to public IPs looked like a possible hijack or leak. The addresses belonged to Squarespace. That was confusing for about a minute — until I remembered that I own the public version of this domain. I bought it during the lab build, through Google Domains before the registrar migration to Squarespace, never built anything on it, and it has worn the default "under construction" page ever since.

Owning the public twin of your internal AD domain is the textbook-correct pattern — this is exactly the failure it prevents. The "stranger" was me, and keeping that registration renewed is what keeps the joke a joke.

It's also getting a job now: this site. A professional home for write-ups like this one that doesn't depend on any employer, and eventually custom-domain email as the later phase of a broader migration off rented platforms.

What I'm taking from it

Check before you patch. A maintenance errand nobody needed to health-check first is the only reason a year-old silent failure was found on a quiet afternoon instead of mid-crisis.
A system nothing depends on can be broken forever without telling anyone. The lab routed around this failure so completely that a dead core service presented as "domain joins are flaky."
When services fail and admins succeed, suspect the machine account. Everything I ran interactively worked. Everything running as SYSTEM was refused. That split personality is the secure channel's fingerprint.
The test tool may need the broken thing. If your diagnostic locates its target through the failing service, an inconclusive result tells you about the tool's dependency, not about your theory.
Read the event's data, not its headline. The actionable detail was in an XML payload and a sub-error code, hours into an event that just said "unable to open AD."
Dead ends are evidence. Every acquittal — partitions, cross-references, ACLs, the role reinstall — narrowed the suspect list until the only untested principal left was the machine credential itself.
Own your namespace. See the subplot.

The domain controller can find itself now. It only took a year, one health check, and fifteen minutes with netdom.