PNOMA - A vSAN Troubleshooting Framework

Diagnostic only — nothing here changes state. Every command on this page only looks.

  1. PPhysical
  2. NNetwork
  3. OObject
  4. MManagement
  5. AApplication

98% of vSAN issues are diagnosed in the Physical and Network layers. Diagnosis flows UP the stack: start at P, clear each layer, move up only when it's clean.

Framework: Brad Mott. Expansion: Daniel Barnes.

No customer specifics, no internals.

Version 1.5. Verified-against date: 2026-10-28.

Triage

Symptom?

98% of vSAN issues are diagnosed in the Physical and Network layers. Diagnosis flows UP the stack: start at P, clear each layer, move up only when it's clean.

P — Physical · N — Network

Scope?

  • P — Physical: the hardware and storage devices themselves.
  • N — Network: host-to-host transport; if it can't talk, it can't cluster.
  • O — Object: are the pieces of the data present, healthy, and placed per policy?
  • M — Management: when things get weird — tasks, limits, and the settings nobody remembers changing.
  • A — Application: what the client shows vs what the hosts know.

Recent change?

M2 — Did somebody change a setting?

Management: when things get weird — tasks, limits, and the settings nobody remembers changing.

I'm seeing…

Start from what is on your screen. Find the line that matches and take the door — each one lands where the diagnosis actually starts, which is not always the layer the symptom shows up in.

Disk Issues

Object/Datastore Health, Stuck Resyncs

P — Physical

↑ Back to top

The hardware and storage devices themselves. This is the layer where data traverses and resides within the vSAN file system. Architecture split: OSA carries disk groups with a cache tier; ESA is a single-tier storage pool — the checks below cover both (P3 carries the split).

Verified-against date: 2026-10-22. Captures: ESXi 9.1.1, vSAN ESA.

What lives here. Host Hardware, Storage Controllers, Physical Disks, LSOM. Key components: LSOM, Cache Tier drives, Capacity Tier drives.

  • Host Hardware — the physical server box and its guts (CPU, memory, NICs, etc.)
  • Storage Controllers — the card or chip that talks to the drives and presents them to the host
  • Physical Disks — the actual spinning rust or SSDs installed in the server
  • LSOM — the local software layer that manages the disks and cache on each host
  • Cache Tier drives (OSA) — the fast SSD(s) used as the write buffer and read cache in the original vSAN architecture
  • Capacity Tier drives (OSA) — the bulk storage drives (HDD or SSD) that hold the actual data in the original vSAN architecture

Doctrine (DDH). An unmounted or evacuated disk is NOT presumed dead. Dying Disk / Device Handling — the vSAN Device Monitor, vsandevicemonitord — proactively pulls disks showing persistent high latency (enhanced in 9.0: ESA latency monitoring with automatic corrective action; proactive cache-drive failure handling in OSA). First question on an unmounted disk: did DDH act on it? Check vmkernel.log / vobd.log for device-monitor actions before condemning the hardware or reaching for an RMA. This doctrine hangs over P4 and P5 below.

Start here. P1, P3, P5.

An I'm-seeing door overrides this list: when a door routes elsewhere, follow the door, not the list. P1 is not mandatory when the door routes elsewhere.

If seeing this issue, see this KB:

P1 — Is this host in the cluster it thinks it's in? Start here

The node's own view of itself: cluster identity, membership size, and maintenance state.

Run 
esxcli vsan cluster get

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

Cluster Information
  Enabled: true
  Current Local Time: 2026-10-22T14:22:51Z
  Local Node UUID: 6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046
  Local Node Type: NORMAL
  Local Node State: MASTER
  Local Node Health State: HEALTHY
  Sub-Cluster Master UUID: 6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046
  Sub-Cluster Backup UUID: 3d8b7a19-5c2e-4f60-b4a9-1e6d8c3f5a72
  Sub-Cluster UUID: 5e2f8b41-6a9d-4c37-8e1a-9b3d7f5c2e68
  Sub-Cluster Membership Entry Revision: 1
  Sub-Cluster Member Count: 3
  Sub-Cluster Member UUIDs: 6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046, 3d8b7a19-5c2e-4f60-b4a9-1e6d8c3f5a72, 9a5e1d73-8b4c-4a2f-a6d1-4f7b2e9a8d05
  Sub-Cluster Member HostNames: esxi01.lab.local, esxi02.lab.local, esxi03.lab.local
  Sub-Cluster Membership UUID: 8c1d4f69-2b7a-4e95-a8c3-6d9f1b5e4a27
  Unicast Mode Enabled: true
  Maintenance Mode State: OFF
  Config Generation: 7a3f5c81-9e2b-4d64-b7a5-1c9e4f6d2b83 35 2026-10-21T12:37:00.476
  Mode: REGULAR
  vSAN ESA Enabled: true
  vSAN Max Client Network Enabled: false

Look for

Interpretation
Healthy Member count matches the cluster's real size; the Sub-Cluster UUID matches what the other hosts report; Maintenance mode agrees with what you know about the host.
Suspicious Member count lower than the real cluster -> this host can't see everyone (keep going in P for the local side, then take it to N). Maintenance Mode State disagreeing with the host's actual state is an M-layer signature (see M) — note it and route it there.
Stop-escalate the host reports a different Sub-Cluster UUID from its peers — you are looking at a partition, not a sick disk; work N before touching P hardware.

Cross-refs: this command is also the instrument for N's membership-churn signature (N9) and M's DECOM reading.

What build is this host running? The exact vSphere build — readings and driver expectations are version-shaped; pin the version before trusting either.

Run 
vmware -vl

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

DEPRECATION WARNING: `vmware -v / --version` are deprecated as of ESX 9.0, and will be removed in a future release.
VMware ESXi 9.1.1 build-25782009
DEPRECATION WARNING: `vmware -l / --level` are deprecated as of ESX 9.0, and will be removed in a future release.
VMware ESXi 9.0 Update 1

Note (from the 2026-10-22 capture): the command prints a deprecation warning on this build — vmware -v and vmware -l are deprecated as of ESX 9.0 and will be removed in a future release.

P2 — Does the host see its storage controllers?

Which controllers are installed, and which driver name and version each is running.

Run 
esxcli vsan debug controller list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

Device Name: vmhba0
  Device Display Name: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less)
  Used By VSAN: true
  PCI ID: 144d/a80d/144d/a801
  Driver Name: nvme_pcie
  Driver Version: 1.4.0.8-2vmw.911.0.25714478
  Max Supported Queue Depth: 2046
  Firmware Version: N/A
Device Name: vmhba1
  Device Display Name: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less)
  Used By VSAN: false
  PCI ID: 144d/a80d/144d/a801
  Driver Name: nvme_pcie
  Driver Version: 1.4.0.8-2vmw.911.0.25714478
  Max Supported Queue Depth: 2046
  Firmware Version: N/A

Look for

Interpretation
Healthy every expected controller present, with the driver name/version you expect for the build (P1 gives you the build).
Suspicious a controller missing from the list, or an unexpected driver version -> the disks behind it are about to be a P5 problem — vdq -iH shows the mappings: what is missing, what is there, and what is healthy. Identify the affected disks before anything else.
Stop-escalate Missing disk, HBA present vs HBA missing: if the controller is listed but the disk is gone, the trail is a disk story; if the controller itself is gone (vmhba absent), the trail is an adapter story. Either way, identify at P5 and stop — do not run IsPDL on a disk the host cannot see.

P3 — What disks does vSAN claim on this host?

The disks vSAN is using here, and whether vSAN is still talking to each of them.

Run 
esxcli vsan storage list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — tail of the output: the host's single ESA disk entry

t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
  Device: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
  Display Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
  vSAN UUID: 4b7e3a92-5d1c-48f6-9a2e-8c5d1f7b3e94
  Used by this host: true
  In CMMDS: true
  On-disk format version: 24
  Checksum: 6823358492999446998
  Checksum Ok: true
  Is Mounted: true
  Is Encrypted: true
  Disk Type: singleTier
  Creation Time: Sat Dec 28 00:19:05 2024

READ THIS — SAMPLE OUTPUT OSA example (from a support log bundle; identifiers changed — two capacity-tier disks)

naa.5000c29a1b2c3d4e01
  Device: naa.5000c29a1b2c3d4e01
  Display Name: naa.5000c29a1b2c3d4e01
  Is SSD: true
  VSAN UUID: 52c81f4a-3d7e-4b9a-a5c2-6e1d8f3b9a04
  VSAN Disk Group UUID: 52f6b3e9-8a4d-4c7f-9e1b-3d5a8c6f2e47
  VSAN Disk Group Name: naa.5000c29a1b2c3d4e00
  Used by this host: true
  In CMMDS: true
  On-disk format version: 20
  Deduplication: true
  Compression: true
  Checksum: 6292486903516222788
  Checksum OK: true
  Is Capacity Tier: true
  Encryption Metadata Checksum OK: true
  Encryption: false
  DiskKeyLoaded: false
  Is Mounted: true
  Creation Time: Wed Oct 7 20:39:00 2020
naa.5000c29a1b2c3d4e02
  Device: naa.5000c29a1b2c3d4e02
  Display Name: naa.5000c29a1b2c3d4e02
  Is SSD: true
  VSAN UUID: 52a94d7c-6e1b-4f3a-8d5c-9b2e7a4f6c18
  VSAN Disk Group UUID: 52f6b3e9-8a4d-4c7f-9e1b-3d5a8c6f2e47
  VSAN Disk Group Name: naa.5000c29a1b2c3d4e00
  Used by this host: true
  In CMMDS: true
  On-disk format version: 20
  Deduplication: true
  Compression: true
  Checksum: 81241688920798380
  Checksum OK: true
  Is Capacity Tier: true
  Encryption Metadata Checksum OK: true
  Encryption: false
  DiskKeyLoaded: false
  Is Mounted: true
  Creation Time: Wed Oct 7 20:39:00 2020

ESA instead of OSA? Run the ESA counterpart — the single-tier pool view.

Run 
esxcli vsan storagepool list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
  Device: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
  Display Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
  vSAN UUID: 4b7e3a92-5d1c-48f6-9a2e-8c5d1f7b3e94
  Used by this host: true
  In CMMDS: true
  On-disk format version: 24
  Checksum: 6823358492999446998
  Checksum Ok: true
  Is Mounted: true
  Is Encrypted: true
  Disk Type: singleTier
  Creation Time: Sat Dec 28 00:19:05 2024

Look for

Interpretation
Healthy every expected disk listed, In CMMDS true.
Suspicious In CMMDS false on a disk = vSAN has lost comms with it. That is flag one of the two-flag verdict in P5 — pair it with IsPDL before calling the disk dead or alive.
Stop-escalate a disk is absent from the list entirely — check P2 (controller) and P5's group mappings (vdq -iH shows missing disks) before condemning it. No device ⇒ no IsPDL: MISSING MEMBER is enough to stop.

P4 — What is the status of the vSAN disks?

The debug view of vSAN disk status on this host.

Run 
esxcli vsan debug disk list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — the host's single ESA disk

UUID: 4b7e3a92-5d1c-48f6-9a2e-8c5d1f7b3e94
  Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
  Owner: esxi01.lab.local
  Version: 24
  Disk Group: N/A
  Disk Tier: singleTier
  SSD: true
  In Cmmds: true
  In Vsi: true
  Fault Domain: N/A
  Model: Samsung SSD 990 EVO Plus 4TB
  Encryption: true
  Compression: false
  Deduplication: false
  Dedup Ratio: N/A
  Overall Health: green
  Metadata Health: green
  Operational Health: green
  Congestion Health:
    State: green
    Congestion Value: 0
    Congestion Area: none
    All Congestion Fields:
  Space Health:
    State: green
    Capacity: 3725.96 GB
    Used: 2034.52 GB
    Reserved: 0.00 GB

READ THIS — SAMPLE OUTPUT OSA form (trimmed) — group states, contrasted

DiskGroup[0]: State: Mounted
DiskGroup[1]: State: Unmounted

Mounted is in service; Unmounted is out of service. An Unmounted group reads under the DDH doctrine in the Look-for below — do not read it as a dead disk on sight.

Look for

Interpretation
Healthy disks reporting healthy status across the host.
Suspicious any disk flagged otherwise — and before calling it hardware, apply the DDH doctrine: an unmounted or evacuated disk may be the Device Monitor's deliberate act (persistent high latency), not a failure. The proof either way is in vmkernel.log / vobd.log.

Cache latency can unmount a whole group (OSA): when the cache device stalls, DDH may unmount the entire disk group, not just the slow disk. Read the group state, not a single disk, before concluding.

(This check also serves M — it is filed there too.)

P5 — Is this disk eligible, missing, or really dead?

Disk eligibility and the two-flag death verdict. vdq is the eligibility reader; no single flag condemns a disk.

Run 
vdq -qH

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — both disks the host reports

DiskResults:
  DiskResult[0]:
    Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001
    VSANUUID: 4b7e3a92-5d1c-48f6-9a2e-8c5d1f7b3e94
    State: Ineligible for use by VSAN
    Reason: Disk in use by storage pool
    StoragePoolState: In-use for Storage Pool
    StoragePoolReason: None
    IsSSD?: 1
    IsCapacityFlash?: N/A
    IsPDL?: 0
    Size(MB): 3815447
    FormatType: 512e
    IsVsanDirectDisk?: 0
  DiskResult[1]:
    Name: t10.NVMe____Samsung_SSD_990_EVO_1TB____________2C11000000000002
    VSANUUID: (empty)
    State: Ineligible for use by VSAN
    Reason: Has partitions
    StoragePoolState: Ineligible for use by Storage Pool
    StoragePoolReason: Has partitions
    IsSSD?: 1
    IsCapacityFlash?: 0
    IsPDL?: 0
    Size(MB): 953869
    FormatType: 512e
    IsVsanDirectDisk?: 0

Reason "Has partitions" beside IsPDL 0: the disk carries partitions vSAN did not place — it explains ineligibility, not death. IsPDL 0 still rules out permanent device loss.

Run 
vdq -q -d <DEVICE>

Replace: <DEVICE> with the device name (from P3). One disk only.

Run 
vdq -iH

Group mappings:

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

SingleTierDisks:
  "singleTier": [
    "t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001"
  ]

(ESA fills this slot with the single-tier pool membership; on OSA the same slot is the cache -> capacity mappings, and where missing disks show.)

READ THIS — SAMPLE OUTPUT OSA form (trimmed) — cache -> capacity mappings; a disk the host no longer sees shows as MISSING MEMBER

DiskGroups:
  "diskGroup": [
    {
      "cache": "t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________5F1E000000000001",
      "capacity": [
        "t10.NVMe____Samsung_SSD_990_EVO_1TB____________2C11000000000002"
      ]
    }
  ]
MISSING MEMBER:
  "missing": [
    "t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________9A22000000000003"
  ]

A MISSING MEMBER entry tied to a missing vmhba (see P2) is a vanished disk, not a dead one — identify it here, then stop (see the vanished-drive door one-liner).

Run 
vdq -iq

Combined query (JSON).

Look for

Interpretation
Healthy disks eligible, IsPDL 0.
Suspicious IsPDL 1 = permanent device loss — that flag is the real death certificate. The verdict is two-flag: read IsPDL together with In CMMDS from P3. In CMMDS false alone means vSAN lost comms with the disk — a comms story (work it here, then N), not automatically a corpse.
Stop-escalate a disk shows unmounted/evacuated with no PDL — run the DDH doctrine first (P4): check vmkernel.log / vobd.log for vsandevicemonitord actions before any RMA language reaches the customer.

No device, no IsPDL. If vdq -qH has no row for the disk and P2 shows its controller missing, do not run an IsPDL determination — the MISSING MEMBER reading is enough to stop. IsPDL answers whether a present disk is dead; it cannot answer for a disk the host cannot see.

If seeing this issue, see this KB:

P6 — What does the device itself report?

The disk's own error counters — the device's testimony, not vSAN's opinion of it.

Run 
esxcli storage core device smart get -d <DEVICE>

Replace: <DEVICE> with the suspect disk (from P3).

Run 
esxcli nvme device log smart get -A vmhba1

NVMe instead: the NVMe-side SMART readout; check the Volatile Memory Backup Device Failure field.

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — both runs

SMART get (the 4 TB device):
  Parameter                 Value  Threshold  Worst  Raw
  Health Status             OK     N/A        N/A    N/A
  Power-on Hours            15323  N/A        N/A    N/A
  Power Cycle Count         39     N/A        N/A    N/A
  Reallocated Sector Count  0      90         N/A    N/A
  Drive Temperature         51     81         N/A    N/A
NVMe SMART log (-A vmhba1):
  SMART And Health Info:
    Available Spare Space Below Threshold: false
    Temperature Warning: false
    NVM Subsystem Reliability Degradation: false
    Read Only Mode: false
    Volatile Memory Backup Device Failure: false
    Composite Temperature: 329 K
    Available Spare: 100 %
    Available Spare Threshold: 10 %
    Percentage Used: 0 %
    Data Units Read: 0x89a3fb8
    Data Units Written: 0xab1ff6
    Host Read Commands: 0x3b911501
    Host Write Commands: 0x3115813e
    Controller Busy Time: 0xa28
    Power Cycles: 0x41
    Power On Hours: 0x3ea6
    Unsafe Shutdowns: 0x28
    Media Errors: 0x0
    Number of Error Info Log Entries: 0x0
    Warning Composite Temperature Time: 0 Mins
    Critical Composite Temperature Time: 0 Mins
    Temperature Sensor 1: 348 K
    Temperature Sensor 2: 329 K
    Temperature Sensor 3: 0 K
    Temperature Sensor 4: 0 K
    Temperature Sensor 5: 0 K
    Temperature Sensor 6: 0 K
    Temperature Sensor 7: 0 K
    Temperature Sensor 8: 0 K

Look for

Interpretation
Healthy error counters flat / zero on the suspect disk.
Suspicious climbing error counters, or the NVMe Volatile Memory Backup Device Failure field tripped -> the device is indicting itself; now the hardware conversation is evidence-based.

P7 — Is the latency real, and where does it live?

Live storage latency at the device, split by who owns the wait.

Run 
esxtop

then press u (disk device view)

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — point-in-time monitor views; values are the instant of the capture, not stored output

Device view (u):
  2:26:11pm up 4 days 16:29, 2155 worlds, 3 VMs, 10 vCPUs; CPU load average: 0.14, 0.15, 0.15
DEVICE                               | PATH/WORLD/PARTITION | DQLEN | WQLEN | ACTV | QUED | %USD | LOAD | CMDS/s | READS/s | WRITES/s | MBREAD/s | MBWRTN/s | DAVG/cmd | KAVG/cmd | GAVG/cmd | QAVG/cmd
t10.NVMe____Samsung_SSD_990_EVO_1TB_ | -                    | 32    | -     | 0    | 0    | 0    | 0.00 | 0.00   | 0.00    | 0.00     | 0.00     | 0.00     | 0.00     | 0.00     | 0.00     | 0.00
t10.NVMe____Samsung_SSD_990_EVO_Plus | -                    | 2046  | -     | 0    | 0    | 0    | 0.00 | 205.99 | 91.55   | 114.44   | 41.98    | 2.46     | 0.25     | 0.01     | 0.26     | 0.00
  (device names as esxtop truncates them in its column)
Adapter view (d):
ADAPTR | PATH | NPTH | CMDS/s | READS/s | WRITES/s | MBREAD/s | MBWRTN/s | DAVG/cmd | KAVG/cmd | GAVG/cmd | QAVG/cmd
vmhba0 | -    | 1    | 249.09 | 86.47   | 162.76   | 36.26    | 1.13     | 0.22     | 0.02     | 0.23     | 0.00
vmhba1 | -    | 1    | 5.09   | 5.09    | 0.00     | 0.02     | 0.00     | 0.37     | 0.02     | 0.39     | 0.00 Bonus: press `d` for the disk adapter view.

Look for

Interpretation
Healthy latency low and steady across devices.
Suspicious one device's latency standing out from its siblings -> the suspect is physical (stay in P, take the disk to P5/P6). Latency everywhere at once is rarely one disk — think congestion (a Look-for at this layer) and take the pattern to N before blaming media.

STOP — boundary of the sheet.

A disk the host unmounted on its own is a symptom, and this layer's job is reading the evidence — the counters, the log lines, the pattern across hosts. Mounting, unmounting, or condemning a disk is the fix, and the fix happens after diagnosis, supervised. It is never a step toward one.

Read this (P logs)

  • vmkernel.log
  • vobd.log

Healthy-at-a-Glance

  • cluster get: member count and Sub-Cluster UUID agree with the peers; maintenance state agrees with reality
  • storage list / storagepool list: every expected disk present, In CMMDS true
  • vdq: disks eligible, IsPDL 0; unmounted disks explained by DDH before they are explained by death
  • SMART: error counters flat

Layer-level Look-for: SCSI sense codes, disk latency, permanent disk failure (PDL), congestion.

Found it? Capture: the check ID, the reading, and the timestamp for the case notes.

Layer clear? -> Diagnosis flows UP: take it to N. A clean P with a live symptom is a transport problem until N says otherwise.

Hands off to: N — the hardware reads clean but hosts still can't see each other: membership and partitions continue at N1 and N9. O — what a sick disk did to the objects living on it is read in Object health.

N — Network

↑ Back to top

Host-to-host transport; if it can't talk, it can't cluster. Diagnosis flows UP from P, and this is the second of the two layers where 98% of issues are diagnosed. Architecture split: OSA clusters carry multicast settings; ESA is unicast — N3/N4 carry the split.

Verified-against date: 2026-10-22. Captures: ESXi 9.1.1, vSAN ESA.

What lives here. RDT, Networking.

  • RDT (Reliable Datagram Transport) — the protocol vSAN uses for communication between hosts over the vSAN VMkernel ports (commands, I/O flow) — optimized to move large amounts of data.
  • Networking — the NICs and switches that move vSAN traffic between hosts so the cluster can stay in sync

Out of scope for this layer: deep performance diagnostics (Humbug, Grafana dashboards) are not cheat-sheet material. Basic esxtop statistics only.

Start here. N3, N5, N9.

N1 — What physical NICs does the host see? Start here

The physical NIC inventory — the bottom of the transport stack.

Run 
esxcli network nic list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

Name   | PCI Device   | Driver   | Admin Status | Link Status | Speed | Duplex | MAC Address       | MTU  | Description
vmnic0 | 0000:02:00.0 | i40en    | Up           | Up          | 10000 | Full   | 58:47:ca:11:22:31 | 9000 | Intel(R) Ethernet Controller X710 for 10GbE SFP+
vmnic1 | 0000:02:00.1 | i40en    | Up           | Up          | 10000 | Full   | 58:47:ca:11:22:32 | 9000 | Intel(R) Ethernet Controller X710 for 10GbE SFP+
vmnic2 | 0000:57:00.0 | cndi_igc | Up           | Up          | 2500  | Full   | 58:47:ca:11:22:33 | 1700 | Intel Corporation Ethernet Controller I226-V
vmnic3 | 0000:59:00.0 | cndi_igc | Up           | Up          | 2500  | Full   | 58:47:ca:11:22:34 | 1700 | Intel Corporation Ethernet Controller I226-LM

Look for

Interpretation
Healthy every expected vmnic present.
Suspicious a vmnic missing or down -> the fault is at the physical path before it is anywhere else; pair with N6's port-localization reading (a dropping vmnic port points at cable, optic, or switch port).

N2 — How are the VMkernel interfaces addressed?

Every VMK's IP, subnet, and gateway — the addressing the vSAN VMK (N3) has to live on.

Run 
esxcli network ip interface ipv4 get

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

Name  | IPv4 Address | IPv4 Netmask  | IPv4 Broadcast  | Address Type | Gateway    | DHCP DNS
vmk0  | 10.20.15.11  | 255.255.240.0 | 10.20.15.255    | STATIC       | 10.20.0.1  | false
vmk1  | 10.20.17.11  | 255.255.240.0 | 10.20.17.255    | STATIC       | 10.20.17.1 | false
vmk10 | 10.20.13.10  | 255.255.240.0 | 10.20.15.255    | STATIC       | 10.20.0.1  | false
vmk50 | 169.254.1.1  | 255.255.0.0   | 169.254.255.255 | STATIC       | 0.0.0.0    | false
vmk2  | 10.20.18.11  | 255.255.240.0 | 10.20.15.255    | STATIC       | 10.20.0.1  | false

Look for

Interpretation
Healthy the vSAN VMK addressed on the expected subnet, consistent with its peers.
Suspicious an addressing or subnet mismatch on the vSAN VMK -> hosts that can see some peers and not others (and a membership story — see N9) without anything being physically down.

STOP — boundary of the sheet.

This sheet checks MTU; it never sets it. Do not re-tag a VMkernel adapter, and do not change MTU on a host or a switch from anything you read here — an MTU change, on the master especially, can take down the entire cluster. If the two sides disagree, capture both readings — then form a deliberate action plan to correct it, with full customer awareness and approval.

N3 — Which VMK carries vSAN, and how is it set up?

vSAN's own view of its transport: the vSAN VMK, multicast settings, and port info.

Run 
esxcli vsan network list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

Interface
  VmkNic Name: vmk1
  IP Protocol: TCP(0)
  Interface UUID: 2e6a9c15-7d3b-4f81-b5a9-3c8e2d6f1a47
  Agent Group Multicast Address: 224.2.3.4
  Agent Group IPv6 Multicast Address: ff19::2:3:4
  Agent Group Multicast Port: 23451
  Master Group Multicast Address: 224.1.2.3
  Master Group IPv6 Multicast Address: ff19::1:2:3
  Master Group Multicast Port: 12345
  Host Unicast Channel Bound Port: 12321
  Data-in-Transit Encryption Key Exchange Port: 12443
  Multicast TTL: 5
  Traffic Type: vsan
(A second capture from the same host in the same sitting shows an additional witness-tagged interface on vmk0 in a fuller listing.)

Look for

Interpretation
Healthy the expected VMK bound to vSAN traffic; on OSA, multicast settings as configured for the cluster.
Suspicious the wrong VMK carrying vSAN, or multicast settings (OSA) that differ between hosts -> a cluster that forms partially, or not at all, with every ping in N5 potentially still passing.

Witness traffic uses the vSAN-tagged VMkernel unless an adapter carries the explicit witness tag. The tag exists for segregated stretched clusters — ROBO especially — where the vSAN adapters have no layer-3 path to the witness and another adapter has to carry the traffic. It belongs on data nodes only: the witness itself never gets it, and needs no tag but vSAN. Broken witness flows? Check whether a tag is diverting traffic that should ride vSAN, or missing where the vSAN adapters cannot reach.

N4 — Does this host know the unicast table?

The vSAN unicast table — who this host believes its cluster peers are, address by address. (The ESA-era membership phone book; OSA leans on N3's multicast.)

Run 
esxcli vsan cluster unicastagent list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

(The capture crops the top of the table — most column headers are cut off; SubClusterUuid and Traffic Type are the legible ones, so the rows below are transcribed in column order as captured, without asserting the cropped header names. The two visible rows are the remote agents: the list shows the OTHER cluster members from this host's view — the local host does not appear in its own table.)
9a5e1d73-8b4c-4a2f-a6d1-4f7b2e9a8d05 | 0 | true | 10.20.17.13 | 12321 | 3F:7A:91:C4:0D:58:E2:6B:14:A9:D7:03:8C:F5:29:61:BE:47:9A:20:5D:E8:13:76:CB:04:92:AD:68:F1:35:7E | 5e2f8b41-6a9d-4c37-8e1a-9b3d7f5c2e68 | vsan
3d8b7a19-5c2e-4f60-b4a9-1e6d8c3f5a72 | 0 | true | 10.20.17.12 | 12321 | A6:2D:84:F9:51:CB:07:9E:38:D4:6A:B5:20:7F:E1:49:93:5C:A8:16:DA:62:0B:F4:87:3E:C9:55:7D:21:9A:04 | 5e2f8b41-6a9d-4c37-8e1a-9b3d7f5c2e68 | vsan

Look for

Interpretation
Healthy the table carries the cluster's member hosts.
Suspicious missing or stale entries -> this host will flap or partition against specific peers; correlate with N9's revision reading before blaming the fabric. Unicast row ≠ same UUID / Member Hosts: a peer present in the table but omitted from Member Hosts is a membership story, not a fabric story.

Membership can also desync with nothing broken in this layer: if a host's /VSAN/IgnoreClusterMemberListUpdates advanced setting sits at 1 — a leftover from a cluster move or a vCenter replacement — it ignores the member list vCenter publishes, and the 'vCenter state is authoritative' health check fires. Confirming that setting's value on each host is the diagnosis. The command lives under M2; the narrative lives here.

Is data-in-transit encryption on, and the same everywhere? Confirm whether data-in-transit (DIT) encryption is on for this host/cluster, and its rekey interval — the first question when DIT is suspected in a comms break.

Run 
esxcli vsan network security get

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22

Sub-Cluster UUID: 5e2f8b41-6a9d-4c37-8e1a-9b3d7f5c2e68
Data-in-Transit Encryption status: true
Rekey Interval (in minutes): 1440
Remote DIT Encryption: None
Remote Cluster Type: None

Run security get on every node when a partition persists — including when N1–N3 and the unicast table look fine. A single host's DIT status is not the cluster's; the mismatch hides on the host you did not check.

Look for

Interpretation
Healthy status true and CONSISTENT across hosts; rekey interval 1440.
Suspicious status disagrees between hosts — especially after certificate or thumbprint changes. DIT key exchange then fails and vSAN comms break even though every L3/L4 check in this layer passes. Different Sub-Cluster UUIDs on the data hosts with a broken witness view: the hosts are not in the same sub-cluster the witness sees — correlate with the witness view before blaming the witness host.
Stop-escalate hosts disagree on DIT status and comms are down. Restoring comms by changing DIT state is a deliberate, state-changing procedure. Confirm status per host here; the change itself is supervised work.

STOP — boundary of the sheet.

If the cluster is partitioned, DIT is the suspect, and vCenter is down, DIT can be turned off from the command line. Work it supervised.

If seeing this issue, see this KB:

N5 — Can this host actually reach a peer, end to end?

L3 proof over a specific VMK — first plain, then at full frame size. The MTU end-to-end test is the second run, not an afterthought.

Run 
vmkping -I <VSAN-VMK> <PEER-IP>
Run 
vmkping -I <VSAN-VMK> -s 8972 <PEER-IP> -d

Replace: <VSAN-VMK> with the vSAN VMkernel interface (from N3); <PEER-IP> with the peer host's vSAN VMK address (from N2's table, on the peer).

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — both runs, to peer 10.20.17.13 over vmk1

Plain run (vmkping -I vmk1 10.20.17.13):
  PING 10.20.17.13 (10.20.17.13): 56 data bytes
  64 bytes from 10.20.17.13: icmp_seq=0 ttl=64 time=0.135 ms
  64 bytes from 10.20.17.13: icmp_seq=1 ttl=64 time=0.141 ms
  64 bytes from 10.20.17.13: icmp_seq=2 ttl=64 time=0.106 ms
  --- 10.20.17.13 ping statistics ---
  3 packets transmitted, 3 packets received, 0% packet loss
  round-trip min/avg/max = 0.106/0.127/0.141 ms
Jumbo run (vmkping -I vmk1 10.20.17.13 -s 8972 -d):
  PING 10.20.17.13 (10.20.17.13): 8972 data bytes
  8980 bytes from 10.20.17.13: icmp_seq=0 ttl=64 time=0.226 ms
  8980 bytes from 10.20.17.13: icmp_seq=1 ttl=64 time=0.264 ms
  8980 bytes from 10.20.17.13: icmp_seq=2 ttl=64 time=0.155 ms
  --- 10.20.17.13 ping statistics ---
  3 packets transmitted, 3 packets received, 0% packet loss
  round-trip min/avg/max = 0.155/0.215/0.264 ms

READ THIS — SAMPLE OUTPUT Jumbo fail modes — same command, vmkping -I vmk1 <PEER-IP> -s 8972 -d

Plain run:
  3 packets transmitted, 0 packets received, 100% packet loss
Capped host:
  sendto() failed (Message too long)

Two jumbo failures, two meanings. Plain 100% packet loss means the packet left the host and died downstream or went unanswered — nothing special in the output; look off-host. "sendto() failed (Message too long)" means the sending host itself is set below the test size (a host at 1500 cannot send an 8972 test), so the packet never left — check the local nic/interface MTU first. Small pings pass + jumbo fails is the split that IS the diagnosis.

Note on the second run: 8972 payload + headers = 9000 MTU — it only passes if the whole path (VMK, vSwitch, physical fabric) is jumbo-clean.

Look for

Interpretation
Healthy both runs answer.
Suspicious plain ping passes, jumbo fails -> an MTU mismatch somewhere on the path (a layer Look-for). Small frames walk through; vSAN's real traffic doesn't. That split IS the diagnosis. If host MTU matches 9000 but jumbo fails, the mismatch is off-host — capture the fabric / intermediate-hop MTU (ask the network team for the segment MTU); stop before changing anything. Point to the nic/interface MTU from N5: read the interface MTU on the path, not just the vmk.
Stop-escalate even the plain run fails to a peer the cluster insists is a member — stop pinging and go to N9; you may be watching membership churn, not a dead path.

If seeing this issue, see this KB:

N6 — Where are packets dying: physical or virtual?

Live drop and throughput reading, with the localization that tells you which side of the host to blame.

Run 
esxtop

Then press n (network view).

Run 
esxcli network nic stats get -n <VMNIC>

Replace: <VMNIC> with the uplink under suspicion (from N1). Cross-check.

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — the esxtop view is a point-in-time monitor view; values are the instant of the capture, not stored output

Network view (n):
  2:33:10pm up 4 days 16:58, 2155 worlds, 3 VMs, 10 vCPUs; CPU load average: 0.19, 0.10, 0.10
PORT-ID    | USED-BY          | TEAM-PNIC | DNAME        | PKTTX/s | MbTX/s | PSZTX   | PKTRX/s | MbRX/s | PSZRX   | %DRPTX | %DRPRX
67108870   | Management       | n/a       | DvsPortset-0 | 0.00    | 0.00   | 0.00    | 0.00    | 0.00   | 0.00    | 0.00   | 0.00
67108873   | Shadow of vmnic2 | n/a       | DvsPortset-0 | 0.00    | 0.00   | 0.00    | 0.00    | 0.00   | 0.00    | 0.00   | 0.00
67108875   | Shadow of vmnic3 | n/a       | DvsPortset-0 | 0.00    | 0.00   | 0.00    | 0.00    | 0.00   | 0.00    | 0.00   | 0.00
67108880   | vmk0             | vmnic2    | DvsPortset-0 | 104.63  | 0.23   | 289.00  | 43.60   | 0.07   | 208.00  | 0.00   | 0.00
67108883   | vmk10            | vmnic2    | DvsPortset-0 | 0.00    | 0.00   | 0.00    | 4.36    | 0.00   | 60.00   | 0.00   | 0.00
67108884   | vmk50            | void      | DvsPortset-0 | 0.00    | 0.00   | 0.00    | 0.00    | 0.00   | 0.00    | 0.00   | 0.00
67108885   | vdr-vdrPort      | vmnic2    | DvsPortset-0 | 0.00    | 0.00   | 0.00    | 4.36    | 0.00   | 60.00   | 0.00   | 0.00
  [further VM port rows omitted]
100663303  | Management       | n/a       | DvsPortset-1 | 0.00    | 0.00   | 0.00    | 0.00    | 0.00   | 0.00    | 0.00   | 0.00
100663308  | Shadow of vmnic0 | n/a       | DvsPortset-1 | 0.00    | 0.00   | 0.00    | 0.00    | 0.00   | 0.00    | 0.00   | 0.00
100663311  | Shadow of vmnic1 | n/a       | DvsPortset-1 | 0.00    | 0.00   | 0.00    | 0.00    | 0.00   | 0.00    | 0.00   | 0.00
100663313  | vmk1             | vmnic0    | DvsPortset-1 | 1120.43 | 9.74   | 1139.00 | 1185.83 | 25.61  | 2830.00 | 0.00   | 0.00
100663314  | vmk2             | vmnic1    | DvsPortset-1 | 0.00    | 0.00   | 0.00    | 4.36    | 0.00   | 60.00   | 0.00   | 0.00
2214592520 | vmnic2           | -         | DvsPortset-0 | 113.35  | 0.24   | 272.00  | 52.32   | 0.07   | 186.00  | 0.00   | 0.00
2214592522 | vmnic3           | -         | DvsPortset-0 | 0.00    | 0.00   | 0.00    | 4.36    | 0.00   | 60.00   | 0.00   | 0.00
2248146956 | vmnic0           | -         | DvsPortset-1 | 1142.23 | 9.75   | 1118.00 | 1190.19 | 25.60  | 2819.00 | 0.00   | 0.00
2248146958 | vmnic1           | -         | DvsPortset-1 | 0.00    | 0.00   | 0.00    | 8.72    | 0.00   | 60.00   | 0.00   | 0.00
Cross-check (esxcli network nic stats get -n vmnic0):
  NIC statistics for vmnic0
    Packets received: 314388720
    Packets sent: 296789063
    Bytes received: 1182890032615
    Bytes sent: 643179955812
    Receive packets dropped: 0
    Transmit packets dropped: 0
    Multicast packets received: 1163983
    Broadcast packets received: 1960209
    Multicast packets sent: 16810
    Broadcast packets sent: 1408
    Total receive errors: 0
    Receive length errors: 0
    Receive over errors: 0
    Receive CRC errors: 0
    Receive frame errors: 0
    Receive FIFO errors: 0
    Receive missed errors: 0
    Total transmit errors: 0
    Transmit aborted errors: 0
    Transmit carrier errors: 0
    Transmit FIFO errors: 0
    Transmit heartbeat errors: 0
    Transmit window errors: 0

Look for

Interpretation
Healthy %DRPRX / %DRPTX at 0.00; throughput nowhere near line rate unless the workload explains it.
Suspicious any sustained non-zero %DRPRX / %DRPTX is a finding. Which port drops localizes the fault: drops on the vmnic port = the physical path (cable, optic, switch port); drops on a VM/VMK port = the virtual side (vSwitch, teaming, config). MbTX/s pinned at line rate = saturation impersonating a storage problem.
CRC split The CRC split (from the cross-check): drops WITH CRC errors = physical layer; drops with clean CRCs = congestion / buffers. Same drop counter, two different owners — this reading is how you tell them apart.

N8 — Port evidence: probe what you can reach, watch what you can't

nc can only test endpoints reachable over the host's routed stacks, and it cannot designate the vSAN VMK path — so the sheet probes exactly one port with nc and proves the inter-host legs by observing their real traffic with tcpdump-uw.

Run 
nc -zv <KMS_IP_Address> 5696

Replace: <KMS_IP_Address> with the KMS / key provider address. Run A (KMS probe).

READ THIS — SAMPLE OUTPUT Read A. The failure line is the 2026-10-22 capture. The success line is the standard nc -zv counterpart and is not a capture.

nc: connect to <KMS_IP_Address> port 5696 (tcp) failed: Connection timed out
nc: connect to <KMS_IP_Address> port 5696 (tcp) succeeded!

First line: captured failure. Second line: standard success counterpart, not a capture.

A successful nc to a KMS host is not witness proof. KMS connectivity and witness connectivity are different peers on different paths — a KMS nc that answers says nothing about the witness. Prove the witness with the witness checks, never with a KMS probe.

Run 
tcpdump-uw -i vmkX | grep 12321
Run 
tcpdump-uw -i vmkX | grep <peer IP/FQDN>

Replace: vmkX with the vSAN VMK; <peer IP/FQDN> with the other host or witness address. Run the first form on the witness appliance, and the second on the Primary and Backup nodes (per Broadcom KB 326958). The examples are verbatim from that KB. Run B (inter-host observation).

Read B: connectivity is verified by observing BIDIRECTIONAL traffic over port 12321 on both sides. In witness topologies, only the Primary and Backup nodes originate traffic to the witness.

DIT: status first via N4 (esxcli vsan network security get); the same tcpdump pattern with 12443 in place of 12321 watches DIT key-exchange traffic between hosts.

Look for

Interpretation
Healthy the KMS probe succeeds, and 12321 traffic flows in both directions between the hosts that should be talking.
Suspicious no 12321 traffic in one or both directions -> the leg is blocked or mistagged; route back to the N1–N3 readings (VMK binding, addressing, tagging) and work it there. Traffic present but membership still churning -> leave N; the problem is above the wire.

N9 — SIGNATURE: is a host flapping in and out?

Catch the intermittent fault that every point-in-time check passes. Membership churn is this layer's signature: the evidence lives in CMMDS, the root cause is almost always here.

Run 
esxcli vsan cluster get

Repeatedly, across time, not once. The instrument is P1's; this reading is N's.

READ THIS — SAMPLE OUTPUT Healthy example. Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — the P1 capture from this same sitting

Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 3

Healthy is a LOW, STABLE number — equal to the number of hosts in the cluster, or just 1 — unchanged across resamples.

READ THIS — SAMPLE OUTPUT Unhealthy example. Illustrative — not a capture

(first sample)
Sub-Cluster Membership Entry Revision: 312
(resample, minutes later)
Sub-Cluster Membership Entry Revision: 319

A revision high in general (300+) is a sign of an unstable cluster; a revision actively incrementing across samples means membership is churning NOW — route to the N-layer cause hunt (the flapping host's path readings in N1–N6). (The same churn also shows in CMMDS entries via cmmds-tool find -t HOSTNAME; that tool is filed in O as deep-end kit — know the door exists.)

Look for

Interpretation
Healthy the revision sits still across samples.
Suspicious the revision climbs — rapidly OR steadily — means a host is flapping in and out of the cluster. Hunt the cause in this layer: that host's network path — link flaps, MTU, congestion dropping heartbeats. Do not let anyone sell you a storage explanation for a climbing revision. Count may stay full while Revision climbs = churn: a steady Member Count of 3 with a climbing Revision is membership churn, not stability. A high number that does not move is history; a number that climbs across timed samples is an event in progress. Timestamp each revision sample. Capture fail and recover timestamps together: an N9 timeline needs both edges, not just the fail. Flap entry (N9 from a flapping host) vs resync-stall entry (N9 from O5/O6): same N9 mechanics, different doors — the entry door names the story; do not merge the two timelines.
Stop-escalate the revision is climbing and the path checks (N5, N6) all pass in the moment — capture timestamps of the revision at each sample for the case notes; intermittent faults are proven with a timeline, not a snapshot. Clean N5/N6 with a climbing Revision hands off toward networking/switch — do not stay in P. The handoff destination is a pointer, not a fix: the sheet stops here.

Read this (N logs)

  • vmkernel.log
  • vobd.log
  • vpxa.log

Healthy-at-a-Glance

  • vsan network list: expected VMK bound; OSA multicast / ESA unicast settings consistent across hosts
  • vmkping: plain AND jumbo (-s 8972 -d) both answer
  • esxtop n: %DRPRX / %DRPTX at 0.00; no port pinned at line rate without a workload reason
  • cluster get, repeated: Sub-Cluster Membership Entry Revision sitting still

Layer-level Look-for: port drops (link-state changes, dropped-packet counters), MTU mismatches, latency between hosts.

Found it? Capture: the check ID, the reading, and the timestamp for the case notes.

Layer clear? -> Diagnosis flows UP: object symptoms go to O. But if the membership revision is climbing, you are not done here — the cause lives in N even when the alarm rings in O.

Hands off to: P — CRC and link errors are born physical: cabling, optics, NICs (P4, P5). O — resync and placement symptoms surface in Object.

O — Object

↑ Back to top

The placement layer: why objects went inaccessible, stuck resyncs, object recoverability, and what-if analysis before touching anything.

Verified-against date: 2026-10-22. Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — identifiers pseudonymized per the sheet's disclosure; readings as captured. Samples marked (trimmed) are shortened under the sheet's trimming rule; the elision is labelled inside the sample.

What lives here. CLOM, DOM, CMMDS — the object life-cycle layer: these components create objects with the configuration defined through SPBM, validate the configuration can be satisfied, and publish the object's type, owner, policy/layout, and child components to all hosts.

  • CLOM — the placement brain: decides which hosts and disks an object's components land on, and whether a policy can be satisfied at all
  • DOM — the object manager on each host: creates and distributes the components that make up objects, tracks object ownership and state, and coordinates I/O and resyncs for the objects it owns
  • CMMDS — the cluster's shared directory: who is in the cluster, what devices and networks exist, and the metadata (policies, RAID layout, ownership) for every object, published to every host

The 100-object trap. debug object list and overview show at most 100 objects by default. In a cluster with thousands of objects, "not in the output" usually means "not in the first 100" — NOT "doesn't exist." --all (or filtering, or --max-number) is how you search the whole cluster. This burns people constantly. It hangs over O2 and O3 below.

Start here. O1, O2, O5.

O1 — Are the objects healthy, in one shot?

Health summary of all objects — the 30-second verdict for this layer.

Run 
esxcli vsan debug object health summary get

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. The healthy row holds all 64 objects; every problem category is at 0.

Health Status                                    Number Of Objects
remoteAccessible                                               0
inaccessible                                                   0
reduced-availability-with-no-rebuild                           0
reduced-availability-with-no-rebuild-delay-timer               0
reducedavailabilitywithpolicypending                           0
reducedavailabilitywithpolicypendingfailed                     0
reduced-availability-with-active-rebuild                       0
reducedavailabilitywithpausedrebuild                           0
data-move                                                      0
nonavailability-related-reconfig                               0
nonavailabilityrelatedincompliancewithpolicypending            0
nonavailabilityrelatedincompliancewithpolicypendingfailed      0
nonavailability-related-incompliance                           0
nonavailabilityrelatedincompliancewithpausedrebuild            0
healthy                                                       64

The healthy row holds the count and every problem category answers 0 — that IS the healthy read: the table names every way an object can be unwell, and on a healthy cluster none of them have anything in them.

Look for

Interpretation
Healthy the healthy row holds the full object count; every problem category at 0.
Suspicious any object counted in a non-healthy category -> go to O2 to name the specific objects; do not stop at the summary count.
Stop-escalate objects are inaccessible — name them in O3 with --health=inaccessible before theorising about cause; cause may still live in P or N.

If seeing this issue, see this KB:

O2 — What objects are there, at a glance?

Overview of all objects.

Run 
esxcli vsan debug object overview

Replace: none — but note the caps: --all, --max-number; default shows at most 100 (see the layer's 100-object trap).

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. 3 of the cluster's objects shown.

Object UUID                           Group UUID                            Version  Size       Used       Effective Used  Snapshot Usage  Snapshot Count  SPBM Profile                                            Healthy Components
06b3d06a-da71-7839-8bf2-5847ca78dc88  06b3d06a-da71-7839-8bf2-5847ca78dc88  24       255.00 GB  0.06 GB    0.04 GB         0.00 GB         0               vSAN-ESA - Optimal Datastore Default Policy - AutoRAID  8 of 8            
07b3d06a-521a-853f-ac46-5847ca78dc88  06b3d06a-da71-7839-8bf2-5847ca78dc88  24       256.00 GB  120.04 GB  80.03 GB        116.78 GB       1               vSAN-ESA - Optimal Datastore Default Policy - AutoRAID  14 of 14          
baf6d36a-2ec1-147e-eab6-5847ca78e6b8  06b3d06a-da71-7839-8bf2-5847ca78dc88  24       16.00 GB   0.00 GB    0.00 GB         0.00 GB         0               vSAN-ESA - Optimal Datastore Default Policy - AutoRAID  8 of 8

Look for

Interpretation
Healthy the object set you expect, in healthy states.
Suspicious an object missing from a capped listing is NOT evidence it is gone — rerun with --all or a filter (O3) before concluding anything.

One object needs opening up — its components, its policy, where it is damaged: that anatomy is O3. Pick the object from this list and take it there.

O3 — The deep readout on an object

Per-object health, owner, policy, and every component's state (ACTIVE / ABSENT / DEGRADED), disk UUID and host. This is where an object diagnosis is actually made: component states inside an object ARE the diagnosis, not a detail.

Run 
esxcli vsan debug object list

Replace: filters as needed — --health=inaccessible (only the objects that are down; also: healthy, reduced-availability, etc.) --uuid / -u <OBJECT-UUID> — one object --vm-name <VM-NAME> — one VM's objects --disk-uuid <DISK-UUID> — everything with components on a disk (the P-layer handoff: sick disk -> which objects bleed) --comp-owner, --guuid / -g, --policy, --sort / -a, --all, --max-number (default cap 100 objects — see 100-object trap)

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 (trimmed). The RAID_1 leg and the first RAID_5 stripe in full; the remaining two stripes elided. 8 components total, all ACTIVE.

Object UUID: 32f72769-00b2-a603-805a-5847ca78dc88
   Version: 24
   Health: healthy
   Owner: esxi03.lab.local
   Size: 100.00 GB
   Used: 46.80 GB
   Effective Used: 31.20 GB
   Snapshot Usage: 0.00 GB
   Snapshot Count: 0
   Used 4K Blocks: N/A
   Policy:
      spbmProfileId: 0212ed07-2dca-45d4-8262-c5cab4c30ac1
      spbmProfileGenerationNumber: 1
      objectVersion: 24
      CSN: 27346
      SCSN: 27239
      spbmProfileName: vSAN-ESA - Optimal Datastore Default Policy - AutoRAID
      autoManagedRAID: 1
      autoRAIDLevel: 5
   Configuration:
      Concatenation
         RAID_1
            Component: 32f72769-9bb6-6f05-7016-5847ca78dc88
              Component State: ACTIVE,  Address Space(B): 273804165120 (255.00GB),  Disk UUID: 52e7cb41-fb2c-7ca4-2a5b-ed3f0af8cabd,  Disk Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________941E000000000001:2
              Votes: 1,  Capacity Used(B): 29765632 (0.03GB),  Physical Capacity Used(B): 29765632 (0.03GB),  Host Name: esxi02.lab.local
            Component: d9d4106a-36de-d8c0-aa54-5847ca78dc88
              Component State: ACTIVE,  Address Space(B): 273804165120 (255.00GB),  Disk UUID: 524dbda3-1a2b-6364-d4bf-456798c68e4c,  Disk Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________8B1E000000000002:2
              Votes: 1,  Capacity Used(B): 32382976 (0.03GB),  Physical Capacity Used(B): 32382976 (0.03GB),  Host Name: esxi03.lab.local
         RAID_5
            RAID_0
               Component: d9d4106a-8c1d-dbc0-0581-5847ca78dc88
                 Component State: ACTIVE,  Address Space(B): 205353123840 (191.25GB),  Disk UUID: 524dbda3-1a2b-6364-d4bf-456798c68e4c,  Disk Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________8B1E000000000002:2
                 Votes: 1,  Capacity Used(B): 8390574848 (7.81GB),  Physical Capacity Used(B): 9508356096 (8.86GB),  Host Name: esxi03.lab.local
               Component: d9d4106a-82d0-dbc0-88a8-5847ca78dc88
                 Component State: ACTIVE,  Address Space(B): 205353123840 (191.25GB),  Disk UUID: 524dbda3-1a2b-6364-d4bf-456798c68e4c,  Disk Name: t10.NVMe____Samsung_SSD_990_EVO_Plus_4TB____________8B1E000000000002:2
                 Votes: 1,  Capacity Used(B): 8390574848 (7.81GB),  Physical Capacity Used(B): 9508356096 (8.86GB),  Host Name: esxi03.lab.local
            (two more RAID_0 stripes elided — same shape, on esxi02.lab.local and esxi01.lab.local)
   Type: vdisk
   Path: /vmfs/volumes/vsan:528bd398cf7f9ffb-40c234e1ea584113/fff62769-2a45-0551-498e-5847ca78dc88/VCSA_11.vmdk (Exists)
   Group UUID: fff62769-2a45-0551-498e-5847ca78dc88
   Directory Name: N/A
Run 
/etc/init.d/epd status

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-23. ESA only: epd persists DOM's object-metadata entries.

epd is running

Look for

Interpretation
Healthy components ACTIVE, policy satisfiable with the current placement.
Suspicious ABSENT vs DEGRADED components — read the state, not just the object's headline health; objects whose policy can't be satisfied with the surviving placement.
Suspicious on ESA, object reads that look frozen or inconsistent across hosts, with epd not running -> DOM's metadata is not persisting; distrust the readout until epd is back.
Stop-escalate debug object list cannot identify the object at all -> O4 (Objtool reads below DOM).

ABSENT → P3/P5 before more object/resync work: an ABSENT component is a physical story first — read the disk (P3) and the eligibility (P5) before spending more time in O. Do not linger in O, and do not reach for O5 while the component is ABSENT — O5 measures resync, and there is nothing resyncing yet.

O4 — Name a damaged object DOM can't identify

The below-DOM reader: when an object is too damaged, or its metadata too broken, for debug object list to identify it, Objtool reads the object's own attributes directly — name, path, policy. The damaged-object identifier. (Its policy use lives in M4.)

Run 
/usr/lib/vmware/osfs/bin/objtool getattr -u <OBJECT-UUID>

Replace: <OBJECT-UUID> with the object's UUID. Flags that matter: -c <COMPONENT-UUID> — reverse lookup: given a component, identify the object it belongs to (how you name an object from its active components). --bypassDOM — skip DOM entirely and read the object straight from disk; the move when DOM's own metadata is what's broken.

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. The same getattr capture that fills M4.

Object Path: /vmfs/volumes/vsan:528bd398cf7f9ffb-40c234e1ea584113/fff62769-2a45-0551-498e-5847ca78dc88/VCSA_14.vmdk
        Disk UUID: N/A
        Group UUID: fff62769-2a45-0551-498e-5847ca78dc88
        Version: 24
        Object type: NATIVE_SNAPSHOT
        Object size: 5368709120
        Object policy:
            spbmProfileId = 0212ed07-2dca-45d4-8262-c5cab4c30ac1
            spbmProfileName = vSAN-ESA - Optimal Datastore Default Policy - AutoRAID
            autoManagedRAID = i1
            autoRAIDLevel = i5
            objectVersion = i24
            CSN = l2819
            SCSN = l2817

Look for

  • The object's own name/path/policy as it carries them, independent of what DOM or vCenter believes.

O5 — Is resync moving?

Is resync moving, how much, how long.

Run 
esxcli vsan debug resync summary get

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. Idle.

   Total Number Of Resyncing Objects: 0
   Total Bytes Left To Resync: 0
   Total GB Left To Resync: 0.00

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. Mid-reconfiguration, two samples from one deliberate policy change, in order. Sample 1.

   Total Number Of Resyncing Objects: 19
   Total Bytes Left To Resync: 205372403356
   Total GB Left To Resync: 191.27

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. Mid-reconfiguration, sample 2 of the same policy change.

   Total Number Of Resyncing Objects: 17
   Total Bytes Left To Resync: 240023112176
   Total GB Left To Resync: 223.54
Run 
/etc/init.d/clomd status

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-23

clomd is running

O5 usually points at a disk — but a stalled resync can stall when P is green. A clean P does not clear O5; when P reads clean and resync still does not move, the stall lives elsewhere — take the N9 timeline (see N9), then the handoff.

Look for

Interpretation
Healthy resync progressing; remaining work shrinking across samples. During a planned reconfiguration the totals can RISE first: objects enter the work queue faster than the earliest finish, so bytes-left grows while the queue loads. Judge the trend after the peak — object count draining and bytes falling across later samples.
Suspicious resync that isn't progressing — judge it from summary get ACROSS TIME, not once. A single reading proves nothing either way; the same objects at the same bytes-left, sample after sample with no planned change loading the queue, is stuck.
Suspicious the resync summary is frozen AND clomd is not running -> the placement brain is off. Nothing will move until it is; the stall is a daemon, not a disk.
Stop-escalate summary says stuck -> O6 names the per-object detail, then work the layer the stuck object points at (see the fence).

Verify P yourself even if storage was already poked: a reported-clean P is not a checked P. Run the P reads before moving to N.

If resync is confirmed not moving, stop digging in this layer — a stuck resync is almost never an object-layer problem. Underneath it there is usually a device in trouble: medium errors (sense key 0x11) or checksum failures on a disk in the path. Go back to P. P6 reads what the device itself reports; P4 shows the state of the disks. The resync is the symptom — find the physical cause first.

Evacuation stalled with little or no resync activity? Before blaming the data path, check what is still powered on on this host. The usual pins are vCLS VMs and anti-affinity rules holding a VM in place — maintenance mode cannot finish while something refuses to leave.

When the maintenance itself runs, it opens with the evacuation precheck (esxcli vsan debug evacuation precheck) — the read that says what would become inaccessible and how much data would move. That read belongs to the work, not to diagnosis, so it is cited here, not sectioned.

STOP — boundary of the sheet.

A resync that isn't moving is not a waiting game — stuck almost always has an address in an earlier layer: a disk going bad, a link dropping, a host struggling. Take the stuck object from O6, read its components in O3, and go work that layer. Do your homework before you assume it finishes eventually.

If seeing this issue, see this KB:

O6 — Which object's resync is stuck?

Per-object resync detail — the stuck-resync investigation.

Run 
esxcli vsan debug resync list

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 (trimmed). Same reconfiguration as O5's active samples. Two objects in full, the rest elided. Every row Intent: Compliance.

Group UUID                            Object UUID                           Component UUID                        To Host           GB Left To Resync  Intent    
fff62769-2a45-0551-498e-5847ca78dc88  32f72769-00b2-a603-805a-5847ca78dc88  6f90da6a-e254-a7cb-364f-5847ca78dc88  esxi01.lab.local  0.02 GB            Compliance
fff62769-2a45-0551-498e-5847ca78dc88  32f72769-00b2-a603-805a-5847ca78dc88  6f90da6a-838b-aacb-4c83-5847ca78dc88  esxi03.lab.local  8.85 GB            Compliance
fff62769-2a45-0551-498e-5847ca78dc88  32f72769-00b2-a603-805a-5847ca78dc88  6f90da6a-6ce9-abcb-9623-5847ca78dc88  esxi03.lab.local  8.85 GB            Compliance
fff62769-2a45-0551-498e-5847ca78dc88  32f72769-00b2-a603-805a-5847ca78dc88  6f90da6a-83ea-accb-8d6f-5847ca78dc88  esxi02.lab.local  0.00 GB            Compliance
fff62769-2a45-0551-498e-5847ca78dc88  32f72769-00b2-a603-805a-5847ca78dc88  6f90da6a-5c21-aecb-3f9b-5847ca78dc88  esxi02.lab.local  0.00 GB            Compliance
fff62769-2a45-0551-498e-5847ca78dc88  32f72769-00b2-a603-805a-5847ca78dc88  6f90da6a-f72a-afcb-2761-5847ca78dc88  esxi01.lab.local  0.00 GB            Compliance
fff62769-2a45-0551-498e-5847ca78dc88  32f72769-00b2-a603-805a-5847ca78dc88  6f90da6a-da5d-b0cb-824b-5847ca78dc88  esxi01.lab.local  0.00 GB            Compliance
* (all 7 components) — 17.72 GB
Group UUID                            Object UUID                           Component UUID                        To Host           GB Left To Resync  Intent    
fff62769-2a45-0551-498e-5847ca78dc88  2bf72769-b60e-ad87-f90e-5847ca78dc88  6e90da6a-fc20-923f-f307-5847ca78dc88  esxi03.lab.local  0.05 GB            Compliance
fff62769-2a45-0551-498e-5847ca78dc88  2bf72769-b60e-ad87-f90e-5847ca78dc88  6e90da6a-dbe7-973f-43ed-5847ca78dc88  esxi02.lab.local  13.19 GB           Compliance
fff62769-2a45-0551-498e-5847ca78dc88  2bf72769-b60e-ad87-f90e-5847ca78dc88  6e90da6a-d318-9b3f-b8fa-5847ca78dc88  esxi02.lab.local  13.20 GB           Compliance
fff62769-2a45-0551-498e-5847ca78dc88  2bf72769-b60e-ad87-f90e-5847ca78dc88  6e90da6a-d0cf-9d3f-b056-5847ca78dc88  esxi03.lab.local  13.21 GB           Compliance
fff62769-2a45-0551-498e-5847ca78dc88  2bf72769-b60e-ad87-f90e-5847ca78dc88  6e90da6a-8e30-a03f-1f73-5847ca78dc88  esxi03.lab.local  13.20 GB           Compliance
fff62769-2a45-0551-498e-5847ca78dc88  2bf72769-b60e-ad87-f90e-5847ca78dc88  6e90da6a-05a7-ce3f-b0fd-5847ca78dc88  esxi01.lab.local  13.21 GB           Compliance
fff62769-2a45-0551-498e-5847ca78dc88  2bf72769-b60e-ad87-f90e-5847ca78dc88  6e90da6a-0104-d43f-c86c-5847ca78dc88  esxi01.lab.local  13.20 GB           Compliance
* (all 7 components) — 79.27 GB
(17 more objects elided — same shape)
* (all 19 objects)  * (all 113 components) — 208.80 GB

The first object is the object from O3's readout, mid-move.

Look for

  • The specific object(s) not advancing, then take that UUID back to O3 for its component states.

Naming the stuck object is not the finish — the reason it will not move usually lives in P (see O5).

O6 names one host; test that peer in N after a clean P. One-host To Host means one peer to prove — take that peer, not the whole cluster, into N5/N9.

O8 — Drill-down: the 9.1+ ESA global dedup corner

9.1+ ESA ONLY. Enabling global dedup (cluster-wide toggle) creates ONE dedup store for the cluster — a real backend object with its own UUID, holding chunk objects (deduplicated data) and hash objects (fingerprint index). All read-only:

Run 
esxcli vsan dedup dedupstore get

The store view; the ONLY place its UUID shows (the Virtual Objects UI hides it — documented 9.1 Known Issue).

Run 
esxcli vsan dedup dedupstore distribution get

-c chunk / -x hash.

READ THIS — SAMPLE OUTPUT Read 1 — feature off. Captured 2026-10-21, dedup-off cluster. Both dedupstore get and distribution get answer this line. Empty output is a reading.

Global dedup is not enabled on current host.

READ THIS — SAMPLE OUTPUT Read 2 — Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 (trimmed). Dedup on. The two object lists are trimmed to their totals.

Dedup Store Information
   Dedup Service Enabled: true
   Dedup Service Paused: false
   Default Dedup Store Uuid: a2f8e762-ce67-11f1-8353-005056a62fca
   Default Number Of Chunk Objects: 32
   Default Number Of Hash Objects: 32
   Dedup Store View List:
      Dedup Store View
         Dedup Store Uuid: a2f8e762-ce67-11f1-8353-005056a62fca
         Dedup Store Manager Object Uuid: 8a8dda6a-702c-3ae8-eeb0-5847ca78e6b0
         Default Dedup Store: true
         Number Of Chunk Objects: 32
         Number Of Hash Objects: 32
         Chunk Object List: 8a8dda6a-7a1a-7cf1-6453-5847ca78e6b0, 8a8dda6a-7506-d409-2ca6-5847ca78e6b0, 8b8dda6a-f1c3-5e21-4b66-5847ca78e6b0, ... (32 total)
         Hash Object List: 8e8dda6a-8a41-d2d4-1471-5847ca78e6b0, 8e8dda6a-461d-79eb-bc40-5847ca78e6b0, 8e8dda6a-9863-9d01-c98c-5847ca78e6b0, ... (32 total)

READ THIS — SAMPLE OUTPUT Read 3 — Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 (trimmed). Object lists trimmed to the first UUID and the count. The same dedup store UUID on every row is the reading.

Host Name         Host UUID                             Dedup Store UUID                      Object Type  Object Count  Object List                                         
esxi01.lab.local  6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046  a2f8e762-ce67-11f1-8353-005056a62fca  chunk        10            8b8dda6a-f1c3-5e21-4b66-5847ca78e6b0, ... (10 total)
esxi01.lab.local  6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046  a2f8e762-ce67-11f1-8353-005056a62fca  hash         10            8e8dda6a-9863-9d01-c98c-5847ca78e6b0, ... (10 total)
esxi02.lab.local  3d8b7a19-5c2e-4f60-b4a9-1e6d8c3f5a72  a2f8e762-ce67-11f1-8353-005056a62fca  chunk        11            8a8dda6a-7506-d409-2ca6-5847ca78e6b0, ... (11 total)
esxi02.lab.local  3d8b7a19-5c2e-4f60-b4a9-1e6d8c3f5a72  a2f8e762-ce67-11f1-8353-005056a62fca  hash         11            8e8dda6a-461d-79eb-bc40-5847ca78e6b0, ... (11 total)
esxi03.lab.local  9a5e1d73-8b4c-4a2f-a6d1-4f7b2e9a8d05  a2f8e762-ce67-11f1-8353-005056a62fca  chunk        11            8a8dda6a-7a1a-7cf1-6453-5847ca78e6b0, ... (11 total)
esxi03.lab.local  9a5e1d73-8b4c-4a2f-a6d1-4f7b2e9a8d05  a2f8e762-ce67-11f1-8353-005056a62fca  hash         11            8e8dda6a-8a41-d2d4-1471-5847ca78e6b0, ... (11 total)
Run 
esxcli vsan dedup dedupstore resync list
Run 
esxcli vsan dedup dedupstore resync summary get

READ THIS — SAMPLE OUTPUT Read 4 — Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. Captured during the O5/O6 reconfiguration: the cluster resyncing 200+ GB of VM objects while the dedup pair sat at zero. The zeros and the empty list are the whole reading.

Summary Of Resyncing Dedup Objects
   Total Number Of Dedup Objects: 64
   Total Number Of Resyncing Dedup Objects: 0
   Total Number Of Resyncing Chunk Objects: 0
   Total Number Of Resyncing Hash Objects: 0
   Total Bytes Left To Sync: 0
   Total GB Left To Sync: 0

dedupstore resync list returned NO output at all.

Run 
esxcli vsan debug object list --dedup-obj-type

chunk / hash / dmo / all — the object-level drill-down into those store objects; they're excluded from --all by design.

READ THIS — SAMPLE OUTPUT Read 5 — Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 (trimmed). Captured via --dedup-obj-type all. RAID_1 leg and the first RAID_5 stripe described; the rest elided.

Object UUID: 8e8dda6a-9863-9d01-c98c-5847ca78e6b0
   Version: 24
   Health: healthy
   Owner: esxi01.lab.local
   Size: 0.00 GB
   Used: 0.00 GB
   Effective Used: 0.00 GB
   Snapshot Usage: 0.00 GB
   Snapshot Count: 0
   Used 4K Blocks: N/A
   Policy:
      CSN: 4
      zdomObject: 3
      dedupObjID: 3
      autoRAIDLevel: 5
   Configuration: Concatenation — RAID_1 (2 components,
   ACTIVE, on esxi03.lab.local and esxi01.lab.local) +
   RAID_5 (3 RAID_0 stripes; first stripe: 2 components,
   ACTIVE, on esxi03.lab.local; two stripes elided —
   same shape; 8 components total, all ACTIVE)
   Type: Deduplication hash
   Path: N/A
   Group UUID: N/A
   Directory Name: N/A

This object's UUID is the first entry in esxi01's hash list in Read 3 — the drill-down path, end to end: store -> distribution -> object. Chunk objects read the same way: Type: Deduplication chunk, with real capacity figures in place of the hash object's zeros.

Look for

  • Empty output is a READING, not an error: no store exists, dedup is off; and a quiet dedup resync (Read 4) during a busy cluster resync means the store simply isn't part of that work. (Verified empirically on Daniel's 9.1.1 lab in both states, Oct 21–22.) An empty dedup output is not a failed command.
Deep End — cmmds-tool. Know it exists; don't live there.

Deep End — cmmds-tool

Advanced, last-resort tooling; Daniel rarely touches it outside very advanced troubleshooting. Know it exists; don't live there. Two doors only:

Run 
cmmds-tool find -t DOM_OBJECT -u <OBJECT-UUID>

An object's raw CMMDS entry and owner.

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 (trimmed). Trimmed to the owner line and the content opening. Host UUIDs render per the committed file's fabrications, on this owner line as on the HOSTNAME read.

owner=9a5e1d73-8b4c-4a2f-a6d1-4f7b2e9a8d05(Health: Healthy) uuid=44f72769-b1be-0bfd-e57d-5847ca78dc88 type=DOM_OBJECT rev=1237 minHostVer=3
  content: (("Configuration" (("UUID" "44f72769-b1be-0bfd-e57d-5847ca78dc88") ... (trimmed — the entry carries the object's full configuration, CSN, and component/disk map)

This object is the M4/O4 getattr object — the same object, read at three depths.

Run 
cmmds-tool find -t HOSTNAME

Membership entries.

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22. One membership entry shown.

owner=6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046(Health: Healthy) uuid=6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046 type=HOSTNAME rev=...
  content: ("esxi01.lab.local")

Signature — checksum errors

"by the time checksum surfaces the P is usually long past." A checksum mismatch surfaces on an OBJECT (read fails verification, vSAN repairs from a good replica) but the culprit is the component's disk — dying media that has usually been dying for a while. The trail, per Broadcom KB 405199, "vSAN detected and fixed a medium or checksum error": the event names the COMPONENT; the host's kernel log names the DEVICE with a Medium Error. Trail: checksum errors in vmkernel -> the object's components via O3 (debug object list) -> that disk in P (Checksum / Checksum OK fields in esxcli vsan storage list). Same cross-layer shape as N's membership churn.

Read this (O logs)

  • clomd.log — CLOM's own log: placement decisions, repair/resync reasoning, why an object couldn't be placed or rebuilt. Plus hostd.log, and vmkernel.log / vobd.log — "they're always important," in every layer.

Healthy-at-a-Glance

  • health summary get: object count in healthy, problem categories at 0
  • object list: components ACTIVE; no object trapped behind the 100-object cap in your search
  • resync summary get: progressing across samples
  • epd status: running (ESA)
  • clomd status: running

Found it? Capture: the check ID, the reading, and the timestamp for the case notes.

Hands off to: P — checksum errors are born on a disk: the event names the component, the kernel log names the device with a Medium Error, and both trails end at that disk in P (Broadcom KB 405199, "vSAN detected and fixed a medium or checksum error"). N — objects don't go absent on healthy networks; membership and partitions live at N1 and N9. M — placement failures with healthy objects are policy and limits (M1, M3).

M — Management

↑ Back to top

Management is when things get weird.

Verified-against date: 2026-10-21.

The task plane: vCenter-driven operations landing on hosts, and the host-side readings you take when vCenter is down or the task itself is the problem.

What lives here. vSANVPD, OSFSD, DISKLIB, SPBM — the host libraries and daemons invoked when a vSAN task lands.

Key distinction. An SPBM profile is only the friendly NAME of a policy. The object's ACTUAL policy is the raw capability spec it carries — and that spec can be READ from the host command line (Objtool, check M4). Changing it is supervised, post-diagnosis work and is NOT on this sheet.

The non-default trap. Non-default does not mean wrong — platform personalities set several advanced options by design. The readings start questions; they don't answer them.

Start here. M1, M2, M3.

M1 — Am I out of room to place things?

Check the cluster's placement headroom: component limits and free disk space.

Run 
esxcli vsan debug limit get

READ THIS — SAMPLE OUTPUT SAMPLE, small test cluster

Component Limit Health: green
Max Components: 562
Free Components: 561
Disk Free Space Health: green
Lowest Free Disk Space: 99 %

Look for

Interpretation
Healthy both health lines green; Free Components comfortably above zero; lowest free disk space healthy.
Suspicious Free Components near zero, or one disk far fuller than the rest -> objects can't place or rebalance; the problem is capacity maths, not a fault. Stay in M for the numbers, then take the placement symptom to O.
Stop-escalate limits are green and placement still fails -> go to O (object view) with the object's UUID.

M2 — Did somebody change a setting?

List every advanced vSAN option carrying a non-default value, in one shot.

Run 
esxcli vsan debug advcfg list

READ THIS — SAMPLE OUTPUT Daniel's lab, real output

VSAN.ClomBgProRebalanceEnabled        1  (default 0)
VSAN.GuestUnmap                        1  (default 0)
VSAN.Vsan2DeltaComponent               1  (default 0)
VSAN.Vsan2DeltaComponentUnplanned      1  (default 0)
VSAN.Vsan2DomMetadataEncryption        1  (default 0)
VSAN.DOMNetworkSchedulerThrottleComponent 1 (default 0)
VSAN.VsanPersonalityConfigured         2  (default 0)
(several of these are set by the platform personality —
expected on ESA, not a finding by themselves)

The two names worth grepping by name:

Run 
esxcli vsan debug advcfg list | grep -iE 'IgnoreClusterMemberListUpdates|DomPauseAllCCPs'

READ THIS — SAMPLE OUTPUT The lab's own reading, captured 2026-10-22. IgnoreClusterMemberListUpdates at 1 is this host's armed state. DomPauseAllCCPs does not appear in the output; on this host it is not in the debug list, so it sits at default. One row is the whole reading.

VSAN.IgnoreClusterMemberListUpdates 1 0 false

Look for

Interpretation
Healthy neither name appears (both at default 0), or IgnoreClusterMemberListUpdates is explained by an in-progress recovery procedure you already know about.
Suspicious IgnoreClusterMemberListUpdates = 1 with no live procedure -> the host is ignoring membership updates; expect phantom-partition symptoms (see N). DomPauseAllCCPs = 1 -> ALL component reconfigurations are paused; objects will silently not repair or rebuild (see O). Both are set-for-a-procedure knobs whose failure mode is being left on. Compare against the N4 unicast table: a unicast row is not the same as the same UUID / Member Hosts — a peer can appear in the table yet be omitted from Member Hosts. Check all hosts for the leftover pattern: a non-default knob on one host, or set on some hosts but not others, is the pattern — a single-host read is not the check.
Stop-escalate either is set and nobody owns why.

STOP — boundary of the sheet.

A non-default value for IgnoreClusterMemberListUpdates or DomPauseAllCCPs starts questions — it is never permission to change it from a cheat sheet. These knobs get turned on deliberately during supervised work, and turning them back off is supervised work too.

If seeing this issue, see this KB:

M3 — Are the management services starved?

vSAN's memory footprint on the host (userworld + kernel). Starved management services make tasks go weird.

Run 
esxcli vsan debug memory list

READ THIS — SAMPLE OUTPUT SAMPLE

Kernel      Usage 106.82 MB
User World  Usage 270.11 MB
Total       Usage 376.93 MB  (Max 3442.36 MB)
Run 
/etc/init.d/vsanmgmtd status

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-23. The status answers under the vsanperfsvc name — that IS vsanmgmtd reporting; a missing "vsanmgmtd is running" line is not a failure.

vsanperfsvc is running

Look for

Interpretation
Healthy total usage well below max.
Healthy vsanmgmtd status answers (as vsanperfsvc): running — the host's vSAN management daemon is staffed; perf and management views have someone home to serve them.
Suspicious usage riding at/near max alongside failing or hanging tasks -> resource pressure on the management plane; correlate with hostd/vpxa (see A).

If seeing this issue, see this KB:

Deeper checks — M4, Read this (M logs)

M4 — What is this object's REAL policy?

Read an object's actual policy spec — not the profile name vCenter shows.

Run 
/usr/lib/vmware/osfs/bin/objtool getattr -u <OBJECT-UUID> | grep -i policy

Replace: <OBJECT-UUID> with the object's UUID (from O's debug object list).

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-22 — the policy block of the O4 getattr capture.

Object policy:
    spbmProfileId = 0212ed07-2dca-45d4-8262-c5cab4c30ac1
    spbmProfileName = vSAN-ESA - Optimal Datastore Default Policy - AutoRAID
    autoManagedRAID = i1
    autoRAIDLevel = i5
    objectVersion = i24
    CSN = l2819
    SCSN = l2817

Look for

  • The capability values themselves (failures to tolerate, striping, etc.) — this is the ground truth the object was built with, whatever the profile is called today.
  • Objtool's identification uses (damaged objects, the -c reverse lookup, --bypassDOM) live in O.

Read this (M logs)

  • vsanmgmt.log (the vSAN management service)
  • hostd.log
  • vpxa.log

Healthy-at-a-Glance

  • limit get: health green, free components > 0
  • advcfg list: non-defaults all explained (personality or known change); the two landmine names at default
  • memory list: usage well below max
  • vsanmgmtd status: running (reports as vsanperfsvc)

Found it? Capture: the check ID, the reading, and the timestamp for the case notes.

Layer clear? -> The fault is above or below: object symptoms go DOWN to O; client/task symptoms go UP to A.

Hands off to: O — policy and compliance land as object health. P and N — when management disagrees with the hosts, the host layers are the truth; start again at P1.

A — Application

↑ Back to top

What the client shows vs what the hosts know.

Captured on esxi01.lab.local and the VCSA (ESXi 9.1.1 / vCenter 9.1.1, vSAN ESA), 2026-10-23 — identifiers pseudonymized per the sheet's disclosure; readings as captured. Samples marked (trimmed) are shortened under the sheet's trimming rule; the elision is labelled inside the sample.

What lives here. VPXD (vCenter Server, incl. the vSAN Health service), VPXA (the vCenter agent on each host), HOSTD (the host management service). Every user-initiated vSAN task — VM/VMDK create/delete, policy create/modify, feature enablement, health monitoring, disk-group and fault-domain creation — flows: web client or RVC -> vCenter API (VPXD) -> VPXA on the host -> HOSTD -> the M-layer libraries that do the work.

  • VPXD — vCenter Server and its vSAN Health service.
  • VPXA — the vCenter agent on each host.
  • HOSTD — the host management service.

RVC: an A-layer tool, cited not sectioned: Broadcom KB 326527, "Using Ruby vSphere Console (RVC) for vSAN".

Start here. A1, A2, A3 — the three status reads, one per hop.

A1 — Is HOSTD alive?

HOSTD status only. No restarts on this sheet.

Run 
/etc/init.d/hostd status

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-23

hostd is running.

Look for

Interpretation
Healthy hostd reports running.
Suspicious hostd not running, or esxcli itself stops answering — that symptom points back at this layer; try A4's bypass to prove it.
Stop-escalate hostd down and A4's localcli bypass also silent -> the fault is below A, on the host itself. Stop working the management chain and go to P.

A2 — Is VPXA alive?

VPXA status only. No restarts on this sheet.

Run 
/etc/init.d/vpxa status

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-23

vpxa is running

Look for

Interpretation
Healthy vpxa reports running — the vCenter-to-host hop is staffed.
Suspicious vpxa down while hostd is up -> the host may be fine and vCenter's view of it stale; that is an A diagnosis, not a host fault.
Stop-escalate vpxa down and the host-side reads from P/N/O stay healthy -> believe the host reads. The chain is broken at this hop; nothing the client shows about this host is current.

A3 — Are vCenter's own services up?

vCenter service status, including the vSAN Health service, on the VCSA.

Run 
service-control --status

Replace: run on the VCSA, not on a host.

READ THIS — SAMPLE OUTPUT Captured on the VCSA (vCenter 9.1.1), 2026-10-23 (trimmed). The full Running list holds 40 services; the sample keeps the decisive names. The Stopped names shown are the expected-off set (no VCHA configured, no image builder, no netdumper) — stopped is their healthy state.

Running:
… vmware-vmon vmware-vpostgres vmware-vpxd
vmware-vpxd-svcs vmware-vsan-health vmware-vsm
vsphere-ui vstats wcp (trimmed — 40 services running in the full read)
Stopped:
vc-salt vmcam vmware-imagebuilder vmware-netdumper
vmware-rbd-watchdog vmware-vcha (trimmed)

Look for

Interpretation
Healthy VPXD and the vSAN Health service running.
Suspicious Health service down -> health data that never refreshed is a service problem, not a cluster finding.
Stop-escalate vmware-vpxd itself stopped -> the client has nothing true to show. Every UI reading is void until the chain is staffed again; work from host commands only.

A4 — The hostd bypass

localcli runs the same vSAN namespaces without going through HOSTD. When esxcli stops answering, that symptom itself points back at A — the bypass both proves the hop and keeps diagnosis moving.

Run 
localcli vsan cluster get

Replace: localcli takes the same vsan namespaces as esxcli — swap in the read you would have run.

READ THIS — SAMPLE OUTPUT Captured on esxi01.lab.local (ESXi 9.1.1, vSAN ESA), 2026-10-23 (trimmed). The same reading P1 takes through esxcli — taken here through localcli, with HOSTD out of the path.

Cluster Information:
  Enabled: true
  Local Node UUID: 6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046
  Local Node Type: NORMAL
  Local Node State: MASTER
  Local Node Health State: HEALTHY
  Sub-Cluster Master UUID: 6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046
  Sub-Cluster Backup UUID: 3d8b7a19-5c2e-4f60-b4a9-1e6d8c3f5a72
  Sub-Cluster UUID: 528bd398-cf7f-9ffb-40c2-34e1ea584113
  Sub-Cluster Membership Entry Revision: 1
  Sub-Cluster Member Count: 3
  Sub-Cluster Member UUIDs: 6f4a2c8e-91bd-4e57-a3c0-7d5b9e1f2046, 3d8b7a19-5c2e-4f60-b4a9-1e6d8c3f5a72, 9a5e1d73-8b4c-4a2f-a6d1-4f7b2e9a8d05
  Sub-Cluster Member HostNames: esxi01.lab.local, esxi02.lab.local, esxi03.lab.local
  Unicast Mode Enabled: true
  Maintenance Mode State: OFF
  Mode: REGULAR
  vSAN ESA Enabled: true
(trimmed)

Look for

Interpretation
Healthy localcli answers with the cluster picture — the bypass works and the vSAN readings underneath remain trustworthy.
Suspicious localcli answers while esxcli does not -> HOSTD is the lying hop; the vSAN readings themselves remain trustworthy.
Stop-escalate localcli also fails -> this is not an A problem. The host's own vSAN stack is not answering locally; go down to P and work the host.

Read this (A logs)

  • vpxd.log on vCenter; vpxa.log and hostd.log on the host.

Look for (the layer's signature)

UI vs reality divergence. The client shows what VPXD believes. When that disagrees with the host-side truth from P/N/O/M, believe the host commands and diagnose the chain: stale inventory, a task that died at one hop, or health data that never refreshed.

Healthy-at-a-Glance

  • hostd status: running
  • vpxa status: running
  • service-control --status: VPXD + vSAN Health running
  • UI and host commands agree

Found it? Capture: the check ID, the reading, and the timestamp for the case notes.

Layer clear? -> If all three hops are staffed and the UI still disagrees, the fault is below: take the host-side truth from P/N/O/M as the record and work that layer.

Hands-off: nothing in this layer gets restarted from this sheet. A dead hop is the diagnosis — restarting hostd, vpxa, or a vCenter service is a remediation decision, made off-sheet, after the chain is proven.