VMware VM Power-On Failure After SAN Recovery and Replication Restoration

Overview

This document describes a VMware issue encountered after an unexpected power outage where several virtual machines (VMs) failed to power on even though the SAN storage and datastore replication had recovered successfully.

The issue was caused by stale VM storage metadata remaining after storage recovery, preventing ESXi from correctly recreating the VM swap (.vswp) file.

Environment

• VMware ESXi
• VMware vCenter Server
• SAN Storage with datastore replication
• Virtual machines using both replicated and non-replicated datastores

Scenario

An unexpected power outage caused all ESXi hosts and SAN storage to shut down abruptly.

After electricity was restored:

• SAN storage started successfully.
• Datastore replication completed successfully.
• Replicated datastores became accessible.
• Most VMs started normally.
• However, several VMs still failed to power on.

Symptoms

Affected VMs exhibited one or more of the following symptoms:

• VM cannot power on.
• Storage Migration (Storage vMotion) is disabled (greyed out).
• Snapshot consolidation is required.
• Power-on fails while creating the VM swap (.vswp) file.
• The VM configuration (.vmx) still contains a sched.swap.derivedName entry referencing the previous swap file location.

Example:

sched.swap.derivedName="/vmfs/volumes/Datastore-Replication/VM01/VM01-123456.vswp"

Root Cause

The unexpected power outage interrupted normal storage operations and caused an unclean shutdown of both ESXi hosts and the SAN storage.

Although SAN replication completed successfully after power was restored and the replicated datastore became available again, some VMs retained stale storage metadata in their configuration.

During VM power-on, ESXi attempted to recreate the VM swap (.vswp) file using the information stored in the VM configuration (sched.swap.derivedName).

Because the VM configuration had not been fully refreshed after storage recovery, ESXi could not successfully recreate the swap file, causing the VM power-on process to fail.

Refreshing the VM configuration rebuilt the storage metadata and allowed ESXi to generate a new swap file successfully.

Solution 1 (Recommended Recovery Procedure)

Step 1 – Remove Snapshots

Remove all VM snapshots.

Step 2 – Consolidate Virtual Disks

If vCenter reports:

Virtual machine disks consolidation is needed

perform disk consolidation before continuing.

Step 3 – Remove and Reattach Existing Virtual Disks

Important

Do NOT delete the VMDK files.

Only remove the virtual disk references from the VM configuration.

Then:

• Add Existing Disk
• Reattach all original VMDKs

This refreshes the VM configuration.

Step 4 – Re-enable Storage Migration

If Migrate is disabled (greyed out), it may be caused by an orphaned entry in the vCenter database (vpx_disabled_methods).

Procedure

1. Identify the VM Managed Object ID (MOID).

Example:

vm-17145

2. Open VMware MOB:

https://<vcenter>/mob/?moid=AuthorizationManager&method=enableMethods

3. Log in as a vCenter Administrator.

4. Enter the VM Managed Object Reference:

<entity type="ManagedEntity"
xsi:type="ManagedObjectReference">
vm-17145
</entity>

5. Enable the following method:

<method>RelocateVM_Task</method>

6. Click Invoke Method.

7. Refresh the vCenter Client.

Storage Migration should now be available.

Step 5 – Perform Storage vMotion

Perform a Storage Migration to the same datastore.

Although the destination datastore does not change, Storage vMotion refreshes:

• VM configuration
• Storage metadata
• Virtual disk references
• Swap file configuration

Step 6 – Power On the VM

The first startup may take longer than usual.

During startup, ESXi automatically creates a new VM swap (.vswp) file.

The VM should now power on successfully.

Solution 2 (Observed Emergency Workaround)

Warning

This workaround has been successfully reproduced in our environment but is not documented or officially supported by VMware.

Use this method only if the recommended recovery procedure cannot be performed immediately.

Observation

Some affected VMs retained a stale sched.swap.derivedName entry inside the .vmx configuration file.

Example:

sched.swap.derivedName="/vmfs/volumes/Datastore-Replication/VM01/VM01-123456.vswp"

The VM consistently failed to power on until a file matching the expected swap filename existed in the referenced directory.

Workaround

1. Open the VM's .vmx file.
2. Locate the value of sched.swap.derivedName.
3. Create an empty file using the exact filename specified.

Example:

VM01-123456.vswp

4. Upload the empty file to the directory referenced by sched.swap.derivedName.
5. Power on the VM.

Observed Result

During power-on:

• ESXi detected the placeholder file.
• ESXi automatically generated a new valid .vswp file.
• The placeholder file was replaced.
• The VM powered on successfully.

This behavior was successfully reproduced multiple times in our environment.

The exact internal VMware mechanism has not been confirmed.

Result

After completing the recommended recovery procedure:

• VM powers on successfully.
• Storage Migration is available again.
• ESXi automatically recreates the VM swap (.vswp) file.
• VM operates normally.

Lessons Learned

• Protect VMware infrastructure with UPS systems to avoid unclean shutdowns.
• Wait until SAN storage and datastore replication have fully recovered before starting production VMs.
• Verify datastore accessibility after storage recovery.
• Remove unnecessary snapshots.
• Consolidate virtual disks before performing Storage vMotion.
• Refresh VM storage metadata if VMs fail to power on after storage recovery.
• If sched.swap.derivedName appears to reference stale swap metadata, the documented workaround may be useful as an emergency recovery method, but it should not replace the recommended recovery procedure.

References

• VMware ESXi Documentation
• VMware vCenter Server Documentation
• VMware Storage vMotion Documentation
• VMware Virtual Machine Swap Files (.vswp)
• VMware Managed Object Browser (MOB)
• VMware Snapshot Consolidation