Troubleshooting and Resolution of Ceph Disk WWN Conflicts Caused by Cross-Host Cloning

Background and Problem Symptoms

When deploying virtual machines in batches based on OVF templates or by directly copying folders across ESXi hosts, we encountered a classic low-level storage identifier conflict issue.

Phenomenon description: When using lsblk When using the command to check node disk information, it was found that the newly deployed node (such as 192.168.8.235,.236,.237) on, all 16GB data disks'WWN (World Wide Name) completely identical.

# 192.168.8.235 node output example
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb      8:16   0   16G  0 disk
├─sdb1   8:17   0  100M  0 part
...

You can see the conflict more intuitively by checking the disk WWN:

# View disk WWN information
for disk in /dev/sd{b,c,d,e,f}; do
    echo -n "$disk: "
    /lib/udev/scsi_id --whitelisted --device=$disk
done

In the output results, the corresponding disk WWNs on the three machines are exactly the same, for example, all are wwn-0x6000c2964f24f32d82a275484614e171.

Compared with a normal cluster: As a comparison, the previously normally deployed 192.168.3.x In a cluster, the WWN of each disk on each machine is unique.

Root cause: Because the virtual machine is Cross-host deployment(copy directly .vmdk files or cloned without processing), the UUID and WWN underlying the virtual disk are carried over unchanged to the new node. When distributed storage systems such as Ceph scan disks, they cannot distinguish which node these disks with identical hardware identifiers belong to, directly causing cluster initialization to fail.

Solution approach

Core principle: Do not attempt to modify WWN/Serial from within the Linux operating system. These identifiers are passed directly to the operating system by the virtualization platform (ESXi) through a virtual SCSI controller. The correct approach is Force regenerate in the vSphere UI.

VMware's virtual disk WWN is bound to .vmdk In the underlying configuration. As long as the disk is removed from the virtual machine configuration and then reattached to that virtual machine, VMware will regenerate a unique WWN and Serial.

Specific operation steps

Step 1: Regenerate the WWN by "removing and re-adding".

Targeting 192.168.8.235,.236,.237 The three machines perform the following operations one by one:

• Shut down the virtual machine: Ensure the virtual machine is completely powered off (not suspended).

• Record existing hard disk information: In "Edit Settings", note the size of each data disk (e.g., 16GB) and the current SCSI node (e.g., SCSI 0:1, 0:2, etc.).

• Remove the hard drive (an extremely critical step):

• Select the hard drive you want to clean and click "Remove".

• Warning: In the pop-up dialog box,Never check "Delete files from data storage"., select "Remove from virtual machine only".

• Remove all data disks one by one.

• Re-add the hard drive:

• Click “Add New Device” -> “Hard Disk”.

• Select "Existing Hard Disk".

• Browse the data store and find the one that was just removed..vmdk Files (such as xxx_1-000002.vmdk）。

• Select and confirm.

• Restore SCSI node: Ensure that after the hard disk is re-added, its SCSI node (e.g., SCSI 0:1) is exactly the same as the original.

• Save and power on: VMware automatically assigns them brand-new, unique WWNs when remounting these VMDKs.

Additional Tips

If, after repeatedly removing and re-adding in the UI, the WWN checked in the Linux system still remains unchanged, you can try to The entire SCSI controller Remove and re-add them together as well; this forces ESXi to re-enumerate the entire bus and generate a brand-new WWN.

Step 2: Verify WWN uniqueness

After restarting, log in to the operating system and check the disk WWN again:

# View the WWN of all data disks
for disk in /dev/sd{b,c,d,e,f}; do
    echo -n "$disk: "
    /lib/udev/scsi_id --whitelisted --device=$disk
done

Confirm two points:

• WWN unique: All data disk WWNs on the three machines must be unique, with no duplicates.

• The drive letter is correct: Confirm /dev/sdb to /dev/sdf the mapping sizes are consistent (all 16.0GiB), avoiding drive letter order confusion.

Step 3: Thoroughly clean up residual disk data

Because the cluster installation failed previously, the disks are very likely to have residual Ceph LVM structures, partition tables, or filesystem signatures.Cleanup must be performed on each machine., otherwise even if the WWN changes, reinstalling will still report an error.

Dangerous operation notice

Before executing the following commands, please be sure to use lsblk Confirm the drive letter (e.g./dev/sdb - /dev/sdf) It is indeed your raw data disk; never apply it to the system disk!

# Assume sdb through sdf are your 5 data disks.
for disk in /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf; do
    echo "Cleaning $disk ..."
    # Erase the file system
    wipefs -a $disk
    # Erase partition table
    sgdisk --zap-all $disk
    # Erase LVM or disk headers that Ceph may have left
    dd if=/dev/zero of=$disk bs=1M count=100 status=progress
done

After cleanup is complete, run again lsblk Confirm that all data disks have no partition or file system information and are in a clean raw disk state.

Experience Summary and Pitfall Avoidance Guide

• Cross-host cloning requires caution. Directly copying a virtual machine folder or cloning across hosts will cause the virtual hardware identifiers (MAC address, disk WWN, UUID) to be exactly the same.

• Correctly Use OVF Export and Import: If deployment must be done via template, it is recommended to use the OVF/OVA format and check the option related to "regenerate UUID" in the export/import wizard.

• Avoid mounting old VMDKs When deploying a new virtual machine, it is recommended to directly add a brand-new blank virtual disk rather than mounting an old virtual disk copied from another host..vmdk file, so that UUID/WWN conflicts can be avoided at the source.

• Must clean before installation: For any distributed storage cluster (such as Ceph), before reinstalling after a failure, be sure to conduct a thorough ... on the raw disk wipefs and sgdisk --zap-all Operation.

After following the above process, the hardware identifiers are unique and the disks are clean. Reissue the cluster installation command, and you can smoothly get past the disk identification stage.