Migration of VMs with RDM Datastores from vCenter/ESXi to OpenStack

[sharma-tapas]

Introduction

The migration of virtual machines (VMs) from vCenter/ESXi to OpenStack presents a significant challenge, especially when those VMs are backed by Raw Device Mapping (RDM) datastores. RDMs are commonly used in VMware environments to provide direct access to physical storage devices, which are either virtual RDMs (VMDKs mapped directly to LUNs) or physical RDMs (direct access to raw block devices on the storage). These setups have specific requirements when migrating to a cloud infrastructure like OpenStack, and careful consideration must be given to the migration strategy.

Key Considerations:

• Type of RDM Setup: Virtual vs Physical RDM

  • Virtual RDM: Mapped to a virtual disk (VMDK) that resides on an ESXi datastore but points to a LUN. These are relatively easier to migrate as they are already in a virtualized format.
  • Physical RDM: Provides direct access to a physical LUN, which can present a challenge when migrating, as OpenStack does not have direct equivalents for physical device mappings in a similar manner to VMware.

• Migration Strategies:

  • In-Place Migration: This involves moving the VM and the RDM disk directly to OpenStack without modifying the underlying disk structure. However, this may not always be possible, especially for physical RDMs, due to incompatibilities between VMware's RDM and OpenStack's storage backend.
  • Block Data Copy Migration: In this case, the data is copied from the RDM datastore into an OpenStack-compatible format. This is more complex but may be the only viable option for physical RDM setups. For virtual RDMs, a block data copy might still be used if required.

Considerations for OpenStack Compatibility:

• OpenStack uses various storage backends like Cinder for block storage and Swift for object storage, but there is no direct equivalent for VMware’s RDM technology. Thus, data from RDM-backed VMs needs to be either migrated into virtual disks (like VMDK to QCOW2/RAW formats) or connected to OpenStack’s storage systems in another way.
• Cinder Volumes: If RDMs are virtualized, you could potentially migrate the data into Cinder volumes. For physical RDMs, a custom approach might be needed to recreate the connection to the physical storage or adapt the data model for OpenStack.
• Do we need to write a Cinder driver that supports LVM, or data copy from one provider to another?

More to Discuss

• How should we approach the migration of VMs that are using physical RDMs? Can we adapt these setups into OpenStack’s architecture, or do we need to re-architect storage configurations?
• For virtual RDMs, is it more practical to perform an in-place migration, or would block data copy offer better results?
• How do existing migration tools handle RDM-backed VMs, and where are the gaps in their support for complex RDM setups
• What would be the impact of different migration strategies on VM performance, downtime, and data integrity?
• Should we focus on automating the migration process as much as possible, or is manual intervention needed for specific scenarios (e.g., physical RDM)?
• Is it feasible to create a hybrid storage solution that retains the benefits of RDMs while being compatible with OpenStack’s cloud-native storage solutions?

[roopakparikh]

Some thoughts on the questions

1. In place migration or a Copy: I suggest we give the choice to the user, I can imagine for most cases a copy may be acceptable, but for the large volumes or products which need direct SCSI access.
2. Other products: Looking at products like Forklift, it seems RDM is not handled, perhaps due to the rare use of that tye in VMware.
3. Manual vs Auomated: its not clear me what a manual process might look like. If there is a way to do that manually specifically given the rare occurence of RDM disks, it may be a viable alternative, but would like to understand what manul process may work.
4. Hybrid storage solution: to paraphrade, it apears to indicate that we may be able to use the Cinder drivers of the LUN provider, if we can find that it is possible, we should definitely proceed in that direction.

[bhavin192]

Disk types for RDM

VMware presents the adapters available on the host to be added as raw disks. These can be SATA adapters, Fibre Channel HBAs, or iSCSI endpoints. In the case of FC and iSCSI the LUNs are made available to the VMs.

References: Characteristics of Raw Device Mapping in vSphere, Introduction to Raw Device Mapping in vSphere, VMware vSphere VMFS

Importing RDM disks into OpenStack

Tough Cinder doesn't seem to have a generic FC/iSCSI driver, it does have vendor specific drivers for wide range of storage arrays. These drivers take care of creating the LUN and making it available to VMs when a volume is created in OpenStack.

Examples:

• Segate driver FC support
• HPE 3PAR

Cinder has zonemanager drivers which take care of setting up target-initiator zones created/updated when the volume is going to be attached to a host.
More details:

• Fibre Channel Zone Manager — cinder documentation
• GitHub: cinder/zonemanager/drivers

Then Nova is responsible for finding the respective LUN on the host and making it available to the VM, and configuring libvirt XML etc. It has support for multipath as well. Reference: connect_volume from os_brick/initiator/connectors/fibre_channel.py

Cinder manage

Cinder supports importing volumes from the storage system which are not managed by it. This cinder manage operation is supported by these FC drivers as well. So we should be able to import the RDM disk's LUN into Cinder and then use that as a volume for migrated VM.

Reference: ManageableVD class from cinder/volume/driver.py. Implementation in Segate driver: manage_existing.

User inputs?

Haven't thought through about this just yet. But once vJailbreak scans for VMs:

1. We figure out which are RDMs
2. Ask user which volume type (and in turn SAN array) this maps to in OpenStack (unless we have some way to do this)
3. Replicate and convert other disks, root volumes etc.
4. Turn off the VM
5. Cinder manage the RDM disk.
6. Create new instance on OpenStack with replicated volumes and imported RDM.

Limitations/requirements

• The source RDM storage SAN array must be available as volume type in destination OpenStack.
• During the cinder manage operation the volume is renamed, so in case of failure, does it need to be renamed back to what it was?
• VM needs to be turned off before the volume can be managed by Cinder. Depending on import operation, we might need more time to migrate a VM.
• Can only be done for the FC and iSCSI RDMs if the SAN vendor/appliance has a supported Cinder driver.
• OpenStack doesn't seem to have generic iSCSI driver. But I'm expecting most users to be using some vendor storage array having Cinder driver.

Open questions

1. How do you handle hardware silos?
   • This seems to be going to be the common case where new storage arrays are getting added. In this case we will have to attach the new storage and old storage to one of the systems be it VMware or OpenStack VMware, then do a data/block copy.
   • Once that is done, new volume can be imported into OpenStack using the Cinder manage command.
2. What about NPIV support, is it there in Nova/os_brick? It is indeed supported in libvirt https://wiki.libvirt.org/NPIV_in_libvirt.html
3. Will there be any performance impact for these migrated pRDM VMs? In case of VMware the SCSI commands are pass-through VMKernel, this might be similar for libvirt. Looking at the libvirt XML from OpenStack would give us better idea.

[roopakparikh]

Thanks for the details notes,

• ManagedVD
  @OmkarDeshpande7 also spoke about the same, given your customer interaction, can you verify if the NetApp appliance you mentioned earlier supports this feature and is there a way to find out who supports and who doesnt ?

• for PRDM - we will need to change the qemu advertised device to be SCSI instead of paravirtualized virtio device