Use PCI and support Firecracker device hotplugging

[richardcase]

Summary

Firecracker now supports PCI transport for virtio devices and Developer Preview hotplugging of virtio block, pmem, and network devices. Flintlock currently starts Firecracker without its API socket, explicitly boots guests with pci=off, and has no API for changing devices on a running microVM.

This proposal makes PCI the default transport for new Firecracker microVMs and adds asynchronous hot-attach and hot-detach operations for block volumes and network interfaces. Guest coordination is performed through the Flintlock guest agent because Firecracker does not currently notify the guest when the PCI topology changes.

This revisits the conclusion of Firecracker Patch Limitations & The Flintlock Update API, but only for explicitly supported device operations. It does not restore a generic microVM update operation.

Goals

• Use PCI transport for every newly created Firecracker microVM.
• Attach and detach virtio block volumes and network interfaces while a microVM is running.
• Represent device changes as asynchronous, observable reconciliation operations.
• Make attach and detach safe to retry.
• Coordinate PCI discovery and removal inside the guest without requiring callers to run sysfs commands.

Non-goals

• virtio-pmem, CPU, memory, balloon, VFIO, or physical PCI passthrough hotplug.
• Hotplug support for Cloud Hypervisor.
• Mounting filesystems or configuring DHCP, addresses, routes, and DNS inside the guest.
• A generic UpdateMicroVM API.
• Converting already-running MMIO microVMs to PCI.

Firecracker and guest requirements

Flintlock would:

• Start Firecracker with --enable-pci.
• Remove pci=off from the kernel command line.
• retain a Firecracker API Unix socket alongside config-file boot so that device PUT and DELETE requests can be made after boot.
• Require a guest kernel with the PCI and virtio-pci options documented by Firecracker, including CONFIG_PCI, CONFIG_PCI_MSI, CONFIG_PCIEPORTBUS, CONFIG_VIRTIO_PCI, CONFIG_PCI_HOST_COMMON, and CONFIG_PCI_HOST_GENERIC.
• Require Firecracker v1.16.1 or newer for implementation and testing.

Firecracker currently describes device hotplugging as Developer Preview. Production adoption should use a tagged release containing the attach and unplug deadlock fixes that landed after v1.16.1 if no such release is available when implementation begins.

Existing MMIO microVMs would continue running unchanged but must be recreated before they can use these APIs.

References:

• Firecracker device hotplugging
• Firecracker kernel policy and PCI configuration
• Flintlock guest agent

Flintlock API

Rather than restoring a generic update RPC, add four narrowly scoped RPCs:

AttachVolume(uid, expected_version, volume)
DetachVolume(uid, expected_version, device_id)
AttachNetworkInterface(uid, expected_version, interface)
DetachNetworkInterface(uid, expected_version, device_id)

Each RPC returns a DeviceMutationResponse containing the current MicroVM. Acceptance is asynchronous: the response confirms that Flintlock persisted the requested operation, not that the guest can already use the device.

expected_version provides optimistic concurrency. An exact retry is idempotent. Only one device mutation may be pending for a microVM; a different concurrent request returns FailedPrecondition.

Requests are rejected when:

• The microVM is not running or is not backed by Firecracker.
• The microVM was created without allow_guest_agent.
• The guest agent does not advertise the hotplug capability.
• The runtime has no Firecracker API socket, as with an older microVM.
• Another device operation is pending.
• Detach targets the root volume, a virtiofs device, or a protected management interface.
• The resource is already owned by another device, including duplicate IDs, MAC addresses, or TAP devices.

Both initially configured and hot-added non-root devices may be detached.

Status model

Extend MicroVMStatus with device-operation status containing:

• Operation type: attach or detach.
• Device type and stable device ID.
• State: Pending, Succeeded, or Failed.
• A failure reason suitable for callers and operators.
• The observed PCI BDF and guest Linux device name.
• Block identity or network MAC address where applicable.

A failed device operation does not change an otherwise healthy running microVM from Created. The operation status records the failure and the reconciler retains enough information to retry or clean up safely.

Guest-agent protocol

The guest agent would expose an additive pci-hotplug-v1 capability and three operations:

• pci-list: enumerate supported PCI devices and return their BDF, kind, Linux device name, block serial, and MAC address.
• pci-rescan: write to /sys/bus/pci/rescan and return the resulting supported-device inventory.
• pci-remove: validate and remove a specific BDF through sysfs.

The implementation should inspect sysfs directly and not depend on lspci or ip. Block removal must reject a device when it or any partition is mounted, based on mountinfo. Network removal must lower the interface before removing its PCI device.

Flintlock should consume the protocol through a reusable guest-agent host client rather than duplicating the vsock framing logic.

The guest agent only discovers and safely releases devices. The caller remains responsible for mounting volumes and configuring guest networking.

Reconciliation

Attach volume

1. Validate and persist a pending operation.
2. Prepare the containerd-backed block resource.
3. Send Firecracker PUT /drives/{id}.
4. Ask the guest agent to rescan PCI.
5. Match the guest block device using the serial Firecracker derives from the backing file metadata.
6. Persist the BDF and guest device name, then mark the operation succeeded.

Attach network interface

1. Validate and persist a pending operation.
2. Create and configure the host TAP resource.
3. Send Firecracker PUT /network-interfaces/{id}.
4. Ask the guest agent to rescan PCI.
5. Match the guest interface using its explicit MAC address.
6. Persist the BDF and guest interface name, then mark the operation succeeded.

Detach

1. Ask the guest agent to validate that the device is releasable and remove its PCI function.
2. Send the corresponding Firecracker DELETE request.
3. Release the TAP or the individual containerd volume resource.
4. Remove the device status and mark the operation succeeded.

Ambiguous guest-device matches fail safely. Reconciliation must preserve the last completed step so retries do not create duplicate Firecracker, TAP, or containerd resources. Flintlock also needs per-volume snapshot and lease cleanup rather than releasing all containerd resources only when the microVM is deleted.

Alternatives considered

Restore a generic UpdateMicroVM RPC

Rejected because Firecracker still supports only a constrained set of post-boot mutations. Dedicated RPCs provide clearer validation, authorization, status, and compatibility semantics.

Make PCI opt-in

Rejected for new Firecracker microVMs. Maintaining both MMIO and PCI paths would increase test and compatibility complexity, while hotplug necessarily requires PCI.

Require callers to rescan and remove PCI devices

Rejected because the host cannot know that guest filesystems and interfaces are safe to release. The guest agent provides a controlled protocol and consistent failure handling.

Perform host-side hotplug only

Rejected because Firecracker provides no automatic guest notification. Unplugging before the guest releases the device can leave stale guest state and unsafe memory access.

Only detach devices that were hot-added

Rejected because Firecracker can detach initially configured non-root devices too, and callers should not need to track how a detachable device was introduced.

Consequences and limitations

• PCI becomes a breaking runtime requirement for new Firecracker microVMs and their kernels.
• Hotplug remains experimental until Firecracker promotes the feature from Developer Preview.
• Firecracker has a finite PCI slot limit, so exhaustion must produce a clear operation failure.
• Guest coordination and host mutation are not transactional. Persisted operation progress and idempotent retries are required.
• Network devices are identified by MAC address. Block devices are identified using Firecracker's derived serial; ambiguous identity is an error.
• Callers must unmount filesystems before volume detach and must manage DHCP or static network configuration themselves.
• The guest agent must be released before Flintlock can advertise this feature.

Validation

Implementation should include:

• Unit tests for request validation, optimistic concurrency, idempotency, pending-operation exclusion, status transitions, guest capability checks, and cleanup.
• Guest-agent tests with injectable sysfs and mountinfo roots.
• Firecracker integration tests for block and network attach/detach, duplicate resources, missing resources, failed guest release, API socket recovery, and PCI slot exhaustion.
• End-to-end tests proving that a running microVM can discover and safely release both a volume and a network interface.
• Regression coverage for creating, starting, and deleting PCI-backed Firecracker microVMs.

Suggested delivery order

1. Add and release guest-agent protocol support.
2. Enable PCI and retain the Firecracker API socket.
3. Add device-operation API and status types.
4. Implement volume hotplug reconciliation.
5. Implement network hotplug reconciliation.
6. Add integration and end-to-end coverage.
7. Document compatibility and the experimental support level.

Feedback is especially welcome on the dedicated RPC shape, mandatory PCI rollout, status representation, and whether the one-operation-per-microVM constraint is appropriate.