[PATCH hci-dev v5 2/2] vfio/migration: don't stop the device on guest shutdown #VSTOR-153976

[Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>]

When the guest shuts down and QEMU runs with -no-shutdown, the VM stays
stopped and can only be reset or quit. A libvirt soft reboot is exactly
that: guest shutdown, system_reset, cont. Today vfio_vmstate_change()
treats this stop like any other and moves the device to STOP, then cont
brings it back to RUNNING.

NVIDIA's vGPU manager (595.71.03) takes that STOP for the start of a
migration, finds no guest driver, breaks its dirty page tracking and
never repairs it when the driver loads again. Every live migration of
the VM after a soft reboot then fails: "Error saving page in pipelined
mode".

Skip the STOP in this case. The VM stop still runs its P2P round, so a
device that supports P2P only stops its peer to peer DMA; a device
without P2P is left running. The vCPUs are stopped and nothing more is
asked of the device until the reset, quit or migration that must
follow, and each of those resets or stops it: the reset by itself, the
two others with the previous patch. A device in the middle of a
migration is stopped as before, the migration needs it stopped.

The case is told apart by the shutdown action: only "pause"
(-no-shutdown) keeps the VM alive. A quit of a running VM, a signal and
a plain guest shutdown use "poweroff" and still stop the device as
before.

With this patch applied, soft reboot + subsequent live migration pass
successfully.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/vfio/migration.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index c391422d601..0341cbeeba2 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -415,6 +415,22 @@ static bool vfio_precopy_supported(VFIODevice *vbasedev)
     return migration->mig_flags & VFIO_MIGRATION_PRE_COPY;
 }
 
+/*
+ * The guest has shut down and QEMU keeps the VM stopped (-no-shutdown). Only
+ * a reset, a quit or a migration can follow. The device is not stopped
+ * here, it only stops its P2P DMA like on any VM stop: the reset resets it,
+ * the two other paths stop it with vfio_migration_stop_after_shutdown().
+ * A device in the middle of a migration is stopped as before.
+ *
+ * The same run state with the "poweroff" action is a quit of a running VM,
+ * which stops the device as usual.
+ */
+static bool vfio_vm_stopped_by_guest_shutdown(bool running, RunState state)
+{
+    return !running && state == RUN_STATE_SHUTDOWN &&
+           qemu_shutdown_action_is_pause();
+}
+
 /*
  * Stop a device that a guest shutdown left running, before it is closed or
  * saved. Only that leaves a device in RUNNING_P2P, or in RUNNING while the
@@ -963,6 +979,11 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state)
         return;
     }
 
+    if (vfio_vm_stopped_by_guest_shutdown(running, state) &&
+        !vfio_device_state_is_precopy(vbasedev)) {
+        return;
+    }
+
     if (running) {
         new_state = VFIO_DEVICE_STATE_RUNNING;
     } else {
-- 
2.47.1