[PATCH QEMU HCI-8.1 v7 35/35] VZ: vfio/migration: don't wait for stop-copy-only device state #VSTOR-141821

[Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>]

A VFIO device reports the state it hands over once the guest is stopped
as pending data, and the migration switches over only when all pending
data fits into bandwidth x downtime-limit.  That state can't get any
smaller by iterating, so when it alone doesn't fit, the migration never
converges until the downtime limit is raised, which Nova does in steps
of minutes.  An NVIDIA vGPU reports 1.5-3 GiB of it, which never fits
the default limits.

Report the stop-copy size as pending only while the device still has
something to pre-copy.  Once it has drained, the rest of the pending
data, which iterating can shrink, decides the switchover on its own.

A device without pre-copy support has nothing to pre-copy from the
start, so its state isn't waited for either.  In both cases the
downtime limit no longer bounds the time that state takes to send:
when it fits, nothing is lost by not waiting, and when it doesn't,
waiting couldn't help.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/vfio/migration.c | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
index 04d0de7cef0..d66de7d31a2 100644
--- a/hw/vfio/migration.c
+++ b/hw/vfio/migration.c
@@ -569,16 +569,24 @@ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy,
     VFIODevice *vbasedev = opaque;
     VFIOMigration *migration = vbasedev->migration;
     uint64_t stop_copy_size = VFIO_MIG_STOP_COPY_SIZE;
+    int ret;
 
     /*
-     * If getting pending migration size fails, VFIO_MIG_STOP_COPY_SIZE is
-     * reported so downtime limit won't be violated.
+     * If getting the pending sizes fails, VFIO_MIG_STOP_COPY_SIZE is
+     * reported, unconditionally, so the switchover isn't taken blindly.
      */
-    vfio_query_stop_copy_size(vbasedev, &stop_copy_size);
-    *must_precopy += stop_copy_size;
+    ret = vfio_query_stop_copy_size(vbasedev, &stop_copy_size);
 
     if (vfio_device_state_is_precopy(vbasedev)) {
-        vfio_query_precopy_size(migration);
+        ret |= vfio_query_precopy_size(migration);
+    }
+
+    /*
+     * Once there's nothing left to pre-copy, the stop-copy state can't
+     * shrink by iterating: don't let it hold up the switchover.
+     */
+    if (ret || migration->precopy_init_size || migration->precopy_dirty_size) {
+        *must_precopy += stop_copy_size;
     }
 
     trace_vfio_state_pending_exact(vbasedev->name, *must_precopy, *can_postcopy,

-- 
2.47.1