[PATCH QEMU HCI-8.1 v5 35/36] VZ: migration: keep the estimate and wait over starved windows #VSTOR-150595

[Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>]

QEMU estimates the switchover bandwidth from the last update window.
Once the guest RAM is mostly clean, the sender runs out of data within
a window and the estimate collapses to the trickle it carried (13 MiB/s
on a 25 Gbit/s link).  Device state which only moves once the guest is
stopped, like that of an NVIDIA vGPU, then never fits into the downtime
limit and the migration never converges.  avail-switchover-bandwidth
would help, but nothing sets it for us.  Meanwhile the migration thread
loops without waiting, re-syncing the dirty logs about once a
millisecond.

Keep the previous estimate over a window in which every device ran out
of data, and wait such a window out, as is done when the rate limit is
exceeded.  Migrations which converge today don't see such a window
before the switchover, so they're unaffected.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 migration/migration.c | 21 +++++++++++++++++----
 migration/migration.h |  4 ++++
 2 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/migration/migration.c b/migration/migration.c
index 57aa0dea264..5f488836ab1 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1839,6 +1839,7 @@ int migrate_init(MigrationState *s, Error **errp)
     s->state = MIGRATION_STATUS_NONE;
     s->rp_state.from_dst_file = NULL;
     s->mbps = 0.0;
+    s->switchover_bw_per_ms = 0.0;
     s->pages_per_second = 0.0;
     s->downtime = 0;
     s->expected_downtime = 0;
@@ -3452,6 +3453,7 @@ static void update_iteration_initial_status(MigrationState *s)
     s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     s->iteration_initial_bytes = migration_transferred_bytes();
     s->iteration_initial_pages = ram_get_total_transferred_pages();
+    s->iteration_starved = false;
 }
 
 static void migration_update_counters(MigrationState *s,
@@ -3481,8 +3483,15 @@ static void migration_update_counters(MigrationState *s,
          */
         expected_bw_per_ms = switchover_bw / 1000;
     } else {
-        /* If the user doesn't specify bandwidth, we use the estimated */
-        expected_bw_per_ms = bandwidth;
+        /*
+         * If the user doesn't specify bandwidth, we use the estimated.
+         * Keep it over a window the sender ran out of data in: such a
+         * window only shows how little was left to send.
+         */
+        if (!s->iteration_starved || !s->switchover_bw_per_ms) {
+            s->switchover_bw_per_ms = bandwidth;
+        }
+        expected_bw_per_ms = s->switchover_bw_per_ms;
     }
 
     s->threshold_size = expected_bw_per_ms * migrate_downtime_limit();
@@ -3601,7 +3610,10 @@ static MigIterateState migration_iteration_run(MigrationState *s)
     }
 
     /* Just another iteration step */
-    qemu_savevm_state_iterate(s->to_dst_file, in_postcopy);
+    if (qemu_savevm_state_iterate(s->to_dst_file, in_postcopy) > 0) {
+        /* Every iterable device had nothing more to send */
+        s->iteration_starved = true;
+    }
     return MIG_ITERATE_RESUME;
 }
 
@@ -3719,7 +3731,8 @@ bool migration_rate_limit(void)
 
     bool urgent = false;
     migration_update_counters(s, now);
-    if (migration_rate_exceeded(s->to_dst_file)) {
+    /* Also wait out the window if the sender ran out of data */
+    if (migration_rate_exceeded(s->to_dst_file) || s->iteration_starved) {
 
         if (qemu_file_get_error(s->to_dst_file)) {
             return false;
diff --git a/migration/migration.h b/migration/migration.h
index ee15ca88461..e2cdb087321 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -350,6 +350,10 @@ struct MigrationState {
     } rp_state;
 
     double mbps;
+    /* Switchover bandwidth estimate (bytes/ms), kept over starved windows */
+    double switchover_bw_per_ms;
+    /* The sender ran out of data during the current update window */
+    bool iteration_starved;
     /* Timestamp when recent migration starts (ms) */
     int64_t start_time;
     /* Total time used by latest migration (ms) */

-- 
2.47.1