From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazon11021092.outbound.protection.outlook.com [52.101.70.92]) by lore.virtuozzo.com (Postfix) with ESMTPS id 471B180CB9 for ; Fri, 23 Oct 2026 16:55:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Nc+Fd3WIiAKc6PyeHMCduW3nthOKTQuXPUtk/7pBuEBhDreJYssEm0ZC0l7Y0DaMC8EHOvRJjWjxMVYyNbLEBk5W3hu1DsuTI4xRrJ+Dw/Mt4GjYnYV9DFi+K0sqPMHYWlqFaga/FEsViOMJCQ/mtsAhlOOgGiq6ELtueHO5rjLMFolQmWjH7gNj2mHCKvsoYXlXgX1EMG34Ewpv1mweUAnpW9eAtBZuuFibRT/P1xj+vcKjs9KppsrbtRLglSVuyXYLDvsq3owy9G3IcGbCpuX0Dgk7h3qDiCVShz3+JLBQrkMC3UOGZCFnCuLdX5byCXikPWFDQZDQnM8wxMYCIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cGOlBnzO9hQeVoUCJtFd+25cuR5nlgyPMGp6cSyvvms=; b=uDq4QdC2foz9L/Xul2QK1uIizdpA4vwkaG8q7AZb6hxWPd9XgvVDn1IQhzF2/qMsj0M6f7Oa1VyVs9KobuTyNa3o5dWVrcYJBVDYAiwYkVIEh0Xv2OMU7q9VxYAgi17Bc8CerbdQHI98JOR/hkCClBGvlAz3EdnzMuOd9XPFhqLULujMN607rFxoEuzPQ0gLKlqs7HC+OX4miJTdKOQwYe6639KOLu0E68FdJYl7V5qsroGfzH64yYlXe1Fps4mSW6tbAAFhsAYyLG1goCHC+ZDPeto/51SMg349h84u2M5YQA3kYuQ73eJmwldqfslhnZ9PvUGDJ9345f1zzU1WKQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cGOlBnzO9hQeVoUCJtFd+25cuR5nlgyPMGp6cSyvvms=; b=crD2EFsIjThALAZe4eZEdS5wyqJATZEcUnbOPshadIA73g0wAkQ/M+Esx3sufTtahMyu+xhYLWWN7Ixvn+hPK763AGvoH360aTwB7Jv9uAwnrih8FO0uN9fPPcKhaSfMQOn5t0XQg+f3+IMjk4dpAq9iXguHOhPjTKh5TqHc4z4iUJHEwXWC25yGfd96YhySWjkD8YvwlduWZPYNCP79vwBF0MJsrKUauV9aR8AUbJdcld5vL6e0caBbcwKJzktCtZtHvdFcUKXClnJ17oeKMQVi3JtZ4XCC1TzvJHCSU0EE18KvjN9y2ys4ROm+8gUd8KWPYJgvSkrI/ay6HwOxRw== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by AS8PR08MB6373.eurprd08.prod.outlook.com (2603:10a6:20b:33a::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.18; Fri, 23 Oct 2026 16:55:22 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%7]) with mapi id 15.21.0546.017; Fri, 23 Oct 2026 16:55:22 +0000 From: Andrey Drobyshev Date: Fri, 23 Oct 2026 19:54:12 +0300 Subject: [PATCH QEMU HCI-8.1 v5 35/36] VZ: migration: keep the estimate and wait over starved windows #VSTOR-150595 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261023-vfio_v1-v5-35-ba0bf3c6d122@virtuozzo.com> References: <20261023-vfio_v1-v5-0-ba0bf3c6d122@virtuozzo.com> In-Reply-To: <20261023-vfio_v1-v5-0-ba0bf3c6d122@virtuozzo.com> To: svt-core@virtuozzo.com Cc: Andrey Drobyshev X-Mailer: b4 0.16.0 X-ClientProxiedBy: VI1PR0902CA0027.eurprd09.prod.outlook.com (2603:10a6:802:1::16) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|AS8PR08MB6373:EE_ X-MS-Office365-Filtering-Correlation-Id: 6eb4961c-f41b-4db4-5275-08df312658cf X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|366016|23010399003|1800799024|5023799004|38350700014|56012099006|261009223027099003|261021123027099003|261021121427099003|261022075727099003|261022105027099003|10067099003|18002099003|22082099003|5113699003; X-Microsoft-Antispam-Message-Info: 5veePTX3jkANmkBFK72vFIDJ6gYrovDuFCxwB1OCatTXPrRa5ftWdg+F7uIKpNmhNNaHPCPozt7otta64fiyWuI/mLKCm39zA9L5tCASj5C6x/9+hKYd37zF3ODhXr2sAAKLcHIDKwOkoYHua7zaWczeGdxFMeo9BnIetm5tXI/q6AKMPoeSgKSW8QYwy6j6EUPMdcoTM3QJqjtPCEWf6eb6Ls1wXfKJI2dVb/tgIA4h11u6Lf3RKkXXHKNEc5K6XPJXBC8lRXyd2Z36f/S0RGMP5X/gPVVz3zQ/vfy+q5fQvqjBzPf4jyBMuvQXTccqRzYudVHRTDYX/SLZ0N/IYzPTPvGLW8GzPD6STYPz50ouNQmkBRt1samz48R/pPX9uN/yTpmRuXXFGB2zKWmm/kvDxBquc8Vx+00Urm7PtFFdQ41udn5xQWIZt9Qazfs6H8HPcXJKo1PPpuKdEocPOOvJDp3uzrsMFStXlV/hUdEwGE8ozEpSA+8M/xk8Or9SSm9Q0J1z5QHGUYaYKmjTWyQDfOPm+EKsipHrvs4T1cOfTHxhTHGWXUpUdlGObNSkpBqai50NPcC/GIZdJQGnyM55ox1HbWUCoAhK4TDwtqmfDS7fRht9BYyddhPY8AF4u3Xj3efhDxlcMunoy+FpZJAGQ5MBjm2IhdIhXzihTKqZwncopjP4+ngkfTqtv27ux85UgwuAj9Dpl+IoEFQCg2yTLfcgPLxE98llExkgCps= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(366016)(23010399003)(1800799024)(5023799004)(38350700014)(56012099006)(261009223027099003)(261021123027099003)(261021121427099003)(261022075727099003)(261022105027099003)(10067099003)(18002099003)(22082099003)(5113699003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UlZmcXlMNXRmTmhLQ1JodEcycmpjTzI3RFREWm9nbWh3SElYcDJuNEJuRE14?= =?utf-8?B?K3lkQU5aSGhZR2V1bnd6TkppYWZQUGJXWXhONW1nbDE1NDRwQUdxM1NJK0l6?= =?utf-8?B?dTF1VjZaNWlsTDVpdHlVdlRvRkRUVTk0RndDa1AzVzZjZCtabElWVFhmTG5a?= =?utf-8?B?WEdnS0VLUGZzZnlsT1dTZ3I4UkFtYWFENDJnQ1ppZ0NjdVFrT0dTb3FrSWY2?= =?utf-8?B?MkhYQVJkdWVHV3VOU0dUUVcxNTZ6ZVhmbUo2dVU1OFZZdWViemVEMTFSbkVE?= =?utf-8?B?YkRBUzJFVEIyN2h0ZWtQcGNnRFdVSmd0RjZDaEdqMWE3ZEpudjVjQnhjUkkw?= =?utf-8?B?ZW5ZWTR5RTlmNUVIeGNRdDZ2bWxHY1JCK0xRblp0eUdYR1hjV1g0U2wwMk1X?= =?utf-8?B?bmxTc2lIT3NPWFZiWDZaSHIvYXNoNDZsQXdtSXJRb3lYN0RaZFdQWHNrUnU1?= =?utf-8?B?V21pckh4SitQYkViK21qTm85ZlVZTzBidDhmN0VGNytQM2dVd0VwaUcrOVJH?= =?utf-8?B?YzRHc2kydFdyeHhRRVhjVE9LV3IwV09pbXpOUmZwbldjWXNOaVUrTTBZa2dZ?= =?utf-8?B?SktROUdQRDZVZ1E5bzRsUFJscVhEVDJMVTFaZW4yb2xkZW5jRXpOY2FHYytS?= =?utf-8?B?bDBLVkIwNXhHSjJqQlhuMk9OQnQ5MHB4UXFaVDJPVUdVdms5ODlBRWdSamdm?= =?utf-8?B?RmZyQ0VRSjJGdCtBYkM4VUlCYW5nejRXdy84THdxUmlxOThRajM5YVlycGZ3?= =?utf-8?B?REkzTG5sWU1tT0QxNFczd1g2Rnp3eDBjRG5tL3J4a2FBaC8zWFZUTWlOampT?= =?utf-8?B?aUdMakora0VaU1E5SmFlM2pkUktTV01WUVVkYjIyUDhycnFIWVdCYUZOOGN0?= =?utf-8?B?NWpWcmIzQUd1UWgxTmtSSkh1NzNxZzFGemNGT2I5d1YyQnVmc2tlRUN6VTAv?= =?utf-8?B?SHZKc2tZdWU5eGVyQ1lpNDlaVmVzcjBycVliRFhtMjJpTlZDcE9WSExoc1RT?= =?utf-8?B?dlVHdUQwQ0tFWE5WL0tBRGtubU9FbU02OUZxQ0JYQXR2TWZmN2xKbkZIbEpG?= =?utf-8?B?SGpQVXlrbGFqYzhJamxaSXNmMTU1WEw0UUdxUEpuRG1SVWlTeStHWmhtYzR6?= =?utf-8?B?MGpuYkxsdUdQZG80S0RGWk9DbGptOGRPWmdXNlNrclRNZVhIVjRYUzVUTzBE?= =?utf-8?B?MFpzYkZnVXBaaWtCSHlUSnNoSTJ6OWRvVmgremM2WHlGczZ4QVVrV0ErMUdY?= =?utf-8?B?U1BLRVNkN2t6czlrdEJBK0dRRExvSG9YMWp5SUsrRnM0bXNVTjB4ODNocnBt?= =?utf-8?B?a1ltSkFGYlZvYjB0OVdGVWJrSE0xV2JaRGRMcXh5eFJNT0c5TUxPTXh6bCtY?= =?utf-8?B?ZHJrakdGR0NCRFhxVUJkc0hVSGdoODNIVlYwL2ZUUkZITDlBemNVRFBVWFR4?= =?utf-8?B?UXlaaytDVExGd0dTOXJ0emVrL3FocnJPemtKS0NqSDl2QllieE80NDlIVkp4?= =?utf-8?B?QjRad0c2bk9DaFUxV2hocFkrNzJJaENvbkZHY1lUUG5qV3hNanJJbFNYZkpM?= =?utf-8?B?ZDlzTVJLR2N1R0dhamp2cXNEMUZHc25OWlNONVRVTkF4SS9Nb0lUK1lrb21R?= =?utf-8?B?djBLcGZHTWo1Z0p5MnhVRGc5RWhtZDYvRFpXS3J1dHRjT3U2NXpSdW1NQ1Vr?= =?utf-8?B?VGVTSlk5a2diTXhrWUN1bXVWYnQyYmgzM2IzZEZrZFpoa2dQN3B1TVBKR2M3?= =?utf-8?B?c2tkMjhwcEVFdGs4V29lZ05uMitwb3ZWSTRMQ3BQTE5lVkdtRngwTWF0dld3?= =?utf-8?B?S216Tk9KdkJqNWxlOWNrVkYvbjJkVTBzSXZCcCtVY01Gc3hmQUd1bUQ0U3hV?= =?utf-8?B?anBBNm9lK0w2T3N4cVR0M24xRUFGMTVhMWJyLzVpbHA3UHB0bEs1QnVqYU5Y?= =?utf-8?B?SXIvS0RMeEZxcjNvdk1SdWhNV2g1QVFOeWJBeXN4WVMxNHJ1ZGExbEoyVE9a?= =?utf-8?B?OWhzNXRGQlNSWVR3M0FrbVRjRFFvMGxNR3FWNkJDT2xrc0g4U1FxcFRkZWVs?= =?utf-8?B?RUx1ZVV4c2pJNG9XU1MzdnIyM2o0Qmx5UitsaG95Y3d2dDlEbDBySXNMOGtC?= =?utf-8?B?a2R4Q2wyUzlWVzhETU9teEdlY3huSWZBNXltT2RHcDc3MGlLdy9SNVpDOGk0?= =?utf-8?B?YklidnVPdUEwQnFia0tNR2R0QThMOEVjay9nUXN4VElBUHIrZVJOekFwT3Ix?= =?utf-8?B?Y2g2VE9xOG5jdmpsTE5JZTdJdHdYRUF3c2swOWlQRzB1Q1h1bG55T2xkVWFY?= =?utf-8?B?QWQ5Si93d2dKeG5rRkpBUDl3N1ZFdDF6SGt3eVNHOWhYcTd5RUxmSlNKUEZa?= =?utf-8?Q?SohK7ew5dUts3RSM=3D?= X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6eb4961c-f41b-4db4-5275-08df312658cf X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2026 16:54:48.0891 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QosT0Vw8BHBVPucb9KnUV07pXWfV3ondQhoe09YiURttBsXd8QAG5pQ4TNdlu5kWxlX6XAVvPINa0NMSZCWQhbrlTyiX9DChHginNIQQE08= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB6373 QEMU estimates the switchover bandwidth from the last update window. Once the guest RAM is mostly clean, the sender runs out of data within a window and the estimate collapses to the trickle it carried (13 MiB/s on a 25 Gbit/s link). Device state which only moves once the guest is stopped, like that of an NVIDIA vGPU, then never fits into the downtime limit and the migration never converges. avail-switchover-bandwidth would help, but nothing sets it for us. Meanwhile the migration thread loops without waiting, re-syncing the dirty logs about once a millisecond. Keep the previous estimate over a window in which every device ran out of data, and wait such a window out, as is done when the rate limit is exceeded. Migrations which converge today don't see such a window before the switchover, so they're unaffected. Signed-off-by: Andrey Drobyshev --- migration/migration.c | 21 +++++++++++++++++---- migration/migration.h | 4 ++++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 57aa0dea264..5f488836ab1 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1839,6 +1839,7 @@ int migrate_init(MigrationState *s, Error **errp) s->state = MIGRATION_STATUS_NONE; s->rp_state.from_dst_file = NULL; s->mbps = 0.0; + s->switchover_bw_per_ms = 0.0; s->pages_per_second = 0.0; s->downtime = 0; s->expected_downtime = 0; @@ -3452,6 +3453,7 @@ static void update_iteration_initial_status(MigrationState *s) s->iteration_start_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME); s->iteration_initial_bytes = migration_transferred_bytes(); s->iteration_initial_pages = ram_get_total_transferred_pages(); + s->iteration_starved = false; } static void migration_update_counters(MigrationState *s, @@ -3481,8 +3483,15 @@ static void migration_update_counters(MigrationState *s, */ expected_bw_per_ms = switchover_bw / 1000; } else { - /* If the user doesn't specify bandwidth, we use the estimated */ - expected_bw_per_ms = bandwidth; + /* + * If the user doesn't specify bandwidth, we use the estimated. + * Keep it over a window the sender ran out of data in: such a + * window only shows how little was left to send. + */ + if (!s->iteration_starved || !s->switchover_bw_per_ms) { + s->switchover_bw_per_ms = bandwidth; + } + expected_bw_per_ms = s->switchover_bw_per_ms; } s->threshold_size = expected_bw_per_ms * migrate_downtime_limit(); @@ -3601,7 +3610,10 @@ static MigIterateState migration_iteration_run(MigrationState *s) } /* Just another iteration step */ - qemu_savevm_state_iterate(s->to_dst_file, in_postcopy); + if (qemu_savevm_state_iterate(s->to_dst_file, in_postcopy) > 0) { + /* Every iterable device had nothing more to send */ + s->iteration_starved = true; + } return MIG_ITERATE_RESUME; } @@ -3719,7 +3731,8 @@ bool migration_rate_limit(void) bool urgent = false; migration_update_counters(s, now); - if (migration_rate_exceeded(s->to_dst_file)) { + /* Also wait out the window if the sender ran out of data */ + if (migration_rate_exceeded(s->to_dst_file) || s->iteration_starved) { if (qemu_file_get_error(s->to_dst_file)) { return false; diff --git a/migration/migration.h b/migration/migration.h index ee15ca88461..e2cdb087321 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -350,6 +350,10 @@ struct MigrationState { } rp_state; double mbps; + /* Switchover bandwidth estimate (bytes/ms), kept over starved windows */ + double switchover_bw_per_ms; + /* The sender ran out of data during the current update window */ + bool iteration_starved; /* Timestamp when recent migration starts (ms) */ int64_t start_time; /* Total time used by latest migration (ms) */ -- 2.47.1