From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from MRWPR03CU001.outbound.protection.outlook.com (mail-francesouthazon11021123.outbound.protection.outlook.com [40.107.130.123]) by lore.virtuozzo.com (Postfix) with ESMTPS id 8E17680CBA for ; Fri, 23 Oct 2026 16:55:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=k8Zrih7DHQNLvKskJXs1OHt7j2kLHwj0Ox3t07jOrro1sqCb/K/Y7IKYP+UgVD0MDYn1XkncxtxQL3IJKvfGpdHYkDBHfTgCASiO2aNW7IcGIVD07ld6r2iWTGVh4Chc4yCvtqPeIC0a7yyDjAm6XiQZqRCWIByu45sft+qMSK/OUwtjzOTXlL7IO2eGHH44b8nOGbPbbM/agkST8W7JvS7+d0u530DikWrumQ+hj0TW/H+SDRl4DX0sH8pEtWyEXATZsmb2rS6a0LvA904hLBRwiCByCjq2tklfivWkxlM4qqNbyWrZf7qhuuPoTBi0/34ar8sWcdZ90LJQ78e3aQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vaFpdbb6ou/AFk7Ai35Cqb6HK4YjG0hkTBkhVZjG2fA=; b=K+X+M/va2YgOxrHtJDeALPVCLgTf+UDMZ6yLApWrIK9GfDHlqEldO3LALdG3GPR1GIDnSdrMXCthw0DO15OuGJj5jWhhetnO6fHSv3WigusZwLBkGx43FqM4jJWuSkBjzNC3g73WWDCxFtYRL5Kq6MbGqNNek10m86SpIFOpM7kN9GSGztdcOzp7T9YNutbQmtWVNhTsHtUOe8nc9VR1GwM7WehyH9C+TUgl8XIBCmggjUU9jiDkq8qqKHBYFDTXufxxUst0Y0Q5kqp55POxEyUeTpti/cvppk1XKuJ3x+Z03T/xvQKBpbfabZcufLiRKgwNQpp0kg4W/bZGVXRs0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vaFpdbb6ou/AFk7Ai35Cqb6HK4YjG0hkTBkhVZjG2fA=; b=d5v+4AE2bhtPfiKfqNEeeUyW7pwLfvL/EYvmTT9LYargdAaVirZr12JNDtAqDpd2Zx+mfVyC2kc+wo80qVnK5ubsJjdlBLsWWVmTUMfn7SRvQBZjC/SUv+WO/5dIkNEMGPhxHjXfM9aXxg/OO0hszs7z9+vFE+sREbniVBRnk9mhAeCK8OzHYVZl++QbwjOhL/bGQJZkBFqr1LX09IyIH9TLbwzuftjyJHLxAm9yfngTeVYe+RHZg0GafbXC+c60hqSwdHGJ/oMaclovXu0YMS4ewrbKCW7gcnCPEJNSE+vlZegyxau80bmGW94CKRLl4fQ4Z1QEkwPmLGSz9uLWOg== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by AS8PR08MB6373.eurprd08.prod.outlook.com (2603:10a6:20b:33a::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.18; Fri, 23 Oct 2026 16:55:22 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%7]) with mapi id 15.21.0546.017; Fri, 23 Oct 2026 16:55:22 +0000 From: Andrey Drobyshev Date: Fri, 23 Oct 2026 19:54:13 +0300 Subject: [PATCH QEMU HCI-8.1 v5 36/36] VZ: migration: don't wait for stop-copy-only device state #VSTOR-150595 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261023-vfio_v1-v5-36-ba0bf3c6d122@virtuozzo.com> References: <20261023-vfio_v1-v5-0-ba0bf3c6d122@virtuozzo.com> In-Reply-To: <20261023-vfio_v1-v5-0-ba0bf3c6d122@virtuozzo.com> To: svt-core@virtuozzo.com Cc: Andrey Drobyshev X-Mailer: b4 0.16.0 X-ClientProxiedBy: VI1PR0902CA0027.eurprd09.prod.outlook.com (2603:10a6:802:1::16) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|AS8PR08MB6373:EE_ X-MS-Office365-Filtering-Correlation-Id: 1acb7997-de12-4956-1717-08df31265929 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|52116014|376014|366016|23010399003|1800799024|5023799004|38350700014|56012099006|261009223027099003|261021123027099003|261021121427099003|261022075727099003|261022105027099003|10067099003|18002099003|22082099003|5113699003|3023799007; X-Microsoft-Antispam-Message-Info: sn37M1GXu1lYUNQbtWB7QydU+HGASo2pB0Lh/Q5pcV37UCTzltz1b51iJMffF5F52SeXXuRF19pnkneNbuXAxDenAbpkpG1Spe3e6N8qRg150gwfMG7JPEfRLno3G0i5/FatxezDPyuJtPPoDVTDFPuEdsO0H17mmINyLCpfVKQqNfTtNITBOF0h4FKSQwXcZcLj6L+hfsd8OwkCeAarKu8P7a4EJMnwLJEKL3jp3NEBp6jQfQG1WU0Lf5Uzo2tlwAUtszZh+E/bolSkM5zC/PpFtTyhGhspgO7gHYj+2dc2yeariS9wJ+VwFfPKdyJAaRUFXoZKHJ4yiesaFPVED5RNGyUUTQhJxIohr7DYYr5dgZbvB30ZORnpe8BNtOC/zAy7RvPg5KQdkXzsaO1D0FghakTzMcwzqe+bDkRbw+yxmimyCbNevPfbG0o0dloBxvD4+3sCM8ezbHA+0FLy/34ksO64LBf4WKzGusEpUu1wk0JQjN0MFNgnv56cLXvBnWfqZ6BTJbG2MzsDW5o6wwKcR5aWRbt7BwQJXhSPwALiSXUtfnGhToVtQkKBP1MblBxqt0okbE78/Xh1Ro+3Zzl159E0Olxr+UdVLBNgaDDadarLFfwDuzIumVskdAGYa8EoYRi0+IkI+cO5IVvkZeTPhV4s9dqDZDPwISFa9cvzcytWmrQwiZIwRM0T5NHZYB/W8yQBInGmvVmoFo5PGxoodaHgn1vN5AvIsT5JXbo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(52116014)(376014)(366016)(23010399003)(1800799024)(5023799004)(38350700014)(56012099006)(261009223027099003)(261021123027099003)(261021121427099003)(261022075727099003)(261022105027099003)(10067099003)(18002099003)(22082099003)(5113699003)(3023799007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Mk9VZFRrQ2UvN3UyZ1p1b1B0YzU3d1d0dS93TlpOcUFOS0pKaWlBSlJQQytM?= =?utf-8?B?UmlYaXR5OE9Db0tYVCtjQjM5Ly85alNnQmFqMkFQNlRjcDVlc2VQR3lIYXJv?= =?utf-8?B?MldDd0xPL05pZkJ5WVZGOGZQT2VZQ2VIV2NuTmRUSVpPR2VxeG5uZlIxdjNH?= =?utf-8?B?azk1NEg4R3BwSlhlbU1hS0RuZ0RlMlllZ0FBRDNGMTFaTmdaZnVhNU9jclZw?= =?utf-8?B?RHYwV0t2RTRPT3J6WU94ZDVwbzNwbGZPZERnc3J0dGxwdWJ6V05XM3NPR0Qw?= =?utf-8?B?NW01L0RXRzYzTktpZWJnbmpWTXpzNG1qQWJocTVRL3FiNmE5OVhYZkE2ZUJq?= =?utf-8?B?RUlMcjdJWWQ3eEtMTEg1R0NjZGM1QXRsUi8vZ3JPT2ZWVlJRNUp5Y3FPZ2U4?= =?utf-8?B?cnhvb3dmTnh6cUVoNW13RmRnVi9Ec0IyTnR2dEs3MG5oVGpJbkxaczJ5SUxj?= =?utf-8?B?MnBUenJiR0I1M0NCdWtZQTlyNE8wbExlbE1WU3Y5a0dDSlBQN21GZ2c2Skk4?= =?utf-8?B?cHVQYysySDNwLzV3bkhQalNJMVI3SlNndUVYejcxbXgwazJOcGRxVGRMMkdT?= =?utf-8?B?ck5pQVRzZS9wNHlqWEZVWFA1OWZHWm41Tk11aEJCTFIzOVg1RjhZVWVQT2VG?= =?utf-8?B?VHMzUGVKYXE3THhzdFpDdnFGQlIyUk9kMXhHSTFObzBxNXlSVkpVeDdhN0I5?= =?utf-8?B?bmZLcW8yRWMxSmE5bHNqZnh4OUJ1WURBVkViWUVBM3VUN29zQ2EvMGZnV3ZO?= =?utf-8?B?OEJidmxkeHVoZGxxWHdkSHFxL2J4SG8xSm1qTmRHYXNoNUE4SkdIMThScFlv?= =?utf-8?B?RHB2QlFVcU1uZVZQS1dSejQrUEtJZzFSZGQ5ODZXb0ticFhzaHJ3ZVp4b1F2?= =?utf-8?B?aERzRmo2NjFobDFiQUN0a2IrZjcyRlNaNVVpWGdRanVBT0lwODlMNU5sbUF6?= =?utf-8?B?UlJwbFFkYVlhTjRYa1ZIQ200NUxOWW8rcFMxcjQ5MHoyeFd3aElZdFp6NVAr?= =?utf-8?B?TDl5M1hySjhoQ0pnaXN4WlFNSVhzQ1lXWGdGblRwT1I2eHQwYUsrSktIQ2VD?= =?utf-8?B?SlQwVEE4SWEvRWdtVmE3SlRrNXNJUkhDVS9VQUlIdFNDdkZoUWxlRlFEN0U1?= =?utf-8?B?V2lsQ0hhMVFHdTZ5cHIyNXAvVE1jdzZyc29ER2VkS0lHTHdCY3BCQ25aTUdY?= =?utf-8?B?bnlCZUx2OEpKVTQxNWNUaE9DZ3dXVEY1Z2p4bnV3alZVQk9nTHRsV2Q1b1lR?= =?utf-8?B?R2tFQ3U2UXArL2pnR3pqTm40MEgxQ2hFVCtkeUplQlJHZ3FVL3AzY3NxaU1w?= =?utf-8?B?SW5tUzNsVW8zWnZsS0R1TkxtTzlsZS96ZHJRTGVLV2Ryb3U0WlU4Q0Rqc0JY?= =?utf-8?B?dld2bjdGOVovbVBGb0VPWXg2YXlvb2trdnVISWRoNXNLbFA0TVd5ZThlREpp?= =?utf-8?B?MEhPRGxVRUVnMDAxSDJhVEwwV3FGcDVGMG5lTTh5S1lHUnVobmdrOTdLdEZm?= =?utf-8?B?RFNxOVpnSnZSdEcvd0ZwbElUOUtwWjBEOForZy9UVGVqRzUvVjIydE84V2N1?= =?utf-8?B?U1ZwZ1p4WFg0VC9KK0Mwc0NPY1RzemJZRHFMVnBnRi9UM1dSVHMrM3FzTHN0?= =?utf-8?B?ZnNxL25sdnoxallTdENvVTZ5S0dTUEtpSllYWUswMVhNdWpiOC9Ra1VQdGJu?= =?utf-8?B?MFZqOXNXQ2Z6VFQ0RnN2enYzK09IQ09TQ1pSTElpbEk1cVNWdGx5L1YvN1Aw?= =?utf-8?B?SnJFQkJIU2xEWmQ2T3FxZkVUejF1OXBBTTYvbDJvQ1pTeVVBbGlxMnkzUU9R?= =?utf-8?B?UTFBSEM2cjJzUzF6ZHVkWUYrVk5aWXhuZmFnL0pTT2tqemZEdjFOVEphUFlK?= =?utf-8?B?VzlONzBSVEVsc2FwdGNuMW42NkdjN005Vno5RzdjRmNUZ251amRFaHM3bENq?= =?utf-8?B?Sm5ocytpUmZyUi96MkxjeGVDN0U5T2lMNG1ILzJ6R1FqVG1sSGFibGRTTDhi?= =?utf-8?B?ZnhXR1I5SVo1a203OGVEZ0NFN1NCaTlVU0tuTmhZZm41NVE4M0txTUxZdUt6?= =?utf-8?B?TGk0dXVCYXhxM09VV2NHMXNQNW1kVXpyRU9QZjFwMnhpUmRIVmROeG0xRm5B?= =?utf-8?B?WGhaYTBJek03QnBBNXJwRzFRVkVFRzFVZkhROVoyeFN0aUNHNjFnK25KVEg5?= =?utf-8?B?c2p0Yk1nT0lvbnIxVUZkOElwaldFejBlOTVRUEJjL0xQcTIvWnlYQTc5R3RL?= =?utf-8?B?TVdoTE8zdW9EdXRTWUJqakNpQUtKbU9HbDVMaVFsVGF6RldSNkRUU01JVnpo?= =?utf-8?B?ME1STHl4aWMweUNYdlQyMUNJdko0ZmlFZGQwcmFaOHd0U1BIdDNCOFB2VVNO?= =?utf-8?Q?+8QJPpclYMI+17yg=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1acb7997-de12-4956-1717-08df31265929 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2026 16:54:48.5975 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: J0Htm0r3M/1VZnFUU9Eb4JfzjDJGiaRw73yf+57VahmPUxQor0u9TNE4SRt1ezlsHZ2fzg1wo7ZYKnnTY5/FjB4CZwoyWhuVDVyuQvZ5v7E= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB6373 A VFIO device reports its stop-copy state as pending data. Once it has nothing left to pre-copy, or doesn't pre-copy at all, nothing done while the guest runs makes it smaller. If it doesn't fit into bandwidth x downtime-limit on its own, the migration iterates until the limit is raised, which Nova does in steps of minutes. An NVIDIA vGPU reports ~2 GiB, so at the usual limits it never fits. Have state_pending_exact handlers report such irreducible data and, when it alone exceeds the threshold, switch over once the rest is within the threshold plus as much again: waiting can't do better, and squeezing the rest further only keeps the guest throttled. Migrations whose pending data fits the threshold are unaffected. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 9 ++++++++- migration/migration.c | 17 +++++++++++++++-- migration/migration.h | 2 ++ migration/savevm.c | 15 +++++++++++++-- migration/savevm.h | 5 +++-- 5 files changed, 41 insertions(+), 7 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 8f2722dc671..c7312d9cb58 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -569,18 +569,25 @@ static void vfio_state_pending_exact(void *opaque, uint64_t *must_precopy, VFIODevice *vbasedev = opaque; VFIOMigration *migration = vbasedev->migration; uint64_t stop_copy_size = VFIO_MIG_STOP_COPY_SIZE; + int ret; /* * If getting pending migration size fails, VFIO_MIG_STOP_COPY_SIZE is * reported so downtime limit won't be violated. */ - vfio_query_stop_copy_size(vbasedev, &stop_copy_size); + ret = vfio_query_stop_copy_size(vbasedev, &stop_copy_size); *must_precopy += stop_copy_size; if (vfio_device_state_is_precopy(vbasedev)) { vfio_query_precopy_size(migration); } + /* Once there's nothing left to pre-copy, iterating can't shrink it */ + if (!ret && !migration->precopy_init_size && + !migration->precopy_dirty_size) { + qemu_savevm_state_pending_add_irreducible(stop_copy_size); + } + trace_vfio_state_pending_exact(vbasedev->name, *must_precopy, *can_postcopy, stop_copy_size, migration->precopy_init_size, migration->precopy_dirty_size); diff --git a/migration/migration.c b/migration/migration.c index 5f488836ab1..4a7da07fa05 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1840,6 +1840,7 @@ int migrate_init(MigrationState *s, Error **errp) s->rp_state.from_dst_file = NULL; s->mbps = 0.0; s->switchover_bw_per_ms = 0.0; + s->irreducible_pending = 0; s->pages_per_second = 0.0; s->downtime = 0; s->expected_downtime = 0; @@ -3576,8 +3577,10 @@ static MigIterateState migration_iteration_run(MigrationState *s) * postcopy started, so ESTIMATE should always match with EXACT * during postcopy phase. */ - if (pending_size < s->threshold_size) { - qemu_savevm_state_pending_exact(&must_precopy, &can_postcopy); + /* Allow for what the estimate lacks, see the irreducible rule below */ + if (pending_size < s->threshold_size + 2 * s->irreducible_pending) { + s->irreducible_pending = + qemu_savevm_state_pending_exact(&must_precopy, &can_postcopy); pending_size = must_precopy + can_postcopy; trace_migrate_pending_exact(pending_size, must_precopy, can_postcopy); @@ -3601,6 +3604,16 @@ static MigIterateState migration_iteration_run(MigrationState *s) * (which was calculated from expected downtime) */ complete_ready = can_switchover && (pending_size <= s->threshold_size); + + /* + * Data which only moves once the guest is stopped can't shrink by + * iterating: if it alone exceeds the threshold, switch over once + * the rest is within the threshold plus as much again. + */ + if (s->irreducible_pending > s->threshold_size) { + complete_ready = can_switchover && + pending_size <= s->threshold_size + 2 * s->irreducible_pending; + } } if (complete_ready) { diff --git a/migration/migration.h b/migration/migration.h index e2cdb087321..d5df6ce495f 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -316,6 +316,8 @@ struct MigrationState { * measured bandwidth, or avail-switchover-bandwidth if specified. */ uint64_t threshold_size; + /* Pending data which only moves once stopped, from the last exact query */ + uint64_t irreducible_pending; /* params from 'migrate-set-parameters' */ MigrationParameters parameters; diff --git a/migration/savevm.c b/migration/savevm.c index 3b768f4f4e0..f4df8308c2d 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1752,13 +1752,23 @@ void qemu_savevm_state_pending_estimate(uint64_t *must_precopy, } } -void qemu_savevm_state_pending_exact(uint64_t *must_precopy, - uint64_t *can_postcopy) +/* Exact pending data which only moves once the guest is stopped */ +static uint64_t savevm_pending_irreducible; + +void qemu_savevm_state_pending_add_irreducible(uint64_t size) +{ + savevm_pending_irreducible += size; +} + +/* Returns the part of the pending data which only moves once stopped */ +uint64_t qemu_savevm_state_pending_exact(uint64_t *must_precopy, + uint64_t *can_postcopy) { SaveStateEntry *se; *must_precopy = 0; *can_postcopy = 0; + savevm_pending_irreducible = 0; QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { if (!se->ops || !se->ops->state_pending_exact) { @@ -1771,6 +1781,7 @@ void qemu_savevm_state_pending_exact(uint64_t *must_precopy, } se->ops->state_pending_exact(se->opaque, must_precopy, can_postcopy); } + return savevm_pending_irreducible; } void qemu_savevm_state_cleanup(void) diff --git a/migration/savevm.h b/migration/savevm.h index 2d5e9c71668..dc37a4ba194 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -40,10 +40,11 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy); void qemu_savevm_state_cleanup(void); void qemu_savevm_state_complete_postcopy(QEMUFile *f); int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only); -void qemu_savevm_state_pending_exact(uint64_t *must_precopy, - uint64_t *can_postcopy); +uint64_t qemu_savevm_state_pending_exact(uint64_t *must_precopy, + uint64_t *can_postcopy); void qemu_savevm_state_pending_estimate(uint64_t *must_precopy, uint64_t *can_postcopy); +void qemu_savevm_state_pending_add_irreducible(uint64_t size); int qemu_savevm_state_complete_precopy_iterable(QEMUFile *f, bool in_postcopy); bool qemu_savevm_state_postcopy_prepare(QEMUFile *f, Error **errp); void qemu_savevm_send_ping(QEMUFile *f, uint32_t value); -- 2.47.1