From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from MRWPR03CU001.outbound.protection.outlook.com (mail-francesouthazon11021107.outbound.protection.outlook.com [40.107.130.107]) by lore.virtuozzo.com (Postfix) with ESMTPS id 695E280B19 for ; Fri, 16 Oct 2026 15:26:20 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HlpIb9fjtZ8H9P9KuQaPCqbdLWnzVa8BcVAbhk1hV23sP1fvWJHBA5k6UEsUhVYYsvJ+nDZUN7LHRjaTjdbVjk5QkSe4qH/RM/omcb8MNT52b7CCBuUrKOSGDiQBBILEpZTZwsZVaizoTOgX5reJLrVoacL8FmXuJ0DLlcpy1izpxd2ccuML7IHcj/sdhlziTVMB3sCLa+QtkJFfBqTXV93UxFkEviCG1K9pvsKY1DEfox2STRo1lZ1kpYmdwU9sW7ClUOCD7Hw2E3XLfeUHtnD68G/0uApxf2EZ6Ef8hx3FxJmJLd5R7XIOXcYjS69BY5nheaig8ydOsClu9iO9DA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WfaaSBKGONMoj3MJ6ODhPyuqVl4UJ4Hf8UPOZsF7KFQ=; b=Tt96UhJWtAlmBHfgH8FQfH6KzkrOQ4WrBQ1XWtcqVRBNfu6kaBpSmkhDNrC2MOvPf0F8RU5fzo78Jrp8K2QH900l7EFNgzgSk3yXHpj7ioTCFaHrTg5VBHaA/+edYIzkk04UNMIeOhiklqHV38jvFavP/G2EOIICMDJddqU6S2adPb20nb0hk6imS4FKtMlPKzAmHWYn4iHidBRWYQR9fbNDATgP3ntav4rUT3Oo6XHge1fIQDOadYV0YnyB4zJdqWj/1oW69BgWnJ8pw15KdtbZycT9iLCXB3h1pzl50yqEIasXFZi2/LTO8lta2NFbjIb9P8zXUyX1LyIKTr9fZw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WfaaSBKGONMoj3MJ6ODhPyuqVl4UJ4Hf8UPOZsF7KFQ=; b=RPYTlh594RAHOg33LuWkbu+RwpIxZzKxkoRWjWpdlBTrsXivrz7J96piDpPt1zlR32uiVsD+kV09ItSKlPTq15btz0EQEUloNZYQ0s0BJVYKvQsuzkqR2WT3Vf9Wf20RqZX4m9nkJzlpLuGTONYiEuVNm3tJa+sqr+nvBTTjgH16A8XksgmeNA59Vb1ZJgiP4pMFcnLwqeR19nT05AfPCDRikz6ebQwFXeVNd2MJ0z6NZp+mw1i0ygZ2QTZErWlOVWjtVWgUrQ3ATb8AymdXD1sPrN6nb7OAcfXRiSjZweVV9eS2sx8V92IZz+0U50E10MaMtA9tV/v9qFYCiRoKdg== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by VI0PR08MB10941.eurprd08.prod.outlook.com (2603:10a6:800:250::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.7; Fri, 16 Oct 2026 15:26:19 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%7]) with mapi id 15.21.0546.006; Fri, 16 Oct 2026 15:26:19 +0000 From: Andrey Drobyshev Date: Fri, 16 Oct 2026 18:26:19 +0300 Subject: [PATCH QEMU HCI-8.1 v3 21/28] vfio/migration: don't migrate device state during CPR Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261016-vfio_v1-v3-21-47487cb5a8e5@virtuozzo.com> References: <20261016-vfio_v1-v3-0-47487cb5a8e5@virtuozzo.com> In-Reply-To: <20261016-vfio_v1-v3-0-47487cb5a8e5@virtuozzo.com> To: svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com X-Mailer: b4 0.16.0 X-ClientProxiedBy: VI1PR10CA0088.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:803:28::17) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|VI0PR08MB10941:EE_ X-MS-Office365-Filtering-Correlation-Id: 159f6bee-fafd-4330-c5d6-08df2b99d101 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|1800799024|52116014|366016|376014|5113699003|10067099003|22082099003|18002099003|261009223027099003|38350700014|56012099006; X-Microsoft-Antispam-Message-Info: nTDGhwI+2gsWgtkmLO/oNOlqb8VIOv8Vky5wbv+Qyp93ofHSXsmbW1fpHeI/Pldv+CXOCHz5ejs7BKdeEqRHcT7MpTO2qXzycDZzm5zG9IIqBOiDelGNPyicGwXjFHONoZyNzTaH8DA3YFYR03iT00AyHyTNzxk9akcGTyJbMNgndWD34WWAQZrG7P9DEvlWt4PdEfdqhOGHrUBdcsGkkv0b9Ob0AZNvyHq6ZUl6qVX5/zSNXUHs3S9gyggHkDVUl5pGFhggKUYWjUk4cVkfmTJb+Eqa0D6E6c4IJPKh66eZiJuUUWlljjpbXlajxgjnmbkd5B0BPHhb8nssgBGVPEAPGCNINUkdP9MZHqRVha8Ypcps+JoKeIVg48JRrhnwSF1RSR9SiBj0PRI8gN7yHSV5N19pcuuijzgfLrZst60O/myRTh8e15EevOHtS2HVy3NLT9EEfZylpLgBjBtu7QAsmmmwGwPuTfLkbhn0CC6LiWVEWad5PbvjiJdP7zWf4ZI0q/Vv/p+NKDAsg5xHMyujEVQLrJyJjJyqHvsB9PAEWJQ0psaeYBzC3J5y/H4VTUcdkV97/f4+jTE0l/pYvySb+hrJ/9xg9CQ5mTAcL1rlLTtkTbOn5uRWmFJqo0kMOt6k6GQ3eQcoTRiTlKkue4BV+H8pM9IrLt4yaxcSnkeKPGv+Dg/Lek+SCfrD7O/C8tX4+ODzVDlAGzcU1Q/D7gUbrHbUllZp9rWxpqp7+tE= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(1800799024)(52116014)(366016)(376014)(5113699003)(10067099003)(22082099003)(18002099003)(261009223027099003)(38350700014)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UncvNFE5djdNQk9yOGlHWDAycHhBcVpQblBreEYyZGtWRFBENlpBU2QyQm9N?= =?utf-8?B?ZnlSbVhVVWdQMWNZbFljT0RlV1dJKzVia2FxMDJaNi9rVEZZWEt3Mi9WemdW?= =?utf-8?B?ZTlBMlB0S0dQODJieDZSb000RzhpalBTUnF1dHdNR1NnR3M1cWlMcnRQa3A1?= =?utf-8?B?dnRranVSNEtlVXBmdmhIYm5QU1lheHRha1g5TVg5ei92a1c1TVY0bGI2YVFQ?= =?utf-8?B?Snk5R2ZoNzJUZkxzRWN6ZXk4SHNhMDAzcys3RUJ3am5DQ2c1NjRSamt0MTlu?= =?utf-8?B?ZGg5bm4rK1l5UEJscjhpUFozVHZyNE9LUEZXMDRnb3ZEbDEzMWNlamJ1a0dY?= =?utf-8?B?WkJwSldlWmN6T0ovZGs0cGlOdE12eGdiOUJZdTRGQVlNWVR4dDAwV0pXTmFV?= =?utf-8?B?YXQ3c1d4SEhOT1ppcXVlbnRlVC9FM0lMUHRyK25oL1pSQVRucEc2dk03K0lD?= =?utf-8?B?QjNrRkRNeTVYNGgreG9nbmlrM29RNTZpWnY0OU5XR3p3NGVGWkxkRG5oRU8r?= =?utf-8?B?TGdFSXcvd28rZ1ZLSlIybDNMQTNvOE93N25kSWdlVFZPL0pmbWFiOGpkM2Yr?= =?utf-8?B?eWJlOXkweHRTRHhnR21XZHB6bWJWMmhMSjJadGtZK1lSOXhUdGVDQmlsc0I2?= =?utf-8?B?dEN6Y0xIa2crazh3QTgxSUJQMWdnNmE4TDd3UG51N0h0dUxSWUd6azN1YkZs?= =?utf-8?B?cXFjVWs0a1ZxVzN5WHBxZEpWMWhEaHpnM3VSU0lXbHNRWFNlZDg3M1haNFc2?= =?utf-8?B?L3JFazFMUm1QZktMa1JSOTNjbUNRZG1pNDl5QUZBSjFkWFlxdkE4T09NOXVm?= =?utf-8?B?OUx3VXl3WFBkRjdiNXl5TWV5bERVZUJyeEZsamlyWGtSK3Z5REsvdzkrNmRM?= =?utf-8?B?aDdsbmZPOUZrYnVhTVpXOGlEdG14cDY3TEJYUmc0V3MzTTRYV2UyZWYwbHRY?= =?utf-8?B?eHpOelBsdWdFWm02SHYzNFVWZHptNVVmdG5aU1JtSGVMTGlqTllvbDFzMThk?= =?utf-8?B?dGdSUjVPTndhY0UrV21SckI5YlhZY09FNE9YWmE2SGtrUGRDRGhnenl1OXFH?= =?utf-8?B?RHRxYVVaTkZNWVpuY25rbS9HWE96K2w0dkRlQ1dQMnkzVjBwdjFyZFZsejJN?= =?utf-8?B?S2FqMWM1LzdXNE1xOTZtOFBkdzBGcmUyOC9VREc2WExkZ2VZWWZxZXBEYzVB?= =?utf-8?B?aCtlYUJlZ3Q1MnhGK1diOUVPQ0JEbllEQ21VempaZ3dDajdNNTl6ejlIdWJS?= =?utf-8?B?OTBYaHpkMTBrVS9aUG1POCt6enloYkhxVktBYllORUFpL3N6NHRxNFFNQk55?= =?utf-8?B?Q3poSHZhdTFUZkZqbThCM1JENTU1RzlES2V6K2QyMW9pMGwrQVBvN0xtUEkv?= =?utf-8?B?Wmc3YTRtV3MwOWUrcmJwcmJabHRDejl1eE5RR3h1RXZwUjBreXltQk9MUmZo?= =?utf-8?B?SVNiOGJadmJPeW1ZRmNVQWdxT0tpWWlLSG5oUDdnWkFaSnhselFYRkNmYlRh?= =?utf-8?B?U2sxTklaZ0xnMnJwV3plbjY2ZWk2M3AxQnNUR3oxc2pvRG5kdGVYbndhbXFl?= =?utf-8?B?UFVzM2pBTG1HdnZrdFh6eEExckRLdnFnSnRzR3lNT21USjR1NGU3YXNkSDho?= =?utf-8?B?SjlUZkpheUU3cWJRUy9YQ1JkcEZveHErNEVJTUFPS1lVa0gyamRkT1JZaHQw?= =?utf-8?B?VXc1cGFrTEpCMm1ZWU1NdWV6SW9mTFhubDBkNnFEMDd0QkdnRitOYWJJaUNs?= =?utf-8?B?aXZjV3pLKzdEbU8rTEIzVTRVTDZ4TDBrSWI4VktRRmxaZkxQcnNDeEFkWHRp?= =?utf-8?B?cUo2WFNEYXg0QnM1SFNoU2Qxa1VHQkw1VXFYUFppUW1sSkFVT012UFc4UG9W?= =?utf-8?B?S0tabHpJWThiaGV5UlVVV3BiMGZoRzk3UURKdDJ3dktWa1RwSXNHYnZaNVFP?= =?utf-8?B?T3VMUUlKYVk0S0ZIRVQ3MUhrRXgzbmRsNmJ3U1N2aFZhR0J6SnFXYjl4RFNU?= =?utf-8?B?eDlkSU1aU0QwaUxydWVFb2J5QmlOT2U5TU5DcDNFZDl6YUFzeGZpd0Q4QVlv?= =?utf-8?B?MWlNRXpCUU8yVlB2WTBBK3p0UTBtbC9CUDZ6NUhXeTQxaWdTdDBMdG9McEI4?= =?utf-8?B?Yzdmc05vd29ZaHVySkRTR1ZFdlU2blRZRUN4L2prcmZCMkNOUE11eldZYnpj?= =?utf-8?B?U0pPTGxNZmN1bXVyRUFOWGc0M1FvT1dCRjh1R0ZxTXErM0k1S1ZYTG1QcHRP?= =?utf-8?B?M0VqaW1LSGNnRlBOWVEreUJCaEhVaUZMRlNsNVYwbGZhQTVBRXlEMXRyUmN3?= =?utf-8?B?UzBYVW0vN0tqdmxla1hFRGNOM3NYc0FDaTk1L0NzVk9QWTcxSzFDQkRLS2Nq?= =?utf-8?Q?BGtE/ScMK1m/YaqE=3D?= X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 159f6bee-fafd-4330-c5d6-08df2b99d101 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2026 15:26:14.7812 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9qqY01XQXkRlw7fRbfRtBqf8YYjFXbznPXaUk3G3snMYnE7q6kWFNvJ+SRO+d087whBZylEX5R2khd74LmnUh1Pxb/9Nj6ULF25TdwzwsK4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI0PR08MB10941 cpr-transfer and cpr-exec hand the running VFIO device over to the new QEMU as is: the device fd, its mappings and interrupts are preserved, and the device itself is never stopped. But if the device supports VFIO migration, its migration handlers still run. The old QEMU stops the device and saves its state, then the new QEMU puts the very same device into RESUMING and loads that state back. On top of that, vfio_pci_load_config() re-programs MSI-X and disables vector 0. For an NVIDIA vGPU this breaks the device after qemu-update: restoring the state into the live vGPU fails and the guest loses the device. Devices without migration support aren't affected, since no migration handlers are registered for them. Skip VFIO migration while a CPR migration is in progress: don't save or load the device state, and don't change the device state on VM stop/start. All of it is gated on cpr_is_in_progress(), not on the mode alone: the mode may merely be set, and then a plain stop/cont must still change the device state, and a snapshot must still save it. Also reset multifd_transfer in vfio_multifd_cleanup(): it's only set in vfio_multifd_setup(), which CPR now skips, so a stale value from an earlier migration would still let the multifd save thread run. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration-multifd.c | 2 ++ hw/vfio/migration.c | 38 +++++++++++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index e4785031a73..452f772dd7a 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -567,6 +567,8 @@ void vfio_multifd_cleanup(VFIODevice *vbasedev) VFIOMigration *migration = vbasedev->migration; g_clear_pointer(&migration->multifd, vfio_multifd_free); + /* Decided anew by vfio_multifd_setup() of the next migration */ + migration->multifd_transfer = false; } bool vfio_multifd_transfer_supported(void) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 4c06e3db936..b142c9f2289 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -19,6 +19,7 @@ #include "hw/vfio/vfio-device.h" #include "hw/vfio/vfio-migration.h" #include "migration/misc.h" +#include "migration/cpr.h" #include "migration/savevm.h" #include "migration/vmstate.h" #include "migration/qemu-file.h" @@ -513,7 +514,6 @@ static void vfio_save_cleanup(void *opaque) Error *local_err = NULL; int ret; - /* Currently a NOP, done for symmetry with load_cleanup() */ vfio_multifd_cleanup(vbasedev); /* @@ -674,6 +674,16 @@ static void vfio_save_state(QEMUFile *f, void *opaque) Error *local_err = NULL; int ret; + /* + * .is_active doesn't cover the non-iterable stage, so the section is + * still sent: leave it empty, or new QEMU would load the config into + * the preserved device and re-program its MSI-X. + */ + if (cpr_is_in_progress()) { + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + return; + } + if (vfio_multifd_transfer_enabled(vbasedev)) { if (vfio_load_config_after_iter(vbasedev)) { qemu_put_be64(f, VFIO_MIG_FLAG_DEV_CONFIG_LOAD_READY); @@ -810,6 +820,11 @@ static bool vfio_switchover_ack_needed(void *opaque) { VFIODevice *vbasedev = opaque; + /* No device state is sent during CPR, so no ACK to wait for */ + if (cpr_is_in_progress()) { + return false; + } + return vfio_precopy_supported(vbasedev); } @@ -824,7 +839,19 @@ static int vfio_switchover_start(void *opaque) return 0; } +/* + * cpr-transfer and cpr-exec hand the running device over to the new QEMU + * as is: its fds, mappings and interrupts are preserved and it never stops. + * Device state migration must stay out of it, else the device is stopped, + * its state saved and then resumed into the very same device. + */ +static bool vfio_is_active(void *opaque) +{ + return !cpr_is_in_progress(); +} + static const SaveVMHandlers savevm_vfio_handlers = { + .is_active = vfio_is_active, .save_prepare = vfio_save_prepare, .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, @@ -857,6 +884,11 @@ static void vfio_vmstate_change_prepare(void *opaque, bool running, Error *local_err = NULL; int ret; + /* The device is preserved, not a plain stop/cont: keep its state */ + if (cpr_is_in_progress()) { + return; + } + new_state = migration->device_state == VFIO_DEVICE_STATE_PRE_COPY ? VFIO_DEVICE_STATE_PRE_COPY_P2P : VFIO_DEVICE_STATE_RUNNING_P2P; @@ -882,6 +914,10 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) Error *local_err = NULL; int ret; + if (cpr_is_in_progress()) { + return; + } + if (running) { new_state = VFIO_DEVICE_STATE_RUNNING; } else { -- 2.47.1