From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazon11021101.outbound.protection.outlook.com [52.101.70.101]) by lore.virtuozzo.com (Postfix) with ESMTPS id 72E568096E for ; Fri, 9 Oct 2026 14:56:28 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=opFPY49HnOe19ZOkpNY0IyfTVLlTGlBqsy/eA/5Q8pT/AwXZZ4NQSnIjVc5APTi6uycLemwF5ZSNPjW9kUCthz3lCf06qipL8JdRhnmZr06KsNB/NGRQwydKjcV0p23PSiHFaS6L131O59pePChSXybc5Sk1YqL1BqUMLFpvUJ45RBYfe7gmX06RT46ni7yFRdZEfbry+3o9zp8glUPTN0IFDd7nMvkj5C25Oww3cphEBEyfoxCrgeddM1f/R36DYF13q8thL9CtQsYMT9njVIKUMhREdPX5yD1uUsFOdM+0g6A2nAGP+gOrVTWTqQcvQSDS4PA9qpCd8NU3JEzr/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Sp/H4J4DCi9BVSDh4Ge/IefzCTL+qos64X2THNE7ytc=; b=YpeCfIP/p/u/z8teV7VSSX2Gyh8HF88fuliZxY6NdJ65ac7Wwzk2GUTG0d8UM1GFCLTH3uQIV8zZRFmg5KGIgp8wDwMltcvTUrqPdPaWw3FeP6es4Ymie9PoOY7prNblrYar0RIsXHcDhBR+O0vPX1tHbXA0HZodRbelVSQ/dwwA2ErInCzIjxrOnJtK2GIXOKEUWSLxdCuf/6uOtrX/Dc2WpwuS67FQYGZc3Ktvm+BkS+XvOq3lfwLAiXIZS43ofQui4GuKL+SEor7QH/mb5hZ1slRADJivoXIlIGnCpOmiw7M/vhocJdzWvgCJvdQ4Ax6y9dJwVH6/ZfUU6XMJvA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Sp/H4J4DCi9BVSDh4Ge/IefzCTL+qos64X2THNE7ytc=; b=x34e4OKD8VRO3ZV/PuCBPeIMsWtHAatBP/0HIg/R7I4udJ82PCRot+7vkq5lRlosGCzSZaVL8G5E+kKOvbUoF9UJVop0nXGfnQcexGurjyjvuMYFOHqp3YtQwp8Z1uYIgRzYpDikafUfMMNDrxhNR6uZEfAEfjrwGGsapodgVQ5ylGmcLruEdUkopybXw9RbS1V51VdbYxXCoBg1eyxwMxdhNqEpR8/gj2qAfgWIHgmwr2ktApc3BD44o3TjucV+InOjcN2c0cem+u3Hgem2a52I4yOoYhIq4xxaKQLuxOmxSgFmjRTmztf+QYdj0PvCFTtq/XVLx8Q0mmZeFgJgpQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by DB3PR08MB9110.eurprd08.prod.outlook.com (2603:10a6:10:428::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.496.17; Fri, 9 Oct 2026 14:56:26 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%7]) with mapi id 15.21.0496.010; Fri, 9 Oct 2026 14:56:26 +0000 From: Andrey Drobyshev Date: Fri, 09 Oct 2026 17:56:15 +0300 Subject: [PATCH QEMU HCI-8.1 16/21] vfio/migration: don't migrate device state during CPR Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261009-vfio_v1-v1-16-2ae91d5b8048@virtuozzo.com> References: <20261009-vfio_v1-v1-0-2ae91d5b8048@virtuozzo.com> In-Reply-To: <20261009-vfio_v1-v1-0-2ae91d5b8048@virtuozzo.com> To: svt-core@virtuozzo.com Cc: Andrey Drobyshev X-Mailer: b4 0.16.0 X-ClientProxiedBy: VIUP296CA0132.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:34a::14) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|DB3PR08MB9110:EE_ X-MS-Office365-Filtering-Correlation-Id: d03c06e9-1a09-4e15-3676-08df26157580 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|366016|1800799024|52116014|23010399003|56012099006|38350700014|10067099003|22082099003|5113699003|18002099003; X-Microsoft-Antispam-Message-Info: 67/IMrm0haka70RnAPHXvcNPivypQVcDG9tSetn3RAJTHaunn9WBPTsbIJSfg48dO2V9xIb6BHPnEfNdIzl993lN9bp8Fjh5GOqF4F0T8Bgk0LYj+4TR24BersXBmYDEVknqpvR2NRzieuQ9x8k6vjb+ie+LrzQ3Ikw+SnvL4PQfYmjN+TZipmjntNtiSBnAR2O88BpmckaAhb8WIs6MQ2FfgF9clqMMkublyPtXlWMxmePD7tU+pgmS46afVMs1XzE3w5+d9TzUX1oIw+OnwHBJgqHFfOYJ3TL3qzxf97UKa7M0+1Fru3Ih6V/5jBHImPAyV27XSGt2SWE10WZPrCSvx6ndVz4AnfQl61xLOigiT77dzcgpCKnTpeINsvbzheYnXwRsMcAvqIwyEn1/6HfGd4ZAyk8lgvj7oo0mXfmqUgspIe9z32qBp+lllXvZbevqL0z7BvzHRGzGYMfxQRzUc+81mIBIpxeDEULsCW48Yu90xBWTjv7xTHaYy+xpGIrpSSrys1AnHpmxUyRJgD6qEJb5rUeQ+VJgBn8+Z6G5rP+8I7mKzDmBDOXdIP2zBZ45/mCkPwKLNTo8DKMr+Ogj7o1npkJ6te1TsJnG/UdyXh8+PSSe2ohr2ahG7sEdormGpgXm/DSIm7X+0/vdeZEq0j7lotX7Bjp7ZM6okFgH3lG05QhP/zGwmmJPp66BqeKDx3ExpNengIzNqIzqWL4DmVZ6rvffczGGS0D83zI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024)(52116014)(23010399003)(56012099006)(38350700014)(10067099003)(22082099003)(5113699003)(18002099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dnpacGJTa2lvOWdpVm44KzdlMWdIMFkyVjhiTUo5Y0RwQ3FYTFZaZllESTlZ?= =?utf-8?B?OTEyMi82b2RWcVRXWXNzak50SjVwTFplbTM3aURDaGdmdWJwYXhvcEhkSXpO?= =?utf-8?B?bm91RmZRYkRhRWNUdndUMTEza0FUOU1GVGZPeW1haERscm1heHJZZTRLYmth?= =?utf-8?B?UFBtQlR2dzNFcEJhNjRlNUM4dEFaZ2VwWGZYcFQ5MGFOZE4yQ0hsRVNzTEgx?= =?utf-8?B?N0YzS1licFpKMG1Ca1ZuMXpCdHg2YUFES01xKzFxeDFYMDZ1VGhza3llQlQr?= =?utf-8?B?VzFSMFlJT1NPdzcwVTJsVmtDZmF2Uklxa05lUmpBR1h5MVYrRE9sRGt6ZEJ6?= =?utf-8?B?RnlRSUhtamM2bXZiSVpabVBmQkh5Mjh6UWpMZjBFWUVEc1lZV2l5OGRSSmNq?= =?utf-8?B?V2VubHlwdDViOXZJekJGTnVmSyt5Mm9Xei9KeDRtQnE5dEFsd1NQdWorYmEz?= =?utf-8?B?N0twVkFJWkVudWtUNzBBUXdKUWN6akI0TVprRnMrNVZMRGhCaDhmbG51b1kw?= =?utf-8?B?RFhjd2FXdDJPTU00WDRma0pxbzM5RWFJTDdUSDIvUVROSll4U1NoR0I3d2tX?= =?utf-8?B?UWhnYmFOcVI0V1gvYXlSTEVlRTVYb0s1cmovSHR5eWRnMUI1K0kydVRWMmZR?= =?utf-8?B?VzBVV2tJM0Q5dVNlQjJmb2VObytYRU9WNDk2R2NwazJ5dXdaQUlENzE2NXhm?= =?utf-8?B?S0JER0xsMEgrUEFQSG1FbjNFWFFvdkpaai81VlB0VzNWM0hQWkdYQjY2azIr?= =?utf-8?B?cENOdUpkMUU2SjdqTnB4ejVQK2tBQzc1ZDF4ZmRzS2tQMHdTeStnUE9SamFT?= =?utf-8?B?WHRnTGhwWlhhVWxrRmFLZUZMUi9QNU13bCtmd2JsNmkzaW4xSU5zcUo1SDZG?= =?utf-8?B?dWxCL0xEWktwR0ZySkRWZlgzTi91RFRxVlM1WnBzU1RtUXB1aFhJdXJzeE5P?= =?utf-8?B?RWpqdkF5RkE2Z0JlK0tIanNwVnlxM29GVnExRm9EZlI1V3FjM244eHlFSEd3?= =?utf-8?B?RTIraDV6NEZlSkZJQmw4TFAvdUovTEh1UnU1TjArWndoK3ZPWGlMRFkyeHBW?= =?utf-8?B?dU9vQ0cvazVDSDZzbXNBRm9lMlY3SUJ1VFBhR0FSbk9NTDEzQnV5NlNMK01u?= =?utf-8?B?QVVjaERHSDNwZUt0eW1RUE45T1BYTkdZSnZEQjB1aTVpTnJuamIvbVVadzhl?= =?utf-8?B?S2J0SzNVVWRrZ3VNOTN4TEwvWFYvWll4aW1jend0enloeUZoU1dJTWIzOWVP?= =?utf-8?B?ZWg4emNqa01jaHNIQ2JhQ0R5U3N3RmsrbnZ0WVozMUM0bjVjVVJmcDBSYmNO?= =?utf-8?B?cEhCY0Y5dVM4eGFTb1ZpY2pOT3NNNGQ0cFdNemhvemhGSGNROTY2YStyOERK?= =?utf-8?B?QUloVnZic1c1Z2NoMmdQQlVlNTRRaDRBV2I3bjIydmRqbnZRRjFXQzJhMlR3?= =?utf-8?B?Vis5dDRkL2JMek9Ha2FtK2dCTEgyOTUrMDBiL2FpSW5oWmVaTlMyaXd3WDNQ?= =?utf-8?B?Mm9pY3JsbDRjQXJCblpyOWRCWDRwc0NYYXBURVJJc1VQV3JRVm5BTmxWaERt?= =?utf-8?B?b3drdVM5RmM3dWZaVm13VHR4T1JaeHl6N1NieXZuMVkvbENCVisrbkJEN2Nl?= =?utf-8?B?eC81Wml3VzEzaU5XQWF4MEwwREI1bjZWdG5meklTYjVtWUhXdFc5NEdIMWpO?= =?utf-8?B?SFFYQ3hEQlNmZVJ6WXdZYnJRNnk5dHlvMVR1Z0ZCN3pVQm5RTnRFU3krcDZO?= =?utf-8?B?WGdObExxMTJJamthM0dmd21MU2tlUXRyRHMwWVZpUnJsUmYvQzN1RXZqVEtS?= =?utf-8?B?N3A2aUpRMVBzbTZUeUVSSmdsUjVlOXJFTWdDd3ZsZkVoeGlrUjhXU2tCTGU3?= =?utf-8?B?cWpwQXBOQmw3NnF3ZkdRZmdTYWJtNVlBR1hzQzRFWnpMMEdaOXpkRXdlTFhl?= =?utf-8?B?Z3huVWJYYlFYWnRpZEdKOUJwUzNEUzBZRGFyUFhiTk1GNUNyRzBTN0lSc1BL?= =?utf-8?B?S3JkSU9td3lJSFcrTXdSOVZVSndPdVNwZzJ3QzJwYnlRTzJqcHhjV2NjOU1M?= =?utf-8?B?T1pDY1ZUdmM2UFV2ajJLTG9rWFJObUNuQis0NVB4RVR3TFhoc2N3LzZ0c1JK?= =?utf-8?B?R2hFVmhJT1hoVWkyNEtoVTBMN0c1Nll4U1dNSkR6d2RsSlJycXdyUmFBTzd1?= =?utf-8?B?Z0phZzV5OHE2MUt0OEZKZWtBS1BHQ0hMOTJ2ZXJkOEpjc3NLNXI2M05hdWkr?= =?utf-8?B?RkFndmhoZHp4RzdYOE1rVzBYMitUQTBpd0ZyNjZyYnFBZERJTkk2b05ERVNn?= =?utf-8?B?S0ljL0t4U3NXNC9MS1Q4cHJQUC9XNXMvUzdiQ2ZtcDJTSmZ2WEEyeVBLTmlD?= =?utf-8?Q?wWDWN412qnibNAt4=3D?= X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: d03c06e9-1a09-4e15-3676-08df26157580 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2026 14:56:11.8413 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ayuN3FwtEu34+/MogWWvY8D2YIkr5F/EZu5OGCWyzlSwtcucBe8vOn6zo6oyNp+n/u7sZq1/PYeHRHa2eGO2hNpt6Hq48PAJDWfsAT2BMlk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR08MB9110 cpr-transfer and cpr-exec hand the running VFIO device over to the new QEMU as is: the device fd, its mappings and interrupts are preserved, and the device itself is never stopped. But if the device supports VFIO migration, its migration handlers still run. The old QEMU stops the device and saves its state, then the new QEMU puts the very same device into RESUMING and loads that state back. On top of that, vfio_pci_load_config() re-programs MSI-X and disables vector 0. For an NVIDIA vGPU this breaks the device after qemu-update: restoring the state into the live vGPU fails and the guest loses the device. Devices without migration support aren't affected, since no migration handlers are registered for them. Skip VFIO migration in CPR modes: don't save or load the device state, and don't change the device state on VM stop/start. Use cpr_incoming_needed(), which tells exactly when the device is preserved by CPR, both on the old and the new QEMU side. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 4c06e3db936..99d285e7b20 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -19,6 +19,7 @@ #include "hw/vfio/vfio-device.h" #include "hw/vfio/vfio-migration.h" #include "migration/misc.h" +#include "migration/cpr.h" #include "migration/savevm.h" #include "migration/vmstate.h" #include "migration/qemu-file.h" @@ -674,6 +675,16 @@ static void vfio_save_state(QEMUFile *f, void *opaque) Error *local_err = NULL; int ret; + /* + * .is_active doesn't cover the non-iterable stage, so the section is + * still sent: leave it empty, or new QEMU would load the config into + * the preserved device and re-program its MSI-X. + */ + if (cpr_incoming_needed(NULL)) { + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + return; + } + if (vfio_multifd_transfer_enabled(vbasedev)) { if (vfio_load_config_after_iter(vbasedev)) { qemu_put_be64(f, VFIO_MIG_FLAG_DEV_CONFIG_LOAD_READY); @@ -824,7 +835,19 @@ static int vfio_switchover_start(void *opaque) return 0; } +/* + * cpr-transfer and cpr-exec hand the running device over to the new QEMU + * as is: its fds, mappings and interrupts are preserved and it never stops. + * Device state migration must stay out of it, else the device is stopped, + * its state saved and then resumed into the very same device. + */ +static bool vfio_is_active(void *opaque) +{ + return !cpr_incoming_needed(NULL); +} + static const SaveVMHandlers savevm_vfio_handlers = { + .is_active = vfio_is_active, .save_prepare = vfio_save_prepare, .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, @@ -857,6 +880,10 @@ static void vfio_vmstate_change_prepare(void *opaque, bool running, Error *local_err = NULL; int ret; + if (cpr_incoming_needed(NULL)) { + return; + } + new_state = migration->device_state == VFIO_DEVICE_STATE_PRE_COPY ? VFIO_DEVICE_STATE_PRE_COPY_P2P : VFIO_DEVICE_STATE_RUNNING_P2P; @@ -882,6 +909,10 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) Error *local_err = NULL; int ret; + if (cpr_incoming_needed(NULL)) { + return; + } + if (running) { new_state = VFIO_DEVICE_STATE_RUNNING; } else { -- 2.47.1