From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazon11020121.outbound.protection.outlook.com [52.101.69.121]) by lore.virtuozzo.com (Postfix) with ESMTPS id E387F80A9D for ; Tue, 13 Oct 2026 15:28:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=C+8Jd1vsvQAYBcq9MW2CAGWex6CZH1oqg/JVw817j8QyKEtstenV1ugAi8/axGv67NUNTSOWlsg3xMdqKAuvRbT9rQ+1u6LaekD8ix95tJbDymwPGMxTLzleaz0Qbmqee8Y5yA1PXdBsTXHZ4pUrhHYR020v2Ac+kQE7kWykp02FFJDSVO37srVBHDkcrrx1eQ9Ys7H8PFxSifNfb4byn0HiBdH+vBAwh2OfBuq0iBK120PMH8DNboFniUC/dmph+OJHc7s9QUEta9Aa/iYweZ6XKrAqop9+AkdVbc/XD78Fyy5dP82jlmv3oJYuPnk3C+HChxD94N1BuGv0cKB64g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gkBgVo0BpguP1FxYFlG9YXZ9cYG24RZi2XdHdfaa0Es=; b=TqVhH9HWetc4hIGed8QhGx6eF9N6vKX4xFONvgCH8cNLFsftyeSx8sEY4SiU9BiDgi7JZlGTVPzTJ2YZTcHbqkvFVPKIZwkaNOB/Khp36qrCrXOvwvMrHLYza4qkIG85GN+BpM2ucvHz90Qu+02EcoYG9CtCqP712UQ2mZr/Iq5fzvgDHKtGmrEjXhiQeoOUphH8B4b0GEi/y2hazwy7PhFVAZh53jLLE86NHyOzuCr0Uxx1s7xmkQZOD1dD5M5/mVfjrYlAGpeFrACcg8eYkbXexLYH6EalRgOALgqFCfT9JABMSI+Qxp/zJOIs7uvBKoi9LcuG7RkBg3nRx2h1pg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gkBgVo0BpguP1FxYFlG9YXZ9cYG24RZi2XdHdfaa0Es=; b=Dq4dWg6g6MQ0/bhmKYT6+sMBpPBokx+M2/j6epkQejGE8MFZq9teqrEeERdW89BRsk8oFxv4XH1ohlpQml5pi560IFIvp4AxdlwKKg2TXarFwVIETWKumH+BAE0iqnlGiDkXesUOglO7GNhyf1f2roBbvP4PJliptrQ4pMAA9zs2DrQM7wBcMcVnFiDkNwcdE9pUoW5ZWQGz4rO83IzeBlS9NrPpf8EtibcXN9v3QAUPS4lyJtlUUOW7aEsrpeY7m/JFAANsjohShsapQ4/PrkbYFvKpzaEPZegaaXtb+4nw8G4u7k4almTLvHUhRvfAL2z5RB9HGz13QwKOa8xtxg== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by WA4PR08MB366192.eurprd08.prod.outlook.com (2603:10a6:1d0:30::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.522.12; Tue, 13 Oct 2026 15:28:16 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%7]) with mapi id 15.21.0522.008; Tue, 13 Oct 2026 15:28:16 +0000 From: Andrey Drobyshev Date: Tue, 13 Oct 2026 18:28:22 +0300 Subject: [PATCH QEMU HCI-8.1 v2 18/24] vfio/migration: don't migrate device state during CPR Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261013-vfio_v1-v2-18-433455c0a294@virtuozzo.com> References: <20261013-vfio_v1-v2-0-433455c0a294@virtuozzo.com> In-Reply-To: <20261013-vfio_v1-v2-0-433455c0a294@virtuozzo.com> To: svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com X-Mailer: b4 0.16.0 X-ClientProxiedBy: VI5PEPF00000939.AUTP296.PROD.OUTLOOK.COM (2603:10a6:808:1::833) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|WA4PR08MB366192:EE_ X-MS-Office365-Filtering-Correlation-Id: fb585f85-ac03-441b-689b-08df293e9a02 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|1800799024|366016|52116014|5113699003|18002099003|22082099003|10067099003|261009223027099003|56012099006|38350700014; X-Microsoft-Antispam-Message-Info: dCG77sagKYnKwOVrXyWI4i6C1WVPvI6W3RLPthptoEqtapTl7Uf6cifRTE+3CD+NXqJdI87ZG6RlbO7NpOf7r4+L0vc3bo9XBazAAoInPhom+FAg2NDn6O+D489aXYw9GikP51iBCx9/0vQd1fbzKHdoGaZi5d9dEZySQpfxf1Pjz+8s4GXjFGVVNOVzsbPInvcZ/VAqBdwb883vjbSAaIQPR/aF99kK/2UNbNc64oynqru2a+z6LbWOR8aXdAsj782t0S+jCZSGMjB3RQArMyq0Q9kYqBJcbHUNL+x1F0uIVgxV4WmnCQwywkuSdV9wG4p2GZNXmPQPYvDL0mH01Rz5yLjo4ZrSoRFU79m5MKrlDYuv4SHuyfkUw5htSHBXRhDcrRrIsfIy0Kuy6KBHmJq0wzMI0slnTusR34iEyhJeOvqEl5ZeTCvxOiS+ALiXbm02kNbKieA7CrMKeIpDSNWZn9mTB2xomypYA8Oppt23FJO7u/8KOCpz8lm8NZLrLg5zOn0+2uqKBLn+YSXRWVwY9XKmpTEVTeBbKzI7uFpHjCwNtfP5nOOvmZtBCytpPDoBO+hYaa4m+wNw6B+GCIP9mz6CYAB3rpeZbqt/UTUNaAbJnCctppzfBm9Nryhuor2KxBEpu4ODGX5wc+5n3ZtoOwhvH3W0llVtGR7EedinyrHIH+nxQbvjEF+OlisacLj5FfmN1TpN+Icg/ydB1xULBJUB7XSQrXtrJtV+rkQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(1800799024)(366016)(52116014)(5113699003)(18002099003)(22082099003)(10067099003)(261009223027099003)(56012099006)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WWtNZXlUVTh1bG5wNFFKWGs1ZnIyZGhUYVNEOTBOZHZGUk55Qlh5bk8yUGZ3?= =?utf-8?B?OXMyVXRGeVpmMjVheUlFT2o3VmhPVzFWQlNnL2RhUXdGQlFRMjJrazRQVXhV?= =?utf-8?B?aDMzQjk2TUdQb0FUb3VRT09oZGFpRVpsSlNPQ2I5UG1BQVVXdENFYmNQcURS?= =?utf-8?B?UnNkUXp6WmJxcDVoc2ZsN1BSQkFoeUNOWEtvQ2pLbFlJRjBXdWx0dGZKM09I?= =?utf-8?B?K2RVS0ZqRkZGODNBVHNXZ2wxU0N0L0ZvZ1JMSk0yaHYyblJKZDlqcUVxWXpZ?= =?utf-8?B?Zk0xbjJrb2FhV0RuWVkxeDBNUURrdXg3QkoyNGFlK2FPdExpOVlpSnB4cEpH?= =?utf-8?B?NEdlQXUxVWRpczNjMzVPSEI4Smh5cDVPZkJoN0NVZExvdEFzajdVK0g1a2ZY?= =?utf-8?B?Z1BqQmRuWWhLSXdHRmc1QmRHdjkxdVViSDRWaUJaRjNIWWZ6NU9SSC9qWG1J?= =?utf-8?B?L3NBRzQ5RGQ0b25GNWRYbm5LaThWR2FBcmh6MjhpWVpKZ2psS0prREN4NCtJ?= =?utf-8?B?aTBLdDVKNkNpU0RjMEtIZnZ3Nmx3YW1NeDlWNHZaVWhWLzVmWDVNejNXRTQ3?= =?utf-8?B?NFpJSDlWM1N0OXhPK2UwaU1nRG80WHErZHphSUpadXdCSjdUbE9acjRVV2hW?= =?utf-8?B?TVFsc1FwYTRNSzlPSXpPN1IrVm9veWFLdWYxcisxelZXeWR3UkxoZEVwZFlD?= =?utf-8?B?Tm5OM3YwV0JBaGtPWTUrZkdkYVBNT05XeStoRS9tb0RIMFlpMVBqODdjY2Iv?= =?utf-8?B?Z3NoWjZRMStYM0hONUlJeGJWNURvMXJPRlBMc1hhaUpoOWdlTGs4REVITk1u?= =?utf-8?B?cGQ5aXJLM2RoeWl6N2FTUDIwWXJaK3FsTHVWZjkxUnhRTVpRVE1UNml0WXg0?= =?utf-8?B?MUVJbUxGenAvVXcwMEt3SVpNcDlRaE55MVkrVDJsVXFKM09ySGZYNzFUSmxG?= =?utf-8?B?djQvNmNsNFZZeGtuSU4vQW5kZWxlZUhYOUpabkdOKzRMaTIwQmhaek0xTmYw?= =?utf-8?B?ejBPc2xmVURRcmpvZjhjaUV4RC9oMCtpSFZydTdmZlJXRWVOYVhrbWJSWjd0?= =?utf-8?B?b0hvZ2wxQVoxQlZjS1IxTXR1dXN2eGRuTitTU1l3N2JFZmtJZVd3MG9Mc1BS?= =?utf-8?B?Y1FVQWorbTZ4SStoUjM1YlBKaUJ5ayt3VXNpYWRYbTd5VmpyWjBCWkpmY0hq?= =?utf-8?B?b3ZLUEwxTXFUbURKMTBxZ3BGVjNFdGJnSmk1RCtIRjVqQ0JYd3dNWUVyYnFS?= =?utf-8?B?VndVeDdOY01CTmFwbm5YMVVKZDllRHdZWkdFaWR5MzFxRFExR21ROWhpVFhi?= =?utf-8?B?Zkw2WnQvLys1SForeHN6cTljKys3MThjWkVsdTYwRWpoTGVQQTdQNlkxOWla?= =?utf-8?B?c1pXY2lvd01xVksxNGV4dzB2NkpWdkdJTnpFWHZlL0FFKzV6U05jaDZKQUZi?= =?utf-8?B?OHQ4SG5wT1VVQVBQNGU4RFZuTGVqMVM4b2FoOG5GelJDL1VqVDB4bXE4di9a?= =?utf-8?B?QjQ4OTNIYXFtNGtrWGRDSmZNSmZEYnZMRXZ4R28wa21HTTN2dnVmeU53bmxF?= =?utf-8?B?eXhEcElUeTVIOEdjcmlmdmNuZDg4czRaVUkyOXBFWXBRUlNybDVUTkUzY21Y?= =?utf-8?B?djZjaVRrZ3Z1ejd5QzZuWkRVNVVkT01oUjJWNUdzY2liTWUvODBFMndmdjFJ?= =?utf-8?B?dlhtenl0R1dDVjBQM1oxMXE0TE9zZ2tiTzE1bThhL2RmQVlZcXBTdWtvRDR4?= =?utf-8?B?eHFnTXd0cFdHZ0loTG5uY1hKQlJPaWJ6MS9QbGh3NUdhQnhweDMydDdaRDNB?= =?utf-8?B?UzV1NnRyQ3hYZ2xsMWhPck1lekN6cjF5R0VJSVQ0VWRsRU1HbnR5YmhYRmZV?= =?utf-8?B?SStFMmRmU3M3dy9LRkNHNmF1bWNGZlo1bDlxM3hCby9vQzE2dzdkRWlVZGxw?= =?utf-8?B?endlVVVvZHZNUGhrdEo5Uk4wN3o2MytKU1NFSlBrVm40WXVYcUVETVZVUXJO?= =?utf-8?B?czAxNGp1eTFKWG80bUtqQVNTakVDV2tCTHJpY2ZrTUYyWFE5U0dJSTlZMHNy?= =?utf-8?B?dGplYTNBUlQ0QkRzeXNVaWN5MW84N0g0bVNlN0NOcnkxU1F6TEpFSXZ0dytk?= =?utf-8?B?MnVUeFBud0ZPejZabkNlcnBVSGY1ZHNMSXJGa1l1Zjk3d2VQaG53WWVzaCsy?= =?utf-8?B?eURBcUJUeDlOc1ZIck1lM1l5RlFjeHlXNGFLcjZ6NFNrdFVuSVhteTA3QUpw?= =?utf-8?B?SHdIRWg2QkNOYitXZXlKUExSc3d4c2x0UittSlVLZmtCMFBnY3g0S095SDdR?= =?utf-8?B?WTNYZHBlMHk3U1FGbTh1eng5OUFmZG1RNkhNRUhpQlkwU0l2WUJVL25aQWIr?= =?utf-8?Q?fBGoAk6WmGgYJQJw=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb585f85-ac03-441b-689b-08df293e9a02 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2026 15:28:15.9538 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xJC+Lilj2igvdvIEHZUcnCBstJ5IKowuad3ErfEsljOeAdMi10miwbeaZ4269T8odjdiAnKdkDlN7Z3E4u0TnsC0RLXuYIemneEhB62VcSo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: WA4PR08MB366192 cpr-transfer and cpr-exec hand the running VFIO device over to the new QEMU as is: the device fd, its mappings and interrupts are preserved, and the device itself is never stopped. But if the device supports VFIO migration, its migration handlers still run. The old QEMU stops the device and saves its state, then the new QEMU puts the very same device into RESUMING and loads that state back. On top of that, vfio_pci_load_config() re-programs MSI-X and disables vector 0. For an NVIDIA vGPU this breaks the device after qemu-update: restoring the state into the live vGPU fails and the guest loses the device. Devices without migration support aren't affected, since no migration handlers are registered for them. Skip VFIO migration in CPR modes: don't save or load the device state, and don't change the device state on VM stop/start. The former is gated on cpr_incoming_needed(), as it only runs during a save or load. The latter runs on any stop/cont, so it is gated on cpr_is_in_progress() instead: the mode alone would also skip a plain stop/cont of a VM which merely has it set. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 4c06e3db936..23e083853da 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -19,6 +19,7 @@ #include "hw/vfio/vfio-device.h" #include "hw/vfio/vfio-migration.h" #include "migration/misc.h" +#include "migration/cpr.h" #include "migration/savevm.h" #include "migration/vmstate.h" #include "migration/qemu-file.h" @@ -674,6 +675,16 @@ static void vfio_save_state(QEMUFile *f, void *opaque) Error *local_err = NULL; int ret; + /* + * .is_active doesn't cover the non-iterable stage, so the section is + * still sent: leave it empty, or new QEMU would load the config into + * the preserved device and re-program its MSI-X. + */ + if (cpr_incoming_needed(NULL)) { + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + return; + } + if (vfio_multifd_transfer_enabled(vbasedev)) { if (vfio_load_config_after_iter(vbasedev)) { qemu_put_be64(f, VFIO_MIG_FLAG_DEV_CONFIG_LOAD_READY); @@ -824,7 +835,19 @@ static int vfio_switchover_start(void *opaque) return 0; } +/* + * cpr-transfer and cpr-exec hand the running device over to the new QEMU + * as is: its fds, mappings and interrupts are preserved and it never stops. + * Device state migration must stay out of it, else the device is stopped, + * its state saved and then resumed into the very same device. + */ +static bool vfio_is_active(void *opaque) +{ + return !cpr_incoming_needed(NULL); +} + static const SaveVMHandlers savevm_vfio_handlers = { + .is_active = vfio_is_active, .save_prepare = vfio_save_prepare, .save_setup = vfio_save_setup, .save_cleanup = vfio_save_cleanup, @@ -857,6 +880,11 @@ static void vfio_vmstate_change_prepare(void *opaque, bool running, Error *local_err = NULL; int ret; + /* The device is preserved, not a plain stop/cont: keep its state */ + if (cpr_is_in_progress()) { + return; + } + new_state = migration->device_state == VFIO_DEVICE_STATE_PRE_COPY ? VFIO_DEVICE_STATE_PRE_COPY_P2P : VFIO_DEVICE_STATE_RUNNING_P2P; @@ -882,6 +910,10 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) Error *local_err = NULL; int ret; + if (cpr_is_in_progress()) { + return; + } + if (running) { new_state = VFIO_DEVICE_STATE_RUNNING; } else { -- 2.47.1