From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazon11021137.outbound.protection.outlook.com [52.101.70.137]) by lore.virtuozzo.com (Postfix) with ESMTPS id 95FAD8017D for ; Tue, 20 Oct 2026 17:01:10 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DFwFZHCY/N1+byT69Qt5LqMY2Nt/rRN0fcx7oiJCRFwWFmP3aFHvlmroudrTutIUfRUBfDebIvo+cEeYWdExvGcpBhF17QmjjPKjOcvCR9Dgd7NFpJv4dKGIMvPmf4imANbULTmXKOKUiGqcWtkS3GmA2tp5b5BYuaeUJVp0d/IxjggjMTujIFJkIyqWn8/QU5xVEqndUEmtHVIqynrzO6zwKVrJYEpeNij1qVCOjZr/nW568U3uU2QsqrHGvXp3Nm8QzFPYdnzOiBMJxf30KeJ3KH+NgRQIHwYvTG+Hzggf1yz+w8mAwSBDhR+AOYkFULZu5CNwQMp8gXe2t2x8Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Stdu9oF42lH92E8s9agNGF5yFoWDw1fFOlAOP9cswMo=; b=FYnCzx2xQsQqifT1H5eUBJOutbooHjL2QvO8Lc029h+6cFd871NI7bJxgQRI//utOxXUka6EJFi6IJHhZ4GVdkB/+D10Ms0bdBxad7YUXqvTPDKIOVLB1lMknpeSob2+eD2+my71nHuHUGZuhzK/koun93gxrreKTXxZl2UGkOYZRKQTM2YOBsr3YBh24wf5gX7zYgb0+CkKdbOQ6qGR2B55sEsMGh3+2EDvPLF5bxg4JkGpi6IhOUFAgIgl6o3PbMvRqXQ7tCWG6TdnSGoYO3DwMeNWdcmcbQ0DNNj7HoWeTNacZX8oWjmcdyaOyUrSJ6mUr/UcUqZQ+JQE4mHf/A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.2) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Stdu9oF42lH92E8s9agNGF5yFoWDw1fFOlAOP9cswMo=; b=To6qC4N8bvhkHlBxPalsBrLv/sapor3JtJE3a3Ov0EUmVv+uwTzAQjePt7HXhWKBBYiwbEcqVFL1QuU51g5r+2cLDPDfQUfPSNM5rFtleqchiQQD87Y8/quNtXzLFXHId3P/UsQqfOdYNmsyq9W9xjJE6gsqvSfYZCen7xkre9o= Received: from MM0P280CA0022.SWEP280.PROD.OUTLOOK.COM (2603:10a6:190:a::7) by AS8PR08MB6005.eurprd08.prod.outlook.com (2603:10a6:20b:298::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.15; Tue, 20 Oct 2026 17:00:59 +0000 Received: from CPH1EPF00000524.eurprd05.prod.outlook.com (2603:10a6:190:a:cafe::a3) by MM0P280CA0022.outlook.office365.com (2603:10a6:190:a::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.522.20 via Frontend Transport; Tue, 20 Oct 2026 17:00:59 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.2) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.2 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.2; helo=b224-2.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-2.smtp-out.eu-central-1.amazonses.com (69.169.224.2) by CPH1EPF00000524.mail.protection.outlook.com (10.167.241.135) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.546.15 via Frontend Transport; Tue, 20 Oct 2026 17:00:59 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1792515659; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=5CTn6ae6ZNUo12yI5FiIEADpggwBfWmIpxmo1FwtcAk=; b=WrVWmBBseumn4i8ajYry+mDVXNXBGqm7lnPzRtVdXo4hF+jhfI47OTqxopfrw1Es 7kX+Ear7Z7Llx9BWfYLVITgSQeUzxioR3ro692/UwT1C9U8T+NdGHu7sqUFtgKqL+ty ex0SNQKZFa/LjkH3EDoDyu18YQsu3ShYFD2oo4knCuP8G/Wp9Y+HSiywArQmRTJ7G1/ RXB4Wn6Ehje4POR4InWPfGs33+GBYSQssAxMPAIxWhMr8b43+mM7bcFvNBbZnUHZaN7 RbNh9ZjosIrlm71KaN4UoAqlmBoHtUJV7E+l9FhB3qsWSVotfOpg4zejQ5INgX4A3/z lfjr0HpdwA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1792515659; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=5CTn6ae6ZNUo12yI5FiIEADpggwBfWmIpxmo1FwtcAk=; b=Ir81BlqzCgRl0fRcbyOw8/e5PMuQhVnpHEhRJD0/tBhKCUeAq95ICYSVnzVOAnKr 7P6bMmG6wq1WAl1kwY7FMDVR9k0+u8CJe3zGVgS/UFZM+j7N3AhYPGGXU7TonGHq79m pAW/9pqrUS+6fodBgP9CiU+cN2EtawqUyZNL8dJM= Message-ID: <010701a15a425549-b889bb9a-81bd-4807-8f0b-8168e3f89af0-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v4 23/31] vfio/migration: don't migrate device state during CPR To: Andrey Drobyshev , svt-core@virtuozzo.com In-Reply-To: <20261020-vfio_v1-v4-23-83a5668230e5@virtuozzo.com> References: <20261020-vfio_v1-v4-0-83a5668230e5@virtuozzo.com> <20261020-vfio_v1-v4-23-83a5668230e5@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Tue, 20 Oct 2026 17:00:59 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.20-69.169.224.2 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CPH1EPF00000524:EE_|AS8PR08MB6005:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 95a1001d-a18f-4037-9c7d-08df2ecbb72c X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|48200799018|9140799003|61400799027|376014|32142699015|5063699009|56012099006|5023799004|4143699003|261009223027099003|10067099003|16102099003|55112099003|6133799003|4113699003|5113699003|18002099003|22082099003|13003099007; X-Microsoft-Antispam-Message-Info: OV+ucpJ3BdmC/1tKPnURMs9iMmx0yVibLrOesKsL4xAgOJf/MZ+0iqC+02o5tWduZPj/efG4NFXYjwfE5P79+PspYoQTs1lu7SylIn06u/af9VdUuwmY7xpM1tTcJo0oQblTAAInbj5DtxlPRRrT1bKBD5rlPq/wPI8ZZYRd1YMKMZbii4S480BBAtYfaJBSuf3Or80ULSIRAA5McLb0Av3VUp1i4DaBThClTUR+n+btIieYuqMVj/+UEhexxFnq+/ffMXbd2an9y02MWHxkfmL2my1/S19Bi4Cz9iB5MboYgDQD0JrBxsPW1nDO56g/8dPsx/MYAWzhdJp96a4+3s5C3aIojATFBB6REZdERmLl3PTtM5kRBQUPOpBFDAPvTHLoYl16FSQjhF2FsnaufkVWdqzUTmsjl0hbt6S3Y58VhonCFJi5yIVNnlb138JcywnLVRuJyl8Ea1CE2qT5MCESAZdKhvo3U/XNe33xG1z+2t6URBMWZ4URnvFB4bQ5bwNur1tOUIx7py0LZ5LFpftXTD9Hfc5qGNhdEmnwZ5Vu+6k7KFgYTJFlUPRJzwCunuqzSZP9MyInYaaGxjXp/Xm8srtNuh6CFXyKTYJJ4vAgmf/YPa2LuPtQvjxRVpUD1Fuke921h9HvDLCQQhESpWslCTd9DM9PGVfh2scLjjbBbGBJ48oK/9snSZx1ScB+vGQe4iH8BcgZr+IPQ9vNKmH7uSPCPG5bwWVbkkNCJfWmA5uwlYRM7yMHNPmbEV1GIJS7/CWqtPGnIhGZUKwLk53RGwbmKnN1zFTIinFnySoU2dAbYQ8VXTUMM1W/0Ts7aZOFJQIh3TPTcc+iXz1sykzU74DjHKusCiO3iM1oZzjfhZj5tRRTs+SQrFbhZFtIYL2CXrrp/ImYaGFaM86iCw== X-Forefront-Antispam-Report: CIP:69.169.224.2;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-2.smtp-out.eu-central-1.amazonses.com;PTR:b224-2.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(48200799018)(9140799003)(61400799027)(376014)(32142699015)(5063699009)(56012099006)(5023799004)(4143699003)(261009223027099003)(10067099003)(16102099003)(55112099003)(6133799003)(4113699003)(5113699003)(18002099003)(22082099003)(13003099007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9yuqaFGcEZXwYFtjESzi8rQ+FQQ1uyxUbCwXIkw16DqMYq7yyiZnArLcqa5SyTD/QhoihTOo/gNyU1bSRgSjULAVD1RccO1vmUxhHhuGDiOg7qKLKXV4JI6z3IhNKo1oPt0DwXtsPIXbXKzurYcD8yBQA4DY+maBAZ2Zwnx9f4kA8aEB7iWlNdlczGuIlR/yVcC1TwGOvR0QeKPe4FjNp0udQasHgNp45cNuhrMrDayB8FGfN5m5YY0iX6jwq32egtkdamjIeA4ZwH7F8w6r/j9G2bpsqTm/R60tC2p6luLq/dD5/GFSObyK5K6BrKlBf/0N6Zj1E+IboV61WY2X5cl2Yks7d3VsglPqgE42Yt6VxYDpSeHagazN7zQxfDSLLRMJa4DqOC1pYAVxwzGJH+qoTeNDbI2ehW8vT6oOzhch3MiSvFc1mRTc3hcn3JPW X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2026 17:00:59.5482 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 95a1001d-a18f-4037-9c7d-08df2ecbb72c X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: CPH1EPF00000524.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB6005 Thank you for your contribution! Sashiko AI review found 2 potential issue(= s) to consider: - [High] vfio/migration: cpr_is_in_progress() is false on cpr-transfer sour= ce, gates ineffective - [Medium] vfio/migration: cached device_state goes stale when a paused VM = is CPR-transferred -- commit ef9b082e80c829a57ff650c4bacfd6d71dd9e216 Author: Andrey Drobyshev vfio/migration: don't migrate device state during CPR This patch makes the VFIO migration handlers skip saving and loading device state, and skip device state changes on VM stop/start, while cpr_is_in_progress() is true. It also resets multifd_transfer in vfio_multifd_cleanup(). > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > @@ -674,6 +674,16 @@ static void vfio_save_state(QEMUFile *f, void *opaqu= e) > Error *local_err =3D NULL; > int ret; > =20 > + /* > + * .is_active doesn't cover the non-iterable stage, so the section is > + * still sent: leave it empty, or new QEMU would load the config into > + * the preserved device and re-program its MSI-X. > + */ > + if (cpr_is_in_progress()) { > + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); > + return; > + } > + [ ... ] > +/* > + * cpr-transfer and cpr-exec hand the running device over to the new QEMU > + * as is: its fds, mappings and interrupts are preserved and it never st= ops. > + * Device state migration must stay out of it, else the device is stoppe= d, > + * its state saved and then resumed into the very same device. > + */ > +static bool vfio_is_active(void *opaque) > +{ > + return !cpr_is_in_progress(); > +} > + [Severity: High] Is cpr_is_in_progress() actually true on the cpr-transfer source while these handlers run? On the source it ends with: migration/cpr.c:cpr_is_in_progress() { ... return cpr_state_file !=3D NULL; } For cpr-transfer, qmp_migrate() calls cpr_state_save(), which sets cpr_state_file. When the target closes the CPR socket, qmp_migrate_finish_cb() runs: migration/migration.c:qmp_migrate_finish_cb() { ... qmp_migrate_finish(addr, NULL); cpr_state_close(); } qmp_migrate_finish() only starts the outgoing connection, and cpr_state_close() then sets cpr_state_file to NULL. The migration thread, savevm setup, the VM stop at switchover and vfio_save_state() all run after that. At that point migrate_mode() is CPR_TRANSFER, cpr_is_incoming() is false and cpr_state_file is NULL, so cpr_is_in_progress() returns false on the source. If so, on the source vfio_is_active() returns true, so save_setup, iterate and complete still run. The vmstate-change notifiers still move the device to RUNNING_P2P and then STOP or STOP_COPY. vfio_save_state() still emits the real config or data, and the switchover ACK is still required. On the target, cpr_is_incoming() is true until migration_incoming_state_destroy(), so vfio_is_active() is false there and load_setup is skipped. That means no RESUMING state, and data_fd stays -1. The loader does not consult is_active, so vfio_load_state() can still receive the source's real DEV_SETUP, DATA or CONFIG sections. Could those fail on data_fd =3D=3D -1, or re-program MSI-X, which is what this patch is meant to avoid? cpr-exec looks unaffected, since there cpr_state_close() only runs in migration_cleanup(). Does the cpr-transfer source need a different predicate, or should the cpr_state_close() call be moved? > @@ -882,6 +914,10 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > Error *local_err =3D NULL; > int ret; > =20 > + if (cpr_is_in_progress()) { > + return; > + } > + > if (running) { > new_state =3D VFIO_DEVICE_STATE_RUNNING; > } else { [Severity: Medium] Can the cached migration->device_state go stale when a paused VM is CPR-transferred? On the paused source, vfio_vmstate_change(running=3Dfalse) has already put the kernel device into STOP before CPR starts. The target creates a fresh VFIOMigration in vfio_migration_init(): migration->device_state =3D VFIO_DEVICE_STATE_RUNNING; Nothing queries the kernel for the real state, because the device fd is inherited as is. With this early return, the target's notifiers do nothing while cpr_is_in_progress() is true. In process_incoming_migration_bh(), a paused source takes this path: } else { runstate_set(RUN_STATE_PAUSED); } so vm_start() is not called and no VFIO notifier runs. After that, migration_incoming_state_destroy() resets the incoming mode and cpr_is_in_progress() becomes false. A later 'cont' then calls vfio_vmstate_change(running=3Dtrue) with new_state =3D RUNNING, and vfio_migration_set_state() hits: if (new_state =3D=3D migration->device_state) { return 0; } so the VFIO_DEVICE_FEATURE ioctl is never issued. Would the kernel device then stay in STOP while QEMU believes it is RUNNING? Should the cached device_state be reconciled with the real device state when CPR skips the transition on the target? --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261020-vfio= _v1-v4-0-83a5668230e5@virtuozzo.com?part=3D23