From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazon11021127.outbound.protection.outlook.com [52.101.70.127]) by lore.virtuozzo.com (Postfix) with ESMTPS id 78C168017D for ; Mon, 2 Nov 2026 11:07:32 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LA9Yczz3kFz8BpXYCL6TLjK9Ls9oE5VjG/4lBhGH8DQZRZL8G7smV6MjBrzCsIsVEdxKVaTlLMnxY8xORBKN1ZhSzNG3WsyNj1q68iJg6V+J3vPAU1JnarKGNcfM0rfXLDRhxQU8meKwO6mNBFKlH9ae5+tW+3k+pMGy3BGoJUD8YBeKLGPqxpemMkRr5zyqf+a7sGWQIHXD361QTzsPy142ZfzL1ZtfBGE2Kx6HwnCqm2MfB2SrRvrwmLViCeCZKPck6M83UfspryJaqpMop9YYfhKSGAtlLAZteJ795dSw91jl1X/QqWUqxOyC/3viSrsrAmSgVy6LD3LRLpBykg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=i4b3rbRDYCKakGIWjYHVvH4omLIa7FMpMVessSR7rps=; b=OiOSfhtNepoxXFFG4HA0A0uFA6XTQIvcDuHRnW5g0jrykxKREkJKpONE+zSCLvZIdSxtn/9nP/SPTELnHK1NWyJhSxlUvaqN/e9hujLKonptPBPHnrvCzHzTIS8SCwrZ1ICpbnC4zldE0KhiY3ean+3PN2sPPG8wA56BASd6lF0LdemJvJRGNnCy6slo5Jua/Z6sEbqwJK6LcjUYT30863vheKaaLtZEAbN/1CGNadj40/fsF0tlJgHOu/RukhSAAtyi+aJ568+NZ2w0glKeE4d3tHvZx7XbvyMSNCF4RL/KIk1GMKvus1tL6OW8q0GgJLBzbeFQbPW4yod9W/AuQA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.1) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i4b3rbRDYCKakGIWjYHVvH4omLIa7FMpMVessSR7rps=; b=gEBNL11E6L6CGmGqq2SycmDNcu8eUjwnZKPR6RYjkSiPobMHy3Uyrsk69QYvmEn/qF6RiysvDJGNRhNxp6Xo9EdyFoqXzDTV8qt7My/PMDdWeCRywnfCqdS2mXEhwWViMTp5G8SMZdM+uDqoa09mEMEtuplEON064so1CredOfA= Received: from DB8PR06CA0066.eurprd06.prod.outlook.com (2603:10a6:10:120::40) by PAVPR08MB9332.eurprd08.prod.outlook.com (2603:10a6:102:302::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.569.12; Mon, 2 Nov 2026 11:07:29 +0000 Received: from MAD0EPF000008B9.eurprd04.prod.outlook.com (2603:10a6:10:120:cafe::15) by DB8PR06CA0066.outlook.office365.com (2603:10a6:10:120::40) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.569.12 via Frontend Transport; Mon, 2 Nov 2026 11:07:28 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.1) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.1 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.1; helo=b224-1.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-1.smtp-out.eu-central-1.amazonses.com (69.169.224.1) by MAD0EPF000008B9.mail.protection.outlook.com (10.167.241.167) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.5 via Frontend Transport; Mon, 2 Nov 2026 11:07:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1793617648; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=TgpHuA5EqOG0N2rRUQOZExBtrWyc8+EY5WW/FfLZydk=; b=McIGTo1sJWAExQJZGdg0phnyq+s2TViUwJv+RQ6lUZuPfSlM/m3Wef1vJqnP7H/H Hmk6P3ZBlHphqnE/fGBv1E60P0CNw1iKRLGB5IUsubF008Re4z2nGrEGb3QWMxIMVVC 6vrdkp16qERAYgQPOvKQqr/H2XmpwOwv+XbIQyggHKEB1GGKQ0spu0h5wkgxHS+URmu FxwwJoU6d0iHc9FQJV8xsR1OeHSgugC69PgVX5xfFhf5d+thocJXrB+3ujqjvG8pSl/ jWJJc2E4veuuyUl5d1V/Ck9n2KJvhnQ0pdmaDh/hy7ehALf4kghbCga31W8mXRdVI3r BUjtzmRkaA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1793617648; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=TgpHuA5EqOG0N2rRUQOZExBtrWyc8+EY5WW/FfLZydk=; b=NZ157EWEwzaqLTRyBGzMzWn2EyJ7/AstPPox1MmyOPIoOKlQkOOdWfPTQAjkI+/K eIGTkcO9I7/nlb4N/3Kj08pYAstYySr+aRWa7yGWkk7QEqx4/6dlHNQ8GbqdYtskWJ1 2z5Fv34S/+zaaiItuepNrTEye/3r0bQIbb9+Z3pM= Message-ID: <010701a19bf1590f-117044d8-f3c5-4aaa-a837-e73551d5f65c-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v6 25/34] vfio/migration: don't migrate device state during CPR To: Andrey Drobyshev , svt-core@virtuozzo.com In-Reply-To: <20261102-vfio_v1-v6-25-9bc7cabc850d@virtuozzo.com> References: <20261102-vfio_v1-v6-0-9bc7cabc850d@virtuozzo.com> <20261102-vfio_v1-v6-25-9bc7cabc850d@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Mon, 2 Nov 2026 11:07:27 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.02-69.169.224.1 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MAD0EPF000008B9:EE_|PAVPR08MB9332:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: d26b2a3a-2730-482b-cc8a-08df38d17bc1 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|376014|48200799018|9140799003|61400799027|32142699015|56012099006|5023799004|4143699003|5063699009|261009221227099003|261009223027099003|261028144727099003|261021121427099003|261026121527099003|261022075727099003|261022105027099003|261026140227099003|10067099003|22082099003|18002099003|16102099003|55112099003|5113699003|4113699003; X-Microsoft-Antispam-Message-Info: Q9hWd55Y6DTSfR7DVs2OJx6MUNw36RXH9waZNT7XIvhnHk5rm6Ar+AoAz4hQLU2XqsIBkZZAI+CIjt9lrshOHHSoUXhSfM6A+DAvdU+vjquuLZTE08MHKij7kNqFwvsy0r3VrVsceMGTTvbnD85Bt/wAHC1eUFhvSJPW14BmZ5MkYG7NUOTdFA4OdRb2/L0wcLM09P3LJTqMnqW8RWCU4VEW/QK977gv31XoDA6bzK+FLZYSQGZLhPSD2g0gkV8F2nQUXWpB530OFE3rRflQxfFoy8xNxA0y70T/UnWk1v+P02MRrpnyd4LB4RaRI7ceIYfLcfaBEwDCw4PVHHvNBYnA5Bcm41ReJYt7KffdD+NnVR/1du8iW+HVPpGrlyQouoMtDB1Ag3VIAYl8Ajjx5J8YKYWTqc3JlWjZDzqFoAL8RDYkQxlXTcewUMBjOWESWPeaRK1wJ9P95jK7wOl2YTcUTgMK8kVLNDFCek+pLZ1137lCuF2McfR8NqAblQD+IUH4k4e/UkWjQSGbtvsaVShpDC0ZXvabL+SykBx/1V3Tj9qtwY0SQo+6JKRcR6CIuHJtuoGhr3Vy10DMqrZXEgMmlE/hiySRJAiJ+Af9DJtmc9/cHYbNAztto2pRL1F0pObHk/ApKnnB4xnLeBq6dK/FqH6kfT5rU7rQToQWFNrlMpoYM9uaQ69mq20IKrVJI+xXSFANyixKWAJRwNPTP7iI80NbyOiJzGmY1uXqZs6dv4mHKKjx2NOlONVWpl3x8x7TKWYmgxd5CudH4/TSucU6B2wmHJ4DZ9sGlDm09wLPQHboSAf5tqoAj2wSZ7UU X-Forefront-Antispam-Report: CIP:69.169.224.1;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-1.smtp-out.eu-central-1.amazonses.com;PTR:b224-1.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(376014)(48200799018)(9140799003)(61400799027)(32142699015)(56012099006)(5023799004)(4143699003)(5063699009)(261009221227099003)(261009223027099003)(261028144727099003)(261021121427099003)(261026121527099003)(261022075727099003)(261022105027099003)(261026140227099003)(10067099003)(22082099003)(18002099003)(16102099003)(55112099003)(5113699003)(4113699003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CvUxk8tUsrybqXxFcDLm1G0F3KtwxJxclMHGHGsHiAdyUcXOD8ENQS8n0DJjV9JNzM8qY0Oi8R3WbJzE1FBsLQ5e77gFoCIX0e2PTdNd/sa69hibZrrstjVukrI1wkzArjmFedNZf1B3ii+3oQt4K1gkmFs57yPuTMgSFH5PoUIWuZ2PID1eTe0nQrBAXoDhdMZVdE7j4Wiz4HwoMIWy9nc7EAdqcBYZnAnf8wz7qPeMme4toNd4ujFgdLU92BqYUB7+hgyJbddFiw28cpV7q+Ua4zc/XEu8ZiMsvC0IqKZACwBFOumizDDrBYowydBaVhTJHXPbnpXVGsMLTCrx4xXQD+uqCY5zuO/A9h0roQUDmSzM6IIOhNqoUzc/lfpxBxIKeN9humW4TZbcNO/HS03wA7wDe0uRKYrF6ZQIIBgaj7N8yFSnQ43bDaetO8kC X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2026 11:07:28.4538 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d26b2a3a-2730-482b-cc8a-08df38d17bc1 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: MAD0EPF000008B9.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: PAVPR08MB9332 Thank you for your contribution! Sashiko AI review found 3 potential issue(= s) to consider: - [High] vfio/migration: cpr_is_in_progress() is false on cpr-transfer sour= ce - [High] migration: cpr_is_in_progress() is false on the cpr-transfer source - [Medium] vfio/migration: stop/cont ignored for whole CPR outgoing window,= not just the handover -- Commit 54a8e2428cc329d383d08292ed034898e8bf828e Author: Andrey Drobyshev vfio/migration: don't migrate device state during CPR This patch gates the VFIO migration handlers on cpr_is_in_progress(). With the gates in place, device state is neither saved nor loaded during cpr-transfer or cpr-exec, and the device state is not changed on VM stop/start. It also resets multifd_transfer in vfio_multifd_cleanup(). > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > @@ -674,6 +674,16 @@ static void vfio_save_state(QEMUFile *f, void *opaqu= e) > Error *local_err =3D NULL; > int ret; > =20 > + /* > + * .is_active doesn't cover the non-iterable stage, so the section is > + * still sent: leave it empty, or new QEMU would load the config into > + * the preserved device and re-program its MSI-X. > + */ > + if (cpr_is_in_progress()) { > + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); > + return; > + } > + [Severity: High] Is cpr_is_in_progress() still true here on a cpr-transfer source? On the source it returns cpr_state_file !=3D NULL: migration/cpr.c:cpr_is_in_progress() { ... return cpr_state_file !=3D NULL; } For cpr-transfer, qmp_migrate() only registers a HUP watch: qmp_migrate() cpr_transfer_add_hup_watch(s, qmp_migrate_finish_cb, addr) When the target closes the CPR socket, the callback runs: qmp_migrate_finish_cb() qmp_migrate_finish(addr, NULL); cpr_state_close(); qmp_migrate_finish() ends up in socket_start_outgoing_migration(), which only starts qio_channel_socket_connect_async(). The completion (socket_outgoing_migration() -> migration_channel_connect()), the migration thread, qemu_savevm_state_setup(), vm_stop() and vfio_save_state() all run after the callback has returned, with cpr_state_file already NULL. If so, cpr_is_in_progress() is false for the whole data migration on a cpr-transfer source, and the gates in vfio_save_state(), vfio_switchover_ack_needed(), vfio_vmstate_change_prepare() and vfio_vmstate_change() are all bypassed. Wouldn't the device then still be stopped and put into PRE_COPY or STOP_COPY, and its config and data state still be sent, which is the failure the commit message describes for the NVIDIA vGPU? The same question applies to the source-side comment in cpr_is_in_progress(), which says the state is open "from cpr_state_save() to cpr_state_close()". For cpr-transfer that window ends before the data migration starts, whereas for cpr-exec cpr_state_close() runs in migration_cleanup() after qemu_savevm_state_cleanup(). I did not find a later patch in the series that changes cpr_is_in_progress() or the cpr_state_close() call sites. [ ... ] > +/* > + * cpr-transfer and cpr-exec hand the running device over to the new QEMU > + * as is: its fds, mappings and interrupts are preserved and it never st= ops. > + * Device state migration must stay out of it, else the device is stoppe= d, > + * its state saved and then resumed into the very same device. > + */ > +static bool vfio_is_active(void *opaque) > +{ > + return !cpr_is_in_progress(); > +} > + [Severity: High] On a cpr-transfer source, wouldn't vfio_is_active() return true here, given that cpr_state_file has already been cleared by qmp_migrate_finish_cb()? In that case save_setup, iterate and complete would all still run for the device. On the target, cpr_is_in_progress() is true because the incoming mode is set, so load_setup is skipped. vfio_load_state() has no CPR gate: hw/vfio/migration.c:vfio_load_state() { ... data =3D qemu_get_be64(f); while (data !=3D VFIO_MIG_FLAG_END_OF_STATE) { ... } Could real DEV_CONFIG_STATE or DEV_DATA_STATE sent by the source then be loaded into the preserved live device? That would re-program MSI-X, or write to a data_fd that was never opened. The two sides would also disagree about whether device state is migrated. [ ... ] > @@ -857,6 +884,11 @@ static void vfio_vmstate_change_prepare(void *opaque= , bool running, > Error *local_err =3D NULL; > int ret; > =20 > + /* The device is preserved, not a plain stop/cont: keep its state */ > + if (cpr_is_in_progress()) { > + return; > + } > + [Severity: Medium] Can this gate skip a user-issued stop or cont while a CPR migration has been started but the VM has not yet been stopped for handover? On the source, cpr_state_save() sets cpr_state_file from qmp_migrate(), before the migration thread starts. For cpr-transfer, the source then returns to the main loop and waits for the target's HUP, with cpr_state_file open the whole time: qmp_migrate() cpr_state_save() cpr_transfer_add_hup_watch(s, qmp_migrate_finish_cb, addr) During that window, a QMP stop or cont reaches vfio_vmstate_change_prepare() and vfio_vmstate_change(), and both return early on cpr_is_in_progress(). The commit message says that a plain stop/cont must still change the device state when the mode is merely set. Does that hold once CPR migration has started but has not yet stopped the VM? A stopped guest could keep a running, DMA-capable device, and migration->device_state would not be updated. The final-state series still has the source-side early return in vfio_vmstate_change(). > @@ -882,6 +914,10 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > Error *local_err =3D NULL; > int ret; > =20 > + if (cpr_is_in_progress()) { > + return; > + } > + [Severity: High] Is this gate effective on a cpr-transfer source? With cpr_state_file already cleared in qmp_migrate_finish_cb(), the early return would not be taken when the migration thread calls vm_stop(). The device would then still be moved to STOP or STOP_COPY, and the same applies to vfio_save_state(), vfio_is_active() and vfio_vmstate_change_prepare(). In contrast, cpr-exec appears to work, because cpr_state_close() is called from migration_cleanup() after qemu_savevm_state_cleanup(): migration/migration.c:migration_cleanup() { ... qemu_savevm_state_cleanup(); cpr_state_close(); } Should the early cpr_state_close() in qmp_migrate_finish_cb() be reconsider= ed, or the predicate changed, so that cpr_is_in_progress() holds for the data migration in cpr-transfer as well? The close in qmp_migrate_finish_cb() seems to be the defect, rather than any mismatch between stop and start values. --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261102-vfio= _v1-v6-0-9bc7cabc850d@virtuozzo.com?part=3D25