From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from OSPPR02CU001.outbound.protection.outlook.com (mail-norwayeastazon11023132.outbound.protection.outlook.com [40.107.159.132]) by lore.virtuozzo.com (Postfix) with ESMTPS id 5CFD28017D for ; Tue, 3 Nov 2026 14:26:56 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MxDT/BToRMmy7Z3JOWjHtlmm8kNsiAy3bkVsRw+mG8M8IDNDoS0qJBhGcl131FhMLfb3GWv05H7Z/2oXcWmTvKKaRhqZ9zyO1hSBeep2SG/s/xvjc/Fh4ue9BBljvl6RzgUIlrOSdyi3qlwnfewhlXpSrFVqU8LcRFuIw4J/VdlLnf3pTvxk9D7N9oMSH6eRBE9+8BeFYaj3qjlIcGw/gqQSWDntkRdS+YyzoGD4zDeY96pHFHUxvBI6JjnR3jQs1VrEnY5MXMxFA83r/wBrIYjmYeDZvUGLHEWK2P/vScjUr5GAfL/IbnqegoCuJukzPi3Z48WQlYXNFanBG+TC9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fcRbJMKqDosX9BauzuaIypESnoysQVd0lrgcyVtywNo=; b=KotIRcwDVLZ84M7IrwOyxgb3IIMsaw+m4V3Tj/0aYC2eKrl4OrLRaDOGbC17WDhwHIs5/E3GN3YKFt9n0eSFV/7QmFG+BQtW4EzJV1msygexkToxVuM7jgtH5aN6wsDFd65WIy7jihv5BJLvgjfXz0wUqvoohfhIGSzV9Te6eo7aBM5uAgYHS/5kKNrysHzEhMPGhIZ2QEnV7TNGf1rSkO6mZonxqfzBxmTCwSIOj9iocRWKw+zHfeT2Y2QT462P423qWMURe6zU4e3Gnct71cVe5swYGIxIQj2pELZKSGpeJfYehGr/Zr0zf7ze0MKJfGptvp4jDNLse2iSCYZa5Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.2) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fcRbJMKqDosX9BauzuaIypESnoysQVd0lrgcyVtywNo=; b=BOBBCZl/MrIj4FirDsOJSv9tA8Y0LE15Pha/3Wpk2XTLmC31H9ZZsBwbx/17vyzXVwcGAMvFImLyJek6ncZ/+VYx1yS6EeQqJkoz5PxxMniwmybc7GFfJKYfgVAzQSbKmotAE48vM1kpy0w18mfFT5QuM0G8e0EdEc96cozF/+c= Received: from PAZP264CA0013.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:21::18) by MI9PR08MB999927.eurprd08.prod.outlook.com (2603:10a6:290:8b::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.569.12; Tue, 3 Nov 2026 14:26:53 +0000 Received: from ZR1PEPF0000E6B1.eurprd05.prod.outlook.com (2603:10a6:102:21:cafe::5a) by PAZP264CA0013.outlook.office365.com (2603:10a6:102:21::18) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.8 via Frontend Transport; Tue, 3 Nov 2026 14:26:53 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.2) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.2 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.2; helo=b224-2.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-2.smtp-out.eu-central-1.amazonses.com (69.169.224.2) by ZR1PEPF0000E6B1.mail.protection.outlook.com (10.167.241.88) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.5 via Frontend Transport; Tue, 3 Nov 2026 14:26:52 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1793716012; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=KDyq5GU2TYFEI7dEvVTvzU7/91GafG7ULKdgten2oYw=; b=uVbUxPyn5tuZOb9RqTSSQvJgy+2kx+LnrEx7F1+RwQXzG9CfqcXFEg3teNe4Figx 2KLDxohTNWR7LpxRwKP+qHs6VnIbUoGy6ZzBYwmvYQmBJSJRXsjfsSug340EbdgxWdl gA1VCclYQoC9D+uK+K1Z1eW/CbzPTyHhALoKm2XJdznZ6bHWTZSoqLRvpqEArU2yv1/ jlMz1HzPxbiu97aj84+qpXc9eKcIyZ6jpBYbc3NrEVMtbElECDrNp6xeeQnipSENk81 3bePsRs7OZGrYOJExvRQFNTtUz8HrL+sMubk23A4Q60STKs3FvU+EwyAzH38WPOIkxl zd/R8j4HuA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1793716012; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=KDyq5GU2TYFEI7dEvVTvzU7/91GafG7ULKdgten2oYw=; b=TCMk4eqpVScSgt1n/oOQ8Hnys9yjDILfSTNpmTBurZ16ZKdn51eU9nQzXb25ALc6 VjPMkk0fxEP8xamz1WrT5jFmR5DYaNBeWUZpZTvGTeDgRjNoMKAmv03eUGjJ4gxf0gJ TI7FRi6wkqHIRlX54wP89a+C56qOJz0CAT3L1mjA= Message-ID: <010701a1a1ce45cb-46be795f-0b17-4f6d-ba66-cc2cb960ae05-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v7 26/35] vfio/migration: don't migrate device state during CPR To: svt-core@virtuozzo.com, Andrey Drobyshev Cc: andrey.drobyshev@virtuozzo.com In-Reply-To: <20261103-vfio_v1-v7-26-4467fdccbbe6@virtuozzo.com> References: <20261103-vfio_v1-v7-0-4467fdccbbe6@virtuozzo.com> <20261103-vfio_v1-v7-26-4467fdccbbe6@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Tue, 3 Nov 2026 14:26:52 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.03-69.169.224.2 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: ZR1PEPF0000E6B1:EE_|MI9PR08MB999927:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 19eb0d2d-df52-455f-c5cc-08df39b68180 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|61400799027|48200799018|32142699015|9140799003|376014|261021121427099003|5023799004|56012099006|5063699009|4143699003|261009223027099003|261009221227099003|261028144727099003|261022105027099003|261022075727099003|261026140227099003|261026121527099003|10067099003|22082099003|18002099003|55112099003|261102123227099003|5113699003|4113699003|16102099003; X-Microsoft-Antispam-Message-Info: 5hByZUNnrA0XkCSK+dec75PZGwMdNZAu0u4QPIrRJkJE5FAXBiVNXu5KHZkMXag/nqDGDUIUS0sfKkFWvk7aU2mpk1+fjt67+OaA5MixL11nA66j9p1ZAhOtTVKSzRYdLyi9ra7um/3ZzEBCGV1PXQFOp39VFnK1wRw2DOOF1B5YW4r9jSZ83DCgVQpqIra7K2W+JK1ep2xtmgFeQshd6QctchUO2LXbEQ/zSE8KLyAY+p2pO7PeBvnhicBl6+hydPQtM+hH8UtMYgMfMsNuVIElZ1OKafcf+a1yqL+WVP0Or9WxGcr1PFxH7ej5A1Qa7qoSd8Y7kZPVtf3+5AvC9MWQegy068S7f4DqWudppE666e87Gor+B3V0MUOOzvSII+hrMpQrbAGlsLelof4cAdH9+gxOR4Ul9piIK+SW7jLLlTtme/ripVal5QjbcX+IxWEPa4IkE7qnt+ixGpVyAF7QL5JNriT/uBMjauWyoYhDqASnzsfI++51LgbvqYfGaBu3phMjyK1wC8oq5v/9oujLo4WwB4NAA/mUaXmMTin7jypYW4OlMzzs7kig8tPf53ZjSClAoEkFspsXPEY13q1xhCtu4Wc4cJNRR5quMfM3/2n2vuTSkZiiRU7ObK1I6MH8MNXudnSCjqeeEg402cARNF5ZEaPQQ7e5t3eQQ8dbO33GHkrYkpGgew1KszrAcQoG/6frNN7rwUfkvdgZ0ZNP7z0AC0cwyPhjRo8H8k7hbTnOJG8dkplDMlnNHo2omh3F2KFrOPRBfxmnl8bPRsnRNlbzAEoZhoRU807t79rFHFsLSzvPwDV+guYvdCb9ShIsrXyxtwO1zOwzwuPvAg== X-Forefront-Antispam-Report: CIP:69.169.224.2;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-2.smtp-out.eu-central-1.amazonses.com;PTR:b224-2.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(61400799027)(48200799018)(32142699015)(9140799003)(376014)(261021121427099003)(5023799004)(56012099006)(5063699009)(4143699003)(261009223027099003)(261009221227099003)(261028144727099003)(261022105027099003)(261022075727099003)(261026140227099003)(261026121527099003)(10067099003)(22082099003)(18002099003)(55112099003)(261102123227099003)(5113699003)(4113699003)(16102099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Vs2TW/sTHG4jGUoZwvOau8oiuXAQIo7pEP4RgP+2AzC9vJOvEg/1kvi/jJuG3luCcAT9IjgftRRPTthr2tedLTZtUXQ4U3Qjf3gU0xWMiXkeRHkJhDbf4i2KFCZDDHNmdP+8SQzdg/Tc7KEVtDpNaDI/vuHAH+CEhw8C9ehdl70RV4Guha6ai861wMVDicuDBLHCQMJE/cjRRTm6aBe2HVojU+zKIcD6BPXZvWK+5oTur3M+hRX/6iy95jw76QR90RYpqHVb4CKAqA+y86/+nWm+2UOu6nMUirnddP/RTiOT2S9R6z90QrjfWvBiemhwplF7VYwtX0g1+3Lz+U6qIPLgHQsQWiQCk5bdKMJ72nR6WrDlQ21RjGRHV+ONIkhyxR8yawYxk6jbrqsrqqn0uXQ6wGpv8DhlfRubmdRCQIdO5KzCN9eB02cI2dq0g28R X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2026 14:26:52.8824 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 19eb0d2d-df52-455f-c5cc-08df39b68180 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: ZR1PEPF0000E6B1.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: MI9PR08MB999927 Thank you for your contribution! Sashiko AI review found 2 potential issue(= s) to consider: - [High] vfio/migration: cpr_is_in_progress() is false on cpr-transfer sour= ce during the stream - [Medium] vfio/migration: cpr_is_in_progress() true before migration start= s skips stop/cont device state -- commit 67f4499a74b760de74e49d3ef3311d53eec749fd Author: Andrey Drobyshev vfio/migration: don't migrate device state during CPR This patch gates the VFIO migration handlers on cpr_is_in_progress() so that cpr-transfer and cpr-exec no longer save, load or change the state of a preserved VFIO device. It also resets multifd_transfer in vfio_multifd_cleanup(). > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > @@ -674,6 +674,16 @@ static void vfio_save_state(QEMUFile *f, void *opaqu= e) > Error *local_err =3D NULL; > int ret; > =20 > + /* > + * .is_active doesn't cover the non-iterable stage, so the section is > + * still sent: leave it empty, or new QEMU would load the config into > + * the preserved device and re-program its MSI-X. > + */ > + if (cpr_is_in_progress()) { > + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); > + return; > + } > + [Severity: High] Is cpr_is_in_progress() still true on a cpr-transfer source by the time this runs? On the source, cpr_is_in_progress() ends with: return cpr_state_file !=3D NULL; For cpr-transfer, qmp_migrate() calls cpr_state_save() and then cpr_transfer_add_hup_watch(). When the target closes its CPR socket, qmp_migrate_finish_cb() runs: qmp_migrate_finish(addr, NULL); cpr_state_close(); qmp_migrate_finish() only starts the asynchronous connect. cpr_state_close() then does: qemu_fclose(cpr_state_file); cpr_state_file =3D NULL; The migration thread is started later, from socket_outgoing_migration() via migration_channel_connect(). So by the time save_setup, vm_stop, vfio_save_state(), the switchover-ack check and save_complete run on the source, cpr_state_file appears to be NULL. The source has no incoming mode, so cpr_is_incoming() is false and cpr_is_in_progress() returns false. On the target, cpr_is_incoming() is true, so cpr_is_in_progress() returns true. The two sides then disagree: vfio_is_active() returns true on the source, so the source sends the full config section, and vfio_vmstate_change() stops the device. The target skips load_setup and vfio_vmstate_change(), but vfio_load_state() has no is_active check, so it may still process the config and buffer sections. That could re-program MSI-X on the preserved device, use an uninitialised data_fd, or leave the device stopped and never restarted. For fd transport the connect is not asynchronous, so the migration thread may race with cpr_state_close(). Can the source-side predicate be made to stay true for the whole outgoing cpr-transfer stream? As far as I can tell, cpr-exec is not affected, since its cpr_state_file stays open until migration_cleanup(). The later patches in the series do not appear to change cpr_is_in_progress(), cpr_state_close() timing or qmp_migrate_finish_cb(). [ ... ] > @@ -857,6 +884,11 @@ static void vfio_vmstate_change_prepare(void *opaque= , bool running, > Error *local_err =3D NULL; > int ret; > =20 > + /* The device is preserved, not a plain stop/cont: keep its state */ > + if (cpr_is_in_progress()) { > + return; > + } > + > =20 > @@ -882,6 +914,10 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > Error *local_err =3D NULL; > int ret; > =20 > + if (cpr_is_in_progress()) { > + return; > + } > + [Severity: Medium] Can these early returns skip a plain stop/cont on the source before the migration has started? On the source, cpr_is_in_progress() becomes true as soon as qmp_migrate() calls cpr_state_save() and sets cpr_state_file. For cpr-transfer, qmp_migrate() then only adds the HUP watch and returns. The real migration starts only in qmp_migrate_finish_cb(), after the target closes the CPR socket. That window is not bounded. During the window the migration state is still SETUP and no savevm handlers have run. qmp_cont() still accepts stop and cont. Both vfio_vmstate_change_prepare() and vfio_vmstate_change() return early. For example, if the VM is paused, the device is in STOP. The user runs migrate with cpr-transfer and then cont before the target connects. vm_start() resumes the vCPUs, but vfio_vmstate_change() skips the RUNNING transition, so the guest runs against a device left in STOP. If the migration is then cancelled or fails, cpr_state_close() clears the predicate, but nothing restores the device state. A stop inside the window can similarly leave the device RUNNING. The commit message says "the mode may merely be set, and then a plain stop/cont must still change the device state". Does that hold for this window, where cpr_state_file is already set but no migration is running? The later patches in the series appear to add only a cpr_is_incoming() branch for the target in vfio_vmstate_change(), so the source keeps the same early return. --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261103-vfio= _v1-v7-0-4467fdccbbe6@virtuozzo.com?part=3D26