From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <010701a135f85947-e6462840-260c-40f1-a397-427470d6640b-000000@eu-central-1.amazonses.com> Received: from DUZPR83CU001.outbound.protection.outlook.com (mail-northeuropeazon11022132.outbound.protection.outlook.com [52.101.66.132]) by lore.virtuozzo.com (Postfix) with ESMTPS id 7DD0180726 for ; Tue, 13 Oct 2026 15:53:56 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=T1UcN7O0Xp2Qc0H6MwdwJqXHa375X26CgdkMt46BExSXZn23xaYIXqj/jDJl9QLcJMiBruT7fxpfEl/cFfjgJSR+xx34EJ8cvClQtMDW9WFReMLfeBYiohbeZR6i61C6LdGh+Kq8lmlDeSaCU7UBeLqIfvKCtb/NT4ANWpS4S1gV7KMfy7h4uLstNicvOlqe9m43P1ZaY8+nD+btnKUavPMJVzDqJ89aruK8rdr7lfaOxf0P0Znfxub411RZz8Q5DcxE/phg/TuikDOT1hVeiYJItA662GfJQ9tuR/ilSEoZ73TcV1eg1c+G/btIyjCApHNgEKKVLDJJAv8e4s6hhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FSWWTGB0pJRf1kYxtdB9Xp/vhE9aIFFtIT8tG1LZXqI=; b=VetVU4QInAZ0F2Z+2/65+hGYhdvcRS5E9186AQE9iOhdEMoiYEuz5I3bypSW+1QlH9Aqnierh8xVgF4qpbESEoL+50T9xDtumtyL8PpaJgxIOaKf5ty6NpLFpwF5pQcUaiPDUSOKrq4XwHa3kcFBo8FImj25Ff5oSsez8jocSMk+tp8E1Qf/eE9lf6bNJvAyicS0KQICHVleKwjx+PmoxykaE/UaZ48e2bVmQZnGVZZ3IzIzGuEZ6MRnR7kj0ywj3tAe3WcWtmm5/SI0LwKmYtvw9FemoiwEWtwjYM6JSRJYD/+n9FbwD2sMn02qcn9e9HvF48mXcRScjI7m30V8Ow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.5) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FSWWTGB0pJRf1kYxtdB9Xp/vhE9aIFFtIT8tG1LZXqI=; b=h2yJw1m/nE6OD8ZPcEJvjrDndeVTj/S5wQ4ZNxOauR3ylwTh765L6KBvlcrNnXPcgJElkB6VuWcJfjX7RjWt6YklZedlfCfAIRSg56cqJQcvumidXlObnSuC67TjEmC2PO489STPrtwdJXaEJNFk1JLc7LXwTra/zRb6LHFxaQY= Received: from OS6P279CA0055.NORP279.PROD.OUTLOOK.COM (2603:10a6:e10:31::13) by VI3PR08MB766121.eurprd08.prod.outlook.com (2603:10a6:800:37b::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.522.12; Tue, 13 Oct 2026 15:53:52 +0000 Received: from CPH1EPF00000525.eurprd05.prod.outlook.com (2603:10a6:e10:31:cafe::8b) by OS6P279CA0055.outlook.office365.com (2603:10a6:e10:31::13) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.496.5 via Frontend Transport; Tue, 13 Oct 2026 15:53:51 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.5) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.5 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.5; helo=b224-5.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-5.smtp-out.eu-central-1.amazonses.com (69.169.224.5) by CPH1EPF00000525.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.522.6 via Frontend Transport; Tue, 13 Oct 2026 15:53:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1791906830; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=m5JpEclJmtGsuiZmwbb3FZnmtFi2ZUVA0amCkpgk9hc=; b=zM0dJode1DZ4iYRkLcC+9XsteeagJJW3nFxNO5ObzvSjNfskILoSqq0t9IkzTHnI CwcyVl6aKEqw5Sn6QTKdDBBRtt63Tu7tjzD4zHxoFpCENggMDv9FTg6WqRdrhIxN9K7 2mkwAyYghtij1bsZrSQx/Wqra38Xjf2jDYk3eVMJaQOU/UJ8gtQukLBHxqss8kJ2nuE p6V0Ys2lbR0uguk9mlXZAfkD19SEnrbOC99Fe4ijEzzzYCpA9VJOekSMte/yyxCkWM6 0l45GszDdcHd1YwWmi+FMVIyQMvX0daLnHQ/fBV8TTT6wujTTuEukDFw3T55RSrteqt 9pjdBeYmZA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1791906830; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=m5JpEclJmtGsuiZmwbb3FZnmtFi2ZUVA0amCkpgk9hc=; b=fU0+w/Kix84uqgPi3WxkxBwEyQ9li/zbX6epl5Dlnp33+AnoSXrI40Njkui1NaDv wPKBN/2eDta6Pz1adXQoWUzUnQqugoLvwF+H9meJsbJONH277cwLHG5BRMAoGgiighX oj3ajYCwK5EMmJ19eN3IsWWW1jtRA6r2VQHzsc4A= Message-ID: <010701a135f85947-e6462840-260c-40f1-a397-427470d6640b-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v2 18/24] vfio/migration: don't migrate device state during CPR To: Andrey Drobyshev , svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com In-Reply-To: <20261013-vfio_v1-v2-18-433455c0a294@virtuozzo.com> References: <20261013-vfio_v1-v2-0-433455c0a294@virtuozzo.com> <20261013-vfio_v1-v2-18-433455c0a294@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Tue, 13 Oct 2026 15:53:50 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.13-69.169.224.5 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CPH1EPF00000525:EE_|VI3PR08MB766121:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 05286d78-6771-4b5c-3704-08df29422d2b X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|32142699015|9140799003|376014|48200799018|61400799027|4113699003|5113699003|6133799003|10067099003|22082099003|16102099003|18002099003|55112099003|13003099007|4143699003|261009223027099003|56012099006|5023799004|5063699009; X-Microsoft-Antispam-Message-Info: M8esgaA/I9PqyjigB3C+LnvIQZDgM3jv4K7Kry6UdH8BIsT3LeU1To5cRRnANDw1ElGRhQR9GVUk8kcMGzVzGzEDOywrnV39kHUMmTsrKhGPSSMj58vNHPd1FBqkp/d9P548N5cnBL27BlM+RoxMvEu18whJwamKzBct3mHdO5LXl1904OXcgTRbgia1zUFasQY+lTJNh76/n6EMcGdqtSLflQ4/8RDUGE0i+DBIDGabHqOYWm328EUUrVHVqAIU8TbkUjxYpvRMgJhGBHFlIughGYZqKKAK705OulxIvosl7ZbLDo0AFR5vIMceMtDCQyafQKLIjC4Upu/ImN5YeN3tS9QUCALkx2y80ecLJKA+NobWTn+c1niWmfzz1eiq54WnuRk2vhDouCD0GTI3xKwx8kGRx0LtjuX6qpPjeM5meT+HLWWQ0pelcZWlBjOwFOQwp3aX4UGdxK2z26YlL0tZvVYumOy7yQRIfAvVZ4UuZ3ZWjyxKZVxeptNBPfP10r4KVhaklaffqhY4YjDLaXY0uLHkcaVgvTlljqTYLiyhBek2D82cXRCEwU8Gd24y24k7Ug7io5NGyz40IWbiLSKVwDw+6DSf4EKoEGmUPvV7zs3q/UT/VxLi6ZnMU6ygK+kwmyX7FDDlg0GuDMrJSrqRXg6Rzmx17uqy4p8tAC+S3uL+no8Wdlj+BVfT8SKbH3YfeFqbjH/7V+g6S4HitX5WFzKvFE9yAxLtKmVvX5uOtgnxSjz+t3TzQ8Y1sIpYOHOrfwVWqA3zDfLol4BtH0nm8B1KCvjPlSEvSjE6cgBDqTh6CycbqWZdmHoc9R9P X-Forefront-Antispam-Report: CIP:69.169.224.5;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-5.smtp-out.eu-central-1.amazonses.com;PTR:b224-5.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(32142699015)(9140799003)(376014)(48200799018)(61400799027)(4113699003)(5113699003)(6133799003)(10067099003)(22082099003)(16102099003)(18002099003)(55112099003)(13003099007)(4143699003)(261009223027099003)(56012099006)(5023799004)(5063699009);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: BVcF1l+ay98FOZT6lQr4BZVuDw1WRTMWTHiTaJtsWm3avNwkr2kpaO2dfh5z1ME/mV2LMZP4hzgYhzhLqn+P99lkR8Ayz5op5ssHOQeCXa0wUgKTchYlkPfe5sV4wOHNn+AO+XMYyJD8dYPQyuobowbo1Sh9FJOZPNrggK1Pl5sWyqM0cmk/42pFmOR70NZ7cxOwzNrmOH1ZRHQCAcptFPZun03BklFTNwesCunX3eTuhhG/OFe7+MLk/b1R216sJ32sdbbRUno+E/Hi4KutMGuomyQecOucW9mFGsX/5RUXysx5yt+aNWWtZts8KA2/9kLcfQHuEXfORHSCfyqHyFbIWVDOgwbWz0pVJ5+TXgcTjCST3N5217zmHrg7lWc3uEBId/c4WmUTDIuDzfGXKIFYrw9Xu03LHsyvSUbWQM1N0PSKNmISW5L8xSXKfN3i X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2026 15:53:51.1411 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 05286d78-6771-4b5c-3704-08df29422d2b X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: CPH1EPF00000525.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI3PR08MB766121 Thank you for your contribution! Sashiko AI review found 4 potential issue(= s) to consider: - [Medium] vfio: migration save/load gated on mode parameter, not CPR in pr= ogress - [Medium] vfio/migration: load_state not gated for CPR, unpatched source s= till hits live device - [Medium] vfio/migration: switchover_ack_needed not gated for CPR, hangs m= igration - [Medium] vfio/migration: stale multifd_transfer lets save thread stop dev= ice during CPR -- commit 27c407560d6e12fc61d8fd9e47b3508c1eaa3fcb Author: Andrey Drobyshev vfio/migration: don't migrate device state during CPR This patch skips VFIO device state migration in the cpr-transfer and cpr-exec modes. It adds a vfio_is_active() callback and an early return in vfio_save_state(), both gated on cpr_incoming_needed(). It also makes the VM state change handlers return early when cpr_is_in_progress(). > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > @@ -674,6 +675,16 @@ static void vfio_save_state(QEMUFile *f, void *opaqu= e) > Error *local_err =3D NULL; > int ret; > =20 > + /* > + * .is_active doesn't cover the non-iterable stage, so the section is > + * still sent: leave it empty, or new QEMU would load the config into > + * the preserved device and re-program its MSI-X. > + */ > + if (cpr_incoming_needed(NULL)) { > + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); > + return; > + } > + [Severity: Medium] cpr_incoming_needed() only tests migrate_mode(), that is, the configured mode parameter, and not whether a CPR operation is running. If the mode parameter is left at cpr-transfer or cpr-exec (for example after an earlier or failed CPR attempt) and a non-CPR save such as snapshot-save runs, what happens? save_snapshot() vm_stop(RUN_STATE_SAVE_VM) qemu_savevm_state() vfio_save_state() cpr_incoming_needed(NULL) is true -> only END_OF_STATE is written In that case vfio_vmstate_change() is gated on cpr_is_in_progress(), which requires migration_is_running() on the source, and that is false for savevm. The device is therefore still moved to STOP, but no state is written, so a later load restores nothing. The commit message says the stop/cont hook uses cpr_is_in_progress() because "the mode alone would also skip a plain stop/cont of a VM which merely has it set". Does the same reasoning apply to vfio_save_state() and vfio_is_active()? Should they use the same predicate as the vmstate handlers? [ ... ] > @@ -824,7 +835,19 @@ static int vfio_switchover_start(void *opaque) > return 0; > } > =20 > +static bool vfio_is_active(void *opaque) > +{ > + return !cpr_incoming_needed(NULL); > +} > + > static const SaveVMHandlers savevm_vfio_handlers =3D { > + .is_active =3D vfio_is_active, > .save_prepare =3D vfio_save_prepare, > .save_setup =3D vfio_save_setup, > .save_cleanup =3D vfio_save_cleanup, [Severity: Medium] This isn't limited to the case above: vfio_is_active() also returns false on the mode parameter alone, so all VFIO save handlers are disabled whenever the mode is cpr-transfer or cpr-exec, even with no CPR handover running. Separately, is the load side covered? .is_active only suppresses setup, save and pending-estimate handling, while .load_state is still called on the destination for any section the source sends. If the source is a QEMU without this patch, vfio_load_state() would still see these sections: vfio_load_state() case VFIO_MIG_FLAG_DEV_CONFIG_STATE: return vfio_load_device_config_state(f, opaque); That would apply the config to the preserved device and re-program MSI-X, which is the failure the commit message describes. A DATA_STATE section would reach vfio_load_buffer()->qemu_file_get_to_fd() with data_fd still -1, because vfio_load_setup() is skipped, and the incoming migration would fail. Is a version requirement on the source intended here, and should vfio_load_state() have its own CPR check? [Severity: Medium] Can this hang a CPR migration when switchover-ack is enabled? vfio_switchover_ack_needed() still returns vfio_precopy_supported() and is not gated by is_active. On the destination, qemu_loadvm_state_switchover_ack_needed() does: if (se->ops->switchover_ack_needed(se->opaque)) { mis->switchover_ack_pending_num++; } without consulting is_active, so the VFIO device is counted as awaiting an ACK. The fallback ACK on MIG_CMD_OPEN_RETURN_PATH then does not fire. On the source, vfio_is_active() now returns false, so vfio_save_iterate() is never called. That is the only place this is emitted: qemu_put_be64(f, VFIO_MIG_FLAG_DEV_INIT_DATA_SENT); The destination therefore never calls qemu_loadvm_approve_switchover(). migration_can_switchover() then keeps returning s->switchover_acked (false) while the VM is running: if (!runstate_is_running()) { return true; } return s->switchover_acked; migrate_caps_check() only requires return-path for switchover-ack and does not reject it in CPR modes. Should vfio_switchover_ack_needed() also return false in CPR mode, or should the capability be rejected there? [ ... ] > @@ -857,6 +880,11 @@ static void vfio_vmstate_change_prepare(void *opaque= , bool running, > Error *local_err =3D NULL; > int ret; > =20 > + /* The device is preserved, not a plain stop/cont: keep its state */ > + if (cpr_is_in_progress()) { > + return; > + } > + [ ... ] > @@ -882,6 +910,10 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > Error *local_err =3D NULL; > int ret; > =20 > + if (cpr_is_in_progress()) { > + return; > + } > + [Severity: Medium] Can the multifd save thread still move the preserved device to STOP_COPY during CPR? qemu_savevm_state_complete_precopy_iterable() spawns save_complete_precopy_thread for every handler that has one, without checking .is_active. The thread's only guard is vfio_multifd_transfer_enabled(): hw/vfio/migration-multifd.c:vfio_multifd_save_complete_precopy_thread() { if (!vfio_multifd_transfer_enabled(vbasedev)) { /* Nothing to do, vfio_save_complete_precopy() does the transfer. */ return true; } ... if (vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP_COPY, VFIO_DEVICE_STATE_STOP, errp)) { migration->multifd_transfer is written only in vfio_multifd_setup(), and vfio_multifd_cleanup() does not reset it. With this patch vfio_save_setup() is skipped in CPR modes, so the flag is no longer refreshed. If an earlier non-CPR migration attempt ran vfio_save_setup() with multifd transfer enabled and then failed or was cancelled, and a later cpr-transfer or cpr-exec is run in the same process, would the flag still be true and the thread stop the preserved device and stream its state? The same flag also gates vfio_switchover_start() without an is_active check, though that is a destination-side hook in a fresh process and is less exposed. --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261013-vfio= _v1-v2-0-433455c0a294@virtuozzo.com?part=3D18