From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AS8PR04CU009.outbound.protection.outlook.com (mail-westeuropeazon11021084.outbound.protection.outlook.com [52.101.70.84]) by lore.virtuozzo.com (Postfix) with ESMTPS id CF3828017D for ; Fri, 23 Oct 2026 17:26:45 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OJ4bXU12XuFFqn304aSs+T32pgMuauEzqHK2SUmyw+CEjBMVu5fPmh/fU7rYObFSXTbcBqbAWt6RnIzBcNeLMmNtzv4DhfflQ7o6CnCVjCwrGXjZqBRKB5fe1IQLW3pQtlDeceEOJ1jGhGon0icckmOLn60+TIy6d1m7jbln8MGny5xmrYGY5b3EnaRBq44hSU0RoioKuTU8A+O48KP82qqpUQD8Bcv2vvcePFk3ATHrszqg4sRuakVIyOsjDBqXfHpHIuYwXnXOafs2qjTqUeBSMfqcWvv3LH3h1DrOJMVvb3T1ygeiO8UPuSTSCQrC/2XZm9JZxZG+cs9FRT2ePQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VAgIFahYAWkZVCZWtUxHbSngPI6rSr9XAuhsRK3qAko=; b=McANZFUTDch/9KsENJRwc6Z919zAN3MQ7ghCq6OVgxdnYE2aSScqYyf0zJF5z1+ztaDDRJFHLr4JK9nzax9XLhFDg8cCOud0A2apysRMS1+1YYqb9f5/FJH0yCCPXCb7pss2M8vZdpK/JJtWeGE3fR3M3j5XMYrDsYidw2JvLCLolM2oCjnhUWTBERwECByn0qCJ32+xvFKcd56GnP/ZqGgoiwMl5YbUZ2k/VHBujYcGHZZfYW+WqzQMax3MrL16ripSY4y0rjnjuRtDTHdpxq86Al7N2pROBexDd2d/uO53R3IfQH3lRFpXwsrnEog+Aq2hciLTfAV46WIddQhpsQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.2) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VAgIFahYAWkZVCZWtUxHbSngPI6rSr9XAuhsRK3qAko=; b=ESmJI3lX7T5gq/2QmN5sfukJ4ARlV/x0GTVJtbfV+wkkHut1AI0pclIJtjSXjN/+NWQomB5o9RwAwDo/hXVtRLECXS+wNq+u43hPcOsc+qVirXVAyMQeB/dvQT4DSRLcBXDJJUvDP9QDulJEAcTKlI3uQ8g95Nv0YynC7mgq1lk= Received: from DU2PR04CA0045.eurprd04.prod.outlook.com (2603:10a6:10:234::20) by GV1PR08MB933119.eurprd08.prod.outlook.com (2603:10a6:150:3bd::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.18; Fri, 23 Oct 2026 17:26:43 +0000 Received: from MAD0EPF000008AE.eurprd04.prod.outlook.com (2603:10a6:10:234:cafe::23) by DU2PR04CA0045.outlook.office365.com (2603:10a6:10:234::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.546.18 via Frontend Transport; Fri, 23 Oct 2026 17:26:42 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.2) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.2 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.2; helo=b224-2.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-2.smtp-out.eu-central-1.amazonses.com (69.169.224.2) by MAD0EPF000008AE.mail.protection.outlook.com (10.167.241.187) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.569.4 via Frontend Transport; Fri, 23 Oct 2026 17:26:42 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1792776402; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=8PHNjLfklMfEzjOOS2FCmsR05CtH6cjcBjMbLZYKaqI=; b=tLIqrlIqFs7771O2V7jPSf47hVGpJxvttSD1tH27RLDqo7tUDtYkcF0X3bIKU+PW D+A7mU6hXNc6klYJXpi5977Slv/R1yWTSBiGjhIjqgGCBD4+tbGX8UzTQj+ZS5wfX9n oIvacL/GHCjDXx8UfyHF8Mh1hP/pIx9SvDZDRytK+lho0K32e8jCUynO9PRYOh5n78E 4CgLgItyw9gwWJMzEFmUyld3gfCzeJdqYxTWkRdomxWRekkOypVG2Y8RYf8iTtC57lk WtxXV7cyrF7sT0C+hxGIhMZJC0Gb3lMuEC+iVqmzk/dlZB489rcKXVOXed5xSUtjUoG Cbe4vQYKhQ== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1792776402; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=8PHNjLfklMfEzjOOS2FCmsR05CtH6cjcBjMbLZYKaqI=; b=Yz41TYBmxshppoo7YDDl3Zt0U6lrsA4s5t+KsNLHApfYFgvmw/Wg5LQm9MYtMTeP zLN6iedI6Y8rHdoLdBsR7SBqM84/YQDxObNQBL3+W8D/VqBHtqiCcjdwKQvIW6d+4/K CnB3N1HvlCq+qEJES4bJUWnJ8RlqnnrPIO9O4kdI= Message-ID: <010701a169ccf478-07c899c0-1d21-4c38-8528-4db80d6786ef-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v5 25/36] vfio/migration: don't migrate device state during CPR To: svt-core@virtuozzo.com, Andrey Drobyshev In-Reply-To: <20261023-vfio_v1-v5-25-ba0bf3c6d122@virtuozzo.com> References: <20261023-vfio_v1-v5-0-ba0bf3c6d122@virtuozzo.com> <20261023-vfio_v1-v5-25-ba0bf3c6d122@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Fri, 23 Oct 2026 17:26:42 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.23-69.169.224.2 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MAD0EPF000008AE:EE_|GV1PR08MB933119:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 64a4ea9c-c718-469a-1bd9-08df312ace1f X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|48200799018|376014|32142699015|9140799003|61400799027|55112099003|4113699003|5113699003|13003099007|3023799007|10067099003|16102099003|22082099003|18002099003|5023799004|5063699009|56012099006|4143699003|261022075727099003|261022105027099003|261009223027099003|261021121427099003; X-Microsoft-Antispam-Message-Info: =?utf-8?B?T01wcy9Qd25BQVY0TG53ZFN1eXlxZUgyQ1Flc2o0QUpVNExEbE1LbWV1K0tS?= =?utf-8?B?VDVRb2ZHQXdMc0FKOVpzd2tDUmRhTi83SXJXSFpiUUJiNWRsTHZMUzlEUkxh?= =?utf-8?B?blYvTWZFYjhmYnNMUWFqaHIrSmJ3T2dUZjEyZ2ZxUXV6MVBhcW1IOFFrNEVn?= =?utf-8?B?YmtSaDdrdzU3MDRkenJXY0ZJalM3TzByODQ4N2Z6cWpaWUlMNXJHRXhDUjFI?= =?utf-8?B?V0VCdUlXdXR0aWxrZmVuOXBlSVc1UTJCMEd5TmorVHdsS3ByT2Z1OThvZHV3?= =?utf-8?B?K0hmbVdRcU1uYTBSd0wzZjZXMGxyUFNvLzVhQXY5ZG0rTFJxMFBoQWdjbFdK?= =?utf-8?B?aU8xWlRSMXR5T2JXTXV6UVYvTmYzWlo1c1ZRaHcrZEtqTzY4Wm5aYmVUbDN5?= =?utf-8?B?TFpBWEVSb3E2TVNBTDJ0Q1RaTE8vblV4Si9EMTJuZ3EyOWFwR1JXUWxyazN5?= =?utf-8?B?YUhFcDNTL3oxbStxclFKMkp2TXBSMy9CSVI0eVVDSEVLRDVWRk52ZnEwVWh2?= =?utf-8?B?UFlDakVtUzZiNmxsTE5jVk1IZEcwUmpweUhvYyt4aEcrZEpxbzh0bXM3Yldu?= =?utf-8?B?QzRQWm9rZzVPcXhrOS9JK0dVNW9NWE9vS25CY0dWSGE4cFhMZ2lMZ1JEdVJ6?= =?utf-8?B?UVpPNTBkQmZZbXBJUVlaQVlmV3kwTzlVakQrUmpqV1poL3hPUUhQRlhDUmVG?= =?utf-8?B?RHpqNCt2TVA0MExyc1BUam1wRUpGcWJDbjdoS09RZk1ob0pvd2E3UzdkYTBO?= =?utf-8?B?czZNaVZtdEZkT3Y1dTFqcWgzY3A3eXNlS0xRMnVTc3pUQjZkOUdBRms4aVhZ?= =?utf-8?B?Q0N4N1B3N2Z0QTN0Y0E0ZGdvRDV1RTh1aWFObjFMR1hWSDNXMkVQRUhGcE1N?= =?utf-8?B?Q2dxdG15Uktsc1dSQk1wVFo1TGdIOGIzT0JMRmtsSmVmOTJDUXUwbEZxQXVS?= =?utf-8?B?RmhKbDIrRUlMZVFuZmJoZmRRNEMrV2N0VnVjSWt3SDd4aytKcW5tbGlPZU9T?= =?utf-8?B?aERWVWRaRDNJN3pTQlVkRFlnSDdidFkwOS9BZ29BV25qaWUwRU0vWWFKaENO?= =?utf-8?B?eGRJczB5L2lvNmF1dW9JWWpoOVRldUwzOTBTUHhycUFWcDV6MkZ2OVJOSmlr?= =?utf-8?B?S3ErV21tSWxXT0U4c2VCTUN0Nk5sMUNEcDJSQ1pwK2NyY3E1aEp1b1djNGRV?= =?utf-8?B?Y1RtR1NsZVBmREllV3ErcS95NFlwekhpOVRnQlNHWGMxcERwMmI4M2xFTU9S?= =?utf-8?B?aUFTa0hZbXhZZ2xienlxZUFxSnA1elhFNFlMenNjb3liYWxrdz09?= X-Forefront-Antispam-Report: CIP:69.169.224.2;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-2.smtp-out.eu-central-1.amazonses.com;PTR:b224-2.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(48200799018)(376014)(32142699015)(9140799003)(61400799027)(55112099003)(4113699003)(5113699003)(13003099007)(3023799007)(10067099003)(16102099003)(22082099003)(18002099003)(5023799004)(5063699009)(56012099006)(4143699003)(261022075727099003)(261022105027099003)(261009223027099003)(261021121427099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uY+GFEGwhdJcYJwcNek3h0owDaXOexseUbPforxD+y3mRokP+L/CA6V1dIG/Fi5lvJm4ps6f5+qVy0XuS8sHSXUqtbAV+UuTB+yxfbpq1//mnoRXNEyxKQWmh3xNIJUdKKX5DVgD3MM7thUgsy0NQcejtboMwNN9xLuIfOtxPDbunYPm+/aa30rF6awSzQeFH+GHaGqalBIZWCHfmWH0rc9lUklAWTK24kem/BfRNTuXggVeenJ2yJbCak8JkKCnXNK7O4QM7BoBUzmzR7gvW/Ya0GEJqR2j7ij1qVQZTbOMlzVPZzO5bl1m7rfrgPKw4XAu0dpKL2NdbZeYaShzmOwH1fMxnJcqkq/Vmnws/8UrHssp0Lta/3Vqg77SqwKSIOwQgthMut73RJc1jJwn+5wgaVRZpDG+V4Y1r+78PmFT0Q8gcLYWMofj5G2k6Kij X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2026 17:26:42.5506 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 64a4ea9c-c718-469a-1bd9-08df312ace1f X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: MAD0EPF000008AE.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR08MB933119 Thank you for your contribution! Sashiko AI review found 3 potential issue(= s) to consider: - [High] vfio/migration: cpr_is_in_progress() is false for cpr-transfer sou= rce after HUP, so the gating is bypassed - [Medium] vfio/cpr: cpr_is_in_progress() is false on the cpr-transfer sour= ce during stream - [Medium] vfio/migration: vfio_load_state() lacks a CPR guard, so a non-em= pty stream re-programs MSI-X -- commit d5eeadfa806615694344ddad118c4979064a63b2 Author: Andrey Drobyshev vfio/migration: don't migrate device state during CPR This patch makes the VFIO migration handlers do nothing while a CPR migration (cpr-transfer or cpr-exec) is in progress. It gates the save, load and VM stop/start paths on cpr_is_in_progress(). It also resets multifd_transfer in vfio_multifd_cleanup(). > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > @@ -674,6 +674,16 @@ static void vfio_save_state(QEMUFile *f, void *opaqu= e) > Error *local_err =3D NULL; > int ret; > =20 > + /* > + * .is_active doesn't cover the non-iterable stage, so the section is > + * still sent: leave it empty, or new QEMU would load the config into > + * the preserved device and re-program its MSI-X. > + */ > + if (cpr_is_in_progress()) { > + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); > + return; > + } > + [Severity: High] Is cpr_is_in_progress() still true on the cpr-transfer source when this runs? On the source the predicate reduces to cpr_state_file !=3D NULL: migration/cpr.c:cpr_is_in_progress() { ... return cpr_state_file !=3D NULL; } For cpr-transfer, qmp_migrate() calls cpr_state_save() and only arms cpr_transfer_add_hup_watch(). When the target closes the CPR socket: qmp_migrate_finish_cb() qmp_migrate_finish(addr, NULL); /* only starts an async connect */ cpr_state_close(); /* cpr_state_file =3D NULL */ The migration thread (vm_stop, the vmstate change notifiers, qemu_savevm_state_setup(), vfio_save_state(), save_complete) runs only after the connect completes. By then cpr_is_in_progress() returns false on the source. Would that make vfio_is_active(), vfio_save_state(), vfio_switchover_ack_needed(), vfio_vmstate_change_prepare() and vfio_vmstate_change() all take the normal migration path on a cpr-transfer source? The old QEMU would then stop the live device and save its config and state. The target has cpr_is_incoming() true, so it skips load_setup but still parses the stream. That looks like the vGPU breakage this patch is meant to fix, together with a source/target mismatch. The comment in cpr_is_in_progress() says "An outgoing CPR migration has the CPR state open, from cpr_state_save() to cpr_state_close()", and the commit message relies on this. Does that hold for cpr-transfer? cpr-exec looks fine, since qmp_migrate_finish() is called directly there and cpr_state_close() runs only on error or in migration_cleanup(). Nothing later in the series appears to change cpr_is_in_progress(), cpr_state_save(), cpr_state_close() or qmp_migrate_finish_cb(). The later patch "VZ: vfio/region: preserve BAR mappings across cpr-exec" adds an mmap_blocker in vfio_device_attach() that blocks cpr-transfer for VFIO devices. That makes the path unreachable at series end, but it doesn't change the predicate. [Severity: High] Is the source-side predicate also an unlatched sample of mutable globals? qmp_migrate_set_parameters() does not reject changing "mode" while a migration is running. Could migrate_mode(), which cpr_is_in_progress() reads, change between the VM stop and the later vm start or vfio_save_state() call? The gates in vfio_vmstate_change_prepare(), vfio_vmstate_change() and vfio_save_state() would then disagree with each other. [ ... ] > @@ -810,6 +820,11 @@ static bool vfio_switchover_ack_needed(void *opaque) > { > VFIODevice *vbasedev =3D opaque; > =20 > + /* No device state is sent during CPR, so no ACK to wait for */ > + if (cpr_is_in_progress()) { > + return false; > + } > + > return vfio_precopy_supported(vbasedev); > } [ ... ] > +static bool vfio_is_active(void *opaque) > +{ > + return !cpr_is_in_progress(); > +} > + > static const SaveVMHandlers savevm_vfio_handlers =3D { > + .is_active =3D vfio_is_active, > .save_prepare =3D vfio_save_prepare, [Severity: Medium] Should vfio_load_state() have a matching cpr_is_in_progress() guard? The patch gates only the save side and load_setup (through is_active). vfio_load_state() has no check: hw/vfio/migration.c:vfio_load_state() { ... case VFIO_MIG_FLAG_DEV_CONFIG_STATE: ... return vfio_load_device_config_state(f, opaque); } Nothing in the loadvm section path consults is_active before calling load_state. If the target receives a non-empty VFIO section during CPR, it reaches vfio_pci_load_config() on the preserved device even though vfio_load_setup() never ran. That would re-program MSI-X and disable vector 0. A DEV_DATA_STATE section would reach vfio_load_buffer() with data_fd =3D=3D= -1 and fail with EBADF, aborting the migration. A non-empty section is reachable when the source predicate returns false, as described above for the cpr-transfer source. The later patch "vfio/migr= ation: don't assume the device is running on CPR incoming" only adds cpr_is_incoming() handling in vfio_vmstate_change() and migration init, and adds no load-side guard. [ ... ] > @@ -857,6 +884,11 @@ static void vfio_vmstate_change_prepare(void *opaque= , bool running, > Error *local_err =3D NULL; > int ret; > =20 > + /* The device is preserved, not a plain stop/cont: keep its state */ > + if (cpr_is_in_progress()) { > + return; > + } > + [ ... ] > @@ -882,6 +914,10 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > Error *local_err =3D NULL; > int ret; > =20 > + if (cpr_is_in_progress()) { > + return; > + } > + The same cpr_is_in_progress() question applies to both of these gates on the cpr-transfer source, since cpr_state_close() has already cleared cpr_state_file when the VM is stopped. --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261023-vfio= _v1-v5-0-ba0bf3c6d122@virtuozzo.com?part=3D25