From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from OSPPR02CU001.outbound.protection.outlook.com (mail-norwayeastazon11023088.outbound.protection.outlook.com [40.107.159.88]) by lore.virtuozzo.com (Postfix) with ESMTPS id 8D9FA80E7E for ; Mon, 9 Nov 2026 19:15:08 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LBSY/0MyBI0uvLeiE39XeVszamgUzV36C9lfLkqdHQ8r5d9kBeGlZyCWU9NNQ6jZ096UMudkkAYQZmDf/r3kAa1zRlXG14rWF6Mr19qGJs8vvjbusHHKjmkR9QDL9R6gXe8/SuNPLfXOVzZA0n0MzmBJAcGwht/RPOwWAt39RM74QS0whqrLaYlnUbNmzfmdn7iu0dy3+k2kdRkn0ZhL78OBRD4F5fqUyRMCNALtwZFWBs3jJBAP0Ibv3nK04zQUCMkvTDjfeQ9b50VMCJdJnLtkEWR4lhS0eFRvRIh5u02Dn2AEGDh9tgzQ4cSTMozx7tfQwopv5X2mUqIb7GZ+Sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QvQheyuk9lpjiqm1evHhEs0UDoCUWM6IoxwLkmQU7ek=; b=b809edRF8V3+QsIN8wUu7wbiJ7aOM1pq6kbWnV1y7to+5i9Nc6xDwHMC2lunPs82OZRkG91/+vEf2zZ/kLUWv1UCRTbI+ETGvoM50bRdiX7tHuejV9YVHyshPC6vo6sQiks0b7KD9ltIyi9+q4ohF0A94cuzFH6I2ljOxodeeISQn1zVnxQ4Sm9ClvGaU9ZVF2zDC4By9Q+tsCP+7K2x3QtzIWQqY02UPTyyM1o9gHzJ9WAH4IKQKqBqorbbdiyRrJLg+7AQnFbEBRmPS98dO8ahAtOEyNNJneuEiOvFJWWL3a4Q0qOmD8JxGrIyKApKNXYp0O5/H/AAFzBX4xQ8fw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.6) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QvQheyuk9lpjiqm1evHhEs0UDoCUWM6IoxwLkmQU7ek=; b=IQx83G11o66g25HGxs4nUYh2n+InZ+kQhjRZjNGxkpsZuOht9bKQyjW0dbtt2tEug4hqpTs14O7zB44225L0sQ4cTrnSkqtR9r2xO2XDp+zYXZPvdz7kpRDUfChMV3lnOBC11eYNDLpuxYw1nEM/KdNc9kiHxDp0PX5a0mttl+g= Received: from AS8PR05CA0026.eurprd05.prod.outlook.com (2603:10a6:20b:311::31) by BR0PR08MB592628.eurprd08.prod.outlook.com (2603:10a6:2c8:57::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.12; Mon, 9 Nov 2026 19:15:05 +0000 Received: from AMS1EPF0000008D.eurprd05.prod.outlook.com (2603:10a6:20b:311:cafe::19) by AS8PR05CA0026.outlook.office365.com (2603:10a6:20b:311::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.13 via Frontend Transport; Mon, 9 Nov 2026 19:15:05 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.6) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.6 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.6; helo=b224-6.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-6.smtp-out.eu-central-1.amazonses.com (69.169.224.6) by AMS1EPF0000008D.mail.protection.outlook.com (10.167.242.84) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.615.4 via Frontend Transport; Mon, 9 Nov 2026 19:15:05 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1794251705; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=x8IbtbfwHDH0lPa/Sk2M3aY+fKAA8cdSiZcNLbb3KCo=; b=phsEMcTt+MxrIptOap8epqL07r4Q/0xdPRJbf5dcX1UJs1SijS+fnEreTn2kIH2N qOyuEANyKdoUadmTkPdV3oUu1yXl/FjBDFkxtvdSFkdQ8J4VPE2qI6dUDNxYBwY/pnG 4LRVHJPZdPyAd0+GTZNgC+dr2qhK1gTSB8v7c9NWfXqR3CAQoQ+8kWB4f4diZ4UZtGU bvGCh0FY8A6DGVEhWWBxAaWWxxgDDxohj6xDJ2laWM0XiAPNmtHtfbjtNzi1r2sxW8A bsiqGVpqBL024Furz3jYKMJ68PIpZLZJN98+56jsyo1b4lweF7Tb8XOU1l7WKvsm9tI tS1+H/SyOw== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1794251705; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=x8IbtbfwHDH0lPa/Sk2M3aY+fKAA8cdSiZcNLbb3KCo=; b=UDaG3vc/mLiotBWhLUoxxLThTl811U1NQpo4vpQ/W81cSs4DuvPtRaTl7rkrjcw4 7VMoMB77m7RW1DQamtwmRqyErK5Hh1I0+o+mOipU/xPMvz6r7XuSkpd3wQnaIwnqxA2 lixLTzBN21zf4x22w1A75gn58KcxFqAUZ0smg9cE= Message-ID: <010701a1c1bc4a67-ffd356e1-5897-49b3-985d-2d44f1efcab5-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH hci-dev v5 2/2] vfio/migration: don't stop the device on guest shutdown #VSTOR-153976 To: svt-core@virtuozzo.com, Andrey Drobyshev Cc: den@openvz.org, andrey.drobyshev@virtuozzo.com In-Reply-To: <20261109190342.200293-3-andrey.drobyshev@virtuozzo.com> References: <20261109190342.200293-1-andrey.drobyshev@virtuozzo.com> <20261109190342.200293-3-andrey.drobyshev@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Mon, 9 Nov 2026 19:15:04 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.09-69.169.224.6 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS1EPF0000008D:EE_|BR0PR08MB592628:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 7aae7821-7cb6-4fe0-ff8f-08df3e95c314 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|9140799003|48200799018|376014|32142699015|61400799027|261009223027099003|5023799004|4143699003|261021121427099003|56012099006|5063699009|261022105027099003|261022075727099003|261026140227099003|261028144727099003|261102123227099003|261026121527099003|10067099003|55112099003|16102099003|22082099003|18002099003|5113699003|4113699003; X-Microsoft-Antispam-Message-Info: ttsaX5Y/+xax6DmbXtUJNtZPVfQqWPsEVVdNkVruclRTFzOasmgGaUQjhrIZ2/tMyn52gacee9neEKaHH7bi/yflz32UOj88lNWa2PLbBkmIBgqayuPnWeY3GhSLhyDb9V+wQceRdeW3eMhBgQ5VloduKFDkNO5yQxjVo+r7/2d4qIT+wgcSq4oXJnp3OAW7Dp6T+vjemRe4NsVOFAwhc0nDIjQCc7OzPDxGY1g8y0TSzVT1VxvDQYCor1BmLOkwm+6yv7boIS6+sIg04hgN6EqxN8QrZUm3ZHFNgP4cQh8w1I/UqgY57NdMEVuOcoxKk1Abg+qYOP6YVJj/QjmJj7lRYVyBFF5b1Ucvls1+xXhk2MlcIxCUcC41LO/n6flvBP3wOTse3GUMaONvNapIU+WQ3GwDkSzw4XVmLjrpv06rKZWHTqPA4uuoxH8hKqctzrJNaLjPrfLC+2busg/aaRT1+ACQZrabk3UtaVr8/R1THFAdgNSE2cLNwV/KIqTH1s631yUaBVtqIgHTiMhtvU2JIVlR3rfz/6lY3RM43/AObZp0WLur8BjLdU3H8VhbRDXQCfwuvgjJr0ulLEXFolC63kwKUevT6VEmBs/6ajCh5nsB+x9v21iggb1b4F7UFKbz7FbwYLYaC+Je2iQ5sGQEAmnQTX/ZsXbYwDevdp0LFuomz4Svf9iUzsxEZzsQtaz1aAnvJBKL3Ftg/nFpdDsbAUprJ8A+TNYniZgqV36b+76Vs3epqpRPth/bcdG1PYhlPfGmIgMAYzklJwMjwdX27Yiu2v0dU87lI9m+9XIi7HekDZLOjVLntECu8Wo7mLw0l+NxDgdsO3WelNadVk/GA1834JNKpfnaCll16d8r3ZFEfMHoQ8iS9EUaZLNL X-Forefront-Antispam-Report: CIP:69.169.224.6;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-6.smtp-out.eu-central-1.amazonses.com;PTR:b224-6.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(9140799003)(48200799018)(376014)(32142699015)(61400799027)(261009223027099003)(5023799004)(4143699003)(261021121427099003)(56012099006)(5063699009)(261022105027099003)(261022075727099003)(261026140227099003)(261028144727099003)(261102123227099003)(261026121527099003)(10067099003)(55112099003)(16102099003)(22082099003)(18002099003)(5113699003)(4113699003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: eHTxRILhSubICWBcNnSFEslJEcSkpZG3sy2+22D+HO85VBjNXORr6j5fQ/U/ztJXb1sHGrj/zxf0W8T9DKIOd/sqDBZP+Tf/FerOLGSwwwEsmJ+Aje3nl2PFlisOoCF5RYeF8AogyKRB6HusfZv1zIFnOudD49cp5ohH2HqS5UtD4L1jx1Nt5RsnvuPw9fhvxp9ksoRsLUrEE3FysIKp47rybe46HmwBfik/7krVCFYwRHxzy1RlA+nNn94XRofdMKvWNJn5wsVV0F9GPSYM6PIxHznDhnOakCPsfZ8pLgPiJA86j9uEmI+xuUFCf7uCaTWpwuTOVcsL9rerFenSDqRw8h8c/+I7bTy4dQcBOzX4G69G602oWvO/v2dXE8EGVrVL3otRqMMNPRCm/psvHhesWxAkbVDkjmBbMG1QVJLLejgp0vQCjfpWa9NDu9x8 X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 19:15:05.3242 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7aae7821-7cb6-4fe0-ff8f-08df3e95c314 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: AMS1EPF0000008D.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: BR0PR08MB592628 Thank you for your contribution! Sashiko AI review found 2 potential issue(= s) to consider: - [Medium] vfio: non-P2P device left RUNNING after shutdown+reset escapes s= top on quit/migrate - [Medium] vfio: shutdown during non-precopy migration leaves device RUNNIN= G at switchover -- Commit 9684fec7cb9ba3f7a56f75e07435bf932dc36779 Author: Andrey Drobyshev vfio/migration: don't stop the device on guest shutdown #VSTOR-153976 When the guest shuts down with -no-shutdown, vfio_vmstate_change() no longer moves the device to STOP. A device with P2P support only stops its P2P DMA, and a device without P2P is left RUNNING. The device is expected to be stopped later by a reset, or by vfio_migration_stop_after_shutdown() on quit or migration. > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > +/* > + * The guest has shut down and QEMU keeps the VM stopped (-no-shutdown).= Only > + * a reset, a quit or a migration can follow. The device is not stopped > + * here, it only stops its P2P DMA like on any VM stop: the reset resets= it, > + * the two other paths stop it with vfio_migration_stop_after_shutdown(). > + * A device in the middle of a migration is stopped as before. > + * [Severity: Medium] Is the claim "A device in the middle of a migration is stopped as before" accurate for a device that does not support PRE_COPY? Such a device stays in RUNNING after vfio_save_setup(), and vfio_device_state_is_precopy() is false for it. Consider this sequence: vm_stop(RUN_STATE_SHUTDOWN) vfio_vmstate_change() vfio_vm_stopped_by_guest_shutdown() && !vfio_device_state_is_precop= y() return; The migration thread later calls migration_stop_vm(FINISH_MIGRATE), which calls vm_stop_force_state(). The run state is already SHUTDOWN, which is not live, so this path only runs runstate_set() and bdrv_drain_all(). No vm_state_notify() is sent. vfio_migration_stop_after_shutdown() is called only from vfio_save_setup(), which has already run, and from the shutdown notifier. So can anything stop the device before the final RAM sync? vfio_save_complete_precopy() and vfio_multifd_save_complete_precopy_thread() both say "We reach here with device state STOP or STOP_COPY only". They would then request STOP_COPY from a RUNNING device, and the device may have been DMAing during the final RAM sync. > +static bool vfio_vm_stopped_by_guest_shutdown(bool running, RunState sta= te) > +{ > + return !running && state =3D=3D RUN_STATE_SHUTDOWN && > + qemu_shutdown_action_is_pause(); > +} [ ... ] > @@ -963,6 +979,11 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > } > =20 > + if (vfio_vm_stopped_by_guest_shutdown(running, state) && > + !vfio_device_state_is_precopy(vbasedev)) { > + return; > + } > + [Severity: Medium] Can a non-P2P device be left RUNNING here, so that it escapes the stop on quit or migrate after a system_reset? With the pause action, this early return leaves migration->device_state as VFIO_DEVICE_STATE_RUNNING. Then system_reset runs qemu_system_reset() and vfio_pci_reset(), which only issues VFIO_DEVICE_RESET, a hot reset or a PM reset. It does not update migration->device_state. main_loop_should_exit() then does: if (!runstate_check(RUN_STATE_RUNNING) && !runstate_check(RUN_STATE_INMIGRATE) && !runstate_check(RUN_STATE_FINISH_MIGRATE)) { runstate_set(RUN_STATE_PRELAUNCH); } The run state is now PRELAUNCH. vfio_migration_stop_after_shutdown() has this check: if (migration->device_state !=3D VFIO_DEVICE_STATE_RUNNING_P2P && !(runstate_check(RUN_STATE_SHUTDOWN) && migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING)) { return 0; } With a cached RUNNING state and run state PRELAUNCH, this returns 0 and does nothing. On quit, vfio_migration_shutdown_notifier() calls the helper, which is a no-op, and vm_shutdown() sends no VM state change because PRELAUNCH is not live. Can the device then be closed while still RUNNING? On migration or savevm from PRELAUNCH, vfio_save_setup() calls the same helper, which is also a no-op. Can the device then be saved from RUNNING, contrary to the assumption in vfio_save_complete_precopy()? Before this patch the shutdown stop left the device in STOP, so this gap is new. The commit message says "the reset by itself" covers it, but vfio_pci_reset() does not stop the device, and it can fall through silently when no FLR, hot reset or PM reset works. libvirt's soft reboot sends cont right after the reset, so that flow is fine. A manual reset followed by quit or migrate would hit this. > if (running) { > new_state =3D VFIO_DEVICE_STATE_RUNNING; > } else { --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/2026110919034= 2.200293-1-andrey.drobyshev@virtuozzo.com?part=3D2