From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <010701a1c219e9a1-c180c4e9-e74d-4b09-95bc-8516357fabae-000000@eu-central-1.amazonses.com> Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazon11020080.outbound.protection.outlook.com [52.101.69.80]) by lore.virtuozzo.com (Postfix) with ESMTPS id 7EBDE8017D for ; Mon, 9 Nov 2026 20:57:26 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=W9IZEW7wYJIx/Stx5m5GsqBut2u2aED6dqJE8Z0Xy4wG67ejzEVpqCuDnDBr6C0AI17o4q+SeJD17r3cNYgXpEFLm0S+sRt8zYgZuAd8arsP67gF1fT6iC34PfBFz+A3fZu2rffMlW5I6CjwCWJkvMe1qkRfXtP0PQiueSuoZCCU6NJ5luUAoi4FkZQUXbnHwRTPkUbgpSlBK6vYfKxCd0NUbUi4wzGEnP6+D4hfY2kcFLpN8OttEIl0WDMzDWiQEV9b2/ppn6B9V1V/pS88CsJw0yia8Umjl7B2qjOcJq6YiHVPdavHDHB/9YIW/st1t2vIvVWe4k18x0nVk41M7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DJKvJ91knNTiW8EoPHMHrNSHP2/Frgh/6A6KCTutteY=; b=uptdrtbVpwCZ4IRxFOz3Po/tz/1D9U3zZ8p7qrk6yLYfqiqO5YEn+/XGfJnx+xVZ9ZbYmHWrUZrhjqYaGi+zwu0FY9SZ9XH2a5KLbrD6jtvfU2PBVOxgqHp+9JFVCZdtUK0F5BaRZFVKqF9mBbxJr/PBoeO89NrCS10Zxm2ul8uhUDhbtFNKh9MsG9X2wyQObMuMFDwK+Ei9tuiqz2sSk5zPiKRKU/W66pVavKhvynUysun2RjXI8me0Mu/bErAuevm4/OvWXSIw5RkOpgp6WVGF5oj7S+q0PtpbvsEDe5Mk6yeqCBCiUIWBp91fbDpWSFe5JOS/XvQgqJ1WPrrd7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.5) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DJKvJ91knNTiW8EoPHMHrNSHP2/Frgh/6A6KCTutteY=; b=Q3tPPT+umeHi7eZWI80AvTIxRu01XAIKiCT7/y1ebTOC2IEMPH4l9T2Z8KZ5KZzZQq9vsLsef7xK+4ojdUX7ODXupgozmZg0zqctwixXbg1k7NTEQjWR5BMZCcXhs76TO3oGYkRoXGukbyJoDJR9Mep30AUW33KakZTmPn32zf4= Received: from PA7P264CA0096.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:348::14) by GVXPR08MB019502.eurprd08.prod.outlook.com (2603:10a6:150:39e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.13; Mon, 9 Nov 2026 20:57:22 +0000 Received: from MAD0EPF000008C3.eurprd05.prod.outlook.com (2603:10a6:102:348:cafe::68) by PA7P264CA0096.outlook.office365.com (2603:10a6:102:348::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.13 via Frontend Transport; Mon, 9 Nov 2026 20:57:21 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.5) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.5 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.5; helo=b224-5.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-5.smtp-out.eu-central-1.amazonses.com (69.169.224.5) by MAD0EPF000008C3.mail.protection.outlook.com (10.167.241.199) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.615.4 via Frontend Transport; Mon, 9 Nov 2026 20:57:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1794257840; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=1Zzzz7wP/1L/ppK+bm8pObz2T5P0A4hT3gTSYCFtFpc=; b=FUhNRaSltKswONwIeEx3sdhA4i5PiJfm1hpY8XRzkoBgTy0JqEs5YBedibpkZSkq GTYHoyAEm+N8tooir7+IRq9seBHbBw6Z0YAhHEXUwjhkgyM7CJ2bqEtnRhwxV+H3yLj XVkmlXVhSkA5oZ8SE8zVsp5fR2lMEHw1Ot8MQc2jjxk3a5uka+xD+ZHFtIcRDjoJ0aX AAIYVHiford6sZNHEvJNyPg3NHSWsnC0oGNhUBaNSXrTLuUjLUbEYyAWabYHri5hFC4 CImEBDKepOOe4JleoAMXWNvbn0Oo+2i3qLRR23ikdWkqX7EJf7Rr8pQ2okbbxNJ6Rj3 aR20GvxLhQ== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1794257840; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=1Zzzz7wP/1L/ppK+bm8pObz2T5P0A4hT3gTSYCFtFpc=; b=HLnllwHgqD3Hvi+QMErxkPvPm1Cnm7Br5LZ677pEJ06+zJmxgRBbVqKbRbbF863t ZuFwSdzP595rKuwRkOjYz0ln0zRWE2jAFlquZnCI/SPn78ru0cQhArbI1GZz2gfR4qG yKOrO/P4zla2jDFNtNZdCOYxjuzYf97F9JGActsk= Message-ID: <010701a1c219e9a1-c180c4e9-e74d-4b09-95bc-8516357fabae-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH hci-dev v6 2/2] vfio/migration: don't stop the device on guest shutdown #VSTOR-153976 To: svt-core@virtuozzo.com, Andrey Drobyshev Cc: den@openvz.org, andrey.drobyshev@virtuozzo.com In-Reply-To: <20261109204518.220617-3-andrey.drobyshev@virtuozzo.com> References: <20261109204518.220617-1-andrey.drobyshev@virtuozzo.com> <20261109204518.220617-3-andrey.drobyshev@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Mon, 9 Nov 2026 20:57:20 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.09-69.169.224.5 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MAD0EPF000008C3:EE_|GVXPR08MB019502:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 4bfe42a4-89cd-47ce-4092-08df3ea40c52 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|48200799018|376014|61400799027|32142699015|9140799003|5023799004|4143699003|5063699009|261021123027099003|56012099006|261026121527099003|261021121427099003|261022105027099003|261022075727099003|261026140227099003|261009223027099003|261028144727099003|261102123227099003|10067099003|55112099003|16102099003|22082099003|18002099003|4113699003|5113699003|6133799003|13003099007; X-Microsoft-Antispam-Message-Info: oaGCgpSgBOGu9A7f/59XsjllGJvfSP7xmsBSy9tE3LJ1Bo3V9vBMeE/MH/FxmYR/T6TpxELfmU02oasLS/atPmYpRNB1GlgYhQDGnGK4B5b0xPi7J2ZT8yI/GKAQiJ6AAuOZX3kiWgdsIqAJKSccwI1CwHXAQbOhIggZ1ykVQzZ37cUM9N8jp2B6vJJH18k/qhGsKUn7v3Tik6KAo9EztDQ8YpsBp3G5y5PfUpqGxaOqCkvPQlJATgW/6pnJHQVyCe8/G0dyBT+HuQ3OKmW1/Nkq4Hh6ifa69t3UjKiYDZvMuluju/OWI11dejvxEu/xcKIiiv8+TD83LQnaIMYE3O1fno3yyrWtvDyPygYcghj4PehIdY89Pu1xWK5CuQ0gFi7WEzkRnS/qzwJJxbv90poeJ7IlUrnM4nn4feAjycJltjq+gBwtnnvO7VRIhFWklmcnN1RSdRDwm72uhTbzUHwhyHZSpXlqkGu0dHrb/KfBiK5wgYgfb1Ldxm94fh4uMp+0TCvceh4InXIIUfj+THsiEv8A8cyCbPFBhTHrvG7ALOtCoRZC3B9M87NXr6SwqhG5ei2J3C8fgT4Oir2fDibF4/0n/ZD354JP0e45PuSCArVM6qBhyciiXhbfEfP/8ZzxCiRsqLiH9VF6xRTa6RsdPZzpOJx+7ZLAZgUIyZFtotvYOUwAiKSpfC8dI7n5P3LcOmLM+ifBKNnLfSq+3y+gaMjkrF5e2HubAvn8hRngZsq7IvYLZXbxi56wwNxCgu3vgQim4lUvIqzupJcbgLbEud7gqZhARekiH5mDCUsL7cErGu4A8lk9AfDURetKdvHHjIYj+YvJJ5ptyf9kxuqQZb1uI6PKpGSKXXTViclRST/263EwjbApsTFMNhg+ X-Forefront-Antispam-Report: CIP:69.169.224.5;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-5.smtp-out.eu-central-1.amazonses.com;PTR:b224-5.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(48200799018)(376014)(61400799027)(32142699015)(9140799003)(5023799004)(4143699003)(5063699009)(261021123027099003)(56012099006)(261026121527099003)(261021121427099003)(261022105027099003)(261022075727099003)(261026140227099003)(261009223027099003)(261028144727099003)(261102123227099003)(10067099003)(55112099003)(16102099003)(22082099003)(18002099003)(4113699003)(5113699003)(6133799003)(13003099007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FXBPqi0cgY3zoI9Cf4JES89CgDP11ruhXA21jjL/HwM96rp0FjxIwDVmh8hGSW6sMxBBueNHSFt5KfgdIKAA38FKaLtF+i/jrLStFTqc8wqeoGfqY+qu5+BYO17fMV8mJfQyKcOrGbs1VbnteId7mPZXzwuUpvQEDFSIIdYGzYr7qpCftk8MCkxf+QkXkx9iut+BNUQfYUB53AMtDgHAQ0ETrVUAZbHNj8que1v0qI88oSnzWSTzMNPnccQ1HE1uDSfP3+eCbuFaf90Au2Qt/9LXDDybtHmJG1U3F5bpwpp84QR7UEKQn0nI85tBRIXFG1D4iErSv9Jsv5VL7xNi4sYeglkUjGhvBAO3T1/uN/PiJ4zDCoy79HxG1H2VS4ltRvYD2n9j5Eq9E9k1HgxI81fVSERXGysQyIiot8AyzswVuQbaF8fxY+ZGpPTpzf+t X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 20:57:21.1109 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4bfe42a4-89cd-47ce-4092-08df3ea40c52 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: MAD0EPF000008C3.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: GVXPR08MB019502 Thank you for your contribution! Sashiko AI review found 3 potential issue(= s) to consider: - [Medium] vfio/migration: device left RUNNING at hot-unplug after guest sh= utdown - [Low] vfio/migration: shutdown-skip keyed on run state and global action,= not the cause - [Low] vfio: P2P round still runs on guest-shutdown stop, so the skip is p= artial -- commit e2760d850f2ea2329cc5d74a55a03aa8958eb853 Author: Andrey Drobyshev vfio/migration: don't stop the device on guest shutdown #VSTOR-153976 This patch makes vfio_vmstate_change() skip the move to VFIO_DEVICE_STATE_S= TOP when the VM stops in RUN_STATE_SHUTDOWN and the shutdown action is "pause" (-no-shutdown), unless a migration is running. The aim is to stop the NVIDIA vGPU manager from treating a soft reboot as the start of a migration. The device is left to be reset or stopped by the later reset, quit or migration paths. > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > index 9c0ba2bde5c6..536fc8b7fbce 100644 > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > +/* > + * The guest has shut down and QEMU keeps the VM stopped (-no-shutdown).= Only > + * a reset, a quit or a migration can follow. The device is not stopped > + * here, it only stops its P2P DMA like on any VM stop: the reset resets= it, > + * the two other paths stop it with vfio_migration_stop_after_shutdown(). > + * A device in the middle of a migration is stopped as before. > + * > + * The same run state with the "poweroff" action is a quit of a running = VM, > + * which stops the device as usual. > + */ > +static bool vfio_vm_stopped_by_guest_shutdown(bool running, RunState sta= te) > +{ > + return !running && state =3D=3D RUN_STATE_SHUTDOWN && > + qemu_shutdown_action_is_pause(); > +} [Severity: Low] The predicate tests only !running, state =3D=3D RUN_STATE_SHUTDOWN and qemu_shutdown_action_is_pause(). It does not look at what caused the stop. In main_loop_should_exit(), any request returned by qemu_shutdown_requested= () reaches vm_stop(RUN_STATE_SHUTDOWN) when the action is pause: if (shutdown_action =3D=3D SHUTDOWN_ACTION_PAUSE) { vm_stop(RUN_STATE_SHUTDOWN); The spapr watchdog expiry also calls qemu_system_vmstop_request(RUN_STATE_SHUTDOWN). Under -no-shutdown, these host-originated stops would also skip the device STOP, even though the guest driver is not necessarily gone. The skip also applies to every migration-capable VFIO device, not only the NVIDIA vGPU case. Is the "guest shutdown" wording in the function name, the comment and the commit message broader than what this check actually matches? A non-P2P device then stays in RUNNING, and a P2P device only reaches RUNNING_P2P, while the VM is reported as stopped. Its DMA and interrupts st= ay live until a later reset, quit or migration. [ ... ] > @@ -964,6 +980,11 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > return; > } > =20 > + if (vfio_vm_stopped_by_guest_shutdown(running, state) && > + !migration_is_running()) { > + return; > + } > + [Severity: Medium] This early return leaves the device in RUNNING or RUNNING_P2P while the VM sits in RUN_STATE_SHUTDOWN. The commit message and the new comment name only reset, quit and migration as the follow-up paths that stop the device. What about device removal? The hot-unplug teardown is: vfio_exitfn() -> vfio_migration_exit() -> vfio_migration_deinit() vfio_migration_deinit() only does the following: notifier_remove(&migration->shutdown); migration_remove_notifier(&migration->migration_state); qemu_del_vm_change_state_handler(migration->vm_state); unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); vfio_migration_free(vbasedev); It sets no device state and does not call vfio_migration_stop_after_shutdow= n(). That helper is called only from vfio_save_setup() and vfio_migration_shutdown_notifier(). vfio_device_detach(), vfio_legacy_detach_device() and iommufd_cdev_detach() also issue no stop or reset. Before this patch the device was always in STOP by the time it was unrealized. Can the device now be closed while still in RUNNING or RUNNING_P2P if it is unplugged after a guest shutdown, with no reset, quit = or save in between? I could not confirm which configurations can complete an unplug in that state, so this may be a narrow window. [Severity: Low] This skip covers only the final STOP. vfio_vmstate_change_prepare() is not gated by the new check: new_state =3D migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY= ? VFIO_DEVICE_STATE_PRE_COPY_P2P : VFIO_DEVICE_STATE_RUNNING_P2P; vfio_migration_init() registers it for every device with VFIO_MIGRATION_P2P: prepare_cb =3D migration->mig_flags & VFIO_MIGRATION_P2P ? vfio_vmstate_change_prepare : NULL; So a P2P-capable device still goes RUNNING to RUNNING_P2P on the guest shutdown stop, before vfio_vmstate_change() returns early. RUNNING_P2P is a migration-flow state. Could a vendor manager that treats migration-flow transitions as the start of a migration still be confused by it? The commit message says the P2P round is deliberate, but it does not say whether the NVIDIA vGPU manager tolerates RUNNING_P2P. It also does not say whether the testing was done on a device with P2P support. --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/2026110920451= 8.220617-1-andrey.drobyshev@virtuozzo.com?part=3D2