From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <010701a1c14692eb-53603ef9-f7d7-4ee6-9afe-8297100aa7b4-000000@eu-central-1.amazonses.com> Received: from MRWPR03CU001.outbound.protection.outlook.com (mail-francesouthazon11021083.outbound.protection.outlook.com [40.107.130.83]) by lore.virtuozzo.com (Postfix) with ESMTPS id 1651481377 for ; Mon, 9 Nov 2026 17:06:37 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=cgc5ZPHbNT3cr1qGm5oDy8lNzLtV3r77wUM5ZhyR9qkk+n69FF5bB0XGrsWUIxiZ4Lggbte7Iu3UqtueVwKYGHoSUyzxXM0DjxeLa3i5D3LBvYs+Tpn+/j9/4pKqyEwZoWMTiVlmVHIVBrR1mpMqMQecz8grS6ukDzKq8qFCxDExkvJfQGojavyfka2LBQ9h2iPXU1l2iiEGuihiNfHahgQBLgiEsIxmR8W5E8p1wOTWbPSID0CxrTuQJs29p3TLFGOT8nx++d92ZBUj06hIOLJGK7LlwZRuxpeFpfp6fV1e1kewtvu4rLDoeY5mJba+hwSq1197mO6dNaP1LRG1Tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Xyph4Crpk3bqhsj9l7ubohFHBd7LfoaFVTWEjRdF4G4=; b=Eee/o1zbTlsHjukcQSGeqsjx20BGF3Z8XpLT87MNzdtjlKhEGYg3ggR8/65KA2ryXc5WPV58dgUk4bDI5uYoCZP3k5L7YRQN4d2DMKNtOeWIohdE+jE7tPw1QWMNX/a7EF4QUPmLV3/EMACwQYddaFUwjcbHAAJVYIUc3+KrYhXYu5fmqjkBuHsxS96enRLITwO8iNT/cIZcOsN6rlhlUtcQZCsKlrJhYZI4urNd5F178IxvpmK6Wr42YHx6mveweqZzYjY0hVB4bFzPJxFXM5vXDJIEHu2XYQwKM+fMSdskRoWTgTxb7T0foZ2Oy5MGJpm/Dj4wpjUt+D8BUnPcig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.6) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Xyph4Crpk3bqhsj9l7ubohFHBd7LfoaFVTWEjRdF4G4=; b=MTZPIptAEvNrkglRR3ZOLBNnGL56wuEtwWCLIsiGKFBOp8AsUnVfegDfOLpOUENYBKTzvLcUbykGuHgSPk1itDYKahhSqK4DqOHVoRxErsy2iL/CO6153B9axKnDMJ9rbT0Qku8r6qrTR7DT7LpDNyf6c+Ve9YwcZNabqWQ+3j8= Received: from DUZPR01CA0160.eurprd01.prod.exchangelabs.com (2603:10a6:10:4bd::20) by GV1PR08MB8691.eurprd08.prod.outlook.com (2603:10a6:150:85::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.12; Mon, 9 Nov 2026 17:06:31 +0000 Received: from MA3PEPF0000040A.eurprd04.prod.outlook.com (2603:10a6:10:4bd:cafe::67) by DUZPR01CA0160.outlook.office365.com (2603:10a6:10:4bd::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.13 via Frontend Transport; Mon, 9 Nov 2026 17:06:31 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.6) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.6 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.6; helo=b224-6.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-6.smtp-out.eu-central-1.amazonses.com (69.169.224.6) by MA3PEPF0000040A.mail.protection.outlook.com (10.167.241.155) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.615.4 via Frontend Transport; Mon, 9 Nov 2026 17:06:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1794243990; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=za1maq1dR4KbG/MnKgs1QPDI1wRG9ZwriywtLcbs2j4=; b=pZ5H8ggvOu3Q8X4TRGK22tow1xtWuRdUAir1MIb38Kj8kxe2rGHufgvPQu+0jzZj nE6JPXjn3D/UPC14mtlTO5hoQOOAwccigLlpm9slFlvo0CsrZaAgGZ8a3q4EJIt1PDZ 1lEXNIOee0r+bKrjwgc/ZRvEZ/WDpmqCf6JmnVoasRwEE6WWnxAAqFIVuKZ3MppwZ4R tv09XK2JMMXqB8r509OkIK0DiBNrVK2gVb7UOoesbA6mv4ft3Zm2FzgJCi6HFUtYN+s cWiD/JgpI/88g86T7fFRt4JRnGFC5dp8qcne1Vs2LO6CDS+BzHRkp3QuFjXX6YuiHps NFn1lIP2jw== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1794243990; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=za1maq1dR4KbG/MnKgs1QPDI1wRG9ZwriywtLcbs2j4=; b=Xg8DGltlGVOB2FlpXHOxwQLGkkkd/6cjc95PQWfjlBIYbGHq2XxzXd/tbcEWfzcc jc+Q9j2aQ54MsHqd94ewgIltPZOOZ2zQScmx9g3hSIKj4nOPpTBUZuWsit45ldy59IQ tXp4uRuv5GtNllcrrQ+tvU601iwNmDJcULmdWpWk= Message-ID: <010701a1c14692eb-53603ef9-f7d7-4ee6-9afe-8297100aa7b4-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH hci-dev v4 2/2]vfio/migration: don't stop the device on guest shutdown #VSTOR-153976 To: Andrey Drobyshev , svt-core@virtuozzo.com Cc: den@openvz.org, andrey.drobyshev@virtuozzo.com In-Reply-To: <20261109165435.175715-3-andrey.drobyshev@virtuozzo.com> References: <20261109165435.175715-1-andrey.drobyshev@virtuozzo.com> <20261109165435.175715-3-andrey.drobyshev@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Mon, 9 Nov 2026 17:06:30 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.09-69.169.224.6 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MA3PEPF0000040A:EE_|GV1PR08MB8691:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: d69e32e9-8f56-4336-f620-08df3e83ccf2 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|32142699015|61400799027|9140799003|48200799018|376014|16102099003|18002099003|22082099003|4113699003|5113699003|10067099003|55112099003|13003099007|4143699003|5023799004|5063699009|261009223027099003|56012099006|261028144727099003|261026121527099003|261102123227099003|261021123027099003|261022075727099003|261022105027099003|261021121427099003|261026140227099003; X-Microsoft-Antispam-Message-Info: Maq2redBYO4X9xs5yE/K1XC9P07/S1lMaX0Ot0b6jIUO8g0GTrffyifSEtZ7GJUoCGZ1o2SXDLVIH8S6uxPgWIS6xiyxajcIiEc3YA8swk7mcOjhkZs6Iee+XarvVclUPYgMp0llS2FiZl0Bu95SLP4INXpXmfyozvt2vx12s/I4GMfMN3qCQT98FWa22mHjfrQoBmLWOptRiozRhIhF7zBfxkZZlM/HXzETFW6NRB1pkrvup8G1sTHeTotAo+19/NQWOdvnNRmw4cKftSu/J+n8cy+NVbd/I6SxTvR89l+kTqNlHUc9EQpwlEtXNVCbhZo0zSzmU6cz/UIy2E6NNhPXu9/j4F+OUOOrMa0Z3bB83zn+4Xf4rxktomZKjAuba4h+uKap96pLT99gjvw4YsI0AdTJ6VPvzi4VhAvdiYG6xPcP3iqO5LldCWqg2hhw83fXE32AGeAKjwLOlhJ8aJH4z3iuupeno4KDhxM1pbpSjEQVpkHqpRMr5rHTGohfVQZdeuCttsB+cxKFSeyR34sAy+1uFFL2xzvGruQtcvzWwWFpylX6/R1FYkqDYWRbMOz+EShQoYHGCAZPoKYWDDEgTynya68+D2DPWfANazral8SI+IVoA8QBoXAffp5gdIM41IKfxZSDaePsLThqfFBxvacLMMbMhOUKON4Hqdvw1tQy7icfigrk7SovwZBoxf0nPuW6gH03s0tfRWql91dMF7m5HgcIzIrR/zieOvLowwuLMnSpxAwDnsWjHBWUqJqCbtykYdmlxWQTb7jdy65OGbDZQuQfrVTbd0YRkyRm7Q21wUsPZmkcSHkTZwW9e+9Y44nqHT1Mzu4z8OhZS5c36VNpZ1crf+02199r05SOOXBlIDdmjPH1Kv3YygVzpDw99ewCCL4kgnTZc0b20g== X-Forefront-Antispam-Report: CIP:69.169.224.6;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-6.smtp-out.eu-central-1.amazonses.com;PTR:b224-6.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(32142699015)(61400799027)(9140799003)(48200799018)(376014)(16102099003)(18002099003)(22082099003)(4113699003)(5113699003)(10067099003)(55112099003)(13003099007)(4143699003)(5023799004)(5063699009)(261009223027099003)(56012099006)(261028144727099003)(261026121527099003)(261102123227099003)(261021123027099003)(261022075727099003)(261022105027099003)(261021121427099003)(261026140227099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: oVsSHeItVTrHgdTAPqHKKPlzfjGBk3BGXuaZQPMr6byAuhDEjO6dWArQsEZJ4ibisc/etUoF5exxDWREfFtb2TLyDWAd+4K8U1IKld74r92St3IoYmYf8wzcCe7PGWUN4u/lVXXNuCfzJcsj3hUwYru74W+fgg6b3f/3DqVwwmrH65/PW1f9AM2Dpsyl1b+wd43yt7DYZ+N3zOLNFRn2gQ0NUCAX3AHfmlDkVM1lDdCHBo9hxymxOBNxyNE6D6Y1u++SZ7Yza8I0xea1yQVSCKSPsf+ljxszhBF4ymC6WDoE7y8J7m40Yd1fIfImdgm5ClshX6OtK4oO8vVcKw0oSZaIzbphiqQihM8ZckhdukTCR4sHFSiqLHIPa2e3yHZ6L3ipagUhshoM6Cbhom5g+cCSinrWcLGH/7TB+Pydg3cJEF8ptPvQDeeemso4OSef X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 17:06:30.9013 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d69e32e9-8f56-4336-f620-08df3e83ccf2 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: MA3PEPF0000040A.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: GV1PR08MB8691 Thank you for your contribution! Sashiko AI review found 3 potential issue(= s) to consider: - [Medium] vfio/migration: stale RUNNING_P2P after system_reset breaks prec= opy save in PRELAUNCH - [Medium] vfio/migration: vfio_migration_deinit() has no stop for a device= left running after guest shutdown - [Low] vfio: shutdown-stop skip leaves non-P2P devices fully RUNNING, unli= ke the commit message says -- Commit f574567e90eb95ac07629b56c1fbc945a61df89b Author: Andrey Drobyshev vfio/migration: don't stop the device on guest shutdown #VSTOR-153976 This patch skips the VFIO device STOP transition in vfio_vmstate_change() when the VM stops with RUN_STATE_SHUTDOWN and the shutdown action is "pause" (-no-shutdown). The aim is to stop NVIDIA vGPU from treating the STOP as the start of a migration during a libvirt soft reboot. A device in pre-copy is still stopped as before. > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > index 064da62ddc26..34aaa35a2cf5 100644 > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > +/* > + * The guest has shut down and QEMU keeps the VM stopped (-no-shutdown).= Only > + * a reset, a quit or a migration can follow. The device is not stopped > + * here, it only stops its P2P DMA like on any VM stop: the reset resets= it, > + * the two other paths stop it with vfio_migration_stop_after_shutdown(). > + * A device in the middle of a migration is stopped as before. [Severity: Low] This isn't a bug, but the comment and the commit message do not match the code for devices without VFIO_MIGRATION_P2P. The commit message says "The VM stop still runs its P2P round, so the device only stops its peer to peer DMA." vfio_migration_init() registers the prepare callback only for P2P-capable devices: prepare_cb =3D migration->mig_flags & VFIO_MIGRATION_P2P ? vfio_vmstate_change_prepare : NULL; For a device without P2P, the early return in vfio_vmstate_change() leaves the device fully in VFIO_DEVICE_STATE_RUNNING while the vCPUs are stopped. Could the commit message and comment say that such devices are left completely running until reset, quit or migration? [ ... ] > @@ -964,6 +980,11 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > return; > } > =20 > + if (vfio_vm_stopped_by_guest_shutdown(running, state) && > + !vfio_device_state_is_precopy(vbasedev)) { > + return; > + } > + [Severity: Medium] Can migration->device_state go stale here after system_reset? With -no-shutdown, the guest shuts down and vm_stop(RUN_STATE_SHUTDOWN) runs. vfio_vmstate_change_prepare() moves the device to RUNNING_P2P, and this early return means it is never moved to STOP, so the cached device_state stays RUNNING_P2P. system_reset then resets the device in the kernel. In main_loop_should_exit= () the runstate becomes PRELAUNCH: if (!runstate_check(RUN_STATE_RUNNING) && !runstate_check(RUN_STATE_INMIGRATE) && !runstate_check(RUN_STATE_FINISH_MIGRATE)) { runstate_set(RUN_STATE_PRELAUNCH); } Nothing in hw/vfio resyncs migration->device_state on reset. The compensating stop is gated on the SHUTDOWN runstate, so it does nothing in PRELAUNCH: vfio_migration_stop_after_shutdown() { if (!runstate_check(RUN_STATE_SHUTDOWN) || (migration->device_state !=3D VFIO_DEVICE_STATE_RUNNING && migration->device_state !=3D VFIO_DEVICE_STATE_RUNNING_P2P)) { return 0; } ... } If a migration or savevm is then started from PRELAUNCH without a cont, vfio_save_setup() calls vfio_migration_stop_after_shutdown(), which is a no-op. For a device with pre-copy support, the switch on migration->device_state accepts only RUNNING and STOP: default: error_setg(errp, "%s: Invalid device state %d", vbasedev->name, migration->device_state); return -EINVAL; RUNNING_P2P reaches the default case, so migration setup fails with -EINVAL. Before this patch the device was in STOP at this point. The soft reboot order (reset, then cont) works, since cont sets RUNNING. Does the commit message claim that "each of those resets or stops it" hold for reset followed by migration without cont? The cached state is also out of sync with the kernel state, which is RUNNING after VFIO_DEVICE_RESET. [Severity: Medium] Can a device be closed while still in VFIO_DEVICE_STATE_RUNNING on a removal path? After this patch the device stays RUNNING after a guest shutdown with pause. The stop-after-shutdown helper has only two callers, vfio_save_setup= () and vfio_migration_shutdown_notifier(), and the notifier returns early when the shutdown action is pause: /* QEMU is quitting: a device left running must be stopped first */ if (qemu_shutdown_action_is_pause()) { return; } vfio_migration_deinit() does not stop the device: notifier_remove(&migration->shutdown); migration_remove_notifier(&migration->migration_state); qemu_del_vm_change_state_handler(migration->vm_state); unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); So an unrealize or unplug in a SHUTDOWN VM would go through pci_qdev_unrealize()->vfio_exitfn()->vfio_migration_exit()-> vfio_migration_deinit() and later vfio_device_detach() with the device RUNNING. The preceding patch says a VF closed while running breaks the next VM on NVIDIA vGPU 595.71.03. Unplug without guest cooperation looks possible, for example pcie_cap_slot_unplug_request_cb() when the slot is already powered off. Is a removal in RUN_STATE_SHUTDOWN reachable, and if so, should the removal path also stop the device? The commit message lists only reset, quit and migration as follow-up paths. > if (running) { > new_state =3D VFIO_DEVICE_STATE_RUNNING; > } else { --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/2026110916543= 5.175715-1-andrey.drobyshev@virtuozzo.com?part=3D2