From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AM0PR02CU008.outbound.protection.outlook.com (mail-westeuropeazon11023138.outbound.protection.outlook.com [52.101.72.138]) by lore.virtuozzo.com (Postfix) with ESMTPS id 4A31281377 for ; Mon, 9 Nov 2026 20:55:20 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=U6o7E3I2WHEMTbNKoyeUfGAAoYSB0HKCTIFACjRvqKhHkp6v1h7eg1Tq9QHXPcpN9xf9XR0PgY27W/+yBLlM7cPj+1Lf7vrJLKSOIg4mVz1Hh4D5EuUdhuZz6RphNMnTf4baMQdrFXDTEVbSrMA8TX8LKAtn+F2xQP0dOlVQMmyLLHxoeQ/BcG5i2WWbZNqxtAfWCohA18n5t1UZaHut1rqU3adUBEhgA8cCJDTDJHwKf9qruPO7G7ia+qkSy4NJLuMCAyvTMvxHR4ibHC45MwWq/6JIRUpvem+xPClvSwb5samezMX/n3Smt+O3xiyHRRew7QGtWuzTeEo3B4BKWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/q5XGkzk10ApoBE0vca1NmaoxEmC1KmgEc3YMKcHfnk=; b=J1Jcq8JaMFEU96P/FE/Fbu50HfOK4e9THjvLF9+i6nH+a9O1RyCS2quks4mmud/iP+buKTuCIFU7isJaHiTIcdF2fhjlQWRaEpDDR7/qgzDJmk9XbTQMktAZgrZ5US2KRilGYr4OC9E2qeWbtlK5M7HYkd5hXzfdMWOcWzeoxO3PaDTSo0qCjPTA8n9ECbvQzzjtiohkzSjLTi8PBZaC3rcgUhoQq6OuS/3xVVe7s9XfwVz2U1s3AYj4lpCvXcm76ccnJbhsDShkWZOraSW0CodHRr2+xgdi24zipT8ffeXizel+uV1O9A3eBpd/KKgSJzt+dSZTOO8iEM0lo3oW2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.5) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/q5XGkzk10ApoBE0vca1NmaoxEmC1KmgEc3YMKcHfnk=; b=PYsIjIdW9VdWjwge43Q1dMU3rUMrkybdBUDZad2aqNdmEkh5UTj1Y4RlP4JGMzcwsKCrZ4dFC4VAz/PIR2VnCuoCpirlFvILRyNdsWL3pHFbaAiqr7/ckXeo5JNalaNbeJBy6WZujb56EvhUeIQDiVho6CUtoourzxD/nQSBm+4= Received: from PR3P189CA0059.EURP189.PROD.OUTLOOK.COM (2603:10a6:102:53::34) by MRWPR08MB11682.eurprd08.prod.outlook.com (2603:10a6:501:9b::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.8; Mon, 9 Nov 2026 20:55:15 +0000 Received: from ZR1PEPF0000E464.eurprd04.prod.outlook.com (2603:10a6:102:53:cafe::1d) by PR3P189CA0059.outlook.office365.com (2603:10a6:102:53::34) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.13 via Frontend Transport; Mon, 9 Nov 2026 20:55:15 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.5) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.5 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.5; helo=b224-5.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-5.smtp-out.eu-central-1.amazonses.com (69.169.224.5) by ZR1PEPF0000E464.mail.protection.outlook.com (10.167.241.57) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.615.4 via Frontend Transport; Mon, 9 Nov 2026 20:55:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1794257714; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=7+8kRzy5mgwhh1D3kjp6WJtxiipEVbTlQiXDbgBAKzg=; b=oSPQb/vOLMUos5CeGQPz97eNdXD1Z6krCfw6uolbMiLc0IENWkU/H9uUQ4mirtsc v5QzHEq67VLD3d90+oixw+7xkao4SZkpxroHn3vlVvYwkz12yDUlfrXXRFs2MrrsU6K pTpjVTN4AepaPZpDAW11o9iQJEX0fJA6rp9p+Q702ZOT6YvTBFvT1AsLDbSrSVCEjnb eA5V71HKseRRx6ssW2e/v1qURsbAroSJy/4VIMdzgW96LjtgjsiRwoMGCy0ChT8DvA3 FQYF0AhYG6oUymOmxrRtNS863V5SVeIhlZ+1Hn8sd/aStGb3LkeBRBQaGqTSF4rugzd RWfidu+gFA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1794257714; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=7+8kRzy5mgwhh1D3kjp6WJtxiipEVbTlQiXDbgBAKzg=; b=Jwy0/iK9tYAo14h/Vc1HODuJghVWSS2j5DKW+1X/HLZBoowQQHCgyiZCIVNE5zj+ SuYefTkzNP0IVx95zTTwx+WdKzAOACuu/uVVih79PdhMIgIfNv6VzGgYvDG91ArAnNB N0oHE73srNCNXQA+P1Y3SY8T1sMPN2/EADF08AP8= Message-ID: <010701a1c217fe1a-e6584213-39bd-4c2d-b5c0-3d03ca1537d2-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH hci-dev v6 1/2] vfio/migration: stop a device left running after a guest shutdown #VSTOR-153976 To: Andrey Drobyshev , svt-core@virtuozzo.com Cc: den@openvz.org, andrey.drobyshev@virtuozzo.com In-Reply-To: <20261109204518.220617-2-andrey.drobyshev@virtuozzo.com> References: <20261109204518.220617-1-andrey.drobyshev@virtuozzo.com> <20261109204518.220617-2-andrey.drobyshev@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Mon, 9 Nov 2026 20:55:14 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.09-69.169.224.5 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: ZR1PEPF0000E464:EE_|MRWPR08MB11682:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 6c50e1e3-3624-4c6f-bcb1-08df3ea3c13c X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|376014|9140799003|32142699015|48200799018|61400799027|55112099003|5113699003|4113699003|10067099003|16102099003|22082099003|18002099003|13003099007|56012099006|4143699003|5023799004|5063699009|261009223027099003|261028144727099003|261021121427099003|261022105027099003|261026121527099003|261026140227099003|261102123227099003|261022075727099003; X-Microsoft-Antispam-Message-Info: WbYoila1nsyem6BlppcCOSyeNWcICYlstxRswi7Zm/PEBJE9iHybhdXtHXC2/VG1ralWAwwRTWT4vsEG7OCUAlsBv8lpYOLwF/m7d21J2OhrQW/LOscpNqPd29z4y4DnwcTWp7pBx+YRinDOJ3myd4l1C27VOthqOgLK3Coo0fjHMhrheU57zdzbv8tUpPsVW+XT3Q0AezBmSWX+zPJ79MWVSbYlsaWOYPz3avnMKXXIePxCcCivDXVU9glbh0ZUcgrJtpAE+FN4FRewEfFyr+2vLlNhRF8tm3msyeLEK0e8bXdoNv5rp8Krffc48ncfPR25L41/yZbFxrDMY3xOfvdW3iE/rdXzRegduVXMxCj6dXP0DF8y/bEmwBKraKf/4MCmbymPyv07j+iWPikMrT2jRvNWyZZCrib12Oukvs/u4rwLoDfKnWLaWKIWTOtY8QSGwHKbTKlbFbmVOa2ubACn2SgF7HOPz5sF4oMjphRk0SB/mFSmFoFHK/jbPqyPFD92O4hoMafu8o9rH7h9N1ZYBZ2x4mSQziY0d5hdJHDfjRZlaVdjIvdAmvKGYeZVcUoIS+j/3gUJyjqFOauUxR0uBKg8ksLAyWHh8PNIy8plGVSR/6vgJMyZKLgtAEUI6ons1pOUD+rHcv7jDYv/V3PIM+9yzXJ09f6/9N6ajvTKmLHXNKgl1vFhXOClD9yqyC27pjwAoeSI/+xdadciZ/c4r68QME4ZW1iiQjPVRIvx8+7Pv1BOwb8yXPqYO/z3gw4Q3r/vLBIeK8Jr6nF0sJeY0ZvRL6aeJrXpg6WHhkxUEoaflmDEPjDsm3UblJ1v1cyEFQLRf4QoJvmC+Tf8vFoxwmSREnGT7yF/q5HQcWlgBQzQg4LLtVOMkZTmRehq X-Forefront-Antispam-Report: CIP:69.169.224.5;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-5.smtp-out.eu-central-1.amazonses.com;PTR:b224-5.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(376014)(9140799003)(32142699015)(48200799018)(61400799027)(55112099003)(5113699003)(4113699003)(10067099003)(16102099003)(22082099003)(18002099003)(13003099007)(56012099006)(4143699003)(5023799004)(5063699009)(261009223027099003)(261028144727099003)(261021121427099003)(261022105027099003)(261026121527099003)(261026140227099003)(261102123227099003)(261022075727099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1mwdeJa9E8PLufwgELO3BxW60hBbjFIiGXHU8uo+UlKZT0aBmSGF5KCX6cnm7HjuqNg8luXerlKvrOfTCvjQ+unsmF4pDpWOXKmyQTR0Z4WQEr+vtvkBWUPKb5/88tJ2UDvSiqXq4X5C1w+5qhLHMUibGvPS4zU34wEjAO4yBw+bu4GBesRNik7d2NMsQhLo04VsPlkI7hx7qPeXYXcDWJEwauFJIqxesUHF3wA4AinaznTILGDaHv127aIbkj/gD/wEjmANCDMCpw0H5fSe9ZMKPd6qqflPg+sBIU3vtUBqOZzoSULrg3sv0LGLCefRWNUhoEavEz6oYtRnsD8BnA6MYfUzfv8bgYSOnMBRp92CvQ49S4CeIcmzRhQDhyivntecdc9JU7e5ntS0kLurCLuyuFZe+yV4BmZTqVAncNiWFH9RC52Xub0N5MWgm9Sl X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 20:55:15.1679 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6c50e1e3-3624-4c6f-bcb1-08df3ea3c13c X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: ZR1PEPF0000E464.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: MRWPR08MB11682 Thank you for your contribution! Sashiko AI review found 3 potential issue(= s) to consider: - [Medium] vfio/migration: cached device_state is stale after reset of a st= opped VM - [Low] vfio/migration: commit message and comment describe RUNNING_P2P sta= te that only exists after patch 2 - [Low] vfio: stop_after_shutdown() stops RUNNING devices to STOP with no P= 2P quiesce round -- commit c3c452e7f3860c3cb5398bd283a1b5d4d73e848f Author: Andrey Drobyshev vfio/migration: stop a device left running after a guest shutdown This commit adds a helper that stops a VFIO device left in RUNNING_P2P, or in RUNNING while the VM is in SHUTDOWN. The helper is called from a new shutdown notifier and at the start of vfio_save_setup(), so a device is not closed or saved while running after a -no-shutdown guest shutdown. > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > +/* > + * Stop a device that a guest shutdown left running, before it is closed= or > + * saved. Only that leaves a device in RUNNING_P2P, or in RUNNING while = the > + * VM is in SHUTDOWN. The devices are stopped one by one: a device in > + * RUNNING_P2P has been through the P2P round of the VM stop, a device in > + * RUNNING was set there by a failed migration, which has no such round. > + */ > +static int vfio_migration_stop_after_shutdown(VFIODevice *vbasedev, > + Error **errp) > +{ > + VFIOMigration *migration =3D vbasedev->migration; > + > + if (migration->device_state !=3D VFIO_DEVICE_STATE_RUNNING_P2P && > + !(runstate_check(RUN_STATE_SHUTDOWN) && > + migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING)) { > + return 0; > + } [Severity: Medium] Can a device still be closed or saved running after a reset of the stopped VM? The stop decision uses only the cached migration->device_state. At this commit, a VM stop with RUN_STATE_SHUTDOWN runs vfio_vmstate_change(), which moves the device to STOP and caches STOP. A later reset goes through qemu_system_reset() -> qemu_devices_reset() -> vfio_pci_reset() -> ioctl(VFIO_DEVICE_RESET). This path does not go through vm_state_notify() and does not update migration->device_state. By contrast, the reset_device path in vfio_migration_set_state() does refresh the cache: if (ioctl(vbasedev->fd, VFIO_DEVICE_RESET)) { hw_error(...); } vfio_migration_set_device_state(vbasedev, VFIO_DEVICE_STATE_RUNNING); If the reset returns the device to RUNNING in the kernel, the cache would still say STOP. vfio_migration_stop_after_shutdown() would then match neither RUNNING_P2P nor (SHUTDOWN and RUNNING), since the run state is PRELAUNCH, and return 0. The device would then be closed or saved running, which is what the patch aims to prevent. The kernel-side behaviour of VFIO_DEVICE_RESET on a migration-capable device was not verified here. [Severity: Low] Is the commit message accurate for this patch on its own? It says: A reset of the stopped VM leaves the device in RUNNING_P2P. Nothing else does, so a device in that state is stopped in any run state. Today only a failed migration of such a VM leaves a device running there. The next patch does it at every guest shutdown. The code comment says "Only that leaves a device in RUNNING_P2P". At this commit, vm_state_notify(false) runs vfio_vmstate_change_prepare(), which sets RUNNING_P2P, and then vfio_vmstate_change(), which sets STOP in the same pass: new_state =3D (vfio_device_state_is_precopy(vbasedev) && (state =3D=3D RUN_STATE_FINISH_MIGRATE || state =3D=3D RUN_STATE_P= AUSED)) ? VFIO_DEVICE_STATE_STOP_COPY : VFIO_DEVICE_STATE_STOP; A reset therefore does not find the device in RUNNING_P2P here. The RUNNING_P2P arm of the new helper is unreachable until the follow-up patch adds an early return in vfio_vmstate_change() for the guest-shutdown case. Also, "Only that leaves a device in RUNNING_P2P" does not seem strictly true, since a failed vfio_vmstate_change() on VM start can leave RUNNING_P2P with the VM running. Could the commit message and comment be reworded, or this patch be ordered after the one that makes the RUNNING_P2P case reachable? > + > + return vfio_migration_set_state_or_reset(vbasedev, VFIO_DEVICE_STATE= _STOP, > + errp); > +} [Severity: Low] Is it safe to move a device in RUNNING directly to STOP here? This bypasses the RUNNING -> RUNNING_P2P step that vfio_vmstate_change_prepare() runs for all devices before any device is stopped. The helper is called per device, from vfio_migration_shutdown_notifier() and from each device's vfio_save_setup(), so other devices may still be RUNNING when one is stopped. For example, with -no-shutdown, a migration of a SHUTDOWN VM fails. vfio_migration_state_notifier() then sets every device to RUNNING with no P2P round. A following quit or a new save moves one RUNNING device straight to STOP while its peers are still RUNNING, so a running peer could issue P2P DMA to a stopped device. The "one by one" comment does not address this. The next patch only adds a skip in vfio_vmstate_change() and leaves this RUNNING branch unchanged. Would moving all RUNNING devices to RUNNING_P2P first, as vfio_vmstate_change_prepare() does, be needed before stopping any of them? [ ... ] --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/2026110920451= 8.220617-1-andrey.drobyshev@virtuozzo.com?part=3D1