From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from OSPPR02CU001.outbound.protection.outlook.com (mail-norwayeastazon11023118.outbound.protection.outlook.com [40.107.159.118]) by lore.virtuozzo.com (Postfix) with ESMTPS id 5BFFB80E7E for ; Mon, 9 Nov 2026 21:07:30 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PKXaHthm75g3XzKD7kM5eo1IsCVpGGIPL4PX37+rTMBXAGbYr0DePBPIV1wwtbZntx2Y4gpDgxYEYzs0im6zj5AlJi5sNOL6VmFx+mmIupWCroFqvFL9ROF2YCadvA6g9mJMJ80di1s0mbws47FTXLGDlFAWLXAW8BEyRQ3eU76/f7MoJfOeixZxNFfbip6ix6GIAWOiuBn3g1odIpvSiJmkboWaew1ovFRv/gQ/3k/Y1xPeiCMq1VCfD65juMS+PpslyTpHiqMPTd4TRNQlckgYJn6SQ/kAArr8ZI+fNrvY45nn1KpixfgVM5htfF7yB8K7nxcFNASipC9XmROyQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RQTnvvUEJ0w2UzP1m6w8aEW73V5XOwnBjJfTPXR6t6o=; b=S2K+5ZQ/YAgxnkv6BEDPWzHxSxT67gsxVmBNuzILePlBbDeMwCQWnDajj/HvgCmf3E/sIRDhIwwbJ6WE9MAonrpBtYorJnNquo2FhNMcFwPIOtUdZLOyVKC2+St2j7ry87Qj9BiltRS4d3AbX5DzYNcMDacAV59ya8dn9SF+ixUqo7/8VaIpfTi+oh1gwK4HpubCsmIXDQCazgTAibkNYmuAHZVRiCnWozLEBT9LsMqAH5fMEz4i5VlYUQ7cw30I0XSF35O1NwJKSCTwRNP5UbszqcwOW+52QozttAvrgBiIQY8wGBl9POCtoTdj/WXdCwwQS2KdExOLjBquI009Hw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RQTnvvUEJ0w2UzP1m6w8aEW73V5XOwnBjJfTPXR6t6o=; b=MUsjFv4Bww2wjR/tjeEJGCrHkNcZ2zJwTHNGoVVK+wwIirhbpiJ+x591asOZPIIGBn4EfeEU9joTV9A2gBAcbByR4V2b/n47B2MbW251veSAnCs1o4hfPrUp8B3qkQyR4jM6Y8B6+WD5FbEW5pkp5tAFxSQ/EMBCQTkURquFnMbSE2imbYQQV/7o26RgArVHIFhgJAldyAm5//GNbiiTqGq9Jed6KPzCgdsWkGoPZkMFuTn286BwuMxZ7LoYd86PefNKzIjquc2ayoXb1/hoP7g2rxRzLzqthr73PX3eGEdgl1qImkgaoDfD6n4jCxw82sVKamMtWgvgRUPhnXWVag== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by DU0PR08MB9608.eurprd08.prod.outlook.com (2603:10a6:10:448::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.13; Mon, 9 Nov 2026 21:07:28 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%4]) with mapi id 15.21.0592.012; Mon, 9 Nov 2026 21:07:28 +0000 From: Andrey Drobyshev To: svt-core@virtuozzo.com Cc: den@openvz.org Subject: Re: [PATCH hci-dev v6 1/2] vfio/migration: stop a device left running after a guest shutdown #VSTOR-153976 Date: Mon, 9 Nov 2026 23:07:20 +0200 Message-ID: <20261109210721.222909-1-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <010701a1c217fe1a-e6584213-39bd-4c2d-b5c0-3d03ca1537d2-000000@eu-central-1.amazonses.com> References: <20261109204518.220617-1-andrey.drobyshev@virtuozzo.com> <20261109204518.220617-2-andrey.drobyshev@virtuozzo.com> <010701a1c217fe1a-e6584213-39bd-4c2d-b5c0-3d03ca1537d2-000000@eu-central-1.amazonses.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: VIVP296CA0101.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:355::18) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|DU0PR08MB9608:EE_ X-MS-Office365-Filtering-Correlation-Id: eebd35a7-7c84-4caa-9752-08df3ea575fc X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|52116014|23010399003|10067099003|38350700014|5023799004|4143699003|56012099006|261026140227099003|261028144727099003|261021121427099003|261026121527099003|261022105027099003|261009223027099003|261102123227099003|18002099003|22082099003|5113699003; X-Microsoft-Antispam-Message-Info: M8Qan4MpmYaCWitTJF+oc07yqgpHY3oEt01Tudu43C2lc48iq5YvDthE+7o1GTiSkQeB9Dln7tK6DsBO7jJtJ9TO7kBAZ7ngJSpDKoY5/MFY/QD17JF4oj/OofpL0ThGiRr4WDJO9KkRNxdS82xoypVHTSimLwU+I3X+9NRCxuS25rpCXERYBFo2KcRel8N4OWUIFSUub7fD6sncQA5XKXtjWyD9/6gENLPdAdL/po+rYOwF1MdGXFSoHjb0DKe6sztOgw6T7iUP8W0QiWjvH2u3suNUTdtX8dMlBQl9WhtVt6IgVcZ4tO5iaqAeusHd89eyOG2c3+M1UZZUogo5QbQF8K3FMTWB6PiaqLQGxYybPiGdrL913VUnycmAe0ZwXfTARkZCbF4DN68JOkElptzG7D2u6D6Lu0dyXjkOGoE46+yYghhbRzPOcqfZCv2IALM1nP4zQhusfI1iRXKwyFQnmjurNS8gmIAgOUWeFuewILvb39HQVut5sIqiczNycCbsMZUqtqCMTENIGTl1ZFD4U08arRnrw7w29Zvvax+E274KRssph41WGmfx5CeeJRHmFDxgPu9O0QTE7QLIOZOWw95cKza/ktHJlLFDmo2zfxLwEWvituyoQuLXcKoKoS6cCH/QdhlsYa0cgXzPLE4xk0CxDRlHUPXlEymQdAWfhJE3tZjVyj0beCX+D4WcS+pUOW9mo+c552QUOtI7vw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(52116014)(23010399003)(10067099003)(38350700014)(5023799004)(4143699003)(56012099006)(261026140227099003)(261028144727099003)(261021121427099003)(261026121527099003)(261022105027099003)(261009223027099003)(261102123227099003)(18002099003)(22082099003)(5113699003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?e3JhY7T2xfOyeW9YxSx+GBAg9YihipRQQaOdlC6du0CfQYhrwfw35Kn4bMTj?= =?us-ascii?Q?Ev5jWmOVh5TaN1p6C6Ffn7C5KLX29mce8z88zMOaJELUP7ZqvjMb3UREQV1M?= =?us-ascii?Q?9hn2uRNpdtsPsWuZuncvuhdlWvTurTKZuhWfsB6U3JbtjP/D55f3EU1+tCzo?= =?us-ascii?Q?fQBggeajyrdBpITPTGKprEDUtfNmeZ32wBq68OznToPGjkSwLpB9Va+6QQij?= =?us-ascii?Q?Oy2BxgBsxvMNMvbRFskdpB0tFL38UPOcSKGJZvSeptck92XgF3YeMzMLbEZ/?= =?us-ascii?Q?4aUtaXkK6FyAyzJjD/xqxJfrO2szwJcaJsWi24fSR7BcNoIDF4jdqabscYcL?= =?us-ascii?Q?zKUHXKqH580gcvhy9Rf37e8ThwsU6pelnlXDBcI8o9y9Jx0tjA+a1w9G+XGC?= =?us-ascii?Q?lhq6NNa6VZjQfS/p/1yAjjC6SK+0aP1AeVo/x1+JufGzNXL42HzuX603XkCt?= =?us-ascii?Q?QX08gu+rY92+kiiM6AnGenmRDgZNh6SNQEvL0oLU5TevSH8SpRdgExkjXr48?= =?us-ascii?Q?WSMrJGtSLzPs35yz5RA+7lsRA6KA/GksoGqtPhMr6hG0J46tVkletLd7OF/G?= =?us-ascii?Q?8wqamCf7clxgTv/FCogRzHhJaeWdB4XpuGqBWBXXwocPRQkrY/qIN3Ox6VPy?= =?us-ascii?Q?ByTV61ssoTxv7lhMn63RE22xBsrp8GyNfu0ejM4zKBhv3hV6m5OZw2T/2diq?= =?us-ascii?Q?GUNbSXwLYZE81IYdYsG7jFU0f0mHaRKKkfxagluo7XD0+x26nYsDSgIdMD6p?= =?us-ascii?Q?TGTMcQERVimZDaz3NCiY5ZZ+sCaq+xZbWPT2J4iLt8eEUsfURASRMpuTYvZ2?= =?us-ascii?Q?f2ufUqIkLa9O3OfYNQQOqu1/+t/wZdI6bI6YyKeW6bA4u0kG4CgQbONjF6cR?= =?us-ascii?Q?+eG2aiERvbQR/1HfLPpKKKQsJ3nfZb9cC0dl/+14Nc5PRFnjoVgH5BE25hSj?= =?us-ascii?Q?gpbqBTK1I1/0STmmOv/aQ+qrns7nA2NfJlPsgvUEFsVlwvjX4lVZCTfDdpO6?= =?us-ascii?Q?LcnjW7wGs7kqHpOpeRwoBuDAAx8O8m8QwDtH9vvJOcpVWkzr4YeXrwIG5XZ2?= =?us-ascii?Q?karGZlwLzhTrLQjYk/AhpeEdEjNxU4uyKfShX/xVOuGvKfkcCLpIiPu6KjTw?= =?us-ascii?Q?SEjiZn/+bFOQNqwxRdH0P0UgH31DvuddeWALXTkwO4TLtwdV8RAVLOctxGn7?= =?us-ascii?Q?zTFsudxlL5pL6HY0L027PYRBgpaOvJd9w2pVdw07KT7b94EfCEoBMjFudap5?= =?us-ascii?Q?2v8fMYiDQMn3WnvpcuUZsaNzbO5EMSBopPlvDK3PbwpqrszzBYFXUmrAlpSp?= =?us-ascii?Q?TqPhbG0x2IMfOW/pgU7nqZJ+LeToE57STRnggLouNanLLD1yEvZXyO82AOU0?= =?us-ascii?Q?FLKAeObJjxoBS1KNSjlc+pKdUzd1VGvFngZmncdMeHlAoMtXqq0TjIr3FDCh?= =?us-ascii?Q?XO2YUs//Vpug5N5ioJWlPKVQ/wSR/PQea3BOwpRp26BATaxM4ZvLX4+5oXpS?= =?us-ascii?Q?3dtXVlPOsvdgu4JkGC+J/ErHPaW5O2iVo7E16u/FstmKqHg/D/Ibvqfv6K0j?= =?us-ascii?Q?LH13Wm0PVvjUISqpOoy3M6W/xU8H+NdNSQ6Tm0UI1WKfpoA+xl8fhZG9mqke?= =?us-ascii?Q?VrnQygxvMV8iixyVtKokAPzWfuhucmpDN5JYvT7M9zxtnkK83Q+z+T6bOh/G?= =?us-ascii?Q?ERhCeHw2YXWPe6T9fd9MmHRcNcO7CEVN9wirUIiZHBoTu8R+URdeuvXWqS/B?= =?us-ascii?Q?QP0ZNh/ydyxWCKJKDK2OZEAhBilZu4s=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: eebd35a7-7c84-4caa-9752-08df3ea575fc X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 21:07:28.1231 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2XpTsh5oLtM+RG9fWn3y7noIZU5Qra2PEBJApA9CuwhsJkqTqTvNTLo3d39CmAwKEHwzqGGdu6F8i8/uO1ozdtRYILpAIxyPwcML2KKaDVc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB9608 On 11/9/26 22:55, no-reply@sashiko.virtuozzo.com wrote: > [Severity: Medium] > Can a device still be closed or saved running after a reset of the stopped > VM? The stop decision uses only the cached migration->device_state. > > At this commit, a VM stop with RUN_STATE_SHUTDOWN runs > vfio_vmstate_change(), which moves the device to STOP and caches STOP. > > A later reset goes through qemu_system_reset() -> qemu_devices_reset() -> > vfio_pci_reset() -> ioctl(VFIO_DEVICE_RESET). This path does not go through > vm_state_notify() and does not update migration->device_state. This is how QEMU behaves today, without this series: the cache says STOP while the kernel has reset the device. The patch does not change that sequence. A device that has just been reset is idle, it has no guest driver and no DMA, so closing or saving it is the same as closing or saving a freshly started device. The patch is about a device the guest shutdown left running, not about the cache after a reset. Updating the cache on reset is not safe either: the NVIDIA vGPU driver keeps its migration state across a reset, so a paused VM that is reset and continued would end up with the device in STOP while the guest runs. > [Severity: Low] > Is the commit message accurate for this patch on its own? It says: > > A reset of the stopped VM leaves the device in RUNNING_P2P. Nothing > else does, so a device in that state is stopped in any run state. > [...] > A reset therefore does not find the device in RUNNING_P2P here. The > RUNNING_P2P arm of the new helper is unreachable until the follow-up patch > adds an early return in vfio_vmstate_change() for the guest-shutdown case. That is intended. This patch has to come first, so that no commit in the series can close a device running. On its own it is a no-op, and the message says so: "The next patch does it at every guest shutdown." > Also, "Only that leaves a device in RUNNING_P2P" does not seem strictly > true, since a failed vfio_vmstate_change() on VM start can leave > RUNNING_P2P with the VM running. It cannot. vfio_vmstate_change() uses vfio_migration_set_state_or_reset(), which resets the device when the transition fails and caches RUNNING. RUNNING_P2P is never left behind by a failure. > [Severity: Low] > Is it safe to move a device in RUNNING directly to STOP here? This bypasses > the RUNNING -> RUNNING_P2P step that vfio_vmstate_change_prepare() runs for > all devices before any device is stopped. > [...] > For example, with -no-shutdown, a migration of a SHUTDOWN VM fails. > vfio_migration_state_notifier() then sets every device to RUNNING with no > P2P round. A following quit or a new save moves one RUNNING device straight > to STOP while its peers are still RUNNING, so a running peer could issue > P2P DMA to a stopped device. The failure handler itself moves the devices to RUNNING one by one, with no P2P round, for any stopped VM, today. The VM has been stopped the whole time, so the guest cannot have started any new DMA. Stopping the devices one by one afterwards adds nothing to what that path already does. The comment states exactly this.