From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from DB3PR0202CU003.outbound.protection.outlook.com (mail-northeuropeazon11020111.outbound.protection.outlook.com [52.101.84.111]) by lore.virtuozzo.com (Postfix) with ESMTPS id A69F181377 for ; Mon, 9 Nov 2026 15:43:56 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gmnmfoj2ZP9LG8a1CgcfkDFCZtvsG5r7as60vIe9Nl7XlwhfCSRtyhncKiOK73TCHnTmk5LonV2Bwrslj3QofsTKQ6YzFZx/EkVbvdd/E6ZcstVVc2mS6na9CroSe3+UT4niqNCq8EsnWNlzLe5v4yWSCQmAC2pHyiSBdyxHjyCtKwnbsdCcXoOFiC95h8xN95R9qwCFwUAxZTKWNV3kYzFLLOsOrgzQBu4Wi053lsqSzG9T9rF7uyj0sunsHqK2mHZq44ecdU+UTJsDkQR3+pZJmsmfhbcYs6E8Vy1KvvA3WEaAZBzgRErKwePb5MTJ+nj+nhImbLikQ+m+Xj4W9Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RF0HDS+N7fArr4ln0MTKcBC2iHzrCrTkNcG0wVtCM9k=; b=o/mCyn2lq6XWmw0VX5cO+YGrkeDuFjT7xrJOAAjQNf8ZinOD2EM2CoTtFKYZ/qaHDyKOESXnZKZ1JNscoMZO0FEroQTpSV1gG8EWz71aUSiODmDydF/e4v77WK1/DG2eVRzbFC1vAUu+eRMFk3tnCTnHkgZnqZISKBWFHwRdofnzEfA2NxaPZ85Q2Y702frc7vFoV+A14j4csiEKZ+7m/GCCOw6rJRhhRFDnFRPGUoM7I5rlLHhjGTm+LLlJ2hKtoeaVeNhEzOq1BYjOXnoszAQKdWiYAaH4SWsqHvcMOK67WPLe1sd34BV+3n9xBDOAwVCTzpo73i6OWi0KlvMNHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RF0HDS+N7fArr4ln0MTKcBC2iHzrCrTkNcG0wVtCM9k=; b=dMts98W2N/Ehe3s+U4e6F//dOzg7uBuPYfhJzCJKzDoDXNQgRwiDCbtJ7tCZBPvG+0bW5mig0A/Urtg06fIZLm2NydoiSaUS+s3MJ8HXTSRpldim4QVoBm/Yt/s49Q4kFuBshjoKl5/XDaF2QIfR8a6IXCOgu8yyTmZ9a7FWfBH3mI5U6TWWHi0+n/jCsP5iuWgyVztgMkiFQtcHWdQjIZGX6hDtWPW/x88nZZ1wEyt34mKWBfx/7WzpSOn1XOYRgz9miWa16BLMoG6HnQGCkZR5VWU4RHqyEWdBFOXGG81+8j/0jH9FOKCfsRcJUEUANr/yC5Jdq34tcJ/oIakXvQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by VI0PR08MB11841.eurprd08.prod.outlook.com (2603:10a6:800:313::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.13; Mon, 9 Nov 2026 15:43:54 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%4]) with mapi id 15.21.0592.010; Mon, 9 Nov 2026 15:43:54 +0000 From: Andrey Drobyshev To: svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com, den@openvz.org Subject: [PATCH hci-dev v3 2/2]vfio/migration: don't touch the device on guest shutdown #VSTOR-153976 Date: Mon, 9 Nov 2026 17:43:46 +0200 Message-ID: <20261109154346.153153-3-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20261109154346.153153-1-andrey.drobyshev@virtuozzo.com> References: <20261109154346.153153-1-andrey.drobyshev@virtuozzo.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: VIXP296CA0017.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:36c::13) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|VI0PR08MB11841:EE_ X-MS-Office365-Filtering-Correlation-Id: 41e63b57-6e38-4521-7a35-08df3e784263 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|52116014|23010399003|376014|22082099003|18002099003|5113699003|10067099003|5023799004|261028144727099003|261026121527099003|261102123227099003|261022105027099003|261021121427099003|56012099006|261026140227099003|261009223027099003|38350700014; X-Microsoft-Antispam-Message-Info: Ea++xZ1LX2bYLff3Cif76aj/yqz3G0oI6T9Jdms3UeXB2MNpP0UibvGV/kX9XNAySjVcTt39PMxXo32NfFysaFvbeHyNMV6En21Y1dJ9gnnMXJN10KlDpu31AjaY2L+oSJSR4GrIz9po+HtnAdhnIJa600R+WqzTONP50WxC8+uHe+I77qyy7sc00DcrHhMC+bnQGtY7M89bZXcho+WaaksKVXxj5nEtTJmTz8UiQMy3rIP4dsmVA1x5E4hESJb+47wXbHOyNTLZuC+9/oGfRoRJnBZzOyLrFwlfmXgDhD2uYxRmp4UTPvrTqYGhxrSbrQ8uNLBKVZ3Oa/cd09N1OxhWNE/zj26uZqtJxzf8s5oL2G5/W68NOuPOAK+KPHPt0NaD/NIcqsYjhXLDmpgtItcZ3Q5u0Wc7dK4mr++c4JHODOuq9akqsGTvBjapOw7iGqZK5GgraMZLsvXPa5BrRVA4tXqFuySS4pxoYfPomXGqTjKC3DzskPv3IlqMMI8l1EAZF2NNzWBBaNi4aM+a03UcZ5s1tNXXo/JK1KQq7F6oOdOoR1kg8XBU9+YC+3Npl5n6uS/dmClMb7rA7sDEzHPQrhMv4ymFhwYPCwH5xX03InjxO5g4/CDmqB/MWdsoMlLID5g1xYj8HafVNA/iGPSheVAsDBKJDN2Ox39H8OYt/JXrgK+vcI1AU7aWeHHls/NCnyz1rwsJdEH0FGTbIGKOUVD4KyoVy/uOWgLegIU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(52116014)(23010399003)(376014)(22082099003)(18002099003)(5113699003)(10067099003)(5023799004)(261028144727099003)(261026121527099003)(261102123227099003)(261022105027099003)(261021121427099003)(56012099006)(261026140227099003)(261009223027099003)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/aW47cFhMgJow7TjFe4AlLp10PxIcNMYdR7KDbZp00x5C+tNALBfs/yPOCXM?= =?us-ascii?Q?3al/9FZCPfwASuX5ui2+9g3PDgZu+OVPx7pdNObGOJ3pDrbYLciRGkTH7+l/?= =?us-ascii?Q?SbQC8Io1hktiqrdmx59NdkEohc0VZePL9izuF2b1IYCIXayAEJlVvGtBeO3a?= =?us-ascii?Q?3O3TPMnFJt597KqIBo4XD2fQ8sP9CYgk0jqNqejuRlhXjtan/2cm0ymaHIT0?= =?us-ascii?Q?LEozsx0f/L7gP8j+9cIPG9fcL3DMdUnwQY7AZvNjWs8oVIqg+xzMwKMSp6Oe?= =?us-ascii?Q?Gjz2VU3euv0ubDw4aGGtlIuWe69PlTP7bu3Dm4ptTVnrvduT6XjtWnZtFjcN?= =?us-ascii?Q?XIOw2Kr7L4XkuL/yfIk/7x5EZFAhl8elU0COwsiK37htDlBJajxsGNSfnF+N?= =?us-ascii?Q?Jf3lJWD/W737+LaKwVET1LeaTdYpKnb6Olwep0fbsmpFI+okkomW3Pjvbdi0?= =?us-ascii?Q?ph3Gkaq8aVgNqlEiwj+NwM83Pc32LqDboaTQWKUNM8p/4Gap0EKfuOV67MyP?= =?us-ascii?Q?CmCz2a/rtuhPZrdkIBm4I9JgoIwtenJxjpwX9Pfe+IED9HWMnt1Bidn376Ny?= =?us-ascii?Q?dwToW5VqnfzKiTdMBmusrL/0D3v7rKT81525i9nzB8Qpme77rqHFvoONxLSe?= =?us-ascii?Q?e4TK/dv+ooBDbzwzQuAjAX7Q8QzBES0Sj7UxvXBw6wvE0ckgADDG/9FmIeg/?= =?us-ascii?Q?gi3N9fl+I8t4fI2CMeQbh+kDHGoBOuSq1YLyTLkY5U9HQe4WaaV9eMYk+b7Q?= =?us-ascii?Q?j3XyND0Zt5ZuW604aq2itlhNDd1PSSY+Sw2soqKiPXdU8MLn/H+SzJjI5QFg?= =?us-ascii?Q?4uySTuVzEF0yIuz7HNtxz9YRSHH5Mno68jRhoERmsN2ZP4zrUX1IAaWsZNeI?= =?us-ascii?Q?hlUDJ/PBUEU+2bL4JRWnO/uDaxz92avdkbV1cK54OF9b7FYWkblb8UAynFSL?= =?us-ascii?Q?KKtZzwMC8PSRuYOKjcfRgr+vo3wlxfbquth3v3D7Ol4iefr7zIb3tUnxN5+a?= =?us-ascii?Q?o79tnpK8SFHu/iddCym+KykZZp4UhnPmO+BEeFN6IxmJBtaBxbSUsvTSZfS2?= =?us-ascii?Q?rgjIUb3NBeQX50XLzfL1UOLs/GVWJRUfvgNAtW/6dLZL8EbRslbG099Joee+?= =?us-ascii?Q?HqSS9GIkH4MGnzx/OIbSvVUUPkNo44lFQKPxQ0UaftSmVfUs0CzuiIvcs8Aa?= =?us-ascii?Q?ai6uIssRB+blp0/C87xU+mTXh6+/SqZSLhIBVpYVWT7jtXR6Xw7ejFCLwkSk?= =?us-ascii?Q?i3g6khU7KWGaNRksst5sim0zSjqgs9MOz+LOYIzwMoi/KROeKHRZklEEpnbA?= =?us-ascii?Q?L2+A6Ej8+f8UJXxaelqLgKaSoaO8DHrQL4b2dixOoKNM8+PhIGFDrEgN+xKn?= =?us-ascii?Q?keGCAU8Ix+BhPRzksGXCHQFNvlI7yyXBqb1x6vSE/q3ZUOI3DlXQfuMHdn8j?= =?us-ascii?Q?cQ2CbT6eP9FS4KEjJe8Il6DYCqSo8VXflzadgbzH258oxMli1u6tk0CnTfPD?= =?us-ascii?Q?4/5L5lzp4yW9Q5U64TALXTQJwZ75/5HC9wrWX97WDJkxiYz3AlaYX9Ssj3v+?= =?us-ascii?Q?zzJoAheg/wgJzaworP9/vG09ofAfThhcq4C9nan9u8SHsQ5Ve+kkU8YDNviN?= =?us-ascii?Q?JbeftkaXgpJG0C7FQhBNY2vgzqnHZ8/GPD/UXyeEpCPGvNxn3qjqyX/DzUH5?= =?us-ascii?Q?So0lJh/8jcu9zmmlwkxsiBcyT3El9rq0/QvDXKoZepeIiSiO5KFXYz9hRTGn?= =?us-ascii?Q?yqL/jrmnvtzN2HahmgcaFf0O2AVJzC8=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 41e63b57-6e38-4521-7a35-08df3e784263 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 15:43:54.1619 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: oMe6fPoz3xnM3pxafzzBZdLXFYz+BuuWS/LErf55DhWprRyWZsN9Xh0jqvn0n+lgpb5kI5JGz8sFBtyl+8FNv2C86p8+V+5gcxPiYJ6ecgI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI0PR08MB11841 When the guest shuts down and QEMU runs with -no-shutdown, the VM stays stopped and can only be reset or quit. A libvirt soft reboot is exactly that: guest shutdown, system_reset, cont. Today vfio_vmstate_change() treats this stop like any other and moves the device to STOP, then cont brings it back to RUNNING. NVIDIA's vGPU manager (595.71.03) takes that STOP for the start of a migration, finds no guest driver, breaks its dirty page tracking and never repairs it when the driver loads again. Every live migration of the VM after a soft reboot then fails: "Error saving page in pipelined mode". Do not touch the device state in this case. There is nothing to quiesce, and the reset that must follow resets the device anyway. The case is told apart by the shutdown action: only "pause" (-no-shutdown) keeps the VM alive. A quit of a running VM, a signal and a plain guest shutdown use "poweroff" and still stop the device as before. A quit or a migration of the stopped VM stops the device first, see the previous patch. With this patch applied, soft reboot + subsequent live migration pass successfully. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index bfafc14e5b9..454c4e3f553 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -415,6 +415,21 @@ static bool vfio_precopy_supported(VFIODevice *vbasedev) return migration->mig_flags & VFIO_MIGRATION_PRE_COPY; } +/* + * The guest has shut down and QEMU keeps the VM stopped (-no-shutdown). The + * guest has released the device, and only a reset, a quit or a migration can + * follow. The device is left running here: the reset resets it, and the two + * other paths stop it with vfio_migration_stop_after_shutdown(). + * + * The same run state with the "poweroff" action is a quit of a running VM, + * which stops the device as usual. + */ +static bool vfio_vm_stopped_by_guest_shutdown(bool running, RunState state) +{ + return !running && state == RUN_STATE_SHUTDOWN && + qemu_shutdown_action_is_pause(); +} + /* * Stop a device still running while the VM is stopped after a guest * shutdown. Nothing else leaves a running device in the SHUTDOWN run state, @@ -924,6 +939,10 @@ static void vfio_vmstate_change_prepare(void *opaque, bool running, return; } + if (vfio_vm_stopped_by_guest_shutdown(running, state)) { + return; + } + new_state = migration->device_state == VFIO_DEVICE_STATE_PRE_COPY ? VFIO_DEVICE_STATE_PRE_COPY_P2P : VFIO_DEVICE_STATE_RUNNING_P2P; @@ -961,6 +980,10 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) return; } + if (vfio_vm_stopped_by_guest_shutdown(running, state)) { + return; + } + if (running) { new_state = VFIO_DEVICE_STATE_RUNNING; } else { -- 2.47.1