From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazon11020107.outbound.protection.outlook.com [52.101.69.107]) by lore.virtuozzo.com (Postfix) with ESMTPS id D65E1814B4 for ; Mon, 9 Nov 2026 16:54:45 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ENKk035qcT/2fSzCJaWPtCLzwRfiQRmN1+JkKrHwjplPveAYrKgvJp+oP8xpmY8XXROwUu9TEeKiuS9s1Q/x8tJmPtBO01dS8KL98D6skcv9bL7GP0c2xBEgX+kHgyjD9ZwHOi0+oX2cZk8ma16wJgcKQHECxGc/th7CTdQwjEfA+PGIgBuDwAdWdfNusml403paWXjEYiyq8iol83hBn3KHGGq3XQ7/R7dR3So9n6LnEuvBrAbXxhJUdAoWPQNY2MDHZkJIR2jh+JUSOlR5pO401Z6ZssWaSt7X90CX3AYC1Tev7iMFiM+b+I761+AS6/44uXDya8d+0Ua4ta9z/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vypt5Kp+ll/x05vLVXwy2d0sPXxCGLjuc0zOhKtO1wI=; b=rsBSCL2oWPfFN1V2zQP1kL9EL8QaZkFuGg6EqWt5bwekVNDmvLd3MC6YVWRFTlHvjFVYmuhs10dXTW1Xog39dyOlfbCdqYO42ZOn1Og9TcL+/H2SiTIgYo8tS1/+wn4c+jZy5met9zbbjwu2fflF+zmw+6So40O1Dd4LZDHUx1vnbY2bpHej4zjmuTlWkNsoRe+xG34Pg+AX5zr0je0zxwRO6mlHnQp/4Asegtkksa4sh6jdL4WeFhX6r6JM5ryGa5c4rohAQ27zVISS5jpycEi0uTAVzrVgdWTAVNgZXIBT9GSlImTMSomQe5pJcOf0HWoJuWz/o0onYmIFZTr4nQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vypt5Kp+ll/x05vLVXwy2d0sPXxCGLjuc0zOhKtO1wI=; b=m9yxMK34nvohnA+NTZLZdwn9WP69eyKAG6H4kWr9HoLOWPT0fV+Oyo7CwVLmgJL0AJqmJCql3yxvh1pV3JvzZXHW1ChfXgKyt9xlhYscEiolzEhNmkW1EV3DPfF8dgOqGbeSd9QaQ4vM+WYoTbcISj7VBFj+1uvWTMejR4K2lQ+u1N5aiMciOtrp5CAExopdivrmblAGaAtCblnW4bPiCyhlBQbOYy4pmxBE6JDm3F+v7akp0PtqZ0h1h76gj8wZC5NX4HaIRhLsVYloF0kFxUtex1qIwzVUa5UNT3HazcMmkFrJfb49SQQ8BowmLHkian73l0i0F+mmpz4jf12zzQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by DB9PR08MB11474.eurprd08.prod.outlook.com (2603:10a6:10:60f::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.9; Mon, 9 Nov 2026 16:54:44 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%4]) with mapi id 15.21.0592.010; Mon, 9 Nov 2026 16:54:43 +0000 From: Andrey Drobyshev To: svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com, den@openvz.org Subject: [PATCH hci-dev v4 1/2]vfio/migration: stop a device left running after a guest shutdown #VSTOR-153976 Date: Mon, 9 Nov 2026 18:54:34 +0200 Message-ID: <20261109165435.175715-2-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20261109165435.175715-1-andrey.drobyshev@virtuozzo.com> References: <20261109165435.175715-1-andrey.drobyshev@virtuozzo.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: VIZP296CA0002.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:2a1::6) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|DB9PR08MB11474:EE_ X-MS-Office365-Filtering-Correlation-Id: 1822a1d0-2764-42ff-6d49-08df3e822695 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|23010399003|376014|52116014|366016|38350700014|5113699003|10067099003|18002099003|22082099003|56012099006|261026121527099003|261026140227099003|261102123227099003|261028144727099003|261009223027099003|261021121427099003|261021123027099003|261022105027099003; X-Microsoft-Antispam-Message-Info: ZbxNP3FFyh5XCaz9WJqNwAh84IRoA+A8HgcJdetL+Cme9VTgOZUMLv3nxUInV7Efv1AQAJitiCb+FDakzDvfAI7P+EqIpDyvpmKWDY+22l6rQJm8dgYCNg3rst+MH1tEhgMcO5uwxgko0TTbF57LgDmsYrqJrBRcQX0GOmOJlgVakm9b2NcUhVklwAttpvA/F72i4LmQAJZel6oGZ8Hr81K73O4OlGMpCtQtzQzw3K7JKF4LcWAvbyytnzCvoZuKAnHFYLYGgDMpz84Ar1jWkOvwwMDmmkz7SZsECw44nwzvjVmzpLpLz2o1qbCwObPLRkaBJtFoU+gJNaPaHXw5NleSJko64iOnz9q3uRrpVMR2QPbX7F/zEQQvcVKltopG/4hylCxA9DnOw4KL1vpzE642z2tzKxlc24FG/kOLYu8z78FrVLH8zEJL8fUCkaO8szfjsKgCVQntTM7pUuxPq1/kN7WmSRVX98TMWp52BTBETw4zK/1BsJmOLJ8Pz8YmYbiXhby+HoIC9ohhvd416FqPfEgSTLud+jkQgOvAZEjTyKS2gxQhADnHDJM9P3amXdEg3RyenbzoXKdQNqY3S5TfiChcrWOQNQf9yBI1KeooBQJ15Js8WLji02M/DbmhlAvYkB1v/Mkz2v4IReJEFk3FlgBjq6uWG54ot/ZiqzL/CJRA5aHfBIXI7onay5IBXvJyjWkKu0JqzsII494kGWB+LgH/0h0jq2f6W2oYdjM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(23010399003)(376014)(52116014)(366016)(38350700014)(5113699003)(10067099003)(18002099003)(22082099003)(56012099006)(261026121527099003)(261026140227099003)(261102123227099003)(261028144727099003)(261009223027099003)(261021121427099003)(261021123027099003)(261022105027099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?sDcQ44gvH4zfBO1QvUB4SZohb/6COpl4kswBUyTQ7olfzB3ftJlu053sS8JC?= =?us-ascii?Q?Ru5qGxgY5FqdDXVsVmIcchdo/YWZ+fcsMxRxeNQUv5ArMXiB9irDBbisEKLU?= =?us-ascii?Q?gv8CaEnRkbbxT+Ao77JMuT/JNq7X0J9e6nkE7WfEcF+OsMsbfS+Dz5YA3aIc?= =?us-ascii?Q?QESEeN8mFPIgh4jxKx5ubDbRp9DWVh8JHqnEP9/y9YXEDMJ5ggPWWdnfyCxg?= =?us-ascii?Q?nBxVCb7UXEJba0R8rNz4NReUS8DF9cIjS3Cdnh/YA27QeYwUSabnvMOvUUcm?= =?us-ascii?Q?05l5xZ99OEA9LG0/EDAEJRbJg4qci3PSAA8qjsczFojf6bKltGu0ScWYq/Ad?= =?us-ascii?Q?FrhtDrEqwZp/KcOYGxut3H8qEU75yn9BtAJalRQ99iIA9uMPTM1ClMM8q+LQ?= =?us-ascii?Q?EJd3q8ARL/hbU6K/5M6eyaftZTkZ0TdO261JLdXXTbbI2fEdutjX/jiLOwH5?= =?us-ascii?Q?t9qFQt6+3ED7C/p5nbmYNJCidhkh8m/vHNFv95/RrhMyz/NtT8k6k5OV9j54?= =?us-ascii?Q?OIeCI3R0+v/MPYsyHKPLvdfbXdUihqUMI6+h1znMKZ42/W23w/JLTvyVf1+w?= =?us-ascii?Q?GX+RC1w9+t12j87VzbLmyoM9M68h4KR4ZHrdI1P9mcOmaEurb1+RZ2pIzwcJ?= =?us-ascii?Q?IO55W/+nqWgmh0DaL2tlFST9UVCBuCAwGzNAbZWUSYD4HMLcZAM+6xeGAZ7D?= =?us-ascii?Q?b2a5lKyYhoalrbGqNqZsqSccArRI2xb0nwV7GJbIUhdlwqdZOMSGDhwGJzYa?= =?us-ascii?Q?f5dOBAXA/GpNeNJaYezLWy1HvxXSEu9gFAbwsPgEN/ptpreGKF4IEIV/pwVD?= =?us-ascii?Q?aKmYZkW4OdIdt5twSm3wdQYhRLDrlV5YEYudc8tYOUpu8vembqbsjzGfRUw1?= =?us-ascii?Q?uXQqjRiZNqVREbOF0z46gGrqALIKwnmRyy+YmsGu5zb7X0SloIA8XiK0tIyL?= =?us-ascii?Q?ZfiuYbfav3iJyroSzWTRbaVRCMHHSspFb9iMbpD+m119Upx35BvbHjAbqjbK?= =?us-ascii?Q?QNRDyzsP/DVAJCGOiMbdk+pGJEmM2jbX9amcZOtUvc2cIVbTsyi+foAgqQG2?= =?us-ascii?Q?dFjQmoUhkTh9y19LobsSVSEu2XFBOlwS00STPPWoNbnM97s9tFhnyHtROx6J?= =?us-ascii?Q?2lnIfsk/lwXmnnpIroukhVAeTZz1gtOfR4MHDPzFsXDFK4suGVQ5atHQBQrV?= =?us-ascii?Q?kQSPqCPrn6dRQCUkvcxOfyrUKSdv8A1Op5aEHChHtyjI8TL1SgyTFuQhSsa8?= =?us-ascii?Q?x9RjCWmFu4HV4PeFy2udEFc+LUe7/mF66T9rADAEfV6C0y9OsGcj8uyrHfIZ?= =?us-ascii?Q?Ogzt4Rtd+y4chOLjAns3wFke9X1Vvg3vAfGvDPiaOYuLySaYY2R3BGT2c9p8?= =?us-ascii?Q?TdyfuzqM5ynSoVfb/id/T7u5DCbOnFQGl5WGIBvBBJL0ug0IciUfoXgcF8+B?= =?us-ascii?Q?VLVeyWBU0JzG9I9BLqHV5pD2lEwK3bZrVbjusNZRC6dEcV2eYTS5Q4VMwbm9?= =?us-ascii?Q?s6DfkSY9X26TqBfbdfmh3O2ygD+KMrHxygqCuIEU9++t7ihtBwZ6ze9AInlJ?= =?us-ascii?Q?vNLl1MSBD+alVNx93Td1f/36R0XAmiqPBfCucJEk4Tg+NuTOgen7XCrpmIqP?= =?us-ascii?Q?rRwdp+EJYRa0F8Z16/O8tBl7KnKfR35DHS/026w8N4/qbCcIqmBepk2ZCzVZ?= =?us-ascii?Q?k5orJOmhwSzHgcRQigYATHrBRothQp5ESEK6t6C3EbHknXEIdReF9Z7g9Xv1?= =?us-ascii?Q?nBUkYLt37CRaUjBnHFHdvpAGHJMDAlM=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1822a1d0-2764-42ff-6d49-08df3e822695 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 16:54:42.5055 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ITJtDLIOsr3HgGbBgqLtRiLKiCQzmU0/rxmmFjlkJj7PFHqFgJQHIcu5yRtpESLT11hkAYqiWp35NR7nXt1iQJdZoXL4VuyIHSLeKd/dGpk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB11474 With -no-shutdown, a guest shutdown leaves the VM stopped. From there QEMU can only reset the VM, quit, or migrate or save it. A quit or a signal brings no VM state change, and neither does the switchover of a migration from a stopped VM. So a device that is still running at that point is closed or saved running. Stop it first: in a shutdown notifier when QEMU quits, and at the start of a save. The P2P round of the VM stop has already run, so the devices are stopped one by one. Today only a failed migration of such a VM leaves a device running there. The next patch does it at every guest shutdown. This matters for NVIDIA vGPU (595.71.03): a VF closed while running, after the Linux guest driver unloaded, breaks the next VM that resumes on it. Its GPU stops answering. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 47 +++++++++++++++++++++++++++++++ hw/vfio/vfio-migration-internal.h | 1 + include/system/runstate.h | 1 + system/runstate.c | 5 ++++ 4 files changed, 54 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 7e7fb496bbe..064da62ddc2 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -415,6 +415,28 @@ static bool vfio_precopy_supported(VFIODevice *vbasedev) return migration->mig_flags & VFIO_MIGRATION_PRE_COPY; } +/* + * Stop a device still running while the VM is stopped after a guest + * shutdown, before it is closed or saved. Such a device comes from the + * guest shutdown itself, or from a failed migration of the VM stopped that + * way. The P2P round of the VM stop has already run, so the devices are + * stopped one by one. This is a no-op for every other VM. + */ +static int vfio_migration_stop_after_shutdown(VFIODevice *vbasedev, + Error **errp) +{ + VFIOMigration *migration = vbasedev->migration; + + if (!runstate_check(RUN_STATE_SHUTDOWN) || + (migration->device_state != VFIO_DEVICE_STATE_RUNNING && + migration->device_state != VFIO_DEVICE_STATE_RUNNING_P2P)) { + return 0; + } + + return vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, + VFIO_DEVICE_STATE_RUNNING, errp); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_prepare(void *opaque, Error **errp) @@ -473,6 +495,12 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) migration->event_save_iterate_started = false; migration->event_precopy_empty_hit = false; + /* A stopped VM gets no VM state change at switchover */ + ret = vfio_migration_stop_after_shutdown(vbasedev, errp); + if (ret) { + return ret; + } + if (vfio_precopy_supported(vbasedev)) { switch (migration->device_state) { case VFIO_DEVICE_STATE_RUNNING: @@ -959,6 +987,22 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) mig_state_to_str(new_state)); } +static void vfio_migration_shutdown_notifier(Notifier *notifier, void *data) +{ + VFIOMigration *migration = container_of(notifier, VFIOMigration, + shutdown); + Error *local_err = NULL; + + /* QEMU is quitting: a device left running must be stopped first */ + if (qemu_shutdown_action_is_pause()) { + return; + } + + if (vfio_migration_stop_after_shutdown(migration->vbasedev, &local_err)) { + error_report_err(local_err); + } +} + static int vfio_migration_state_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { @@ -1083,6 +1127,8 @@ static int vfio_migration_init(VFIODevice *vbasedev) vbasedev->dev, vfio_vmstate_change, prepare_cb, NULL, vbasedev); migration_add_notifier(&migration->migration_state, vfio_migration_state_notifier); + migration->shutdown.notify = vfio_migration_shutdown_notifier; + qemu_register_shutdown_notifier(&migration->shutdown); return 0; } @@ -1150,6 +1196,7 @@ static void vfio_migration_deinit(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; + notifier_remove(&migration->shutdown); migration_remove_notifier(&migration->migration_state); qemu_del_vm_change_state_handler(migration->vm_state); unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); diff --git a/hw/vfio/vfio-migration-internal.h b/hw/vfio/vfio-migration-internal.h index 814fbd9ebae..e81d854d3c4 100644 --- a/hw/vfio/vfio-migration-internal.h +++ b/hw/vfio/vfio-migration-internal.h @@ -40,6 +40,7 @@ typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; NotifierWithReturn migration_state; + Notifier shutdown; uint32_t device_state; int data_fd; void *data_buffer; diff --git a/include/system/runstate.h b/include/system/runstate.h index 929379adae4..29a919348da 100644 --- a/include/system/runstate.h +++ b/include/system/runstate.h @@ -148,6 +148,7 @@ void qemu_system_vmstop_request_prepare(void); bool qemu_vmstop_requested(RunState *r); ShutdownCause qemu_shutdown_requested_get(void); bool qemu_force_shutdown_requested(void); +bool qemu_shutdown_action_is_pause(void); ShutdownCause qemu_reset_requested_get(void); void qemu_system_killed(int signal, pid_t pid); void qemu_system_reset(ShutdownCause reason); diff --git a/system/runstate.c b/system/runstate.c index 6178b0091a3..1cdecde0fcf 100644 --- a/system/runstate.c +++ b/system/runstate.c @@ -433,6 +433,11 @@ bool qemu_force_shutdown_requested(void) return force_shutdown; } +bool qemu_shutdown_action_is_pause(void) +{ + return shutdown_action == SHUTDOWN_ACTION_PAUSE; +} + ShutdownCause qemu_reset_requested_get(void) { return reset_requested; -- 2.47.1