From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from PA4PR04CU001.outbound.protection.outlook.com (mail-francecentralazon11023101.outbound.protection.outlook.com [40.107.162.101]) by lore.virtuozzo.com (Postfix) with ESMTPS id 92CFA814BD for ; Mon, 9 Nov 2026 19:03:52 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TKT/OE7sz4H9ryGUJyAgdBvr/jde9b4OagUtElN5lirE+uxdqQ2qR7A1HALShnyMItCbdHWxAp20NJX0UQDSR0bk0JLNSvPimG23mAe+AvWzYap4zz7nA2G5PaTWLPdvrHvqTlTON8i2gGMYlGZcCMtqFpDZVZN3517hcwundBAX56A7/NuTYafphR3/fqGFTgA7w+nzwaxdNsWgjc9bmF95rGMpb/4xodnAGL1ICOOcwhf229dbvIhUa1bAr5RWSFqCValG7dZHi/Xm+W3H6yrFvYr/v2EjB6/aULhNCnqJAIR3qOg87j0Qpz0zv8OgB8+bOZXApcBd6S0U+LmHSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=N5GJsuUXk5FiaUUIjIk8qyyFfmMcRQJNpPsSD5GdFuw=; b=sNJqp+aFacNuND487sXFTyh72yaFLyA/VAyC6USl9nGnetuRi7NML9ZOaCAo3ZB2Z0yTBfg1BNeZbaks+ALC/g16ZRiDCjiYKj8rorsNU4njOYQIFQlmcnKOYzJOm2WpkVnRO/RHMkEmbjnjJ3oZcSUQqoLv842CNHw4h5CsBpWmPeEacIicxm5eO1JpId4XMagv5KP78xxNPbDEMDIuqv1pMGcpzJ4lW/9sNo6XLOBIux/qW1KY6sUElrfd9WFXDbYtsHs9yMbqr21XiH8CaixHjT5//tODOTr04Z1Z2JzKlKS2QGNIxNoR0KiTwM4kaSAtbJnYwTx9/0kRjH+YgA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=N5GJsuUXk5FiaUUIjIk8qyyFfmMcRQJNpPsSD5GdFuw=; b=Km+gB5tM+WnsTUmDxq8fRURnvu2j4YmG+QiY/2dOJ0MoMTI4SCCprEwY+32Ipy2XnsX8fnYQwWeMVnj60jA8Kov//btC6c4Zt6d/Am4yne+rcqh6ZP4pwx2WHGaFPgNEKQeK6CDmJ7lK9bKf1JgChyXh1MFpbkvjVF4281XMc5Of5cOJHDuoQITkJU8rhyRrCIhLCImBdPo5D4jAyL+qzgzLbW3aLw8VuHNhhnAS49AfxjuV8dp0V1DYd1H8cC8qrDJmJQcLJvVO14obK+NNIepQ0hxOV8YdPwqKYnXk43+s9UsW1M9pye6qQJZdWwlU0Kb977EW9O/E8+W+gVBvbQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by DB9PR08MB8627.eurprd08.prod.outlook.com (2603:10a6:10:3d1::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.9; Mon, 9 Nov 2026 19:03:50 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%4]) with mapi id 15.21.0592.012; Mon, 9 Nov 2026 19:03:50 +0000 From: Andrey Drobyshev To: svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com, den@openvz.org Subject: [PATCH hci-dev v5 1/2] vfio/migration: stop a device left running after a guest shutdown #VSTOR-153976 Date: Mon, 9 Nov 2026 21:03:41 +0200 Message-ID: <20261109190342.200293-2-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20261109190342.200293-1-andrey.drobyshev@virtuozzo.com> References: <20261109190342.200293-1-andrey.drobyshev@virtuozzo.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: VIXP296CA0002.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:2a9::12) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|DB9PR08MB8627:EE_ X-MS-Office365-Filtering-Correlation-Id: abc0b927-6a79-41ee-64fc-08df3e94309c X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|23010399003|52116014|1800799024|5113699003|10067099003|22082099003|18002099003|261021121427099003|56012099006|261026140227099003|261102123227099003|261026121527099003|261009223027099003|261021123027099003|5023799004|261028144727099003|261022105027099003|38350700014; X-Microsoft-Antispam-Message-Info: nWsYg+9vGW0UFrWnOL6LszFko2KYbnMr9rIZ5buhkvEzO20Tpwjgaplvtm2mU7VIm9P94hzUh3IFX+JvT/NN2iofu2XKwuTt4v1yqyLcZ6uzI39yFO1JbPiOLa9G8Xeu3yDZPNRjiiB55wP92poprwGg2vW0b6fMiWgyCPl6SPZNibTJSzxShAm92C+v6GExxyYqb8/632wODl8+Qad1dKXpn5XHnRYtqnp3ChlkGCzgHTaiZ3Lb+rOyCE3Nh966GhbrgdFCDIjhSePh4Wh6TalMbCRX2nouOKaHysx/8GKZmItiN60eJdO8mzk3UftvhdHO8qZ7CjgpE42I+9KP8osulAlIjQhQ/7elpIY6yllPdVQ8F0ZLJjTCibEPkiQeYQvvEddKNAbK6rraA+3cuijzVUqmjx8R3nU8Pg3IT98wkN7OtYewTSA087DXhw+ZExcqsMqRoFrEmfD3zgYc3g6io+7MzUsTk7pLLAo/ptYiCG3bwGCxBZtrfxoTCETOJtqZG6eN1NmfCw0XtOd96jCCGHeuZ6hxC0ThWK06BtjW6VO0UWZ/Czz3RW1m9NeCF3PHFTCRG+MHxm/z9PR9erXjyuY+qNUGcktrrs+12WfAzpjJfOfwJ14yGw2dkdtEdx1JwJUi9Z1Pp5M60kVyPfHcL5gxznTmWs9dS5bFckMl8IB9ESnH4J4r5oZavzseMHvXLOLPattXTAWqDdD1KDAE9/yY/qPebHHJ6miNOJ0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(23010399003)(52116014)(1800799024)(5113699003)(10067099003)(22082099003)(18002099003)(261021121427099003)(56012099006)(261026140227099003)(261102123227099003)(261026121527099003)(261009223027099003)(261021123027099003)(5023799004)(261028144727099003)(261022105027099003)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?r18kwkVr9q7+8BHLVyGz6026/CwQNLN9e2C9S5KbaxYRdO3vi7slrIvJTNDo?= =?us-ascii?Q?39HxdSbQ3DaTa96pxPw83O9SEE/8xUNsZQDVx8a0pH2Y+zcwFarsICLujUPE?= =?us-ascii?Q?x05zTQMu38YtZYGwmQFYh9Ei08hdGoUBX/WNLNEfNpWcWO1dvrMQsjsYphcW?= =?us-ascii?Q?VFJsncZ5ygKP0mJ0ni5rxeNZ7wAJkF/HxrQ/Drp0sXKuxGFZTq/Q1OmczX+T?= =?us-ascii?Q?4h1oL1yvrQFz2/U41VxucDfryvKFInx7eDeXKye0pTYqiBw5XcJzL/7kuyiy?= =?us-ascii?Q?nptP29tz8C2oITJIX0zof5fAN22ulTD4LOI00OB3DOfaENOhEEeykwz3Bifw?= =?us-ascii?Q?eqETj+fiASlcw60c4at60U+IkkCBbUAZYFtVwkOrnCrTcyfEbamHgEuYnzDA?= =?us-ascii?Q?jUBkQKimKQpz1H/I/BaXOJmA1k/XShjHBrpH45IK6m8ifA6fR8ke/2RArAJG?= =?us-ascii?Q?XBHblG+hYmiR0dpy578LwkpiiMRrqKGfONELYTSrjdmtJ3Db327BycmdLwhP?= =?us-ascii?Q?efdwM/thO/vJlCvpArz4PTWPuDe1ELA28/2CFbKQeMYREVdnijp1pHtloOay?= =?us-ascii?Q?G1rGf7HguUMFp+pzMXRmz8nV5cFYIcM2mVWSmV6lPepF5TamnuZ9uAS+BABc?= =?us-ascii?Q?VxLf91t9qyF2a6sW0CwDemOUbLCLM9hrex+BGkovIK3nUTSMyQ2GLQNNAjDL?= =?us-ascii?Q?xYNs2aTk5laFCD8oxQc+YuUDtlbbeEwvfrARR0rmNmosEHcYXN4dEeLiD5um?= =?us-ascii?Q?nzfvw+h0ReMcp2hdHbZ+sg6/YC2tSOt+3pQw4HK4RJgMrDo4IqZl6WpDo26R?= =?us-ascii?Q?c51YHxDeMeovyOehE1HkC3MpNcyRo+skYQDCxIzOcC0S1xjrrjp7ZhFwfilj?= =?us-ascii?Q?XZlpDNqzlb+8RcciqXYt/ewowg/X21mg2moE3EAEeNVPyrgd4Q9iOAtmQPD+?= =?us-ascii?Q?yx63VRvIuV9YYO0iEvYK5vgTAOcOKFsL98jl9XTaA8sHFiW5aE/utrUFnSCv?= =?us-ascii?Q?bqgMXj90iGd0/HizpaN6j5sSytNP4OwrC+HWpsgiEY/2iveU88aPoqoGmm+P?= =?us-ascii?Q?PgJFDtcs5YyIOIdaMei9GY49unsNavIOs8GRhzKdGXeORpoZB1RqPd6r3PbD?= =?us-ascii?Q?Ve6Y+oQvnkDyeCnmUfuAhF8uQrW64l0+FT6TvHKuDGwy+dFbJk1oFa4q50Ow?= =?us-ascii?Q?manylCoHLlFOjITxyGSBZx3Rx1FvPUAWRwxCLKnsit49j1ki/X97i+9QqBSO?= =?us-ascii?Q?IXNV41M0fgPtTKYJMjaOCjTUUwbvmDzctoN6BDt6OwBOt/pFoQu9e0SNKhyl?= =?us-ascii?Q?pQMrSgDTa3V/UuFdoZZprIzajgXE5WzEZo/IVxqsKk06ppSJCBgZAmMkhmri?= =?us-ascii?Q?D5PPMi23hwcV39XIWRsCGaiMUDpH+wRAjlzlYbzJ57DgrDlzwvg56ysW3KKT?= =?us-ascii?Q?+y1QcQlD3xkwq6EXrkqJrE/EMGFXp9HdFE1gg97fQ9WjleASJxUXvFq0qODr?= =?us-ascii?Q?oGnjwklscHqWQvLzXbP/O853NY3knMEo936Bsq6I/3PcPKL57KS/JAuwVO+h?= =?us-ascii?Q?DvJJBYD42nFnLgCUYGBQzpnRKfmwFSX5E41cpEghNzQKWXAf9sI4fBfRJCtS?= =?us-ascii?Q?sf8FV3ceghmNpRSdYR+modU9hxJx/c/5TQsMsV3EMWaKSWkPR2GHnhtMvUQW?= =?us-ascii?Q?Cgyi3Lk0MxUytbVjKuV+r+NVeroVdr+x1A+sHkCmcWBlLsFqM3RvhZBelR7v?= =?us-ascii?Q?A5PIV6wSFfYZOqnnEHE0XmTEy5DzVps=3D?= X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: abc0b927-6a79-41ee-64fc-08df3e94309c X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 19:03:50.2711 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SWr7TQBlZeS1wO+FnGtp029sNQ3SSUumEJ+4jfklX8oh9x8m8jvDPBIcsM1cfQ8fZGOy8tU8DOuoWjUq4wCCxlA+rgo6QucxsRhW56t+ilc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB8627 With -no-shutdown, a guest shutdown leaves the VM stopped. From there QEMU can only reset the VM, quit, or migrate or save it. A quit or a signal brings no VM state change, and neither does the switchover of a migration from a stopped VM. So a device that is still running at that point is closed or saved running. Stop it first: in a shutdown notifier when QEMU quits, and at the start of a save. The P2P round of the VM stop has already run, so the devices are stopped one by one. A reset of the stopped VM leaves the device in RUNNING_P2P. Nothing else does, so a device in that state is stopped in any run state. Today only a failed migration of such a VM leaves a device running there. The next patch does it at every guest shutdown. This matters for NVIDIA vGPU (595.71.03): a VF closed while running, after the Linux guest driver unloaded, breaks the next VM that resumes on it. Its GPU stops answering. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 46 +++++++++++++++++++++++++++++++ hw/vfio/vfio-migration-internal.h | 1 + include/system/runstate.h | 1 + system/runstate.c | 5 ++++ 4 files changed, 53 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 7e7fb496bbe..c391422d601 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -415,6 +415,27 @@ static bool vfio_precopy_supported(VFIODevice *vbasedev) return migration->mig_flags & VFIO_MIGRATION_PRE_COPY; } +/* + * Stop a device that a guest shutdown left running, before it is closed or + * saved. Only that leaves a device in RUNNING_P2P, or in RUNNING while the + * VM is in SHUTDOWN. The P2P round of the VM stop has already run, so the + * devices are stopped one by one. + */ +static int vfio_migration_stop_after_shutdown(VFIODevice *vbasedev, + Error **errp) +{ + VFIOMigration *migration = vbasedev->migration; + + if (migration->device_state != VFIO_DEVICE_STATE_RUNNING_P2P && + !(runstate_check(RUN_STATE_SHUTDOWN) && + migration->device_state == VFIO_DEVICE_STATE_RUNNING)) { + return 0; + } + + return vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, + VFIO_DEVICE_STATE_RUNNING, errp); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_prepare(void *opaque, Error **errp) @@ -473,6 +494,12 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) migration->event_save_iterate_started = false; migration->event_precopy_empty_hit = false; + /* A stopped VM gets no VM state change at switchover */ + ret = vfio_migration_stop_after_shutdown(vbasedev, errp); + if (ret) { + return ret; + } + if (vfio_precopy_supported(vbasedev)) { switch (migration->device_state) { case VFIO_DEVICE_STATE_RUNNING: @@ -959,6 +986,22 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) mig_state_to_str(new_state)); } +static void vfio_migration_shutdown_notifier(Notifier *notifier, void *data) +{ + VFIOMigration *migration = container_of(notifier, VFIOMigration, + shutdown); + Error *local_err = NULL; + + /* QEMU is quitting: a device left running must be stopped first */ + if (qemu_shutdown_action_is_pause()) { + return; + } + + if (vfio_migration_stop_after_shutdown(migration->vbasedev, &local_err)) { + error_report_err(local_err); + } +} + static int vfio_migration_state_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { @@ -1083,6 +1126,8 @@ static int vfio_migration_init(VFIODevice *vbasedev) vbasedev->dev, vfio_vmstate_change, prepare_cb, NULL, vbasedev); migration_add_notifier(&migration->migration_state, vfio_migration_state_notifier); + migration->shutdown.notify = vfio_migration_shutdown_notifier; + qemu_register_shutdown_notifier(&migration->shutdown); return 0; } @@ -1150,6 +1195,7 @@ static void vfio_migration_deinit(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; + notifier_remove(&migration->shutdown); migration_remove_notifier(&migration->migration_state); qemu_del_vm_change_state_handler(migration->vm_state); unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); diff --git a/hw/vfio/vfio-migration-internal.h b/hw/vfio/vfio-migration-internal.h index 814fbd9ebae..e81d854d3c4 100644 --- a/hw/vfio/vfio-migration-internal.h +++ b/hw/vfio/vfio-migration-internal.h @@ -40,6 +40,7 @@ typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; NotifierWithReturn migration_state; + Notifier shutdown; uint32_t device_state; int data_fd; void *data_buffer; diff --git a/include/system/runstate.h b/include/system/runstate.h index 929379adae4..29a919348da 100644 --- a/include/system/runstate.h +++ b/include/system/runstate.h @@ -148,6 +148,7 @@ void qemu_system_vmstop_request_prepare(void); bool qemu_vmstop_requested(RunState *r); ShutdownCause qemu_shutdown_requested_get(void); bool qemu_force_shutdown_requested(void); +bool qemu_shutdown_action_is_pause(void); ShutdownCause qemu_reset_requested_get(void); void qemu_system_killed(int signal, pid_t pid); void qemu_system_reset(ShutdownCause reason); diff --git a/system/runstate.c b/system/runstate.c index 6178b0091a3..1cdecde0fcf 100644 --- a/system/runstate.c +++ b/system/runstate.c @@ -433,6 +433,11 @@ bool qemu_force_shutdown_requested(void) return force_shutdown; } +bool qemu_shutdown_action_is_pause(void) +{ + return shutdown_action == SHUTDOWN_ACTION_PAUSE; +} + ShutdownCause qemu_reset_requested_get(void) { return reset_requested; -- 2.47.1