From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from PA4PR04CU001.outbound.protection.outlook.com (mail-francecentralazon11023083.outbound.protection.outlook.com [40.107.162.83]) by lore.virtuozzo.com (Postfix) with ESMTPS id F40F18146A for ; Mon, 9 Nov 2026 20:45:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=XnIULIgXW19Yv3goaaffvI7p5d8A24lt4y7rHMjN6xx6SJOf3AdX4B1yj+As0UPrMVgoPuQI53VD7xbT7kc35oxRyttjr2Xhk7nZ2oMZOUJt57lwvbGduMop8KlIutJLvSlXegsrlUe98lf9al16daEEWjGLo1igtigknbuk7paSa+M9xHzt1kY6+5oQOubV+rqKy1ZrGQEeRV188W+VhluTQATdLZad4oJ21PDhHI7Sh9krep8DipwhDeSGR1VGngBx5Q3PPF5c/O7MoWGiiH8eqJhwYkcpyEdL993uKuZjs9T2qi+8EJePgEmycY6yYXcp9suvzo9vXG4UO7y7CQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TyqE2fBnxZn1ckCbs39PwuUe7joJILzwbp0YepLuWsY=; b=rQBA2/B+FlUk1ijEu0zzckqMevdCn0DG9OmaFHaqTDcNbI68LYiGVOXVQolhFY6UOD8vgXvzOjhHqaXDLjwaB8gcfENPS7Pc1NKAcRMOs44Rw0ZJqPSBLr4mEVSYuqdTZKOfIwXNXijnF5t3qzNFwUK3TlOTb8/QnhrajxEgJcU2IwOD8R20YDZ0PY1jmPEYKhvquoRzxzpWOGWtT3ZjJ7pqSDpqgT0cCnP+dCkaB+hJp8C2AsEjybNKhL1xAToKql5jf2bxysrWlpnVNYOGgLu4BolaX8+hFSdBYcul7b6gh74ycC0Hg+yuCkj0IEWQeOh65eiEZM3l6SReGLDrAw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TyqE2fBnxZn1ckCbs39PwuUe7joJILzwbp0YepLuWsY=; b=mu1nj1GsYYeE+mXjR5/AkPdWABldHBhXBvLtIK5jwWwMduXVI28IeYqr6ykjEzmhJVVomhO9Pq6Y1O21zJ/mXNxoTddaSnjUAoRGP86hVaq6IRRs6w+zYqqCwVcFrlBF0CeFDGRo662PqGa2O0RGMs0Jo8hPYPZdJMo4HFot+2Dr+aw/zgDFJa1bpnYN9aGj+nW02gRVcwSHjeyDEFOlLVWMDnQzx3Hgrn4qi00Ba9FLJ6sjVGp7x9h84tqoLAa9dCPOwn23aO2B1tRQ3cv87ATdfML7jXT8z3/uNTfs+znYsnGcv65IRH8RBq5ttpTHgzsjhuemiXsQ+taIcXbdRw== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by AS1PR08MB7611.eurprd08.prod.outlook.com (2603:10a6:20b:476::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.11; Mon, 9 Nov 2026 20:45:26 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%4]) with mapi id 15.21.0592.012; Mon, 9 Nov 2026 20:45:26 +0000 From: Andrey Drobyshev To: svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com, den@openvz.org Subject: [PATCH hci-dev v6 1/2] vfio/migration: stop a device left running after a guest shutdown #VSTOR-153976 Date: Mon, 9 Nov 2026 22:45:17 +0200 Message-ID: <20261109204518.220617-2-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20261109204518.220617-1-andrey.drobyshev@virtuozzo.com> References: <20261109204518.220617-1-andrey.drobyshev@virtuozzo.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: VI1PR06CA0216.eurprd06.prod.outlook.com (2603:10a6:802:2c::37) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|AS1PR08MB7611:EE_ X-MS-Office365-Filtering-Correlation-Id: f1a78417-89da-4774-130e-08df3ea261c2 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|52116014|376014|23010399003|18002099003|22082099003|5113699003|10067099003|38350700014|261009223027099003|5023799004|261028144727099003|261026121527099003|261102123227099003|261022105027099003|261021121427099003|56012099006|261026140227099003|261021123027099003; X-Microsoft-Antispam-Message-Info: /Xy2/q64FmbV3Mj8UjTzQIRch+yEyedn68p/8uxHIIkDBK33TCAvQAi9PNKldu25r0BEptNwiuotsGjI7HWDs+tM3V+CvSEiMN5TloB4yaAcMnQLveQf+IrJB/8Z2tItKkcEbXVHLvAnzJxvbLvM9HOY7GT8+vRfxKJ5+NYnvhgreFAexWMeJn1YdKP0QyhgvTTbVuPltTh0G9iVZjHMpLPAi1eqHCzcpFwF2AO1tROiPGckp9Sd9ZIJR61FgZGJ+pA0J4ftqe9V2ElvWZQ97NfaXOnkjlmP2qqCWu3A50LM+hHSouCdmxZZLlG9F7WXdrNsY0h1oYfOg0JoeTt5BV9/EofEGbuxxVLOlOQ2AFu9F0jiX6zQ363gLrxdz/hlu8gMtx4hA5MEmSX2HfglCsjIfYQMcSlOo2tsBO9+I4CqvPDG1Lp10gtzEoTVRORnpEvYARqBhEX9yi0evxfy9io65ukYbT3lXIgSwqaiTcuxB5Px6amZkqpBj8SMjjWag+SzTWnJw34WtzsCui3S/voinV6KN77/uKc1TzUwneznysqv5UOPmkf1AvETobNX7eAxiIJjsUc9mNFue/m63mY1jIzMHx+MPa1gyD8zndmpeOARH+O+3LX82wmLJD13aYKM2fGOTHVPounCR0Z++uKI1LUstiwKOnFB9hj60TX11++wCd1NuYUpzHPQwoE1yKepD+3o7FImMA962Ru1H/pQgrWlngx5Zdd3f/IrHNk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(52116014)(376014)(23010399003)(18002099003)(22082099003)(5113699003)(10067099003)(38350700014)(261009223027099003)(5023799004)(261028144727099003)(261026121527099003)(261102123227099003)(261022105027099003)(261021121427099003)(56012099006)(261026140227099003)(261021123027099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?WPTPpo11N7HGZJygVr+bAlPpT/3k9vsEHges40jPElksnhrdIJWcbkoSFyA8?= =?us-ascii?Q?2L0z8SHT0WY3rSwRiqHhgSghlZDEJfwiyykBPm/842fNDO8Zpr7ql54WXo5J?= =?us-ascii?Q?6+wI/y31dKtxLmOE8DQ4A8/RyfWMMupgK//lkT90qLAdc1RZhY6dSkqgOqvF?= =?us-ascii?Q?3rLnWEbq3RBUtJWV1uObPdMhRuRQ4WZu4gTwkwiH73N4uFRzc0kqpvYe2/1r?= =?us-ascii?Q?R8LGwPvznpall3vvYud4OMOaGDzBciAIlIF8oCGbKsvUmoGKbGMhrQWNePMo?= =?us-ascii?Q?O8ykc+YbjwwMaoC+etjp3kgXZ1sc1zWI2AhLFHn4+wNI6GMQ/2DBTAtg8ePd?= =?us-ascii?Q?/hA+q81+kiEp5q1buaVW9bxEjSgkKAa+wl8VIa++BgrwzePfL+cjZLwllub1?= =?us-ascii?Q?QSytKJ8G9tL92NRZYwj24+ahERcMGGHNCHGWWCAbeVlmx8WkYlRqp6uhHxWI?= =?us-ascii?Q?vTAOv7PHpqjs/TpBv+rGZRsgIBKKb6ukhG6O1EVu7F8PVkWzxAGPPYb7fkRX?= =?us-ascii?Q?MLAU3CnvWSPvcZB6mdv8eWFcGQjlNovWuMoj2bCuutJ/Qca1qyq7kWlmy3yw?= =?us-ascii?Q?t205dbnFu7iUd7wYZyewoPhVhqj0uNkWr/37DLCBkqT5HqKPAI6tgZw19zG3?= =?us-ascii?Q?Cyd4wauynrJD5aWxgjJQ4g3vYwoahtQaQkMNQXIo3z+E/7PsuUddqUIoUqD6?= =?us-ascii?Q?RC4FDGCZTbnXaxG0QFROVG5gkeQfLjP3G7OcTu6bUT9srlm98OxHugmwh4A7?= =?us-ascii?Q?Ftt2nhkQuTtFb12C+MqUbi2CNgDYCQJR4qDX3b2icAvubkXtwpgPJEBLOuqe?= =?us-ascii?Q?iSzEwrSaPKs1najrWLLTINzLiMDF4f86h3gW4R5lIjettNgM+GUiq19IwSQL?= =?us-ascii?Q?j3ttcvUEtsq6d/P45ujY7pT97fHmPMx6duia6uvE51TETBOil/R6vYn9jndQ?= =?us-ascii?Q?2EAky7qREkHLv52OFFUqNqTKqRkpU1TRBwpR50F3KYl3Lfrms0XH9RA2HVFO?= =?us-ascii?Q?Ihpv5skMf5zUPaWuFbbpbsfXTAum7jm+ZOOTDCBwUwkXHZuRTWFkpni4Opj+?= =?us-ascii?Q?hyEv13EBKbrHIFCx/a3CMeeq4K7/aAJ9PTwWGdCpR916SX2XL6Ck37bSLtWP?= =?us-ascii?Q?eK+tACb6i5DIxb0z0IWge2j/ooO+7+gR6znYU2vKlWuTcP8ALZUWLIxiBggo?= =?us-ascii?Q?Tt0+gGEB1M0fpkHU12TlZqi4Ra9oc1C7sHWIgbWWjY3qv1b151KaoolkXPNW?= =?us-ascii?Q?PnXKdDlBZKg427M2P5VRv5waUcA9H3TphR0ksFUtD9lwWpCPUhYbWNvwJCpt?= =?us-ascii?Q?bL2QU63S+OJ7kuxVSgc4ZZrUrmQXadO8TKh4dsDk5ctlzdg61/CIFkDBe15w?= =?us-ascii?Q?c6M0hZw4BKImRQObDk0fNB2Ir8Uw36923YKNDqFvTakv+yJV8CkjDlW1oU6Y?= =?us-ascii?Q?aP/mKWZSBtAoIGkL2wrbc+KltaqyI8AsTpywD0z2Q8z06t4r8YEQMLuhgJW2?= =?us-ascii?Q?cgi+JE/B3HKZbw1/n2vFUSNGR7ZaQPm73FjFPv77OiY4OUMK8Bbr/7mtuAqa?= =?us-ascii?Q?TZQ2KR+zdcE27WZcsB3bCE/hSrucsbJ2RY2CNxCFsojr80eLaqa6MBLQMj2I?= =?us-ascii?Q?v2M2fBckg8RaDCP1cEIWjHIHcDF+Jy+OWPA2j/bH0jqy4yhhollXHsXt5NQ2?= =?us-ascii?Q?E6snbuzw2BsfzeFu6Vldu4mbYyyHE+vP1Nm5/1jsHyf8BoOUClSsAcLio9RA?= =?us-ascii?Q?VsqRysVqBdW659E3WrtEAVYANdC2IPw=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: f1a78417-89da-4774-130e-08df3ea261c2 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 20:45:25.6635 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: o5fmAkhz9Q5d/eOUj/HVd+XImJnBRAMs2exdq4I1qXX8zM3FhPzjheeeFoPvvh1Nx8RdDBIugNoTqq7je5UBxrtoVaIy9VFLp+6UVSiiJiw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS1PR08MB7611 With -no-shutdown, a guest shutdown leaves the VM stopped. From there QEMU can only reset the VM, quit, or migrate or save it. A quit or a signal brings no VM state change, and neither does the switchover of a migration from a stopped VM. So a device that is still running at that point is closed or saved running. Stop it first: in a shutdown notifier when QEMU quits, and at the start of a save. The P2P round of the VM stop has already run, so the devices are stopped one by one. A reset of the stopped VM leaves the device in RUNNING_P2P. Nothing else does, so a device in that state is stopped in any run state. Today only a failed migration of such a VM leaves a device running there. The next patch does it at every guest shutdown. This matters for NVIDIA vGPU (595.71.03): a VF closed while running, after the Linux guest driver unloaded, breaks the next VM that resumes on it. Its GPU stops answering. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 47 +++++++++++++++++++++++++++++++ hw/vfio/vfio-migration-internal.h | 1 + include/system/runstate.h | 1 + system/runstate.c | 5 ++++ 4 files changed, 54 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 7e7fb496bbe..9c0ba2bde5c 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -415,6 +415,28 @@ static bool vfio_precopy_supported(VFIODevice *vbasedev) return migration->mig_flags & VFIO_MIGRATION_PRE_COPY; } +/* + * Stop a device that a guest shutdown left running, before it is closed or + * saved. Only that leaves a device in RUNNING_P2P, or in RUNNING while the + * VM is in SHUTDOWN. The devices are stopped one by one: a device in + * RUNNING_P2P has been through the P2P round of the VM stop, a device in + * RUNNING was set there by a failed migration, which has no such round. + */ +static int vfio_migration_stop_after_shutdown(VFIODevice *vbasedev, + Error **errp) +{ + VFIOMigration *migration = vbasedev->migration; + + if (migration->device_state != VFIO_DEVICE_STATE_RUNNING_P2P && + !(runstate_check(RUN_STATE_SHUTDOWN) && + migration->device_state == VFIO_DEVICE_STATE_RUNNING)) { + return 0; + } + + return vfio_migration_set_state_or_reset(vbasedev, VFIO_DEVICE_STATE_STOP, + errp); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_prepare(void *opaque, Error **errp) @@ -473,6 +495,12 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) migration->event_save_iterate_started = false; migration->event_precopy_empty_hit = false; + /* A stopped VM gets no VM state change at switchover */ + ret = vfio_migration_stop_after_shutdown(vbasedev, errp); + if (ret) { + return ret; + } + if (vfio_precopy_supported(vbasedev)) { switch (migration->device_state) { case VFIO_DEVICE_STATE_RUNNING: @@ -959,6 +987,22 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) mig_state_to_str(new_state)); } +static void vfio_migration_shutdown_notifier(Notifier *notifier, void *data) +{ + VFIOMigration *migration = container_of(notifier, VFIOMigration, + shutdown); + Error *local_err = NULL; + + /* QEMU is quitting: a device left running must be stopped first */ + if (qemu_shutdown_action_is_pause()) { + return; + } + + if (vfio_migration_stop_after_shutdown(migration->vbasedev, &local_err)) { + error_report_err(local_err); + } +} + static int vfio_migration_state_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { @@ -1083,6 +1127,8 @@ static int vfio_migration_init(VFIODevice *vbasedev) vbasedev->dev, vfio_vmstate_change, prepare_cb, NULL, vbasedev); migration_add_notifier(&migration->migration_state, vfio_migration_state_notifier); + migration->shutdown.notify = vfio_migration_shutdown_notifier; + qemu_register_shutdown_notifier(&migration->shutdown); return 0; } @@ -1150,6 +1196,7 @@ static void vfio_migration_deinit(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; + notifier_remove(&migration->shutdown); migration_remove_notifier(&migration->migration_state); qemu_del_vm_change_state_handler(migration->vm_state); unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); diff --git a/hw/vfio/vfio-migration-internal.h b/hw/vfio/vfio-migration-internal.h index 814fbd9ebae..e81d854d3c4 100644 --- a/hw/vfio/vfio-migration-internal.h +++ b/hw/vfio/vfio-migration-internal.h @@ -40,6 +40,7 @@ typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; NotifierWithReturn migration_state; + Notifier shutdown; uint32_t device_state; int data_fd; void *data_buffer; diff --git a/include/system/runstate.h b/include/system/runstate.h index 929379adae4..29a919348da 100644 --- a/include/system/runstate.h +++ b/include/system/runstate.h @@ -148,6 +148,7 @@ void qemu_system_vmstop_request_prepare(void); bool qemu_vmstop_requested(RunState *r); ShutdownCause qemu_shutdown_requested_get(void); bool qemu_force_shutdown_requested(void); +bool qemu_shutdown_action_is_pause(void); ShutdownCause qemu_reset_requested_get(void); void qemu_system_killed(int signal, pid_t pid); void qemu_system_reset(ShutdownCause reason); diff --git a/system/runstate.c b/system/runstate.c index 6178b0091a3..1cdecde0fcf 100644 --- a/system/runstate.c +++ b/system/runstate.c @@ -433,6 +433,11 @@ bool qemu_force_shutdown_requested(void) return force_shutdown; } +bool qemu_shutdown_action_is_pause(void) +{ + return shutdown_action == SHUTDOWN_ACTION_PAUSE; +} + ShutdownCause qemu_reset_requested_get(void) { return reset_requested; -- 2.47.1