From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AM0PR83CU005.outbound.protection.outlook.com (mail-westeuropeazon11020105.outbound.protection.outlook.com [52.101.69.105]) by lore.virtuozzo.com (Postfix) with ESMTPS id C1AB98149B for ; Mon, 9 Nov 2026 15:43:56 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LUCFrPi6G7OT4leJSKmoBEF+5nNPYlPPjrM+cXg06XfUmQVjzA57+ONqba9Tt7YolFpUqjHNzQ/pZRqXyJ07/tnCB5ZzcHM6Qoh/Ab7xPjlRTt1aFEdtLu3fQ8KmhrBqyH84C1+LH3HbQy0vkiDe0bXlEWLyqVH/3w/nsznZhXj8pgnHidC/8+teW+qkBCBgJRpSud4vcVmCH56XGqHsU6mi92tUvSWf+jMGLNzvX5tPQP+edHk7Qp5MyHcbLyZqrbM5eUxtVz9dOwgjAtmSn3s97inqI9awhFD9C022Q6cBvWCCgFxwxO9oVH6MoaiXzGXFeb+okM9TeXaDW46mvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=84Zbkl13jkrsiO5pmCK1y46LPY8Jmz3dyj+24uY7eUE=; b=D3/ALKRBGIhaanG/4ilKUM0l7vPem2mY8FZObiebXDPFZfy8F2MDf53WJo+61EkPsRO2axCg+NAeji6T7SBI7EF9YlZq7bKwjImr/HBoFAWu2HhSVH0/KJNNIvZhcb2BOeqZh5ecieNR+byI8FZhO+EOCl6UOnxK/+fOiYppOjuU2Wn4qH/cNBHCrzmD14FEidrlNueGgvLGP2EfSj3VDdBuayeTM1/p5R7okh7u7QBkbvLHqhzhQ3hMEtNTzCTAKMlCoPKNeOCEey8znRnPH0+mEjIKhwGaC7ngUPsnwZNcYTLBxu1Qjdbt74lU/rzQta/h2sRIUtmjCspDrNLcVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=84Zbkl13jkrsiO5pmCK1y46LPY8Jmz3dyj+24uY7eUE=; b=NblgHZA0Zgz4YT0T855o5uiezqYErUpKCCtcDNsRMhseebU05EFr+idNviOXjuEU5ndJFFHctWzDl7CgR5i6hiS7qemEJfoNUpwJbKlAJK/buf3hHoDkHkoJoted+zUThAGf8mZchfv4wjYK0x0WHookBoJgHYJhkcYbnDh+kHDhOJ6X5IIjhIONI/o6e7XbOKu7mS42MhVMrV73WM6bgVnJqxajCgUiqbKL/VeCpvtkp/LEh5kMLTvPhYpltn8dHxCM8Paj2SMXWg3nVijF8J60VRflbniAiraDzhoWnd4xyRHvOZS6nWTJxQ3F1AOBsYftiblA28eHO8pVbE1+uQ== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by VI0PR08MB11841.eurprd08.prod.outlook.com (2603:10a6:800:313::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.13; Mon, 9 Nov 2026 15:43:54 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%4]) with mapi id 15.21.0592.010; Mon, 9 Nov 2026 15:43:54 +0000 From: Andrey Drobyshev To: svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com, den@openvz.org Subject: [PATCH hci-dev v3 1/2]vfio/migration: stop a device left running after a guest shutdown #VSTOR-153976 Date: Mon, 9 Nov 2026 17:43:45 +0200 Message-ID: <20261109154346.153153-2-andrey.drobyshev@virtuozzo.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20261109154346.153153-1-andrey.drobyshev@virtuozzo.com> References: <20261109154346.153153-1-andrey.drobyshev@virtuozzo.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: VIXP296CA0017.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:36c::13) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|VI0PR08MB11841:EE_ X-MS-Office365-Filtering-Correlation-Id: b7cd6668-f77a-41af-f7f1-08df3e784230 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|52116014|23010399003|376014|22082099003|18002099003|5113699003|10067099003|5023799004|261028144727099003|261026121527099003|261102123227099003|261022105027099003|261021121427099003|261021123027099003|56012099006|261026140227099003|261009223027099003|38350700014; X-Microsoft-Antispam-Message-Info: qCB2WhJkJtWhHbT5o5wJxSkwsMsay55ovYGk1xb4oz3lw8/HYESQ7g1o+taTYye/HlIcA7XhLLr++6ilxo5zIPd8MqHcJGGIaf0GIUJjDDzTYy1XrTVHZzSDn+oY2tF7LPnQLfOSmQ7U8mptL159jQPqzAgCSGVtmtvIyle7JF2QOC+mvKZi0gWJepQO/SW919KH4eYg/xA1TtyMLHUPCjrG+iIOim8750TcXz1K8Kk4HryT6QKuOsWsxBSH7rScmX6W7TjN3w9xgBvp9wQsZMv05xsM9FD7zm0eqvIuCXWi4VyqniSbUgKjRff2JTIY5tgDlvgtnCHmMuSqV+WKBdF9lzBddBwh3XsqnhX8jiKOeBFqUw4tZQ/XjK2hFVxOeQR9j50zQUNluaxLTKjd141eBXZ1EuIWw6asmdTeubbig5dBw+LfeYzwosu4rO7P4aV+dNI60LDu7DtaIUp9bJJzI6gOAJDZZWngjszOYbioToe+PwvpprQ1y9hPHt8H4+JXYMIOLzM8HN/zHGC6LZwRaM2g1Ayw1ISAHiup9ivBbGS5SaITW8bgzrNZ8HrERaVK8aHJwrgqWiVfSB1kY4A3IOscMaLoJ1qYE31FJXC6PtHTEVsybHWt30N2CZt4mlwxdGbLFmBYiQrt5stnguXPTHdzk4otwdP2EvSVRKRpSk7bPUnpFYUigwYNp7l7iMB4m430kLd9khAq9r2UOWRM0O4QpwXWHuFcQ+AWA9A= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(52116014)(23010399003)(376014)(22082099003)(18002099003)(5113699003)(10067099003)(5023799004)(261028144727099003)(261026121527099003)(261102123227099003)(261022105027099003)(261021121427099003)(261021123027099003)(56012099006)(261026140227099003)(261009223027099003)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CC2htztxdRQSVVHUUp5kepmBaPIlzasz+gf8Xq2DAD4rulRKzFT0O8SpUq4J?= =?us-ascii?Q?ji1kai38DngJw+ypSviKPkwyNSosALwtWU5HhwIMa0fAdnEvACYbul4ZReEK?= =?us-ascii?Q?a+3jdp1/fHki4IE/GlysgExRLQYrBSIWveMB5bPvbJYeYt8Quz9a5woG1lpS?= =?us-ascii?Q?UlZmAqf6PWUu+2X1lxEgRAlpIrUR+BIwClJb04duxAeh0/GjNwp18jbmMv28?= =?us-ascii?Q?8aKrf7ZvHWyuBU8YwuLPk5MCJg6CISelCGd/TNAKUnI6TuqUD2hflUJqaGD4?= =?us-ascii?Q?sp/WM/V2tIRrd8PCXd0dpqkVoMoTqSlh20XfqMVosM9KJTqzSzdq4zHTL0hL?= =?us-ascii?Q?aH/rpGg1dWuAYAqcSaGzj6WwHLrn9xZIkH2ImEjWmU/nOmG7ZEd2aWTPa//Z?= =?us-ascii?Q?fl/22bbw48zdNKRMfhDOgLZkKFdbSdt1BPIh0ts9WiZasFXZiRWzwKySme67?= =?us-ascii?Q?04QmowaBxV+UttPei63aNYnRkIK5R2fppEwarKUdmYgP1t81MqHYdEXVmcEq?= =?us-ascii?Q?VjYGUQtistgxI0EhwjNnMZnB0tJTofme4GQFEh+3pYCkJGahivZja2BoFa/F?= =?us-ascii?Q?WyrgV8Qn2wXeKvDmPuvlGdSjjUVY021P99YMZWZ/cUf7W4TbpFxzMDFAm0F2?= =?us-ascii?Q?6iZnDI893811owH5E0ILoB1l/UdR1aH/QYqMBFDw2p2A0ZPxTtid4qyOoTaU?= =?us-ascii?Q?eSnzlk8H+KUJxQ8QU5kgt7S+X89yuh/FnIyG6y1WVb87hub6XAnvw2YUgRxJ?= =?us-ascii?Q?/h6ShvWl8FfNbSgNaCQ2u01GqWn0y1JZKJkoUq2xe5gj9Qu0zYp0cltXfZKQ?= =?us-ascii?Q?ltTCnP363yI6KrmJiVZ6D0QadHT+wchc6BZnosSXpo8TL793+2lcvZxYmhSX?= =?us-ascii?Q?9lriMw25u3KIaoBmZzfkwq/0VVnszUqStSBlvxsKpjSLZZ/od+48Wzi4olQM?= =?us-ascii?Q?Ywdu7WlyVkgbRhkE2SquWCJdoSGDfo81CD+PS+JWze0RYMRzGH35Ch7tarfw?= =?us-ascii?Q?CfjXieXqfmb7Zq4YkCkiNH0f2j3786NvEEtUuWdsgiF9Ayb385nlEH9qnXww?= =?us-ascii?Q?EPReJEgl2tZoknquoW2mzb0wSiM2D/DFpL/EBgVH/wlCnmMZzAVvRocBv0D8?= =?us-ascii?Q?VFhOf56QUNLHjJwB9Ldvg+HhOQ44nvTF/HYMYLxT+/9XXMqyqcwnvg8kYQzy?= =?us-ascii?Q?4NeiuV86mKWo7CGvZfBgcgCHD/KK2pUfEMH0534/o2XWLY1cx07nUWiroKa8?= =?us-ascii?Q?Mwa3m8X5A6qpm7IIydQxW/gjEdn7LipBJKXGzE7RCgovSj1fbDglHDP4TLGu?= =?us-ascii?Q?gBv/oS+YpjDnzinqgWqKFR1siovlw8NiK4iS/lpySRKd4xDd80c8ugZNln/b?= =?us-ascii?Q?o4PMCsRm9N24rVQKaKYeDKcr3BHJ7FT2WkIMZy4Ke/wNm6vqYodbj9PiIqCL?= =?us-ascii?Q?qtO5t0nKiTUw5MVceqQLS5bOXe5NagS5/uQoVHxvRy+Y5+w9S5TgBNKRaKsu?= =?us-ascii?Q?6+aAU+/ZoLD9cYzEC46CJXadyuiZ9hh9SkBCwHpGprLMTPEgocyy03uVk22Q?= =?us-ascii?Q?Iffbbomw75auhU2kcmbFXefzoxRH5gGIRKCy13T0craBz+0GTH7MCATy5zdn?= =?us-ascii?Q?ypQ4jsMSME717gFBhBJATgg+16j38nqUsZnmzw9957hD9p4uTCNIp6NYLylZ?= =?us-ascii?Q?zp5zfrNod5Pdf5uUtMGpvqfCLzpRw5P0VudLHpcePQ34+vVFt2qoLKwqewy3?= =?us-ascii?Q?gZXIKJVjGJE7zpxipXrdvf/9Y0ljWTE=3D?= X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: b7cd6668-f77a-41af-f7f1-08df3e784230 X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 15:43:53.8385 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VUayv7XYnQNLgxZMdfuK2gmsG3m7/ExwkhsEeGs7jIKd6kR1CFhzJe6ae/WskLMgDfebHCQS6remf7uOe09j4OMrLH/KY6c0Jp/L7IyqNfE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI0PR08MB11841 With -no-shutdown, a guest shutdown leaves the VM stopped. From there QEMU can only reset the VM, quit, or migrate or save it. A quit or a signal brings no VM state change, and neither does the switchover of a migration from a stopped VM. So a device that is still running at that point is closed or saved running. Stop it first: in a shutdown notifier when QEMU quits, and at the start of a save. Both are no-ops today, since the guest shutdown already stops the device. The next patch leaves it running there. This matters for NVIDIA vGPU (595.71.03): a VF closed while running, after the Linux guest driver unloaded, breaks the next VM that resumes on it. Its GPU stops answering. Signed-off-by: Andrey Drobyshev --- hw/vfio/migration.c | 44 +++++++++++++++++++++++++++++++ hw/vfio/vfio-migration-internal.h | 1 + include/system/runstate.h | 1 + system/runstate.c | 5 ++++ 4 files changed, 51 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 7e7fb496bbe..bfafc14e5b9 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -415,6 +415,25 @@ static bool vfio_precopy_supported(VFIODevice *vbasedev) return migration->mig_flags & VFIO_MIGRATION_PRE_COPY; } +/* + * Stop a device still running while the VM is stopped after a guest + * shutdown. Nothing else leaves a running device in the SHUTDOWN run state, + * so this is a no-op for every other VM. + */ +static int vfio_migration_stop_after_shutdown(VFIODevice *vbasedev, + Error **errp) +{ + VFIOMigration *migration = vbasedev->migration; + + if (!runstate_check(RUN_STATE_SHUTDOWN) || + migration->device_state != VFIO_DEVICE_STATE_RUNNING) { + return 0; + } + + return vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, + VFIO_DEVICE_STATE_RUNNING, errp); +} + /* ---------------------------------------------------------------------- */ static int vfio_save_prepare(void *opaque, Error **errp) @@ -473,6 +492,12 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, Error **errp) migration->event_save_iterate_started = false; migration->event_precopy_empty_hit = false; + /* A stopped VM gets no VM state change at switchover */ + ret = vfio_migration_stop_after_shutdown(vbasedev, errp); + if (ret) { + return ret; + } + if (vfio_precopy_supported(vbasedev)) { switch (migration->device_state) { case VFIO_DEVICE_STATE_RUNNING: @@ -959,6 +984,22 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state) mig_state_to_str(new_state)); } +static void vfio_migration_shutdown_notifier(Notifier *notifier, void *data) +{ + VFIOMigration *migration = container_of(notifier, VFIOMigration, + shutdown); + Error *local_err = NULL; + + /* QEMU is quitting: a device left running must be stopped first */ + if (qemu_shutdown_action_is_pause()) { + return; + } + + if (vfio_migration_stop_after_shutdown(migration->vbasedev, &local_err)) { + error_report_err(local_err); + } +} + static int vfio_migration_state_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { @@ -1083,6 +1124,8 @@ static int vfio_migration_init(VFIODevice *vbasedev) vbasedev->dev, vfio_vmstate_change, prepare_cb, NULL, vbasedev); migration_add_notifier(&migration->migration_state, vfio_migration_state_notifier); + migration->shutdown.notify = vfio_migration_shutdown_notifier; + qemu_register_shutdown_notifier(&migration->shutdown); return 0; } @@ -1150,6 +1193,7 @@ static void vfio_migration_deinit(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; + notifier_remove(&migration->shutdown); migration_remove_notifier(&migration->migration_state); qemu_del_vm_change_state_handler(migration->vm_state); unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); diff --git a/hw/vfio/vfio-migration-internal.h b/hw/vfio/vfio-migration-internal.h index 814fbd9ebae..e81d854d3c4 100644 --- a/hw/vfio/vfio-migration-internal.h +++ b/hw/vfio/vfio-migration-internal.h @@ -40,6 +40,7 @@ typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; NotifierWithReturn migration_state; + Notifier shutdown; uint32_t device_state; int data_fd; void *data_buffer; diff --git a/include/system/runstate.h b/include/system/runstate.h index 929379adae4..29a919348da 100644 --- a/include/system/runstate.h +++ b/include/system/runstate.h @@ -148,6 +148,7 @@ void qemu_system_vmstop_request_prepare(void); bool qemu_vmstop_requested(RunState *r); ShutdownCause qemu_shutdown_requested_get(void); bool qemu_force_shutdown_requested(void); +bool qemu_shutdown_action_is_pause(void); ShutdownCause qemu_reset_requested_get(void); void qemu_system_killed(int signal, pid_t pid); void qemu_system_reset(ShutdownCause reason); diff --git a/system/runstate.c b/system/runstate.c index 6178b0091a3..1cdecde0fcf 100644 --- a/system/runstate.c +++ b/system/runstate.c @@ -433,6 +433,11 @@ bool qemu_force_shutdown_requested(void) return force_shutdown; } +bool qemu_shutdown_action_is_pause(void) +{ + return shutdown_action == SHUTDOWN_ACTION_PAUSE; +} + ShutdownCause qemu_reset_requested_get(void) { return reset_requested; -- 2.47.1