From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <010701a1c1bc4764-0cc8f804-2d7e-4f03-a926-97407c81d235-000000@eu-central-1.amazonses.com> Received: from AM0PR02CU008.outbound.protection.outlook.com (mail-westeuropeazon11023093.outbound.protection.outlook.com [52.101.72.93]) by lore.virtuozzo.com (Postfix) with ESMTPS id CFEE481499 for ; Mon, 9 Nov 2026 19:15:08 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uwOVv4SYFMbSK3s/tMbz0L9J+OBqF/GD1AmihvxjMbvTTB4uyJcAhr3I0WcpDTAunZ7j6pJy4EN5vd6UfcqkWk1iHFeeM3UxN7tIZU7K8ROOJTg3V24udSsEajUp18GS78icG2BZouG16v5udLrvIinOMJhGlrG9k0aZyzTvYSlR0E9uk4Vejcb2Fy4/uZTc4JCXjYboZYjdGn9T7ScudmkFS9S9iLJHQFdXVzWWYgrygrcm2krLGyh3zlaqOi0ceCKcpRP+1uw3OOHaqPPOnI65yR4ZMvJuXk+gPUGHVPG8XYuDtUDU5qqqwS3VUYyjax4iK65MnnzM+UrToyDnqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SMoIcPhWWdOqaUTjX6f0Uxkt6qBFTCAIPCWykYCKSh0=; b=DKvYTCdA5uXvsw5x2DZHAfAlvNa7dzoeZ+EnIYjE5JSzz3PRqIlTmsozrTt+YbFQYEg7BJ3C8JemVcgzQFRBn6Zo06PndZRwYMpX7WrmUun1D/auEWH8j9jZwzOWKvlAEilL9eArqy2a8yLx1q9wQQGkDgb8qTV8XTr62aOaLS86pnshTRqusvqddd21oFwIP4g8Z133DZN3PyqCTbhMfk6rEuheMaSth+MW7fxqM/eTYxtZoEkbQkIV1FtfXxi64Spdwv5ZlfwDtqDpcP4KxJMO+tA5PY/hbQYlXiATzpblmdYhpvOzFouArAFsgIaSq8fAB3kNLkkPx/4zdQzxEQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.1) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SMoIcPhWWdOqaUTjX6f0Uxkt6qBFTCAIPCWykYCKSh0=; b=P0rB+8jFwmVKu8csbjmXlki16AckA4aPwEXGH+9DgpGuXHxva6HiUof/qKO49lLiljL6zGA95ZtLgPd/mKtbD/8tOd+s7MJAzEd9a5SLJLVUQpruf7KuZvvo2MoZO66zRBlKK/np2I9iCTp2k4Etox3VWFD2Y3TTR7j0KiN2alo= Received: from PAZP264CA0006.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:21::11) by PA6PR08MB10419.eurprd08.prod.outlook.com (2603:10a6:102:3c8::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.12; Mon, 9 Nov 2026 19:15:05 +0000 Received: from ZR1PEPF0000E45D.eurprd04.prod.outlook.com (2603:10a6:102:21:cafe::ad) by PAZP264CA0006.outlook.office365.com (2603:10a6:102:21::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.13 via Frontend Transport; Mon, 9 Nov 2026 19:15:05 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.1) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.1 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.1; helo=b224-1.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-1.smtp-out.eu-central-1.amazonses.com (69.169.224.1) by ZR1PEPF0000E45D.mail.protection.outlook.com (10.167.241.53) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.615.4 via Frontend Transport; Mon, 9 Nov 2026 19:15:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1794251704; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=rJ0c4O1mSabsacDlGUHtG8G7XIS6xb91dbIwd+WYpo8=; b=ZNIx/WR7hnt9+Vq2+QI+ruAASXbJwAuN1Wptci6takNtZEmQ7O1dD+aEjyYAQ7Pl HrijInz1tpUtJl9b0u/cemqtMvmKxT1gX31MRUUast+5olV/0MRWFmYWzlkaUBjxn3x RXOJmonUKG1nhNJ2krvA3s3Z0iUO183aBd//9HM120xanZktvWwnCsxZg8ylPs0bHzI bJ0b1+8dsmMkzXQlnEDo93V8kJ+5IjGShmKK8F6ZBfM8Zh1V1/c+CgEwu9jhHd276rP vUNLyedfyb9+tinzgpvde90MMy3k7VnreyyJW/Yk9F7YVc3rMZ4yr07LW/Ly3UXp80b +d8wo7OTqQ== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1794251704; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=rJ0c4O1mSabsacDlGUHtG8G7XIS6xb91dbIwd+WYpo8=; b=ZjDXiTVuYQotckSI2DPz/QdrgVt74MCU1gtLkEaxFyrilJ7Ch8BhY7ojYkTcqLcC yub0BkgTBE/GxeFpCY2L2ifnNaqIaOZaG7sp8i8dAx6lh3wpAlNLiCyhtNFJyGrqQJ3 vgx7n7vuvzyyXyQduyZpVE5HvOsXryxpXGl5o8zk= Message-ID: <010701a1c1bc4764-0cc8f804-2d7e-4f03-a926-97407c81d235-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH hci-dev v5 1/2] vfio/migration: stop a device left running after a guest shutdown #VSTOR-153976 To: svt-core@virtuozzo.com, Andrey Drobyshev Cc: andrey.drobyshev@virtuozzo.com, den@openvz.org In-Reply-To: <20261109190342.200293-2-andrey.drobyshev@virtuozzo.com> References: <20261109190342.200293-1-andrey.drobyshev@virtuozzo.com> <20261109190342.200293-2-andrey.drobyshev@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Mon, 9 Nov 2026 19:15:04 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.09-69.169.224.1 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: ZR1PEPF0000E45D:EE_|PA6PR08MB10419:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 7cdb4547-c074-43d6-ab60-08df3e95c2ab X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|48200799018|32142699015|61400799027|376014|9140799003|5023799004|4143699003|261028144727099003|5063699009|261021121427099003|261022105027099003|56012099006|261026140227099003|261022075727099003|261026121527099003|261009223027099003|261102123227099003|10067099003|55112099003|22082099003|18002099003|16102099003|4113699003|5113699003|13003099007; X-Microsoft-Antispam-Message-Info: 1RMaflewYJzXE1DxKhsbskA8DNkd5L7kHAAT42fR3bAZx4f3Y83oC9ckLh9425kurzmN0P0e3ZTQiwM308ObvvR5c3FlVhxcX0F3TcfHVU+eT0NAOeT1o8b1tFhhPV7JmvG68qfznqvRpqGObtL0vXxC4NbkbvRBbHWdAeNUpuk1cXo11KpWxESOFz8OnR5XtJ8c5QrFLjOWosQumTW00BmEiPkLp7XkRnNZsZpLRrWhB+QdvtUT0QS4p9TnGoBC+VJ4iTpND1HB/13rIf+cNcSyhrqETc/IQ3QVxnYEzB0G4DDCbSbJMza5Qh73zUisJzHjVKLCkIdkhkMXPIsHismtXtdGrR+AbJxKuxqegq7Yktc04oR98LWCJrHkeNcc9H4pchyvVfSO76n79pOzueAERCEUM11mpsKd/42/n58x0g6lu0hyu7GPQvyW0BdmdfCk8rgD1yp+HbUXoYOopdoxzn0BHBdbnZiL5ivLyuOj1apJpLQvQzCyjbMZhEPBF7W3b4K6egVcnzKqs6Y1zoJHavlRWOgq2bQBgPG4R2pYaLXy/OgVc8EhuqcVXvZLAuGIHhZ4RU1m2zfppA/AMoJ5O/WQh7DlraFUS15MZR2eXKYxUfmrIn4VvbqSaZnvygF0VpO5VwebVClMhliwVPmSMVC8x8dkyLyG8mMOAkrjEK+9MikpOYYnB8BSId6+rinD+qejBIUNwMikCZugN1kDV+mFi37QLYYA83/nhQgeX4FckMJRpbk8DFILcjBCAUIVA4V30Wn3tJnwt8Vp7RiXxXZI9SvG96XjoMIZALBnKInk4M/KNJv2gXv8hjtupGqJ6F3zIgmwNkj2cplrM3QmCIzOr1tGroj1MZh1ygpc+8FWuZp7226As7ORzpGu X-Forefront-Antispam-Report: CIP:69.169.224.1;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-1.smtp-out.eu-central-1.amazonses.com;PTR:b224-1.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(48200799018)(32142699015)(61400799027)(376014)(9140799003)(5023799004)(4143699003)(261028144727099003)(5063699009)(261021121427099003)(261022105027099003)(56012099006)(261026140227099003)(261022075727099003)(261026121527099003)(261009223027099003)(261102123227099003)(10067099003)(55112099003)(22082099003)(18002099003)(16102099003)(4113699003)(5113699003)(13003099007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mRSi19xlT4Dx7WVLk91w6sy3C2G0cPei7cu7D1+8VA2UhEsd5M3I0FsROgS6pl8g3F5JvPyZOmhQKYR8mU+vIjsqOa7Dia0wrEJ6b+//V+zdL4IW/+16MRxT3EjQWfWYuELebQh/OOKRoXMA67lXAM62430dWmNZuf2i28Kpw8XmovMeKEf3IXv8Pc5gCDQ2GaYB1MiBoHCwW8/08rcUAsylmkR4ScdjTjIMg73LHsD6YERVAiQ5h2vIJv8PyC9+mOdetaOASQybfYNRiizrwkypRweObgYM8M9gCBdVHgKlOphXAP+AfRREq+flbFnwomAo5o8zvtlXSAwbY3vtrnuspV5MB7qHiiecE9yxt2yV1dGLxqL5dmGYDYdWkRG/u5f3JaUAwOLWosIIrxHM7mGUy7f4WLoq651D4PHlvd4hZRLKXtDgoyNC/Jg+PHCB X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Nov 2026 19:15:04.6370 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7cdb4547-c074-43d6-ab60-08df3e95c2ab X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: ZR1PEPF0000E45D.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA6PR08MB10419 Thank you for your contribution! Sashiko AI review found 2 potential issue(= s) to consider: - [Medium] vfio/migration: stop helper misses RUNNING device left in a PAUS= ED VM - [Low] vfio: stop_after_shutdown() stops RUNNING devices one by one withou= t a P2P round -- commit 011d0ac85a859eefe6fc0507fe391eb5b7768f9c Author: Andrey Drobyshev vfio/migration: stop a device left running after a guest shutdown This patch adds vfio_migration_stop_after_shutdown(). It stops a VFIO device that is still in RUNNING_P2P, or in RUNNING while the VM is in SHUTDOWN. The helper is called from a new shutdown notifier and from vfio_save_setup(). > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > +/* > + * Stop a device that a guest shutdown left running, before it is closed= or > + * saved. Only that leaves a device in RUNNING_P2P, or in RUNNING while = the > + * VM is in SHUTDOWN. The P2P round of the VM stop has already run, so t= he > + * devices are stopped one by one. > + */ > +static int vfio_migration_stop_after_shutdown(VFIODevice *vbasedev, > + Error **errp) > +{ > + VFIOMigration *migration =3D vbasedev->migration; > + > + if (migration->device_state !=3D VFIO_DEVICE_STATE_RUNNING_P2P && > + !(runstate_check(RUN_STATE_SHUTDOWN) && > + migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING)) { > + return 0; > + } > + > + return vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP, > + VFIO_DEVICE_STATE_RUNNING, errp); > +} [Severity: Medium] Can a RUNNING device be left behind in a PAUSED or PRELAUNCH VM here? The helper stops a RUNNING device only when the run state is exactly RUN_STATE_SHUTDOWN. One path that seems to reach this state: VM is PAUSED, VFIO device is in STOP migration_stop_vm() vm_stop_force_state(RUN_STATE_FINISH_MIGRATE) (VM not live: runstate_set() only, no vmstate-change notification, device stays in STOP) migration fails or is cancelled migration_cleanup() migration_call_notifiers(MIG_EVENT_PRECOPY_FAILED) vfio_migration_state_notifier() vfio_migration_set_state_or_reset(RUNNING) migration_iteration_finish() if (runstate_is_live(s->vm_old_state)) { ... } else { if (runstate_check(RUN_STATE_FINISH_MIGRATE)) { runstate_set(s->vm_old_state); } } For a non-live old state, vm_start() is not called. The device ends up RUNNING in a PAUSED VM. On a later quit, vfio_migration_shutdown_notifier() calls vfio_migration_stop_after_shutdown(). The device state is RUNNING, not RUNNING_P2P, and the run state is PAUSED, not SHUTDOWN. So the helper returns 0 and the device is closed running. A retried migration reaches vfio_save_setup(), which calls the same helper and also returns 0. The commit message says "Today only a failed migration of such a VM leaves a device running there", and the helper comment says "or in RUNNING while the VM is in SHUTDOWN". Both seem to understate this case, because the same failed-migration path also leaves a RUNNING device in PAUSED and PRELAUNCH VMs. The follow-up patch "vfio/migration: don't stop the device on guest shutdown" only adds an early return in vfio_vmstate_change(). It doesn't change the PRECOPY_FAILED handler or the run-state check in this helper. Should the helper cover this case, or should the PRECOPY_FAILED handler avoid setting RUNNING when the VM is not running? [Severity: Low] Is the comment "The P2P round of the VM stop has already run, so the devices are stopped one by one" accurate for the RUNNING branch? A device in plain RUNNING in RUN_STATE_SHUTDOWN did not go through vfio_vmstate_change_prepare(). It is sent straight to STOP with no all-device RUNNING_P2P round first. The path to that state is a failed migration of a -no-shutdown VM: vfio_migration_state_notifier() /* MIG_EVENT_PRECOPY_FAILED */ vfio_migration_set_state_or_reset(RUNNING) This moves every device to plain RUNNING while the VM stays in SHUTDOWN. A later quit or a new save then reaches vfio_migration_stop_after_shutdown() for each device in turn. With two or more P2P-capable devices, one device can go to STOP while a peer is still RUNNING and able to DMA to it. I haven't shown that this causes a concrete failure. The comment is only correct for devices found in RUNNING_P2P. The follow-up patch makes RUNNING_P2P the common case, but it leaves the RUNNING branch and this comment unchanged. [ ... ] > +static void vfio_migration_shutdown_notifier(Notifier *notifier, void *d= ata) > +{ > + VFIOMigration *migration =3D container_of(notifier, VFIOMigration, > + shutdown); > + Error *local_err =3D NULL; > + > + /* QEMU is quitting: a device left running must be stopped first */ > + if (qemu_shutdown_action_is_pause()) { > + return; > + } > + > + if (vfio_migration_stop_after_shutdown(migration->vbasedev, &local_e= rr)) { > + error_report_err(local_err); > + } > +} [Severity: Low] Each device is stopped individually by its own notifier, with no coordination between devices, and a failure is only logged with error_report_err(). Could a device that fails to stop here still be closed running? [ ... ] Please let me know if I have misread any of these paths. --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/2026110919034= 2.200293-1-andrey.drobyshev@virtuozzo.com?part=3D1