From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <010701a1456d5473-599ca796-a11b-4317-b61f-f922285ec6ed-000000@eu-central-1.amazonses.com> Received: from PA4PR04CU001.outbound.protection.outlook.com (mail-francecentralazon11023101.outbound.protection.outlook.com [40.107.162.101]) by lore.virtuozzo.com (Postfix) with ESMTPS id A012780460 for ; Fri, 16 Oct 2026 15:55:59 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PA0BumbRyj+hDIh7mXKfueapAVoYG64qOy6Y1hyKgXax8Zq0jPfQ1PqQ+QbZIIpkkJ48jntJ00zX+bNz3yN9zuH4tO1za5V2liQEXf5/0xeIuqx0AG+UIjB65dktablHrxFwAotHy6/wwR6cD9w/obamrnFryUcEeM8ap73KG9M7J1johzijbLAki6leTs8Ifn7ndSGRPxZ4GzuA4nLj9jV2HDNV9guficxN3xCRXfa4hTyJsf+ECdzihoThyTi+2mNMAQVJ3faQUNhJTeUZE97SPr9vs3AlmzEkwAp1lJohqK/t1y7w/Iha8q6b14imtt/zd2TpIQuMnDuA8WYCUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OnSne17p7NeJAJSvI/py9xqBIEDrGwyu+tTgE2TL5ec=; b=ZjRsJ/VHbVZxTlxBEL+hkd00EcQ1QP5lI5rQn/l8CYesPvqB1Ogn8e2PNcWPSe60/Xmvj6MJpu9EEXiOASQfMOXDyAOL0bmhHrK9DDrThduiT8HaJymwjtKfshyLszMI/8Ooc2jMx8ZkppH4bDjAaDp+aqbzu1+NqSrGV6AGNoixCBjWiC0DC8Wxn1JGPolQfzUwLLUScHKcU1qTPAxRnMCfn4aDcFNW+kuifNgoTtigYZNxcqF58Wy7RUtPmvOkB6Blt60YIhKcQuRSz2IOa5vZPKLFPEW1JS+FxUB6sQ5xCW/smqfD0/pItiK8bCxb24+9Q9aOHHqTW0vewHCmIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.1) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OnSne17p7NeJAJSvI/py9xqBIEDrGwyu+tTgE2TL5ec=; b=OZzf1xVoimDbAYxfKHA5Rqt1JJMRa8j1uGM9nlOlcqR94fckm4ykRbaVzfFd17SJ0T7alLXA7IwmRhMB/UGgOr676Y4HUn2oOI2TYLXLUSGa96WS2qt7BJiigSz/Ad1H/pT4H+InjVICsIm8kWDSQxwBTdgAUBznfOqIuQFq4eI= Received: from DU2PR04CA0062.eurprd04.prod.outlook.com (2603:10a6:10:232::7) by DU0PR08MB8954.eurprd08.prod.outlook.com (2603:10a6:10:465::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.496.14; Fri, 16 Oct 2026 15:55:56 +0000 Received: from DU2PEPF00028D0E.eurprd03.prod.outlook.com (2603:10a6:10:232:cafe::19) by DU2PR04CA0062.outlook.office365.com (2603:10a6:10:232::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.522.4 via Frontend Transport; Fri, 16 Oct 2026 15:55:56 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.1) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.1 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.1; helo=b224-1.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-1.smtp-out.eu-central-1.amazonses.com (69.169.224.1) by DU2PEPF00028D0E.mail.protection.outlook.com (10.167.242.22) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.522.6 via Frontend Transport; Fri, 16 Oct 2026 15:55:55 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1792166155; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=epZVra3EAi7sV+TVsyXS+MJQHlSV68cjNDnDWJGac/U=; b=xUiPUI8pKeRMWbFHqKc7s98ZpVh7ogl1esHQSX9zw8eAdgxvd0fZtAMKXbmh216F 4tOryoWXYTx7UfUf+BFOentmPop9afaYHNLX1/4vH8N7UV+v4QM1a/J8fK5XMDT8dmR P3qwJD68OX0rAY4nyCkgwElqrBL21hwKB0ICnO65pA4rOQhkNr8j4ilKjJE4acaosEG NkTatnqKDL3GAUm9G1MnoArhFAtLTZeAHljWYgsC5+nVEfMEoVyZpjlH75XGL3lfTWU akg0bLDTDulqeOs7qdjA1XqizPRDXCNIpJ/ttXqW4C0ipetC5R5z6TtTBkssczY/oUh nNLNGP0glw== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1792166155; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=epZVra3EAi7sV+TVsyXS+MJQHlSV68cjNDnDWJGac/U=; b=Gh2IjPrSF7EQYnA4OOaXQRq/ak0RTmoRp5KOcJa/bJ+P6QUj5bVlxmaBTdUjkcfg v9yCgZYL6iWrUmwc0PKcA7DkRX1je/J+YMUpH02o7RjbcMlud70QIiObh81xTJB1nDp Tmdzo3U2Gr3XgJogcuKw8s/jlpYtCzgJH3hrRrV8= Message-ID: <010701a1456d5473-599ca796-a11b-4317-b61f-f922285ec6ed-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v3 21/28] vfio/migration: don't migrate device state during CPR To: Andrey Drobyshev , svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com In-Reply-To: <20261016-vfio_v1-v3-21-47487cb5a8e5@virtuozzo.com> References: <20261016-vfio_v1-v3-0-47487cb5a8e5@virtuozzo.com> <20261016-vfio_v1-v3-21-47487cb5a8e5@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Fri, 16 Oct 2026 15:55:55 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.16-69.169.224.1 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DU2PEPF00028D0E:EE_|DU0PR08MB8954:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 8923cb07-8123-4d99-70d3-08df2b9df6c2 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|32142699015|9140799003|376014|61400799027|48200799018|6133799003|56012099006|16102099003|5113699003|4113699003|18002099003|22082099003|261009223027099003|13003099007|55112099003|10067099003|5063699009|4143699003|5023799004; X-Microsoft-Antispam-Message-Info: lWf4PQtVOm5vAzvc4w8vtwj5iz6HGrocR5z73k+w1WY9di4CdEjmi5ZLk6lHD3Hlbe4pGOKGdTl3MeRp2/gCFVP9PbiX2adPwyQyUWVKPNCwzARRKf4YDXh5QwEoQjkcDAiAICZKdmWyd+TWNpY+AgKpHbdefm9JG2V4HdubYBnHcznQrmuLhz6aMmev6JDWOKv5mmkjyQVmwxYKxNBPB4RjJ4olU5lNG5N91yLgWDqg1sFlF7nDWEKL/nLgC8WhaX0THMxtr2OBkbZ5PAFNkc7Rj0P27Bg0aqoYCgEkEKbfWnOgHhRxkibVikckMkfarImekQ+GylLgistj8kPVq+o+zYcT2JyN3aLUiopTKssVEfVf/lvanZAQiEmqq37vGuICUkk/yerVaDL487OlXXqM+9hsd1FJYEza0kA5ia+iJZBIzPJQHVh+ZUm6lRXLhZKk9x/MHeIIAE4u6pPl53aZKTrAqoifI3aaO1Wy9cE4gDB7UE0qHXDRjpkXY6MwBiObrU+u67gxhmVezS1DBX/vaVbPvCc4wSdoguSaH0TwxyA1qCE3c9zlaW1aBRGiwJLMgpxbhcY0ang/1J62Ht4lKezYCTJHuSflgk6pW8qYkX5QBwL0FijhXrLG+Epo3/tP1J4DvsVcMeGsHSIJZFpZt+t78xLvAJ1chG2x/avJf5SB/bu9m/Jn/smlRZVq7chpbnTf2wO8h+mn11FkKQ1K5PLqABlT06/RxjdCrgtwcrzysbQUJbTwozg8s5wK8s7PXu/6rJuNUB3q1xrSSt4eoeATVVksCKGF/WkpgOpqaNc0wgV0Xp20oyllXzhi4tzBToBYGNMdgSQ7rOO8+gvQ7XCT2BbrYm3/01Tv2BogCansZurfxjtm2eNLweX8KFgtauscviHji8YrZ4hRaA== X-Forefront-Antispam-Report: CIP:69.169.224.1;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-1.smtp-out.eu-central-1.amazonses.com;PTR:b224-1.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(32142699015)(9140799003)(376014)(61400799027)(48200799018)(6133799003)(56012099006)(16102099003)(5113699003)(4113699003)(18002099003)(22082099003)(261009223027099003)(13003099007)(55112099003)(10067099003)(5063699009)(4143699003)(5023799004);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ZZU6Mz5X+T4Eo6xTOCQAVPBccKdPnONNpUDPf9Y53bzArDYynpFh0UygBx5Kf4F2xwjj8fOZjB4ejJbHB6K9zWkfz33/GoNqRGLcehN7HyMApEDqk+dpZ0eFoodST8OOfWSHiLk35ZX8vnocZ63g0d6AmCc+P/Saq7rc9/mRoTmpst4jfkKkNC/RRj1W0PfTkPOhqKamiT0JPv8hwSlDYRd2/ezaHru/NDFEcMSM7+y5LvDJWP6a2VPVO0yEElhVVQuHefytBi8JknPHlHhwQ0WpJjdbqUwKLz9q2x6niLBE9yoB9urh66z8QRqwKnkm7DIpAhB2jvT/n38yRTujGUl/OAGL0PJrDPAsVmaNLsUS39BCrjgHWtAqa+s+czcHUTPiAX0m19be+Q/PaEfrMUsNm1QWhlv6jP3bno1n5RZWgp10rCQzMAMu+f9Hm9rb X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Oct 2026 15:55:55.8829 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8923cb07-8123-4d99-70d3-08df2b9df6c2 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: DU2PEPF00028D0E.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8954 Thank you for your contribution! Sashiko AI review found 3 potential issue(= s) to consider: - [High] vfio/migration: stale cached device_state when CPR starts from a p= aused VM - [Medium] vfio/migration: CPR skips VFIO state using local-only checks, br= eaking old-source to new-target streams - [Medium] vfio/migration: cached device_state not synced after CPR when gu= est was paused -- commit 5d90a1a10c891a1bfb1b58a35b36f468c5d28350 Author: Andrey Drobyshev vfio/migration: don't migrate device state during CPR This patch skips VFIO device state save and load while a CPR migration (cpr-transfer or cpr-exec) is in progress. It gates the save, load and VM stop/start handlers on cpr_is_in_progress(). It also resets multifd_transfer in vfio_multifd_cleanup(). > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c [ ... ] > @@ -674,6 +674,16 @@ static void vfio_save_state(QEMUFile *f, void *opaqu= e) > Error *local_err =3D NULL; > int ret; > =20 > + /* > + * .is_active doesn't cover the non-iterable stage, so the section is > + * still sent: leave it empty, or new QEMU would load the config into > + * the preserved device and re-program its MSI-X. > + */ > + if (cpr_is_in_progress()) { > + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); > + return; > + } > + [Severity: High] Can the new QEMU end up with a stale cached device_state when CPR starts from an already paused VM? If the user stops the VM before CPR, vfio_vmstate_change(running=3Dfalse) runs while cpr_is_in_progress() is false. It moves the kernel device to STOP and caches STOP in migration->device_state. Once CPR starts, vfio_save_state() emits only END_OF_STATE, so no device_state reaches the target. The new QEMU then runs vfio_migration_init(), which does: migration->device_state =3D VFIO_DEVICE_STATE_RUNNING; It never queries the kernel. The preserved device is still in STOP. The target stays paused, because process_incoming_migration_bh() calls runstate_set(RUN_STATE_PAUSED) and then migration_incoming_state_destroy(). That clears the incoming mode, so cpr_is_in_progress() becomes false. A later 'cont' then reaches vfio_vmstate_change(running=3Dtrue), which requests RUNNING. In vfio_migration_set_state() this hits: if (new_state =3D=3D migration->device_state) { return 0; } No ioctl is issued, and the kernel device stays in STOP while the guest runs. Could the cached state be synchronized with the preserved device in this case? The commit message says the device "is never stopped". Is that accurate for a source that was paused before CPR? [ ... ] > @@ -810,6 +820,11 @@ static bool vfio_switchover_ack_needed(void *opaque) > { > VFIODevice *vbasedev =3D opaque; > =20 > + /* No device state is sent during CPR, so no ACK to wait for */ > + if (cpr_is_in_progress()) { > + return false; > + } > + > return vfio_precopy_supported(vbasedev); > } [ ... ] > +static bool vfio_is_active(void *opaque) > +{ > + return !cpr_is_in_progress(); > +} [Severity: Medium] Is it a problem that the decision to send or load VFIO state is made only from local runtime state? Nothing in the stream tells the peer whether VFIO sections are present. Consider a CPR incoming migration from a source that does not have this patch: vfio_is_active() returns false on the target, so loadvm_state_setup() skips vfio_load_setup() and migration->data_fd stays -1. The old source still sends VFIO_MIG_FLAG_DEV_DATA_STATE. vfio_load_state() has no CPR guard: case VFIO_MIG_FLAG_DEV_DATA_STATE: { uint64_t data_size =3D qemu_get_be64(f); if (data_size) { ret =3D vfio_load_buffer(f, vbasedev, data_size); Would that call qemu_file_get_to_fd(f, -1, size) and fail? The config section would also still be loaded into the live device. The reverse pair (new source, old target) looks similar. The old target runs vfio_load_setup() and puts the live device into RESUMING. Its vfio_switchover_ack_needed() has no CPR check, so it waits for an ACK that the new source never sends. docs/devel/migration/CPR.rst describes the incoming side as a new QEMU binary, so mixed-version CPR seems to be a documented use. Is a stream flag or a compat property needed here, or is mixed-version CPR intentionally unsupported? I found no later patch in the series that adds a CPR guard in vfio_load_state(). [ ... ] > @@ -882,6 +914,10 @@ static void vfio_vmstate_change(void *opaque, bool r= unning, RunState state) > Error *local_err =3D NULL; > int ret; > =20 > + if (cpr_is_in_progress()) { > + return; > + } > + > if (running) { > new_state =3D VFIO_DEVICE_STATE_RUNNING; > } else { [Severity: Medium] Does this early return leave the cached migration->device_state out of sync with the preserved device on the target? On the target, vm_start() runs while the incoming mode is still set, so this handler is skipped. For a guest paused before CPR, the kernel device is in STOP, but vfio_migration_init() has cached RUNNING. After the incoming migration completes, the later 'cont' calls vfio_vmstate_change(running=3Dtrue): vfio_vmstate_change() vfio_migration_set_state() if (new_state =3D=3D migration->device_state) return 0; The request is silently skipped, so the kernel device is never resumed. Should the cache be resynchronized, or the real state queried, when the device is inherited through CPR? --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261016-vfio= _v1-v3-0-47487cb5a8e5@virtuozzo.com?part=3D21