From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from GVXPR05CU001.outbound.protection.outlook.com (mail-swedencentralazon11023100.outbound.protection.outlook.com [52.101.83.100]) by lore.virtuozzo.com (Postfix) with ESMTPS id 163D980474 for ; Fri, 23 Oct 2026 17:40:23 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zFdihc/2KlXqskPTXe9ZrkVUIZW2vuOgz1rq3KQXjvLsQO3yDTTmSib121TKSOYP1ENulRBJlF1QYLuWb6GW/CStfpPTmhr9Eus6IVYNq7A+8amrGWhBKYwS0dAKusogCFHpSvSTTJlPivOkigjRe3/zopr5AuPDcbXHFA9Nt5FzLUXxthT36UuwnaWwDIsiKR1yfMR23zRZdheEqx7cPlHbBXmiBNODVzjzQkFBo6b3H5iiiHUSbCAXsAfEhRpDoyGSvqoMbZJH+iFhRaxiRrDULmpcowOJ+j4uJBnCUQWjf2I8TsD2NeyW1zo/v7HZOg+L5DWlbT6kJzzxGROxEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JGVhlpqzMBsoInT7EDmlGkHqjBRJin61UVB+xJX3AsA=; b=hG9odO1TRwMHHeFd9XlNSQL9N9RPsOTnvhAxVNLmhdcyiOIS6Iz7p7yjxhRqJJXvVV8j3cYikHS62dA4Bt/OY7hP3HDoUtkVDRc2Z5eBVnZCxk55ZD2aP/IRbf6uQIT+puz31otICwUF+3ESobziW26XQ+p1rqYieM1ZMRB3NPWEfDsQ4+3IaDVDS2rHpWVG5sgOiwS/NkOktbyK4V25T5A4uwEbiVCzwndklTPX9rSY7V6c/NP7U1jpgeiTyUrl7rJIOxX6dAhiDHOnb9bCYC8OnVd1J67PxzXlVbvF3H9dtM7LezQgLj7RFrXqTyjvlJWwhzFsHjTSezziotJXzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.6) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JGVhlpqzMBsoInT7EDmlGkHqjBRJin61UVB+xJX3AsA=; b=CXcZ2gfuh034f8q5HEC13ZQWjlWZA0XZj0Pe+yWvy97uo0zYh23zwCnNKqo7kDy63sEa2EeFzdni4krGAu0KpAn2AeknW/86J1Dwq/HXOlzMJikhxLwZoZnkHH7Jx/f0RrtEXQomsmV8BLTxnTO9Br0aEtNJPaoTyeZG4PMgtlg= Received: from VIUP296CA0091.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:35c::7) by AMDPR08MB11365.eurprd08.prod.outlook.com (2603:10a6:20b:71a::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.18; Fri, 23 Oct 2026 17:40:19 +0000 Received: from MI3PEPF0000847E.eurprd02.prod.outlook.com (2603:10a6:800:35c:cafe::d) by VIUP296CA0091.outlook.office365.com (2603:10a6:800:35c::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.546.19 via Frontend Transport; Fri, 23 Oct 2026 17:40:19 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.6) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.6 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.6; helo=b224-6.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-6.smtp-out.eu-central-1.amazonses.com (69.169.224.6) by MI3PEPF0000847E.mail.protection.outlook.com (10.167.240.165) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.569.4 via Frontend Transport; Fri, 23 Oct 2026 17:40:19 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1792777218; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=CiDWUmiECQKk1YSLl/aL7KnxhE0l69i/g4lisDrQVHQ=; b=UPEQXv+VQvm9lgomnUnBuXqoVzAbTmGGeLvE5v8+IFP5veUOBzpxofHA63WJNLwR VV5R7+HmC+4FG6yIhnuBpiSIat7SvHxnh4QOzIJ5juUPlvDqX5OAYPGXLJKSHUZj2Ji 15NvYC9zOzwIhWLtwVPkErARg7cETRxHq7XQh53Z0bbSz8Fg5HEDerP8iQjdglExFAy +WyV0dTuiuEwa5M/EXxoOSgpYu4Q1PCSC5/zEtpB5gox50lHRKNxDhzCP4it8pAM7ZH ays8unQNJurwkIZx/Qoi5fTRf9XXSA06H0uDNp/pc+MLLQIs9LaIUxcNHDGlIjpTdnV RIR50HKAyg== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1792777218; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=CiDWUmiECQKk1YSLl/aL7KnxhE0l69i/g4lisDrQVHQ=; b=DkPw7JCEjo+3bO81sDuQXGRnUqdvVSWEb/nvp4n539AXR+k5t2tgBs3j2on1cmUz RyO5eaiSBQ3ozOgG4c+EiwAaeMEeSsGQ33dtC7R67WndfO8R9Z3MzpUuhIfBsr4DC74 TPYY3nyUz3cIEj2QdtkGMH8KOuqiLfKjjkVuPTvw= Message-ID: <010701a169d96a76-8ca71f65-df46-4590-9bd3-7d5de3967465-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v5 36/36] VZ: migration: don't wait for stop-copy-only device state #VSTOR-150595 To: Andrey Drobyshev , svt-core@virtuozzo.com In-Reply-To: <20261023-vfio_v1-v5-36-ba0bf3c6d122@virtuozzo.com> References: <20261023-vfio_v1-v5-0-ba0bf3c6d122@virtuozzo.com> <20261023-vfio_v1-v5-36-ba0bf3c6d122@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Fri, 23 Oct 2026 17:40:18 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.23-69.169.224.6 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MI3PEPF0000847E:EE_|AMDPR08MB11365:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 4c7cc047-db3e-4e0b-d05d-08df312cb4d1 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|9140799003|32142699015|61400799027|376014|48200799018|22082099003|18002099003|4113699003|5113699003|55112099003|16102099003|3023799007|13003099007|5023799004|5063699009|56012099006|4143699003|261022075727099003|261009223027099003|261022105027099003|261021121427099003|10067099003; X-Microsoft-Antispam-Message-Info: s29ClbWxvvdJdx096iFtShzUT15LzigjhJhsYeWYfISI6grj5G9xTO4dwFTzJeScbRbj661fKqY82snauENSpsiFHdTlVE0XzAgUyGcyVqSKo4NswK6UJHZwojjeQXoG7Ojw06rGkBC76FJvZMlW3hnFJXBFiPpChEHHfWhkMXbHsqpj1uShHrJHNXL7QUaRT3Opyl6qbkosSnqNrtfQX2Ycs5GS7jL03Eapy4bfLdLLf1uZxGthID0eDkScv6opTbme+ZkFidRUdQOUBBPA/cQyAEVMLP9uBJtcOytsT8tZqPqEfBqfePG03s5lNDxlrQOp30VgOhZmJYyce1KLLYh4TyFQ1RXIXqBr5S4C2Rbx4yfEJyb057bi8KrgB7a/BIzXjd/7wnLVhO5ui0Kj2ZCXlc0bFm/nQH6cIkUxpN1yDdcTRJCfrfMmFHMdrKT3JciU3xX3kCMqjyXI5fRnGRiiC0XM3C79h5jibgI/j9mPR0CvZQ/lsf9UUmzWCc+zAwNyggyuSmTa0BWYm965gN2/060znYVSMmiy/0VyIOQ/0m9zx6h6ZdZi0Pcl+KQwJGIAygiD67TSa+flBJJwA0zHpdRPpyh+ql+5itZ8v2w6/xFko5bk96QkcewXELPPdM0wq9e2dpP/lp181TAKrhH9gsKsRiQDAscMve2vCssMzj1JPeFIs6JyPSwxR81Su8+zhXIeYFKKvfd5dgXNQull6aOufYGApEr/eaxRI+2wiWqrUso0Pl2WOYOnbKR4AbTCqdOiR0ssCjKM+pWZvgf45vb79SYEQHFVy2Gg6g3OwIyhA8NuzQiyouqKPeNKt5++DKosJuq+YRZN4LopIRQ2ukgjLpOXGjsCy/gBtxx11ruDsSCW9KOOMRlUkAWo X-Forefront-Antispam-Report: CIP:69.169.224.6;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-6.smtp-out.eu-central-1.amazonses.com;PTR:b224-6.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(9140799003)(32142699015)(61400799027)(376014)(48200799018)(22082099003)(18002099003)(4113699003)(5113699003)(55112099003)(16102099003)(3023799007)(13003099007)(5023799004)(5063699009)(56012099006)(4143699003)(261022075727099003)(261009223027099003)(261022105027099003)(261021121427099003)(10067099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2dJjZdlEWWI/DobNye8u8oVUtqsUMx9qoKd73z0HxUwrgge/gzshNTyktuMSkPaNG2Qv2MV28ZCraWPRECIsdDvZ82FcAG7WrhJomvBmzsCjf09nbPpsFEDLTYUHcDN/E1m2mdgcuA7Dj130Sl42HkodlOdex3/SoiS+/ual7NKRJWnmMueYom28QDW0zDhUcv+KwYTvPZ8mjpQLwofIlu5N5aHczK2yokIiuai+pBPRPxq35/k//KQ0zLnsdqPIB5lCb+76qgodVavb8B8dKQXXTAMLZ9T637Ao+npLn9sukBEMDPdTPYtbugyIjHg5u5Q3ELjtuSD98bkHcUw4ncajFDJ/Tb3tlP09xcTBaN9hlOjMBySw6MS0PcgYLafp4320UHHN1LI2IA+yzYvvmIcMqsxCz+jvOPNifdwEvy6BZYgaYtGcm8LxO7nqlmtO X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2026 17:40:19.1165 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4c7cc047-db3e-4e0b-d05d-08df312cb4d1 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: MI3PEPF0000847E.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: AMDPR08MB11365 Thank you for your contribution! Sashiko AI review found 5 potential issue(= s) to consider: - [Medium] vfio: vfio_state_pending_exact() ignores vfio_query_precopy_size= () failure - [Medium] migration: irreducible-state switchover bound exceeds downtime-l= imit and commit message - [Medium] migration: irreducible_pending only learned after RAM estimate d= rops below plain threshold - [Medium] migration: irreducible-data switchover overrides a pending migra= te-start-postcopy request - [Low] migration: irreducible size passed via file-static global in savevm= .c -- commit f62ee5bb362c0d23b60102c51ee61dedc887ca84 Author: Andrey Drobyshev VZ: migration: don't wait for stop-copy-only device state #VSTOR-150595 This patch lets state_pending_exact handlers report pending data that only moves once the guest is stopped (the VFIO stop-copy size). When that data alone exceeds the threshold, migration_iteration_run() switches over once the rest is within the threshold plus as much again. > diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c > --- a/hw/vfio/migration.c > +++ b/hw/vfio/migration.c > @@ -569,18 +569,25 @@ static void vfio_state_pending_exact(void *opaque, = uint64_t *must_precopy, > VFIODevice *vbasedev =3D opaque; > VFIOMigration *migration =3D vbasedev->migration; > uint64_t stop_copy_size =3D VFIO_MIG_STOP_COPY_SIZE; > + int ret; > =20 > /* > * If getting pending migration size fails, VFIO_MIG_STOP_COPY_SIZE = is > * reported so downtime limit won't be violated. > */ > - vfio_query_stop_copy_size(vbasedev, &stop_copy_size); > + ret =3D vfio_query_stop_copy_size(vbasedev, &stop_copy_size); > *must_precopy +=3D stop_copy_size; > =20 > if (vfio_device_state_is_precopy(vbasedev)) { > vfio_query_precopy_size(migration); > } > =20 > + /* Once there's nothing left to pre-copy, iterating can't shrink it = */ > + if (!ret && !migration->precopy_init_size && > + !migration->precopy_dirty_size) { > + qemu_savevm_state_pending_add_irreducible(stop_copy_size); > + } [Severity: Medium] Can a failed precopy query be treated as "nothing left to pre-copy" here? The return value of vfio_query_precopy_size() is dropped. That function zeroes both sizes before the ioctl and returns -errno on failure: migration->precopy_init_size =3D 0; migration->precopy_dirty_size =3D 0; if (ioctl(migration->data_fd, VFIO_MIG_GET_PRECOPY_INFO, &precopy)) { return -errno; } So if VFIO_MIG_GET_PRECOPY_INFO fails while the device is in PRE_COPY or PRE_COPY_P2P, the new test "!ret && !migration->precopy_init_size && !migration->precopy_dirty_size" still passes, since ret only comes from vfio_query_stop_copy_size(). The whole stop-copy size is then reported through qemu_savevm_state_pending_add_irreducible(), even though pre-copy data may remain that iterating would drain. The existing code fails closed when the stop-copy size query fails, but this path fails open. Should the return value of vfio_query_precopy_size() be captured, and irreducible data reported only when it succeeded (or when the device is not in a pre-copy state)? The same logic also seems to treat a transient dirty_bytes =3D=3D 0 on a precopy-capable device as irreducible. > diff --git a/migration/migration.c b/migration/migration.c > --- a/migration/migration.c > +++ b/migration/migration.c [ ... ] > @@ -3576,8 +3577,10 @@ static MigIterateState migration_iteration_run(Mig= rationState *s) > * postcopy started, so ESTIMATE should always match with EXACT > * during postcopy phase. > */ > - if (pending_size < s->threshold_size) { > - qemu_savevm_state_pending_exact(&must_precopy, &can_postcopy= ); > + /* Allow for what the estimate lacks, see the irreducible rule b= elow */ > + if (pending_size < s->threshold_size + 2 * s->irreducible_pendin= g) { > + s->irreducible_pending =3D > + qemu_savevm_state_pending_exact(&must_precopy, &can_post= copy); [Severity: Medium] Is irreducible_pending only learned after the RAM estimate has dropped below the plain threshold_size? irreducible_pending is set to 0 in migrate_init(), so the gate above is the old "pending_size < s->threshold_size" until an exact query has run. The pending_size here comes from qemu_savevm_state_pending_estimate(). vfio_state_pending_estimate() returns early when the device is not in a precopy state, and otherwise reports only: *must_precopy +=3D migration->precopy_init_size + migration->precopy_dirty_size; It never includes the stop-copy size. So the estimate is roughly the RAM dirty estimate from ram_state_pending_estimate(): uint64_t remaining_size =3D rs->migration_dirty_pages * TARGET_PAGE_SIZ= E; If RAM dirtying keeps that at or above threshold_size, the exact query is never made and irreducible_pending stays 0. The relaxed completion check is guarded by "s->irreducible_pending > s->threshold_size", so it is never evaluated. Would a guest whose RAM remainder sits between threshold and threshold plus the irreducible size, which is the case the commit message targets, then keep iterating as before until the estimate happens to dip below the threshold? > pending_size =3D must_precopy + can_postcopy; > trace_migrate_pending_exact(pending_size, must_precopy, > can_postcopy); [ ... ] > /* Should we switch to postcopy now? */ > if (must_precopy <=3D s->threshold_size && > can_switchover && qatomic_read(&s->start_postcopy)) { [Severity: Medium] Can the new override below discard a pending migrate-start-postcopy request? must_precopy includes the VFIO stop-copy size, so when that size exceeds threshold_size, the postcopy gate above is never satisfied and postcopy_start() is skipped. > @@ -3601,6 +3604,16 @@ static MigIterateState migration_iteration_run(Mig= rationState *s) > * (which was calculated from expected downtime) > */ > complete_ready =3D can_switchover && (pending_size <=3D s->thres= hold_size); > + > + /* > + * Data which only moves once the guest is stopped can't shrink = by > + * iterating: if it alone exceeds the threshold, switch over once > + * the rest is within the threshold plus as much again. > + */ > + if (s->irreducible_pending > s->threshold_size) { > + complete_ready =3D can_switchover && > + pending_size <=3D s->threshold_size + 2 * s->irreducible= _pending; > + } Before the patch, pending_size <=3D threshold_size implied must_precopy <=3D threshold_size, so a requested postcopy always took precedence over precopy completion. The new override does not look at s->start_postcopy. Once it becomes true, migration_completion() runs with s->state =3D=3D ACTIVE and t= akes the migration_completion_precopy() branch, stopping the VM and sending the whole device state during downtime. Should this override be skipped when start_postcopy is set (or postcopy is enabled)? [Severity: Medium] Does this bound match the commit message and the comment? The commit message says "switch over once the rest is within the threshold plus as much again", and the comment says the same. But pending_size already includes the irreducible bytes, since vfio_state_pending_exact() does "*must_precopy +=3D stop_copy_size" and then also calls qemu_savevm_state_pending_add_irreducible(stop_copy_size). So the condition pending_size <=3D s->threshold_size + 2 * s->irreducible_pending is equivalent to "rest <=3D threshold + irreducible", not roughly twice the threshold. For a ~2 GiB vGPU, around 2 GiB plus the threshold of RAM and other state may still be pending at switchover, so about 6 GiB in total. This replaces the bandwidth x downtime-limit criterion from migration_update_counters(): s->threshold_size =3D expected_bw_per_ms * migrate_downtime_limit(); with one that scales with the device-reported size. s->expected_downtime is not updated, there is no capability, parameter or compat property, and the downtime-limit documentation does not mention it. Is that intended, and can the wording be made to match the actual bound? > diff --git a/migration/savevm.c b/migration/savevm.c > --- a/migration/savevm.c > +++ b/migration/savevm.c [ ... ] > +/* Exact pending data which only moves once the guest is stopped */ > +static uint64_t savevm_pending_irreducible; > + > +void qemu_savevm_state_pending_add_irreducible(uint64_t size) > +{ > + savevm_pending_irreducible +=3D size; > +} [Severity: Low] This isn't a bug, but the irreducible amount is passed from device handlers to the migration core through a file-static global. It is reset only inside qemu_savevm_state_pending_exact(). Nothing checks that an exact query is in progress, so a call from any other context would be wiped by the next query or leak into it. The state_pending_exact documentation in include/migration/register.h was not updated to describe the new contract either, and the global is not tied to MigrationState. Would it be cleaner to extend the handler interface, or at least document that the function may only be called from within a state_pending_exact handler? [ ... ] --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261023-vfio= _v1-v5-0-ba0bf3c6d122@virtuozzo.com?part=3D36