From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <010701a169d7e117-7ef74e61-6792-4cc2-93db-a5a3b1d7c6df-000000@eu-central-1.amazonses.com> Received: from DB3PR0202CU003.outbound.protection.outlook.com (mail-northeuropeazon11020124.outbound.protection.outlook.com [52.101.84.124]) by lore.virtuozzo.com (Postfix) with ESMTPS id 21B7D8017D for ; Fri, 23 Oct 2026 17:38:43 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AlagDkDVSz4HsJFEeljHwVnk17wkjRGMJMZAsGTYbc4oOOHZcNqyAxdLeKRo94tJ16vNpSjNMJg676sYdJFyuLl6veLJkPo0hFIcF0V7mynBE1jQgRFTtvIpjGUvVYJAEQ9EVN1gbu336Q6nexHUpemGXdoJWWMN3xd+4UHio6EDL1yU2PaP6aRl0cBI5jBcw/bMFaYYZi3TW2V+u0Swnj0yLnu5NnYlyaLWqb+wl8xbNoIzclQPWbu5WTRiLNUQwLyFVA0YSZX4Ox2unAHjhJk9wzY+8/UPFBne+5kTlab4Jv0LvXOhn3UPwWM9WrPahcsXpjRGhfPN7JELItdDnQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yeIg8g2BY+qjsH3hpoiYAgJrTsSS4u6dRLNDIyACcQU=; b=t6WHGXX97ImgYk13kjyJxm9DL41lkJamnDm7x/szUvNb+Hmiv9/aQ3ST2M5QofVwFOnVWfmSba8kI+pQRAel51Honkvvv3J4Dyxt51CHDjVueNoieFZ0Q0g+J3X0UiZF8+uPad0u7HDaY5Pkl+vMgvjV0TGFxx56onrRsDEu9bs6AsY6hRxzEr/pdHqNwwyMDij/uHVc9x30m3Nm5muMvab38BIpcM0HRCpxqOLRzYGRIF+Gmi3jLZ7TmolmwN58o30sghnR83DC5uD5xaSGPynr3Y/OMpvOybDnRgq63etWxQbm33pqY3cRy7AnXDRDhjqNeBAVB8EipZcWbpzfbQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.1) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yeIg8g2BY+qjsH3hpoiYAgJrTsSS4u6dRLNDIyACcQU=; b=ETsvZBp2RseabFcElxR7JovH3EcXoBnPIK+JKkc4fC3sDQNZuFe+eLWig0hPG1DrAa8KuiZ2R6qyYq2uk3VL7LDve3+VFd33Q67ZSmoutwaVrR5no1aw8LipRGacU3/00qukUw24P0wdSy6mZs/4Zk5GoDAlca/bALLG2Gvq+f8= Received: from DU2PR04CA0002.eurprd04.prod.outlook.com (2603:10a6:10:3b::7) by MRWPR08MB11825.eurprd08.prod.outlook.com (2603:10a6:501:97::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.18; Fri, 23 Oct 2026 17:38:38 +0000 Received: from DU2PEPF00028D11.eurprd03.prod.outlook.com (2603:10a6:10:3b:cafe::3a) by DU2PR04CA0002.outlook.office365.com (2603:10a6:10:3b::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.546.19 via Frontend Transport; Fri, 23 Oct 2026 17:38:38 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.1) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.1 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.1; helo=b224-1.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-1.smtp-out.eu-central-1.amazonses.com (69.169.224.1) by DU2PEPF00028D11.mail.protection.outlook.com (10.167.242.25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.546.15 via Frontend Transport; Fri, 23 Oct 2026 17:38:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1792777118; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=dNx/3Fb6fborhc2aKZRKPRjQIzLrIwT3AJDeP4I5pFU=; b=0np64JxsroV4c82TcQAZCVegXAIjoyhBQBTlYRYixGWbcdaTeXNyErhFDoUFb3if +KCuFZjeCOJtuOWxluFoIEH1z3TLCpgnQdAusy9KO37ThN5fUAHdAAx2HWhwA7YzF91 7TarerLg+z1vo8/BnG2JFtnD8mpaf+4m/WdFGKdSvwPRLqUdxpg2XZ9UEm1V+saQkJ7 XaV56yB1E5uGu2s2oK91gps3UqCokFMu+XloYqbkReet3dl6YnnvxZK4Yh077kHTADw elEIuCeoLFgLy5zarlLVpWig8CPzAMPw/vCIzZwFS4BYmAfkSgtK3zIO9VxkfxMWgbO +/0PHQXgPA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1792777118; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=dNx/3Fb6fborhc2aKZRKPRjQIzLrIwT3AJDeP4I5pFU=; b=mGmENi1fo1elaEdqhRjfGaWx5yZxZnlMyAIAWNnhevOgRVplVvMw/+UclZ9hQu81 Ya6/8KNnbDrAPOiea7OLQZk3qo9TLRunumHJ6i08MJLY5NYdfB5dgVSp9CQCJc7QrzI 1paqQL1j4+Ad5en7uMpQ8UOmFIxnJGj85JAx/mss= Message-ID: <010701a169d7e117-7ef74e61-6792-4cc2-93db-a5a3b1d7c6df-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v5 35/36] VZ: migration: keep the estimate and wait over starved windows #VSTOR-150595 To: Andrey Drobyshev , svt-core@virtuozzo.com In-Reply-To: <20261023-vfio_v1-v5-35-ba0bf3c6d122@virtuozzo.com> References: <20261023-vfio_v1-v5-0-ba0bf3c6d122@virtuozzo.com> <20261023-vfio_v1-v5-35-ba0bf3c6d122@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Fri, 23 Oct 2026 17:38:37 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.23-69.169.224.1 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DU2PEPF00028D11:EE_|MRWPR08MB11825:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 64638fbc-065c-402d-95f8-08df312c78d3 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|32142699015|61400799027|376014|30052699003|48200799018|9140799003|55112099003|4113699003|5113699003|10067099003|16102099003|22082099003|18002099003|5023799004|56012099006|5063699009|4143699003|261022113327099003|261022075727099003|261009223027099003|261021121427099003|261021123027099003|261022105027099003; X-Microsoft-Antispam-Message-Info: qpuen8Ygh6F0pBVMHNSKQ/KbeAcvD4r8iXoa1Fq0MOf4DVz0xftpiiJAu9v0pNqzjAfAn1tsay4Zx6paNO7UroBPJj62IKLV22Yt5ZQWv4/Ft1qFzsMcw4wdx1fL4QgQDtpEHwb849oWSLJvmA6pGTDXt2aLqpmY5BFmZ4KIBCe/FwbvaX5p9hFkPlpZzUrPmQkgqsQkCIlSVWOqcX16RuTRXboQZN87Sl6Po30gQMercek3Ii+IrBNXyTfrvntW3hnUiQqUB3q2nh2k+/6jkfCFKl0E3ttjhCTUfajlhx5SI51yiCRwT7+SxFIbRJpMT2hxnGVIvb0K8b3tFb6hQDAZIXXKX4/jhVEa/JgpxozNVYxf0Oe+M3xHmlYbq2AyERak/FOthmK7S/BYpUFeIX5KWA4wtokDGgS7EF5++HipXkxWO5KUCDFYOroBUQuy+rTRH9XH5wkmqH5ayQf1KtJZk0eUlcb2aSGhvrbtsibxrGdONGgtM8tH+Be6LCzl2ijbjvbiTOSJd6/QqeRjF0Ce+bTy0JRF9H9PksE0JChBtpKwAlc7za+nqqBA8rv4UshYYEf/K9Y4/iIcxtmPeAsehBwSf5pberBqxvF5S24BDTYOIeW/B8DsZDxtgfPHq4RfYeiaey1Q9rNHut8NOkXpIMNC0PL5wCuX3HpMsysn4misuP46IoVY22QWhSV2br5YOE2G+XbQs6ekEDv4/57jyeapog6STUbym36tjRR7/R3G1HPDRJsH7gwmTJNE9lsA+vdI/lOQX+MticNynWXY4CHGfpvrtTj657eOPg4= X-Forefront-Antispam-Report: CIP:69.169.224.1;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-1.smtp-out.eu-central-1.amazonses.com;PTR:b224-1.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(32142699015)(61400799027)(376014)(30052699003)(48200799018)(9140799003)(55112099003)(4113699003)(5113699003)(10067099003)(16102099003)(22082099003)(18002099003)(5023799004)(56012099006)(5063699009)(4143699003)(261022113327099003)(261022075727099003)(261009223027099003)(261021121427099003)(261021123027099003)(261022105027099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mSvxqW8kr261b4jitVf88a0MU+yyFboQCdXdpJ9/VGgOd2G3akB8xvXglPSFLQvRBQr7q8zh1/Z/QTADbYIwr3V+/P5tq4OCyX0dBrfHOhOSvVPlwtoozlJm78LsqkdlrUcalpBTn3h7ZoVAH+VzC/sczrvzYF2ClTj5I5jPiVbjJj/EkF5vP4V84dafW4miUlTcyvGb1FLtFycnordhm09axhba1LLyL00IcVWk2QFpcJxYQ51K8hzj4R3j0Zu3CcboClC+NKhOCTYZpRd3OW2kie+lrG/sIX2M8naxwstrxYJU1vEGoHTpHpwMg09xz0TcgleSX39k+cft3mm1Hb1h4d567f/7X/q4JEETUlgki0Izl/1QQsiXnvJrVdieA/sBI96jpqUc+jP8Zy/FqeazHtmjXqTgohAEg+8dLxqWB0a8GHdAz7D80qGKzn+A X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2026 17:38:38.4466 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 64638fbc-065c-402d-95f8-08df312c78d3 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: DU2PEPF00028D11.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: MRWPR08MB11825 Thank you for your contribution! Sashiko AI review found 2 potential issue(= s) to consider: - [Medium] migration: iteration_starved set from iterate() 'done' even afte= r a busy window - [Low] migration: starved-window wait in migration_rate_limit() delays com= pletion up to 100ms -- commit b0713b8ba87d723db3d095914dcfe10764c0314e Author: Andrey Drobyshev VZ: migration: keep the estimate and wait over starved windows #VSTOR-150595 This patch keeps the previous switchover bandwidth estimate over an update window in which every iterable device reported it had nothing more to send. It also makes migration_rate_limit() wait out such a window, the same way it does when the rate limit is exceeded. > diff --git a/migration/migration.c b/migration/migration.c > --- a/migration/migration.c > +++ b/migration/migration.c [ ... ] > @@ -3481,8 +3483,15 @@ static void migration_update_counters(MigrationSta= te *s, > */ > expected_bw_per_ms =3D switchover_bw / 1000; > } else { > - /* If the user doesn't specify bandwidth, we use the estimated */ > - expected_bw_per_ms =3D bandwidth; > + /* > + * If the user doesn't specify bandwidth, we use the estimated. > + * Keep it over a window the sender ran out of data in: such a > + * window only shows how little was left to send. > + */ > + if (!s->iteration_starved || !s->switchover_bw_per_ms) { > + s->switchover_bw_per_ms =3D bandwidth; > + } > + expected_bw_per_ms =3D s->switchover_bw_per_ms; > } [Severity: Low] The cached switchover_bw_per_ms is maintained only in this branch and is reset only in migrate_init(). If avail-switchover-bandwidth is set and later cleared during a migration, could the cached estimate be stale, or zero? In the zero case the code falls back to the current window bandwidth, so this looks minor. [ ... ] > @@ -3601,7 +3610,10 @@ static MigIterateState migration_iteration_run(Mig= rationState *s) > } > =20 > /* Just another iteration step */ > - qemu_savevm_state_iterate(s->to_dst_file, in_postcopy); > + if (qemu_savevm_state_iterate(s->to_dst_file, in_postcopy) > 0) { > + /* Every iterable device had nothing more to send */ > + s->iteration_starved =3D true; > + } > return MIG_ITERATE_RESUME; > } [Severity: Medium] Does a return value greater than 0 from qemu_savevm_state_iterate() really mean that the window was starved? qemu_savevm_state_iterate() returns all_finished, which stays true as long as no device returned 0: migration/savevm.c:qemu_savevm_state_iterate() { bool all_finished =3D true; ... } else if (!ret) { all_finished =3D false; } ... return all_finished; } That value says nothing about how much data was sent in the call or in the window. ram_save_iterate() can send many pages in its loop and then hit: migration/ram.c:ram_save_iterate() { ... pages =3D ram_find_and_save_block(rs); /* no more pages to sent */ if (pages =3D=3D 0) { done =3D 1; break; } ... } So it can return 1 right after a large send. The result is also 1 when every device was skipped as inactive or non-postcopy. iteration_starved is cleared only in update_iteration_initial_status(), when the window rolls over, so later productive iterate calls in the same window do not clear it. Then in migration_update_counters(): if (!s->iteration_starved || !s->switchover_bw_per_ms) { s->switchover_bw_per_ms =3D bandwidth; } would skip the update for a window that carried, say, 80% of line rate and then ran dry. Is the comment "such a window only shows how little was left to send" (and the corresponding statement in the commit message) accurate for such a mostly busy window? Could the flag be set only when the window actually transferred little data? > @@ -3719,7 +3731,8 @@ bool migration_rate_limit(void) > =20 > bool urgent =3D false; > migration_update_counters(s, now); > - if (migration_rate_exceeded(s->to_dst_file)) { > + /* Also wait out the window if the sender ran out of data */ > + if (migration_rate_exceeded(s->to_dst_file) || s->iteration_starved)= { [Severity: Low] Can this wait delay completion by up to BUFFER_DELAY (100 ms)? The call chain is: migration_thread() migration_iteration_run() qemu_savevm_state_iterate() /* returns 1, sets iteration_starved */ migration_rate_limit() qemu_sem_timedwait(&s->rate_limit_sem, ms) The next migration_iteration_run(), where complete_ready is evaluated, is then postponed until iteration_start_time + BUFFER_DELAY unless an urgent request wakes the semaphore. Examples are an idle guest whose pending data was just above the threshold and was fully drained by this iterate, or the case where can_switchover is false because the destination has not yet sent the switchover ack. Before the patch the loop re-evaluated completion immediately. The commit message says "Migrations which converge today don't see such a window before the switchover, so they're unaffected." Is that strictly true? The same wait also applies in postcopy, where complete_ready is !pending_size. It also applies when avail-switchover-bandwidth is set, where the estimate is unused and the wait seems to serve no purpose. Should the starved condition be limited to the case where the estimate is in use? The later patch "VZ: migration: don't wait for stop-copy-only device state" does not change iteration_starved, migration_rate_limit() or this condition, so the behavior remains at the end of the series. [ ... ] --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261023-vfio= _v1-v5-0-ba0bf3c6d122@virtuozzo.com?part=3D35