From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from GVXPR05CU001.outbound.protection.outlook.com (mail-swedencentralazon11023118.outbound.protection.outlook.com [52.101.83.118]) by lore.virtuozzo.com (Postfix) with ESMTPS id 63E3F8017D for ; Mon, 28 Sep 2026 17:06:09 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ej+KWge4BxZfT2vAX2bJbsMYbpZ7gyJilu2KF0UUie3jvydxGOrPXOH62ZjyHABVCdK215ALy7SVj8npFFihEssshB8/vq0T6iHIHRdOSo7MBHBBrTUM5HN/2TXy/wf98NL6P7Vao88f0Fg2AFxoIEKJlIhu8qFCDUyQGgTd2S4+pkK0eA8Em3VMTHu8Mol9gu5N5d51dFyzfUIuTnyPLvJV93SC63/FBWk9k1iwoP5lWn37+8JEXyEohYvicr4oqIMNsDl7OO2Ir74JPK24bbBrieMzroHabrVBBadFe9efyuUzl1Ug9kWd9ekDv9+0FVZBdk7eISqHerxyUvB9DA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=H6kQ4kJbdzXax4WCeszEy5Xp+D82RLXgatknneJWdhg=; b=jaWijbCizKWzT6mgbsG0ySvJNueEVMezDVwqd8mEzmzhYBMjGdyvjx83ykSwZyfxEtkwvEY7kIrAREz9nQiX5zJJ4lFjLJfjC8Uy8nBrwrt4nDQprJT/fQ4dHV+7Ibo8QiBPy94oSaZwRyCGp00x0BERaANW+ON2qH4eVHl7QXs/KOaN2a2zLtPvGqB4ZUlovOHP+IgG8Tw7ih2T9OngU+0sPBP5XrZO7hrJaRuvNj0dWmua3o8XdwcVQvBhfdX+ydjsD2LFlmvmueRD19UsoPXYEZ7g9fA6Hc70hjfxGzwNBbQ0X00vZf4oohLJtHqeX5/Ah4OwZSBtxvmky7pSIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H6kQ4kJbdzXax4WCeszEy5Xp+D82RLXgatknneJWdhg=; b=BSnmeq9LOKAcJK86WTnn11zKd/JdDRNa3nJbFPEkfshNfwcuEaw4b+547/0to5mmrfi4ingNJFJd1bmzXsrzDdfQJLhyjZxPpA2CDRTVjCxCzFGggtrxwfS7CDBd1fFPx6/3hmC2OQGdohoCEmokR/LRdOFMKvqFXi69AuETk4l0fXP1qWuaY0MUHHXJmCPjKpTNUDlyV9YkQ4DKy2OAhzyPDP+K+yNfTwSDYJ7Ya8dIFke0NcCC1Mxn/+sSRpF1g3jqT69HsWRkSTGSeAFaz3frXWAeGcdya0XXvvii07TxGAqfesErzriw/6Fb3tuZl9RgnTy4Z6QTLWtkpjuU0g== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from AM9PR08MB5892.eurprd08.prod.outlook.com (2603:10a6:20b:2dd::16) by DB9PR08MB7891.eurprd08.prod.outlook.com (2603:10a6:10:39e::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.472.7; Mon, 28 Sep 2026 17:06:06 +0000 Received: from AM9PR08MB5892.eurprd08.prod.outlook.com ([fe80::94bb:633f:1f55:4bbd]) by AM9PR08MB5892.eurprd08.prod.outlook.com ([fe80::94bb:633f:1f55:4bbd%6]) with mapi id 15.21.0451.014; Mon, 28 Sep 2026 17:06:06 +0000 Message-ID: Date: Mon, 28 Sep 2026 19:06:05 +0200 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH QEMU HCI-8.0 v5 00/13] usb-host: support CPR migration To: Andrey Drobyshev , svt-core@virtuozzo.com Cc: den@openvz.org References: <20260925-usb-host_v3-v5-0-e5b21724db4d@virtuozzo.com> Content-Language: en-US From: "Denis V. Lunev" In-Reply-To: <20260925-usb-host_v3-v5-0-e5b21724db4d@virtuozzo.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: VIVP296CA0125.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:358::15) To AM9PR08MB5892.eurprd08.prod.outlook.com (2603:10a6:20b:2dd::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM9PR08MB5892:EE_|DB9PR08MB7891:EE_ X-MS-Office365-Filtering-Correlation-Id: 6693d322-68df-4978-c428-08df1d82c8a0 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|376014|10070799003|23010399003|6133799003|5113699003|22082099003|18002099003|56012099006|10067099003; X-Microsoft-Antispam-Message-Info: Ts5PM2IRzv7gCNjGBCfJepZE/7mGrhMEpVdySsOEe2BLud0lCNkwt9ybX3LZxxjpJpQK1J7dEYz2LlkJqhb04Cp03RqWvgy1cK8lnn8dRTutbvNlKfO95JAGDQEx0Efpgn6dJgNsGZM/vK04AK85a2603RpQXmc/ChbBA63C3CJQfO65+eotO+mFnGNPMNsSyFHOTVxH98j2ZfiHtoFsI6BtVY8mOfX2Z2U6zL8ziIxU9NxA9Vf0h0ILkcs1K48L2M8fWm6xJlbfwWHfU58mrfWzCsq2bQqlqW+2sWgul3cZN8l9rErJEkS3dsDmZoyI+IvsAJzX/gEfRzGv3cf1opHefhLeGFIr+hOKfQ/RxujxF7hfP0fqqVu0jLdwEm9SZiimXeABtuYryPjByeH03prGn3oPicMLDRhAIv2knJu08UcIsjwyaH2UOkJhkzYiRvR7+mxFxDbwiRwZ7twv0jN35GXW9hlo/P8MFoKIhPtVsgI+m4xHn2KtvsbN2AB93dRW1c27DlD2RRo+eIbtoIDytY5kRG+r6e9Uv98Dl1yudY1z1xzDwV6ieD0ZgBRo6XRqBBq37kbkPjoRnqXHicJtA9/4Lgmh8XDgAQq98+t//dXy8udtuGz0YqItAqCacl6IfELNzu9y7M/8CFoyxH2oLBi7cdaIX9RfXPflWkk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AM9PR08MB5892.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(376014)(10070799003)(23010399003)(6133799003)(5113699003)(22082099003)(18002099003)(56012099006)(10067099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bDdzRktCUSswWDJZb1NLMlRCMGloOVptZ09LR3VqY0luMEp3c2hEdkFRalNS?= =?utf-8?B?eVBjd1NwVGxBZkJ1dEFXTjBPOVMvM1VxWDFiTnBpZXFrdG5wYWRUV3p1OWoy?= =?utf-8?B?UVlaeTZJaytnUVFjd0tqV3RQTVkrUVVGbDVJeWg5Q1RVY3IyNmZ6SlBMR2hJ?= =?utf-8?B?NU95RzdtTmFrbi82WjVFS25vSzA2allITmVwT3FhbnUvVUIxeWdXSThWR2hZ?= =?utf-8?B?eXpmSXJDOExublFrUEZEdDhXdzFNcXMydFZ2TFRtVnZnT013ZWtKM2tiNVlK?= =?utf-8?B?ZHRMaVJqMVdFOEZheW1rZHQxV0t1UFNMVUVlUUNjVXMvTkxQcXRKTnJwdFhl?= =?utf-8?B?VWFZNm9sSEpET0Z3eU53aWVyd3hmWG9JKzRxb0dVSnRZZjR1Rk9xRENhcDRD?= =?utf-8?B?QXNCWFE5RkVnZDNlUTNVUmZJNXM1dE1KRUlVRGRpZ084L1N3bk42bmwyN205?= =?utf-8?B?OUNrS2I4K29PM2NvZkZ5cGpWa05TSzNNcTg0ZGhhK0lRRjdCRHNrQ05kU0l0?= =?utf-8?B?SU1yR1NUck8xTEQvTUJLWmY5dXViczZJRm1vY3ZkU2NyNFRCYXEwcEZEbk1N?= =?utf-8?B?clQ2NStmS2lUU1g0Q1RRVVhjQktldlF0UWtoamovUGNrUUtnMW9HYjlBbUlI?= =?utf-8?B?dGJLNG5FYmZuMFBML1VESWM1VGZMRnRTcWduRzdsRGZIQW5TYzF0QUlraDdi?= =?utf-8?B?WTNVZ2RadklBQm5hK1JzajJ2Tm8zaGJvWlkrazZmS3hWNlVkMDJjYlVkWm10?= =?utf-8?B?VFRLMkcvb3VyYjhnb3IyTkhmRVl1TFdEa3ZLK2prbUVvdDRUbWIzdWxvM3l6?= =?utf-8?B?QUZ4WUh2NCtiK2JVdng2cHMxZXNIZWdFZFJzV0Y2Sk1yVVl3UEpmZjlpeGQ4?= =?utf-8?B?eFBSZUhMdkRVRnN2Tk1VTG1UN2lsV1BDTDdQOFNYYlhMb1RxV0xKVm14akE4?= =?utf-8?B?UUhpWm5yZ0tDR2EwUHJ1NTN0MFhHRktIenpiRDBnOWpZWUF0WnYrRS9RSk5m?= =?utf-8?B?M3pySzNNeHV3RjMzQlNDdS9vK3M0ZEZqU3BuSTFGL2J1dVdYcDE4ckZwdWh5?= =?utf-8?B?V1NZcDJVdXpXRGNvM0hmZW1jK2FqSnl0OXhXNHI5QTdYa29oeTFrczI1YWxx?= =?utf-8?B?SG1xcEtUdTkrYUZiQ0E1eVNrWkFLOWdxMk9XVjB0MTVlc211Yk5halB2NWNW?= =?utf-8?B?cEJldzBJTklMbG5vSXF5andhM2JwZ2plQlUyRy9TNk5EL3BPREd1VVo2VzdR?= =?utf-8?B?OTVsdVdyYklrLzZaUkxYMVY0YWhyd2dGUElEZDg2NFcvWi9CdlBlOG1YZVk1?= =?utf-8?B?K3FNeUo3RERlblZtemVNK3BmdlUwR2ZSdCtlMXpkaVM4RnY3aTNlUnI5emNQ?= =?utf-8?B?ZDZManpObnRMdk85OElmNFBrYTJLblZFZFhCWGY1emI5eGZ6TEJRdWFlZnhG?= =?utf-8?B?TGRGV2pudmdvNlZETzFVRlFvbVVXVGdFcVZYVHpJcHVRWUdCNEc1Q0tGcU9m?= =?utf-8?B?RmNLUTNGVmZHVkl5WXozZ1ZWSUVzNC9WMjI3dW5Sbyt0NnkxbU5XcWszUW01?= =?utf-8?B?a2tndmpvYVJ2WC9PMFB2d3lVN25EL2VKQTJEOVBzK214OHpIclN3Mmd4QTZm?= =?utf-8?B?SUZVeVZla1VPQU16VHRteVAxVWR1T1BkODhoNFFGSlFqUGx1NDhPMlhCTGFX?= =?utf-8?B?YjR1dVJ0QWVsU2dkTXBrTVhaQ1QvdnNsYS9SeUtaRkdITjRiSTRWeXgvOHpn?= =?utf-8?B?TFh5SGh1elZRTGJFZHR4VCtQM1h2RFZ2ZitpT3c4R3ZVTG1OazJhZ2hDZUNZ?= =?utf-8?B?ang5RHFIeC9iV3FsdTFPS1Y5bTJPaUF6Umo0NjNsdS9FM0lqNjNETGcvMjJY?= =?utf-8?B?YVVzY1NSUktVMlU1VWlvUkx5YW9VUVY3OWxPWlYvcWtzZ3pkK3VuV3B2T2tE?= =?utf-8?B?cVBMeTJUc2FZRHAxbVphK0NyWERZNHMxN3ZyVUV3KzdVTmhmK1VITXFYMnF1?= =?utf-8?B?NnQ3bllPOHN1c2U0NXI3TTJzNFJSNU5tNkRHdHJFcFVIb2FBYk5CbDNvRERt?= =?utf-8?B?bFRZcVlleTZ4NldsYUltS2hqTnV6RHJEck85bHRXRjBZRjJSckNOZmU4SXBD?= =?utf-8?B?anJvSlJ5QVpvYWR0R2ptZGR4aUhrRjF3UkxUOC84SFE5MzJpWHY0L1pOT09y?= =?utf-8?B?ZjVadVh6ZjFOYk9mVWM4bDE2aGhHL0RSZE1CRmtrQWp4OWEreXYzSklDZzNP?= =?utf-8?B?bHJvcjVON0puUnRsRzlhVGFNbmwvRy9HY01DdnFOKzZZU3hBTnI3MkpYUjc1?= =?utf-8?B?UE9NVDdocVNCZzBsVlZEckwwS043a1BCODIvaTBvY2FDTEJpMlFNZGorbVIz?= =?utf-8?Q?Vw0g2xldIwAiSemznqL/wau7dCaZ7C9ZEc94xfvNdqCDM?= X-MS-Exchange-AntiSpam-MessageData-1: Y7CVPVR5veq9bA== X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6693d322-68df-4978-c428-08df1d82c8a0 X-MS-Exchange-CrossTenant-AuthSource: AM9PR08MB5892.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2026 17:06:06.0905 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: S//ES43BRQA7KJRD801gfb147Q07PeE5xA4wX1+bygA76IjsGD5yP9F89lhcrQAa8ZK5SV92NqE5Kwx0gPpcnA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB7891 On 9/25/26 16:11, Andrey Drobyshev wrote: > v4 -> v5: > > * Patch 5: deliver URBs that already completed in the kernel instead > of dropping their data, and error out any a drain can't reap in > time; > > * Patch 6 (NEW): drain each usb-host before its host controller is > migrated, so a shared-context completion can't advance xHCI after > it was saved; > > * Patch 7: remember the packets to replay from the reap callback, so > only the genuinely in-flight ones are re-issued. > > v4: http://lore.virtuozzo.com/qemu/20260924-usb-host_v3-v4-0-1cc48bf630e7@virtuozzo.com > > Signed-off-by: Andrey Drobyshev > --- > Andrey Drobyshev (13): > usb-host: don't leak hostdev FD on open failure #VSTOR-137800 > usb-host: add migration blocker for CPR modes #VSTOR-137800 > usb-host: preserve hostdev FD during CPR migration #VSTOR-137800 > usb-host: factor out usb_host_reap_xfers() #VSTOR-137800 > usb-host: drain in-flight URBs across CPR #VSTOR-137800 > usb-host: drain before the host controller is migrated #VSTOR-137800 > usb-host: re-issue drained URBs if CPR is aborted #VSTOR-137800 > usb-host: skip product-string read on CPR incoming #VSTOR-137800 > usb-host: hand off the device across cpr-transfer #VSTOR-137800 > usb: migrate the interface altsetting #VSTOR-137800 > usb-host: reconfigure endpoints on CPR incoming #VSTOR-137800 > hcd-xhci: resync isochronous endpoints after CPR #VSTOR-137800 > usb-host: make CPR migration blocker conditional #VSTOR-137800 > > hw/usb/bus.c | 36 ++++ > hw/usb/hcd-xhci.c | 20 ++ > hw/usb/host-libusb.c | 512 ++++++++++++++++++++++++++++++++++++++++---- > include/migration/vmstate.h | 1 + > 4 files changed, 530 insertions(+), 39 deletions(-) > --- > base-commit: f2f4b3de1a1be3360a77ca6ab79762466d238804 > change-id: 20260914-usb-host_v3-bbfea86d1dad > > Best regards, > -- > Andrey Drobyshev   What v5 fixes from v4:   - Draining now happens before the xHCI controller's state is saved (new patch 6).   - Transfers that already finished in the kernel are delivered to the guest instead of dropped.   - If migration fails, only the transfers that were really cancelled are sent again, so none go to the device     twice.   Blocking:   1. The guest can lose an interrupt and USB stops working on the target. The CPU state, including the KVM      interrupt controller, is saved (migration/savevm.c:1668) before usb_host_pre_save() drains. Any transfer      completed during the drain makes xHCI send an MSI (message-signalled interrupt) after that snapshot, so the      new process never sees it. The saved xHCI state still has ERDP_EHB set, so xhci_intr_raise() sends no more      interrupts until the guest writes ERDP, which it only does from the handler that never runs. That kills the      whole interrupter, not just delays one event. v5 makes this more likely than v4, because it now delivers      completed transfers during the drain.   2. A transfer cancelled halfway is re-run from the start. For a bulk IN transfer, the bytes already read from      the device are lost. For bulk OUT, bytes the device already accepted are sent again. This is the normal case      whenever the VM stops under I/O. The fix is to deliver a cancelled transfer that has actual_length != 0 as a      short transfer, and replay only the ones that moved no data.   Proposed direction: drain from a VM state change handler when the VM enters FINISH_MIGRATE in a CPR mode, and   re-issue from the handler that runs when the VM starts again. That runs before the CPU state is saved, which   fixes item 1 and makes patch 6 unnecessary. It also closes the window from the v2 review where vm_start() runs   before the failure notifier. That window now has a second problem: with pipelined bulk endpoints, newer OUT data   can reach the device before the re-issued older data. I left one question open for Andrey: a VM already paused   before migration starts never gets that handler call, so it still needs some fallback.