From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <010701a15a497e88-17499b09-190a-4242-a21f-73d0a45590b7-000000@eu-central-1.amazonses.com> Received: from AM0PR02CU008.outbound.protection.outlook.com (mail-westeuropeazon11023080.outbound.protection.outlook.com [52.101.72.80]) by lore.virtuozzo.com (Postfix) with ESMTPS id 1749E802A5 for ; Tue, 20 Oct 2026 17:08:56 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=JM7+GltIgGXDMvM8A+7DNjtOtjZIEkUVYvCqFmYlRGrbLOG91hcyZUl/qC1cKBY1e1enmrhGQjgReM+ArgoohJcOE5TUG6w+8/oByE6AdXl7C/V/wUyUSwVkfYzJ8pwgFpk5BtUJtIt9PPdQYkuOXHwwXSi09qpls7h9ybdZOjwfOqSerYRD6gGnCQLpmIF4GtbPXFCZ7rk7+WwM0jJrZVwhFnL3CZ7iOEiJa2+1CK71XtlTQzuQV58H6+uo6TQsY8oFgS1Q5+iX6BQ6x1Xn5vW8Dog5aYGjixwV8/gDaPVY6EPiHEK9hhawEfaUC1wg/rTzg2Na5BFsktE6kBuUww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tS5Dto3YznMF9NWymSxMKXaDaMTLKf2DA25pJMWkScg=; b=qab9ncmDKF0BfFUESY0SsH+7/85QxkUvQ5X4CvOIrbts4J1Dt6f77vqfvBW7c3verL0uAVY8X1sE1wYvB6V4948jYQemi86K+qn73KcVZldLON10Fqv4b2S2qR5K0cfXx1AYXJo961oCCYi+RPrKMarAc2VHmT6Srjbrkke20Qu7NiYH/ZFYPrBqscnOdJRZGqOtsrs+nC4tcrt1GNSydWrILGN+qEuaDU5QnP3ZBHnaiueV3E0GIZSQ0X1a14wHP7+OVkN4JSHaL4YsCf3pQJ7NwajwKhdJRNQZgqt8FR4QlslT3T6SkZEglSivKXVDkVWbCOebKwQ1lcm1icdDlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.5) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tS5Dto3YznMF9NWymSxMKXaDaMTLKf2DA25pJMWkScg=; b=j6EGKCuByC8eObXkKpmLEEI6hSDyLWSoyZm/QDr4M1jXNaO6mNuMEJQNVXeJ9cn1OOGKLY+/cqdNKa8fPPhxDY3KJGWQ3ppPPGMico2HtD7NFpeVx4Jd3XezE6LIzF6B52YHJDrsFq1A+mZSMKmjQ/qJzeimY1d9XBT9kBCPjfc= Received: from PA7P264CA0172.FRAP264.PROD.OUTLOOK.COM (2603:10a6:102:36f::8) by GVXPR08MB10497.eurprd08.prod.outlook.com (2603:10a6:150:156::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.15; Tue, 20 Oct 2026 17:08:49 +0000 Received: from ZR1PEPF0000E6AD.eurprd05.prod.outlook.com (2603:10a6:102:36f:cafe::7c) by PA7P264CA0172.outlook.office365.com (2603:10a6:102:36f::8) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.522.20 via Frontend Transport; Tue, 20 Oct 2026 17:08:49 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.5) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.5 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.5; helo=b224-5.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-5.smtp-out.eu-central-1.amazonses.com (69.169.224.5) by ZR1PEPF0000E6AD.mail.protection.outlook.com (10.167.241.84) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.546.15 via Frontend Transport; Tue, 20 Oct 2026 17:08:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1792516128; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=05tOScDD03VIjDm61YYrimGh3pHBDNKUQ1t6XAUnuAA=; b=IYbEWL9HdClq4kscYEMTWOVD4VDLb5cqy4r+XvPfAcP3bmOoIjzWRqOJgpcq0MEq d+GuPKVbi+wmA+Sb4F6Bbb2VKgD/8jEMHCpreeF2BE4BAEZajrTCZOmMYY80n6tIMxT XDuTUz5eovfYznBao+GvGIge0M97hO9qCooVwkHrZLjau1gTzG1xH9WR6rW4uDqjUbY 5he/UYwh9ahlKyyYQbVMnk7NK14OuPFLKWLWTLEOFUp5u8AuUCaATKXALyf0eAh8IVY Q9ReU2dg0uY5eFrgBMrECbf2/qjY5wb6a8QtEOBRV0VGK5nFKrK8kWwb8ZX05nERqm8 U7KDJGnDvw== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1792516128; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=05tOScDD03VIjDm61YYrimGh3pHBDNKUQ1t6XAUnuAA=; b=OKFsJhrMpjeAD1HGZGmRMi7GThzZSVLWg19MtvQg2TYi8eIMQzNmBCMNBm2XRhvZ GczGXEky+Bm7poWNZtiGnNyEwGWSbfL/9y0SxnD2DPlbPyMswrS0WfUjxn0Oh25ehGw 6d98gxgR0NEq2Ji1kl6RwYtx2MhV6BYBna5t8Pag= Message-ID: <010701a15a497e88-17499b09-190a-4242-a21f-73d0a45590b7-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v4 29/31] VZ: vfio/region: preserve BAR mappings across cpr-exec #VSTOR-103860 To: svt-core@virtuozzo.com, Andrey Drobyshev In-Reply-To: <20261020-vfio_v1-v4-29-83a5668230e5@virtuozzo.com> References: <20261020-vfio_v1-v4-0-83a5668230e5@virtuozzo.com> <20261020-vfio_v1-v4-29-83a5668230e5@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Tue, 20 Oct 2026 17:08:48 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.20-69.169.224.5 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: ZR1PEPF0000E6AD:EE_|GVXPR08MB10497:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 42f5497c-8480-4d6f-7f94-08df2ecccedd X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-Transport-Rules-Loop: 1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161411799003|9140799003|32142699015|376014|48200799018|61400799027|55112099003|5113699003|4113699003|18002099003|22082099003|10067099003|16102099003|13003099007|4143699003|261009223027099003|5063699009|56012099006; X-Microsoft-Antispam-Message-Info: uh0glfcxsZpHOVminlDHExFZcnBE8hLvU7fzBg5zgnwDCuY0STW/b/gs/+ylThlehvrLmPyn9tddShYUp+DZcbTxjDNk9h6hdCXx4Cfngw4rfJMLGx31f4oAohaVW9efPfaahO2O+iXpKUUNjLcoOGsR+BEzGlsEguc4cIKW1nCttUH2yPsEQI8i90PlVAbbDI81H2jidQjeMr05vomhEb13g9HFKvTP1vQqfajG1t9V1EA/fossd/7cTQqWk4sotysG4rM4yaqyz0Jslxz2SV9JElHD6nOQAzGLL43bJ6wa7QlH6E0mj2Bmk2gBkvSfb31ItQZDYqrszF+Ycxc5sjsz5rnpqN6LNAPL1UUpygfCFULELe5Rc/l1gFyYV4fH73tBwrs/qT7VreX7mr2uonewQDQsqzS1nTJj/o7LKDGCE36tSFn+4H8U+Lo5QiSavQ7JEEv5ES45c/QuLZ2BIq1ctY+3lh6mwmzhE+Q7H0RxXy7ogRp2RDA1JBgb9KmCKmAn+bYvhcqVZVg2dUAVsAIpnS9JTajVOUabniTHWo+8HIrFguJF+l8Zs5JIWehsMpyAV0/vVN9KjQ7nLCkHx/voQT2yqvnv9eIldRHCh7uZaFsf/HNJDLkALeCi/IVumCHkYNUxUWQtqn35aE9WPODj+F2aF0aQAPO+LGS9QMuxbl9ds+HSl6nZXzTt9z/Exk5LYxRtP3+9mo87t/AdxbiALBfbT03LcRXtMXD/cKziVQ7oXTkXI0zY5G8dmi2OG1a7k0yTVQOizPsGfGzqY0n1tbpyVMMnb+IvpbgJ5RA7kdcBcMfk489ryPd4/gSCkJM3rykgZcHNtgBocCLHcWPES0Byr4VkqqlkXvi6V3DfaBGqrUWMiuH/BCNMb3rM X-Forefront-Antispam-Report: CIP:69.169.224.5;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-5.smtp-out.eu-central-1.amazonses.com;PTR:b224-5.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161411799003)(9140799003)(32142699015)(376014)(48200799018)(61400799027)(55112099003)(5113699003)(4113699003)(18002099003)(22082099003)(10067099003)(16102099003)(13003099007)(4143699003)(261009223027099003)(5063699009)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FyLfVmLpisw4/seLfFKf4rcnwoCZmc2OkR16pecydVvVhBTrZ6rjDaXqfEE6yKKLLKu1MYEgoC2q48hH3XUQC8Mpqok0y2cayRRQQLHkh40eAVWbwCiEB9Yr7iyqYK5k6JHJamOQ7HbNLFgt9A/xogcUwYhhkmDnesxC0US5+DXuu+Oce+nPg8PZAKOIOBLyz2XvDX5TkiLjVq17eeQiWoKAs2NaY3v9pBugUUHswCW9GzoTQokxRJVjmK0DqzMio4Tx0dqK4BLjSlj9wm1oP3S0tTSVtc0ludvdDVXy/y49yGdGaYjg3hPdIh7qZzjSvn2WrqCrrdDnjJkM1Kfm8i54sFnSmoRstG5JW2SwpbbA8uUZdSXwJBLYD8zrmtmcXCpelYvD0Kbo3dCKxkL5z4gGE16mAzlBmCbB8qpXXjWWI1Ud5msHnCg7xEm9jXbJ X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2026 17:08:48.8103 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 42f5497c-8480-4d6f-7f94-08df2ecccedd X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: ZR1PEPF0000E6AD.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: GVXPR08MB10497 Thank you for your contribution! Sashiko AI review found 5 potential issue(= s) to consider: - [Medium] vfio: vfio_device_attach() now fails hotplug while migration/sna= pshot runs - [Medium] vfio: unconditional cpr-transfer blocker in vfio_device_attach()= for all VFIO devices - [Low] vfio/region: vfio_region_cpr_find() aborts on mmap size mismatch vi= a assert() - [Low] vfio: BAR mappings joined to CPR ramblock list hit abort()-on-failu= re mprotect in cpr-ram-rdonly mode - [Low] vfio: preserved display-region mapping not released if never remapp= ed after cpr-exec -- commit af431a92f0ea91854f19f143a605e436ef6246ae Author: Andrey Drobyshev VZ: vfio/region: preserve BAR mappings across cpr-exec #VSTOR-103860 This patch saves each VFIO BAR mmap in the CPR ramblock list so that cpr_exec_cb() marks it with MADV_DOEXEC. New QEMU then reuses the saved mapping instead of calling mmap() again. It also adds a blocker that stops cpr-transfer for all VFIO devices. > diff --git a/hw/vfio/device.c b/hw/vfio/device.c > --- a/hw/vfio/device.c > +++ b/hw/vfio/device.c > @@ -464,12 +464,32 @@ bool vfio_device_attach(char *name, VFIODevice *vba= sedev, > TYPE_VFIO_IOMMU_IOMMUFD : > TYPE_VFIO_IOMMU_LEGACY; > =20 > - return vfio_device_attach_by_iommu_type(iommu_type, name, vbasedev, > - as, errp); > + /* > + * Only cpr-exec keeps the address space and so the region mappings, > + * see vfio_region_mmap(). A cpr-transfer target would have to map = the > + * regions anew, which some drivers (NVIDIA vGPU) refuse. > + */ > + error_setg(&vbasedev->cpr.mmap_blocker, > + "cpr-transfer is not supported with VFIO devices; " > + "use the cpr-exec mode"); [Severity: Medium] The blocker is installed here for every VFIO device, before no_mmap, the device type or the region state is looked at. Does this also disable cpr-transfer for devices that never map a BAR (no_mmap=3Don, vfio-ap, vfio-ccw) and for drivers that accept a re-mmap? The stated reason is the NVIDIA vGPU re-mmap problem, which only exists once vfio_region_cpr_save() has actually saved a mapping. Could the blocker be added only when a BAR mapping is saved, or be made opt-in per device? Also, docs/devel/migration/CPR.rst still says in the cpr-transfer mode section that VFIO devices are supported: This mode supports VFIO devices provided the user first puts the Should the documentation be updated along with this change? > + if (migrate_add_blocker_modes(&vbasedev->cpr.mmap_blocker, errp, > + MIG_MODE_CPR_TRANSFER, -1)) { > + return false; > + } [Severity: Medium] Can this make vfio_device_attach() fail during a migration or snapshot, whatever the CPR mode? migrate_add_blocker_modes() runs is_busy() no matter which mode mask is passed: vfio_device_attach() migrate_add_blocker_modes() is_busy() if (runstate_check(RUN_STATE_SAVE_VM) || migration_is_running()) { error_propagate_prepend(errp, *reasonp, "disallowing migration blocker (migration/snapshot in progres= s) for: "); ... return true; } This returns -EBUSY, so realizing or hotplugging any VFIO device (PCI, AP, CCW; legacy or iommufd) while a migration is running or RUN_STATE_SAVE_VM is set would now fail. Is that intended? The later patches in the series do not change this call. > diff --git a/hw/vfio/region.c b/hw/vfio/region.c > --- a/hw/vfio/region.c > +++ b/hw/vfio/region.c [ ... ] > +static void *vfio_region_cpr_find(VFIORegion *region, int index) > +{ > + g_autofree char *name =3D vfio_region_cpr_name(region, index); > + cpr_ramblock_t ram =3D cpr_find_ramblock(name); > + > + if (ram.addr) { > + assert(ram.length =3D=3D region->mmaps[index].size); > + } [Severity: Low] If the preserved length differs from the size that new QEMU computes for this mmap slot, this aborts after old QEMU has already exec'd. Could this report an error with error_report() and exit(1), as the ramblock reuse path in system/physmem.c does under cpr_exec_is_incoming()? Only the length is compared here. Offset, prot and fd_offset are not validated, so a layout change that keeps the same length would silently reuse a mapping of the wrong window. Is a check of those intended? [ ... ] > +static void vfio_region_cpr_save(VFIORegion *region, int index) > +{ > + g_autofree char *name =3D vfio_region_cpr_name(region, index); > + void *addr =3D region->mmaps[index].mmap; > + size_t size =3D region->mmaps[index].size; > + > + cpr_save_ramblock(name, (cpr_ramblock_t) { (uintptr_t)addr, size }); > +} [Severity: Low] This puts device BAR mappings into the list that migration/cpr-exec.c walks. vfio_region_mmap() can map a BAR PROT_READ only, depending on region->flags. With the debug-only -cpr-ram-rdonly option, cpr_exec_cb() calls mark_ram_rdonly() on every entry. On the exec-failure path, and when cpr_exec_is_incoming() && cpr_ram_rdonly, mark_ram_rw() requests RW on every entry: if (qemu_mprotect_rw((void *) ram.addr, ram.length)) { error_report("qemu_mprotect_rw() failed on memory region %s:" ... abort(); } Can this abort() QEMU if a driver refuses mprotect on a shared PFN mapping, for example by clearing VM_MAYWRITE? Would it also make a read-only BAR mapping writable? I could not confirm the driver behaviour from the repository, so this is a question about the interaction. [ ... ] > @@ -427,6 +470,8 @@ void vfio_region_finalize(VFIORegion *region) > =20 > for (i =3D 0; i < region->nr_mmaps; i++) { > if (region->mmaps[i].mmap) { > + g_autofree char *cpr_name =3D vfio_region_cpr_name(region, i= ); > + cpr_delete_ramblock(cpr_name); > munmap(region->mmaps[i].mmap, region->mmaps[i].size); > } > } [Severity: Low] Can the preserved entry for the lazily mapped display region be leaked here? After cpr-exec, if new QEMU has not yet set up and mapped the display region, vfio_region_setup() and vfio_region_mmap() are never called for it. region->mem and region->mmaps[i].mmap stay NULL: hw/vfio/display.c:vfio_display_region_exit() { if (!dpy->region.buffer.size) { return; } vfio_region_exit(&dpy->region.buffer); vfio_region_finalize(&dpy->region.buffer); } Here the early return, and the early return in vfio_region_finalize() when !region->mem, skip the cpr_delete_ramblock() call. The entry then stays in the list, so cpr_exec_cb() marks the VMA with MADV_DOEXEC again on the next cpr-exec. The VMA also keeps a reference on the device file. A re-added device with the same name could also pick the stale mapping up through vfio_region_cpr_find(). Should the preserved entry be released when the region is never remapped? --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261020-vfio= _v1-v4-0-83a5668230e5@virtuozzo.com?part=3D29