From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AM0PR02CU008.outbound.protection.outlook.com (mail-westeuropeazon11023125.outbound.protection.outlook.com [52.101.72.125]) by lore.virtuozzo.com (Postfix) with ESMTPS id 0F3C280726 for ; Fri, 9 Oct 2026 15:36:29 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gDDu0bT0xntIN9ujnyakgnSAK1F2vXZ/gon4k0IG9smI8V/bQfZQAxWyo4E4AZIbvNSLr5uiaA28eh3msRNbtT4LCjqNqNRXWolepJTyKQrvl8HOPWKq45xbX4u+0nuTK1nx7+M1X6ywszwoUECinecjSYUkwIwxTCdQ7c/Mw7LaTgGG521nIREtcN1cfgt6mi9ZsxBqAq3jID0Up3LIH9DrmzWrXc7v+GlVSYZH+yztOZXCFNm2Xh6Z7sPpyXrUbG+m60Zoz97EmMkFG4auSV7RkZDT6x1ywY9hIGNNPlc8IaZzgOgGffQWRwGsogV+SDOWXX6OStCR8rQlngL3pQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=b+30jRtmyP0cHkVQwW725/NNGfPuBpp3d6jBTyemdSM=; b=u/QUrtNI1ckUlbSIHdYei91t72K0CBWBgClWBnKTTb032fm/P2ri8ubC+saMl2VHEerunhprNjS2h3H1mvpXzwP+7G50lRktqVCqMSOfqLE6Nm4ReSNcHAYQFvZR09DQzadqJYjPsgw2B9t7UsSzAu7Jhxewz22MU6s3/ARX0Aw5l1aN4bFlh52CgJxccdgzy1wbWGWi+jtl+mSbJWRffOpkv7JS5b55VNh6hr6uoE3GlX0e42TtvIc9aeHtv2TJ4xzb41nZ2QyufidEm0V1kfTWBT6anLAvOmF4pub9PhZyptB40MShtgqUe8heIU4e84MHaHcYao5cM+eqzRCDMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.2) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=b+30jRtmyP0cHkVQwW725/NNGfPuBpp3d6jBTyemdSM=; b=kHrJvL/nYg2dMfaV2BE3By0O/VW9iIKtSLdmFFOGzTqLGJ6LGC5O71PEvNkYm0U5cp5V9WgPSvIcRCU6uaBueFNpCzRyrmnDudnVGCMdSZbfryLt2dHw6hLUrbhKKN01JoE2nQydiGbElBJ4uN4qZOfW2d9B/5UkoF120SC0kc0= Received: from AS4PR10CA0010.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:5dc::7) by PA4PR08MB6144.eurprd08.prod.outlook.com (2603:10a6:102:e3::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.522.6; Fri, 9 Oct 2026 15:36:24 +0000 Received: from DU6PEPF0000A7E0.eurprd02.prod.outlook.com (2603:10a6:20b:5dc:cafe::76) by AS4PR10CA0010.outlook.office365.com (2603:10a6:20b:5dc::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.472.21 via Frontend Transport; Fri, 9 Oct 2026 15:36:23 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.2) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.2 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.2; helo=b224-2.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-2.smtp-out.eu-central-1.amazonses.com (69.169.224.2) by DU6PEPF0000A7E0.mail.protection.outlook.com (10.167.8.39) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.522.6 via Frontend Transport; Fri, 9 Oct 2026 15:36:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1791560183; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=3BLVKk2vpDV2h6S8icxT12ptx1i76ZgCaRzj+cCPvpU=; b=isa7Q+rvbfAxFu4PZs77/UT5M/McDFGhci1PlKTaVosuNoHX4RCPPj61XrAYgm7M 9LtL05VZH0Au+Wg014CHlZl3ssLfPONvcp/klsIdPi2jQpYQ4RRIrRy5PHa2P/MqmMp 53zrAvb/nId60wMJqd1GIvZTS0FoSuSo048GElhnR7z7GwIAzJ0uf2bfaIIbMYRM7eT KeobrvNq1GKGS2/mCjPfBR1jy1UP9NZaG6vOKnzN0W7REg6E4DVbTlVjXVUwf+k+44u MJOtGmaAX9gzvOCjZVQpIt1TbXURMhjkzJbzxKavIP7rESjKcXY+WONgtZUR0CqZs4l os/FKD5FRA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1791560183; h=Message-ID:From:Subject:To:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=3BLVKk2vpDV2h6S8icxT12ptx1i76ZgCaRzj+cCPvpU=; b=fjnOpXUuyTSjrn69yjenetrJA22W+kWyCgRfACqhPm+/+8nHUgjRMWMZNYbcCOiQ A+di6Og9kHBoNqgQ9ozUIPSCFw+EPo/twddy08OX2J1TbakLZjEtbBVHlRXgCCwUJ4C W1CHjz2up4iCLb8tLfUVKq1KO8p64NvzCwe4iyNs= Message-ID: <010701a1214eed04-25d41288-19fa-4f8e-8948-16a3812a343c-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 20/21] VZ: vfio/region: preserve BAR mappings across cpr-exec #VSTOR-103860 To: Andrey Drobyshev , svt-core@virtuozzo.com In-Reply-To: <20261009-vfio_v1-v1-20-2ae91d5b8048@virtuozzo.com> References: <20261009-vfio_v1-v1-0-2ae91d5b8048@virtuozzo.com> <20261009-vfio_v1-v1-20-2ae91d5b8048@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Fri, 9 Oct 2026 15:36:23 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.10.09-69.169.224.2 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DU6PEPF0000A7E0:EE_|PA4PR08MB6144:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: a66b0764-7a9e-40c1-c03d-08df261b1318 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|704161211799003|48200799018|9140799003|61400799027|32142699015|376014|10067099003|56012099006|5063699009|4143699003|18002099003|18092099006|22082099003|55112099003|16102099003|6133799003|5113699003|4113699003|10086099003|13003099007; X-Microsoft-Antispam-Message-Info: bDmBLWbfgsJA8bdqE0Fz6w3ng4Ui01m3RKqegXpQFz8H+nNtboVaRxhM9UgbZ2/Vdt0shcvuUiTIrHqfjC2K80YuTX0gst98kUjFP/MBXsXG5UpxkMkYBaMJJ3dENqp3WSbtr9GjvPqiDHYFKRlEBR4Ts1gTux3+Lf3op7g5s5zr1Ll8oB1Lkeu7F0WHVVPQ772Bi8XvJluEzI8X4Z7Iqu6N1qbwQtsHg/ucp4K3bH+MptI3kBhiwvy4vEQUq/wHY4YY1hGYUdO/F9lQSPmsxYuGWrEgUnb8VFzgxaHrhGs3NytkUoKCP91srMk/43mQscRAbQkVcz7jDHVGLFxx75farx8FmpUq4WA/2DRaH71VN+YhYtTTPVzyVqE26Ap4WWZfJzTGPKpzVasClM7DVgn66YkTPBgbN5TBOwCsLqvs6ptkZ6bdOahXqllqISoq2HBB27LKIPRCj5CZvGeJ39kQM6F6q0MQ4PtlRMSH9gcQdTm9sMvs0K8WNM/se/tL1LOEbAzkBwpLLGX541ILRCqVmkEi/Y1vqCNgeHmjHFcA5ApZJXI3xyQdwIRmHko6IuF1hDSEAy/5d/BFPAjXB2rAcsIBxaveTql2YhlUCO6XmxBl8L1nthcfY9KtOVnKd5TiX7b1GQb3ThfkbiZOON6pXaFiZJwgAN7fSRA1qgXNNwQZD55pDgCN1OsOpXJhblR6/J33B9vilRZkkFV1TsUZhPLd2OTrFbvsO0/ZiW/O5YZ3uLp4AVyF1hdWSSNu9vsaPI5u9P+7OqCBdFIRYTOtfZJXFDB1urw0uYNKtCyAhqmSsPtEUGuGeM7UD2Jh X-Forefront-Antispam-Report: CIP:69.169.224.2;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-2.smtp-out.eu-central-1.amazonses.com;PTR:b224-2.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(704161211799003)(48200799018)(9140799003)(61400799027)(32142699015)(376014)(10067099003)(56012099006)(5063699009)(4143699003)(18002099003)(18092099006)(22082099003)(55112099003)(16102099003)(6133799003)(5113699003)(4113699003)(10086099003)(13003099007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PdcTUNJQtPV/PIoNZbEWIJosfV1W/EjNM3Z1BOcc64n1FMhl6ZzkZXz6Zb7jz7QweX/QWYhAUQk59+jBuhUpQu4hg/RJot84Ph+ap9K2Zq6MDW6J8HVd5vx/ggiE4nEyLR73LsOaVrIxGCwOl+Tpn7XA2rJ7BzD6Mk4wMlUDI0i8uTm2GKRN094ALpRB/J2ivo9AUyvmC24G7ozxK3/qCRnXxvXm9wvAGbFLWKgA9x1ZVM+TgmOb1yc0S6xEdKiIEQDPYXRcrI0NXrKyiDkbu88ssTNYtjZYvrqeCh1BVC0ST4UqO2HhxCyLSS24ejG4pKJ06jTBH2EOuMfDWFy9JR3N0SgkGd+11wxUva24M9S42uc1YkexS0jsyfuleNFiK7GeXDxCEPi6gJsCHI/4wR0UiodMbSni1fIZNWDai2nzgmj2+43wyCbaj6a+nPsr X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Oct 2026 15:36:23.5320 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a66b0764-7a9e-40c1-c03d-08df261b1318 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: DU6PEPF0000A7E0.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR08MB6144 Thank you for your contribution! Sashiko AI review found 3 potential issue(= s) to consider: - [High] vfio: region BAR reuse via CPR ramblock is not gated on cpr_exec_i= s_incoming() - [Medium] vfio: stale CPR ramblock entry left after region finalize, reuse= d by cpr_find - [Low] vfio: BAR mappings in the CPR ramblock list get RAM-only mprotect h= andling -- commit d9465df038b744893fa658afc7f09871b9b52f12 Author: Andrey Drobyshev VZ: vfio/region: preserve BAR mappings across cpr-exec #VSTOR-103860 This patch saves each VFIO BAR mmap in the CPR ramblock list, so cpr_exec_cb() marks it with MADV_DOEXEC. The new QEMU then finds the entry by name and reuses the mapping instead of calling mmap() again. This avoids the NVIDIA vGPU driver refusing to map a BAR that old QEMU still has recorded as mapped. > diff --git a/hw/vfio/region.c b/hw/vfio/region.c > index 1deaef12fd0b..07027ee79f3c 100644 > --- a/hw/vfio/region.c > +++ b/hw/vfio/region.c [ ... ] > +static char *vfio_region_cpr_name(VFIORegion *region, int index) > +{ > + g_autofree char *path =3D qdev_get_dev_path(region->vbasedev->dev); > + return g_strdup_printf("%s%s%s mmaps[%d]", path ?: "", path ? "/" : = "", > + memory_region_name(region->mem), index); > +} > + > +static void *vfio_region_cpr_find(VFIORegion *region, int index) > +{ > + g_autofree char *name =3D vfio_region_cpr_name(region, index); > + cpr_ramblock_t ram =3D cpr_find_ramblock(name); > + > + if (ram.addr) { > + assert(ram.length =3D=3D region->mmaps[index].size); > + } > + > + return (void *)(uintptr_t)ram.addr; > +} [Severity: High] Should vfio_region_cpr_find() be gated on cpr_exec_is_incoming()? The lookup is by name only. cpr_state_save() serializes the ramblock list for both cpr-transfer and cpr-exec, and cpr_state_load() loads it in both modes: migration/cpr.c:cpr_state_load() { ... } else if (channel) { mode =3D MIG_MODE_CPR_TRANSFER; cpr_set_incoming_mode(mode); f =3D cpr_transfer_input(channel, errp); ... } In transfer mode cpr_exec_is_incoming() is false, but the entry is still found. The address from the old process would then be passed to memory_region_init_ram_device_ptr() in vfio_region_mmap(). Guest BAR accesses could then fault or hit unrelated memory. In the opposite case, an exec-incoming QEMU with a missing entry silently falls back to mmap(). That brings back the NVIDIA "BAR already mapped" failure with no clear error. ram_block_add() in system/physmem.c checks cpr_exec_is_incoming() and reports an explicit error in both mismatch cases: if (cpr_exec_is_incoming()) { ... } else { if (ram.addr || ram.length) { error_report("CPR mode=3Dexec wasn't requested, but ram block %= s was saved..."); exit(1); Could the VFIO path have an equivalent check? [Severity: Low] Also, vfio_region_cpr_save() puts device MMIO BAR mappings into the generic RAM preservation list: > +static void vfio_region_cpr_save(VFIORegion *region, int index) > +{ > + g_autofree char *name =3D vfio_region_cpr_name(region, index); > + void *addr =3D region->mmaps[index].mmap; > + size_t size =3D region->mmaps[index].size; > + > + cpr_save_ramblock(name, (cpr_ramblock_t) { (uintptr_t)addr, size }); > +} Every cpr_walk_ramblock() user now treats these BARs as guest RAM. With the debug option -cpr-ram-rdonly, cpr_exec_cb() runs: if (cpr_ram_rdonly) { cpr_walk_ramblock(mark_ram_rdonly); } This mprotects the BAR mappings read-only. Neither the fail: label nor the execvp failure path restores RW. cpr_ram_restore_rw() only runs on the incoming side. After a failed cpr-exec, the next guest MMIO store through a ram_device pointer could SIGSEGV the old QEMU. On the incoming side, mark_ram_rw() does this: if (qemu_mprotect_rw((void *) ram.addr, ram.length)) { ... abort(); } Could that abort QEMU for a BAR mapped without PROT_WRITE? I did not confirm that the kernel's VFIO mmap path rejects the upgrade, so this part is speculative. The behavior is the same as guest RAM already has in this debug mode, but the commit message does not mention the interaction. > static void vfio_subregion_unmap(VFIORegion *region, int index) > { > + g_autofree char *cpr_name =3D vfio_region_cpr_name(region, index); > + > + cpr_delete_ramblock(cpr_name); [Severity: Medium] Can a stale CPR ramblock entry outlive its BAR mapping? This is the only place the patch deletes the entry. It is not reached from vfio_region_exit()->vfio_region_finalize(), which munmap()s the mappings directly: hw/vfio/region.c:vfio_region_finalize() { ... for (i =3D 0; i < region->nr_mmaps; i++) { if (region->mmaps[i].mmap) { munmap(region->mmaps[i].mmap, region->mmaps[i].size); } } ... } Two paths seem affected. First, vfio_display_region_update() calls: vfio_region_exit(&dpy->region.buffer); vfio_region_finalize(&dpy->region.buffer); It then calls vfio_region_setup() and vfio_region_mmap() again under the same name "display". vfio_region_cpr_find() returns the stale address, so mmap() is skipped and a dead pointer goes to memory_region_init_ram_device_ptr(). If the new size differs, assert(ram.length =3D=3D region->mmaps[index].size) aborts QEMU. Second, PCI device unplug and re-hotplug goes through vfio_pci_bars_exit()/vfio_bars_finalize()->vfio_region_finalize(). The stale entry stays in cpr_state.ramblocks, and the next cpr-exec applies MADV_DOEXEC to an unmapped or reused range. The fallback cleanup in qemu_ram_free() does not seem to cover this. In the display case, vfio_region_finalize() never unparents the mmaps[i].mem child, so that MemoryRegion is not finalized and qemu_ram_free() never runs. On unplug, qemu_ram_free() runs from object_property_del_all() during object finalize. By then the qdev path is already unset (see the VSTOR-1002= 54 comment in qemu_ram_unset_idstr()). cpr_name() then yields " mmaps[i]" without the device path, which does not match vfio_region_cpr_name(). Should vfio_region_finalize() also call cpr_delete_ramblock() for each mapping? > int vfio_region_mmap(VFIORegion *region) > { [ ... ] > + region->mmaps[i].mmap =3D vfio_region_cpr_find(region, i); > + if (region->mmaps[i].mmap) { > + goto mapped; > + } The reuse decision here depends only on whether an entry is present. It does not check the incoming CPR mode and does not check that the old mapping still exists. See the comments above on vfio_region_cpr_find() and vfio_subregion_unmap(). [ ... ] > + vfio_region_cpr_save(region, i); > =20 > +mapped: > name =3D g_strdup_printf("%s mmaps[%d]", > memory_region_name(region->mem), i); --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261009-vfio= _v1-v1-0-2ae91d5b8048@virtuozzo.com?part=3D20