From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from OSPPR02CU001.outbound.protection.outlook.com (mail-norwayeastazon11023091.outbound.protection.outlook.com [40.107.159.91]) by lore.virtuozzo.com (Postfix) with ESMTPS id 1AE468045D for ; Tue, 3 Nov 2026 14:38:13 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=RUV6wvIWrhUXXhaU/TDbirZgBj94Me9+FiVDW9Egd3Zm2jZ0d6pXbUx+xHklB67VUjS1xlkYFeHa0jI1kKM2p+9+OvAhHojaPXExtZkZf/ApBY4GLBI1Hb76UdUSZliQGcWshir4RHweolFA3UWU2IcrktUaGSJgc2rDKZHEeI1gbOU/IT/9XGMpMTEHDMV5DCSPzn+XyUZLo3CR29di70p9vI249FfcHAt/bOFarUepqQ+B/8Xy5vfjOP/J3qQwcAwnsNP+djdDaAJ9tJ0oaIrpfGjmbaUog7kvb5uZPiWPIE7GFMVHkJlolSZtrOPDoasc0+W1SEIn2ARxkNfnCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=F50YEw3E7Z55OVtP7MjboyKTnc7kfJKx6jdEc77hWwU=; b=ZVX/uJW3cjygAD3JDa4nHfTS74Xou5GjF+S3uuThfanmfABOG1kbKLZ8aR4GJhWT102BVkD22Y5GYVxy6fzkrpX8AVeRDP1gr0SJPrRH+tAQPNLG5/yS1CCsTUBrYZoK+TOUfYF8QtsnwbM7SPzYkBzUWtX4K/usm615rYLVn2r3IXDNzJjIv+yziPNBBb9u0u+jeyal7lzBRVAc1uAMU3kkYtXnIDC9CRwL/56/fwrNjbLUiAenPXalQsfjkXCLxFSNgwzHcQDiaCDIGOgfYD6Vbs4Giyrj32OkZ3MboXVXJvbNsmN3c0bJANlcP7U9GSPlxl75VXLOZKFGxwoykQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 69.169.224.5) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=eu-central-1.amazonses.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com; dkim=pass (signature was verified) header.d=amazonses.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.onmicrosoft.com; s=selector2-virtuozzo-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=F50YEw3E7Z55OVtP7MjboyKTnc7kfJKx6jdEc77hWwU=; b=ZvzCHdoGL+wHy5y+gTBAwaB+1HpmWjrD56mzblyYhRzSs5CwaQWa8+l/cfynY2xxJ4AC34fu/wLMszWpzstDfYHf1gbg0+DEpCWgRLraBTDNLzXaInbXuPSDsIy+SoGs0vqrj6hTtkjnhAtCoCphKjfdVAsWs9WlR+TvkpvVadg= Received: from OS6P279CA0155.NORP279.PROD.OUTLOOK.COM (2603:10a6:e10:39::10) by VI1PR08MB10102.eurprd08.prod.outlook.com (2603:10a6:800:1cf::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.569.12; Tue, 3 Nov 2026 14:38:04 +0000 Received: from CPH1EPF00000523.eurprd05.prod.outlook.com (2603:10a6:e10:39:cafe::a0) by OS6P279CA0155.outlook.office365.com (2603:10a6:e10:39::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.8 via Frontend Transport; Tue, 3 Nov 2026 14:38:04 +0000 Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 69.169.224.5) smtp.mailfrom=eu-central-1.amazonses.com; dkim=pass (signature was verified) header.d=sashiko.virtuozzo.com;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none header.from=sashiko.virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of eu-central-1.amazonses.com designates 69.169.224.5 as permitted sender) receiver=protection.outlook.com; client-ip=69.169.224.5; helo=b224-5.smtp-out.eu-central-1.amazonses.com; pr=C Received: from b224-5.smtp-out.eu-central-1.amazonses.com (69.169.224.5) by CPH1EPF00000523.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.592.5 via Frontend Transport; Tue, 3 Nov 2026 14:38:04 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=23yjbh4k34hwzfubhuiim7m4inbd3ick; d=sashiko.virtuozzo.com; t=1793716683; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date; bh=fNUnfqVjbd1Lqg7r191MKP8NMsjDIZNgAdKpjIW5RZQ=; b=kArMC/0ma8uqxur4DGhGo7TsPuDBmd5RSRMFoXSyWWWZC0q2JhZy6vIrJphIRoLf b9u4gPimnMUQC7znLl8fqvz3PQ7Kl7rV8+zOUix3NjxLXiRhyEPwdliBvLimGabSeup iLq9DjaixUW4OWCKVXSGZRo6PshxIy36D6ekx8/DQ+rVN68Adn4CK9ftF4fHUdv1p4q yES1hcncd710MnwOK+klCp8VONclXwAG87BpvgpDbBxAJRDCmK2URAz12tf3NJ6Iesw gmB6QxK59/kWZU7IxRNYDadwh6FlKruUSUxItzcJFwiHoerGpmm3NbdGRCP0p6hBXT9 o5/gucNUHA== DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=qftdzk2dqsatjnlrq4r5brjbihpfcrsh; d=amazonses.com; t=1793716683; h=Message-ID:From:Subject:To:Cc:In-Reply-To:References:Content-Type:Content-Transfer-Encoding:Date:Feedback-ID; bh=fNUnfqVjbd1Lqg7r191MKP8NMsjDIZNgAdKpjIW5RZQ=; b=WwzwHzJdtfaskhetnwqGhBLXrprjFjr4HDDZDFnhLL0narVAZQo10svAENvieiF5 YW/7qUH/Czoj4pzQCgjuOEL6tTn48W2pl6/+qVlOXmWloj8hXzWMVB1NKQqcmuQKD6V +nX2OpUkm8OK9msCM5Ph5JIR//WVhLDlWU8JxbFA= Message-ID: <010701a1a1d88344-56f27a4c-ab30-4a82-8740-c48017502bb1-000000@eu-central-1.amazonses.com> From: no-reply@sashiko.virtuozzo.com Subject: Re: [PATCH QEMU HCI-8.1 v7 33/35] VZ: vfio/region: preserve BAR mappings across cpr-exec #VSTOR-103860 To: Andrey Drobyshev , svt-core@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com In-Reply-To: <20261103-vfio_v1-v7-33-4467fdccbbe6@virtuozzo.com> References: <20261103-vfio_v1-v7-0-4467fdccbbe6@virtuozzo.com> <20261103-vfio_v1-v7-33-4467fdccbbe6@virtuozzo.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Date: Tue, 3 Nov 2026 14:38:03 +0000 Feedback-ID: ::1.eu-central-1.2wRQJBpg9g1fGRYexx3dUuYGS25Bux+2TEOx72/m6U4=:AmazonSES X-SES-Outgoing: 2026.11.03-69.169.224.5 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 0bc7f26d-0264-416e-a6fc-8352af79c58f:0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CPH1EPF00000523:EE_|VI1PR08MB10102:EE_ MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 029f8ed4-fc27-4ec1-44e7-08df39b81195 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr X-MS-Exchange-AtpMessageProperties: SA List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|61400799027|48200799018|9140799003|376014|32142699015|5063699009|261021123027099003|261021121427099003|5023799004|4143699003|56012099006|261028144727099003|261026140227099003|261022105027099003|261026121527099003|261009223027099003|10067099003|16102099003|18002099003|22082099003|261102123227099003|55112099003|5113699003|4113699003|13003099007; X-Microsoft-Antispam-Message-Info: n2+mEZEEJSqEk/S6rtYVC8+MewlhDE/lSxGg+18Lp49reBV0vuHuWOseeWDEkiBZ16Boieiur0dOIWQqAGSF062V3aD/720ORDrRO0MMu4RI5AdZ4o4I1NgUdtP5qB1SdpLKMITdpSNFc2upXpvxefX8awowDtcYmEmr3HOzMgluE36fgFBGNkco9cKJqRu7xF8mhE2uXR1v05Ct8rRWvn1T7/TqM1v3Zyx5beyI+CaNjZBCuwJnt4TBKNE7B4EEV2lpu6VpsHINB01OnkhwLAJME/xte1g05FlrbgpoP90/UgmB7SPYTH7BGQH1lPaXj1/5v8AnhfX8gcPjTd+PdrxrJN0byJ0miYYxDMCdXKahPFToU79oyIUgNUdTBsGBRG4R23aCU201/y/NWpq6UfjjAvajvF6AhwNHGiO/i93XHM7Tl+nawjhmfgunnmWdzAFJ0YXTGSqhF9u2DF9huQErXv1h3NWYEqLjL+KKL+5hPxK/wiC/i5Trqj+xWGvATZqD/Sk9sexuLRBc3v9dW//RvCa+Wq0K+P5VvLtPLCm7Slcnksf546LPL9GtbaGNegRTzW1ipnI9x6tmoHWohau4gnYApqozGFkM1Wr5qFebs0W6OFYXiViOwGfKVhsgO5YdCv3umAxXp27sF9VL4PtvjWZpQnFdcEXdZB82+zDvHZ2u2fCaFXceAtcBPJesljJnOPOl3Kor66Lv/s/9nTbPzvmJ0O8RvUd58FTy2IhtgRGdom4cVH4yEzIEXLsx93qRniZtd66gH6obTN04g+LhlaSDAM7R4P5VAtlgDg2GVyQ6M9O4PMAF7+SX7tXMR7IlVI4TVulqMGbZNO6K+Cqj1mYZhHWkKslib967AmO+w3qZDOH5iT/H3W4mQw2X X-Forefront-Antispam-Report: CIP:69.169.224.5;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:b224-5.smtp-out.eu-central-1.amazonses.com;PTR:b224-5.smtp-out.eu-central-1.amazonses.com;CAT:NONE;SFS:(13230040)(61400799027)(48200799018)(9140799003)(376014)(32142699015)(5063699009)(261021123027099003)(261021121427099003)(5023799004)(4143699003)(56012099006)(261028144727099003)(261026140227099003)(261022105027099003)(261026121527099003)(261009223027099003)(10067099003)(16102099003)(18002099003)(22082099003)(261102123227099003)(55112099003)(5113699003)(4113699003)(13003099007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0aHuQ2WyszR8CCCbyP1LRl0nAbVhdW3toVphOPImPd61QNVKY1SQFAwf/lL3CQitdhpvYS/DE7kAtOwXhuoEzuIgHd6tepgugFTQKF/r6ETZFduKDvb26bhTiautR8mu7UJVZ294aO3ss/1/vG47fIE20eSIX40su/VnIavQ2djQfk/R4zsL/fJNY0fa53nbPRLdnlaTZkdLnL3bJhmvr7BhpnG4aakh7hTPjVS7S4C4dHRID1fXZ+u6lg+QSBw61xijRJ2jd4OyZw2/rfa6vlVqNF0vgCEVLJBAmNrIa5xktUCgO0hypOuIc88YxjhoqNYabfKHep5UpZIKMWqQy7fAfGs65TFokApTVq1JGS7oMXOioVRq9sQ0DdG6/4cXOqOIsJFCbPa9Su2Xxez0J5B/CXNMvP4vaWil8aOfeD/Pg6Z2dGaA4NBJEH7O/Q6d X-ExternalRecipientOutboundConnectors: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2026 14:38:04.0741 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 029f8ed4-fc27-4ec1-44e7-08df39b81195 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-AuthSource: CPH1EPF00000523.eurprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB10102 Thank you for your contribution! Sashiko AI review found 4 potential issue(= s) to consider: - [Medium] vfio: cpr-transfer blocked for every VFIO device, not only those= with saved BAR mmaps - [Medium] vfio: cpr-transfer blocker skipped for vfio-user, stale BAR mapp= ings reused - [Low] vfio/region: BAR mappings enter CPR ramblock list walked by cpr-ram= -rdonly mprotect - [Low] vfio: unclaimed BAR mappings preserved by cpr-exec are never cleane= d up -- commit 8715bbf752f1d56051fe39f5336635bb75b99caa Author: Andrey Drobyshev VZ: vfio/region: preserve BAR mappings across cpr-exec #VSTOR-103860 This patch saves each VFIO BAR mapping in the CPR ramblock list so that cpr-exec preserves it with MADV_DOEXEC, and the new QEMU reuses it instead of calling mmap() again. It also blocks cpr-transfer for all VFIO devices and updates CPR.rst to say so. > diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst > --- a/docs/devel/migration/CPR.rst > +++ b/docs/devel/migration/CPR.rst [ ... ] > -RAM in place, albeit with new virtual addresses in new QEMU. Devices > -and their pinned memory pages are also preserved for VFIO and IOMMUFD. > +RAM in place, albeit with new virtual addresses in new QEMU. VFIO and > +IOMMUFD devices are not supported in this mode: use cpr-exec for them. This documents the new policy, but only this paragraph was changed. Are the other cpr-transfer and VFIO descriptions in CPR.rst still accurate? > diff --git a/hw/vfio/device.c b/hw/vfio/device.c > --- a/hw/vfio/device.c > +++ b/hw/vfio/device.c [ ... ] > @@ -464,12 +464,32 @@ bool vfio_device_attach(char *name, VFIODevice *vba= sedev, > + error_setg(&vbasedev->cpr.mmap_blocker, > + "cpr-transfer is not supported with VFIO devices; " > + "use the cpr-exec mode"); > + if (migrate_add_blocker_modes(&vbasedev->cpr.mmap_blocker, errp, > + MIG_MODE_CPR_TRANSFER, -1)) { > + return false; > + } [Severity: Medium] Is it intended that this blocker is installed in vfio_device_attach() for every VFIO device? The blocker is added before any backend-specific work and without checking device type, no_mmap, or whether any region will ever be saved through vfio_region_cpr_save(). Devices such as vfio-ap, vfio-ccw, x-no-mmap devices, devices with no mmap-able BARs, and PCI devices whose drivers allow re-mapping would also lose cpr-transfer. The commit message gives the NVIDIA vGPU re-map restriction as the rationale. Would it be narrower to block only when vfio_region_cpr_save() actually saves a mapping, or only on the PCI path? The series does not appear to narrow this blocker later. The final state still has the same unconditional mmap_blocker in vfio_device_attach(). Also, the cpr-transfer specific VFIO code now looks unreachable for cpr-transfer, but it is left in place: the id_blocker with MIG_MODE_CPR_TRANSFER in device.c, the blockers in cpr-legacy.c and cpr-iommufd.c, and the ones in cpr.c. Should that code be cleaned up? [Severity: Medium] Does this blocker cover vfio-user devices? vfio_user_pci_realize() in hw/vfio-user/pci.c calls vfio_device_attach_by_iommu_type() directly: if (!vfio_device_attach_by_iommu_type(TYPE_VFIO_IOMMU_USER, vbasedev->name, vbasedev, as, errp)) { That bypasses vfio_device_attach(), so vbasedev->cpr.mmap_blocker is never installed for vfio-user. hw/vfio-user/ has no other CPR blocker. vfio-user regions can be mmapped through server-supplied region fds, and vfio_bar_register() in hw/vfio/pci.c calls vfio_region_mmap(). That now reaches vfio_region_cpr_save(), which puts the mapping in cpr_state.ramblocks. That list is serialized for both cpr-exec and cpr-transfer. On a cpr-transfer target, vfio_region_cpr_find() looks up "name> region N mmaps[i]". For vfio-user, vbasedev->name is "vfio-user:", so the name matches the saved entry. > +static void *vfio_region_cpr_find(VFIORegion *region, int index) > +{ > + g_autofree char *name =3D vfio_region_cpr_name(region, index); > + cpr_ramblock_t ram =3D cpr_find_ramblock(name); > + > + if (ram.addr && ram.length !=3D region->mmaps[index].size) { [ ... ] > + return (void *)(uintptr_t)ram.addr; > +} Nothing checks that the incoming mode is cpr-exec here. Can a cpr-transfer target take the old process's address from this function and pass it to memory_region_init_ram_device_ptr() in vfio_region_mmap() as if it were a local mapping? If so, the first guest access to that BAR would touch an invalid or unrelated address. Should vfio_region_cpr_find() be gated on cpr_exec_is_incoming(), or should the vfio-user attach path also install the blocker? > +static void vfio_region_cpr_save(VFIORegion *region, int index) > +{ [ ... ] > + cpr_save_ramblock(name, (cpr_ramblock_t) { (uintptr_t)addr, size }); > +} [Severity: Low] Is it intended that BAR mappings, including read-only ones, now go into the same ramblock list that cpr_exec_cb() walks for the -cpr-ram-rdonly debug option? With that option set, cpr_exec_cb() calls cpr_walk_ramblock(mark_ram_rdonly), and mark_ram_rw() later calls qemu_mprotect_rw() on each entry and abort()s on failure: if (qemu_mprotect_rw((void *) ram.addr, ram.length)) { error_report("qemu_mprotect_rw() failed on memory region %s:" ... abort(); A BAR that vfio_region_mmap() mapped read-only (no VFIO_REGION_INFO_FLAG_WRITE) would be turned into PROT_READ|PROT_WRITE by cpr_ram_restore_rw(). If a driver rejects mprotect on these device PFN mappings, QEMU would abort. I did not confirm that any driver refuses mprotect. Should these entries be excluded from the mprotect walks? [Severity: Low] Can a preserved BAR mapping that new QEMU never claims be left behind? The only place a saved mapping is claimed is vfio_region_mmap(): region->mmaps[i].mmap =3D vfio_region_cpr_find(region, i); if (region->mmaps[i].mmap) { goto mapped; } The only cleanup is cpr_delete_ramblock() in vfio_subregion_unmap() and vfio_region_finalize(), and both run only for regions that were set up on the new side. If new QEMU never reaches vfio_region_mmap() for a region (for example a different x-no-mmap setting, a changed region layout, or a failed realize), the mapping stays mapped with its MADV_DOEXEC mark and its cpr_state.ramblocks entry. A later cpr-exec would then apply MADV_DOEXEC to it again through cpr_walk_ramblock(preserve_ram). Because it is never unmapped, a driver such as the NVIDIA vGPU driver would still see the BAR as mapped. Is there a sweep of unclaimed region entries planned? None of the later patches in the series appear to add one. --=20 Sashiko AI review =C2=B7 https://sashiko.vzint.dev/#/patchset/20261103-vfio= _v1-v7-0-4467fdccbbe6@virtuozzo.com?part=3D33