From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from GVXPR05CU001.outbound.protection.outlook.com (mail-swedencentralazon11023092.outbound.protection.outlook.com [52.101.83.92]) by lore.virtuozzo.com (Postfix) with ESMTPS id 7787680BA5 for ; Tue, 20 Oct 2026 16:31:05 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PJbazGGB9rn+ZT1a/WqRPSmZv6VG4bIrNJBFW5mqm17NQsaKKEBlHntiQk182zKeur34b+990qF39tJA+TVYZzKDHoRi5AZfrzGvrbfnNJE5AKDuYSIoQLuR5y4/C5YU20nxZ/9XpXPKEMGU03kwLakzaCFxAZyIZO0G1qXEAoGnZcZiCMDnrqe4s0JmI/5Am2ccmUXeDOo1osjNzXuuNyQFOwjM6pWYFgelR2LXDUd0RP26JQExNLoJv1Qq+0V8UTOrrcSfCb1DxaveRelQlMnxFCfH3Y7UXEtBC7ElCwb47iJAM+fho4jn/oIZmJVxKKNp7zinfMcW96mfCHkjNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Jkr8L7fjFqCYJ7AuahOiyo/nN9ct+vJa2oA5RjKcCo4=; b=KW2qZb2WDDK51l9NV1/x0xh6Ev0CpyIMsylAJMyCzo3BdTIVCPvELd9oU1zxhtIuhTPxSLf4oUk0peb0SIQjK9Sl94ABZFaFRaV5flVd+hEemRIU1pwKtY3isX5OFfU5DqENBy0zeGjxJBG/texlGliUvkzvLnoSGeYHu8+qwyyiXI0/H59M0N07UMNmmi+VR2/gC+TJj40s29uOgf/Syp9TEMxEfS0qXKEDwYgPsXVRp+QcQeKSKA14N7c9DohIRcawiF7162TAjlZBzrPkojx4DiqsG3Nt1F/fSkhaoXyOAXlI3z5WiVKr8MW9FSH+3WZCV306mlgc3E5Y2HIcDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Jkr8L7fjFqCYJ7AuahOiyo/nN9ct+vJa2oA5RjKcCo4=; b=eZ4qWWTis1wjNU8FpZbkRH3v/VpgzgT9AWRQeRl5xKpRHW93/O53wOYtYydvSxkfGSPSSMXzW4y7xYeS93zHnPxb9mWatR7sRfXp9t6hCt3UsxY+Ii0Q4+IDfP4ouO+56bp4FHP8xmbrqsdQMZCdbU7SZguKewq/xC8RPXmZa6npziCY1iuLzo1/bzeUVfnXr/B+W/lt9WQJk5Llc8S5PNXKvwVMdMTT0rN1qdvciljD9J3lMupGUtj/JQLAzYfgbS4c6a36UEB+/4w7WEhfm3J45PjaaD1TP25CK0rgsED/I04qxxX8nS3D6oNdY0fGS9mMRdkXoT8JnBXPJowvnw== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) by AS4PR08MB7553.eurprd08.prod.outlook.com (2603:10a6:20b:4fb::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.15; Tue, 20 Oct 2026 16:31:04 +0000 Received: from VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8]) by VI0PR08MB10656.eurprd08.prod.outlook.com ([fe80::4e37:b189:ddcd:3dd8%7]) with mapi id 15.21.0546.012; Tue, 20 Oct 2026 16:31:04 +0000 From: Andrey Drobyshev Date: Tue, 20 Oct 2026 19:31:09 +0300 Subject: [PATCH QEMU HCI-8.1 v4 29/31] VZ: vfio/region: preserve BAR mappings across cpr-exec #VSTOR-103860 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20261020-vfio_v1-v4-29-83a5668230e5@virtuozzo.com> References: <20261020-vfio_v1-v4-0-83a5668230e5@virtuozzo.com> In-Reply-To: <20261020-vfio_v1-v4-0-83a5668230e5@virtuozzo.com> To: svt-core@virtuozzo.com Cc: Andrey Drobyshev X-Mailer: b4 0.16.0 X-ClientProxiedBy: VIUP296CA0056.AUTP296.PROD.OUTLOOK.COM (2603:10a6:800:34b::14) To VI0PR08MB10656.eurprd08.prod.outlook.com (2603:10a6:800:20a::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI0PR08MB10656:EE_|AS4PR08MB7553:EE_ X-MS-Office365-Filtering-Correlation-Id: c4c73369-28ce-487c-3495-08df2ec786ba X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|1800799024|376014|52116014|366016|5113699003|10067099003|22082099003|18002099003|261009223027099003|56012099006|38350700014; X-Microsoft-Antispam-Message-Info: A4f6kHHbbcUd8tAmYB+8BBuQnLlQqBlg8DRJg9IKDL6t3LYF3GWBNeeN4V+lAbs6I17Nid0IVREdwE5/lzSVRoQ7cmDeIIwxgxCjdgDlMrIM67fZsx2BHwZGqDEtQSPLADHtnwTWYRotQ+6pR5csRR8KXBFtKYrrsHtqBWIgFzmLZJ1335YM6nXL4VuDIslUtLJOol3gyb2P7ZfOJE5gvAv3BkjYgVaDtM2tWPYf70Ye9lT7M8J05I/4Vy6Z0vu3El/CG8Bx8+SvxdYhWCqmWb4usNSVqXzyntLoPYPm4t+OzJ0mqOLHOcQTPcigffoMTDl5IpRjP99YfwWVW+jMkwqjWUT4v6G3CbHDJexGdK+l5md5vmz9QvvhEQnoSeam2oF27uggLCbKCzwEZAqMryIQqtQh0I/TWHZof/O5QRYpHWbTJtkFIhKbqlXnuamM9oSb1aaHJjhiDEJYzOHFCZ5XikKGGOo87hKdkS3GvU1wB0ipSt0oaXbRnJeG9EDH4nvpaCAOD3EbB7OmR8JX3lPsbN2we9oAlYfKx61ipUg3PD3WOm7NmFzdSutXliUk3hTAFKczYQmylaRH0ub4YrcYXs2wOGq8F7wwCPBUtt/Enp7D1NlhmixkCTK2NoWuHPQT+J4YCXEZQp9pAbmXU092OYNxKv4xqVvYSkbJ7ckL5NHeSrmnPXc9YXgNILf3HfHGS4Do8GncTtLnLgivk5kSLD5pLFyl48Os7OAQDFM= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI0PR08MB10656.eurprd08.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(1800799024)(376014)(52116014)(366016)(5113699003)(10067099003)(22082099003)(18002099003)(261009223027099003)(56012099006)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OFRJT2JhdDBQbU5oZlVNYnpkdXZlVjE2RzZlUjgyd0tkWnBrNmVWdG05UjJD?= =?utf-8?B?RXJJSzdDZXBsRmk4VkxJQkYxYnduTDFHa1dKYjlUcTZ4VVdXb1dFUzFDOXEy?= =?utf-8?B?OXNyZUxiRlpaZHJLYUMrZWhjQWF6QzV2dW1yS0trVSt2dXdxOG1CbndqQS9K?= =?utf-8?B?TjFHSWUyK0xLMCtBcC9vYmt6cFQrd0VwbHd3Ym1aRHgrc2pYTXQ5dS9ibWpT?= =?utf-8?B?aUhEZ1ZVVEFiMVBOSk5nbWRFb0JhU3FkTmkzaDF3QzZtaDMrWWpmNXY4RVlr?= =?utf-8?B?dGk0d2NCU3czNEF3SzUrcUhXOUd4TnAxOFFFT0hJdnhhaXlZMHVXVjh3OXdD?= =?utf-8?B?Q0dJWmN0YUg1Y1N4bEdFaUNQNkZ2OTZXUGhPdGJQN3RlMEVwd3ppWEtyc1lo?= =?utf-8?B?emlqVVNtdjFKR3FuSGhHNGdkelhDODhuOE12STd0M1Z4Z1ZzWUhKQkt1U1RE?= =?utf-8?B?aDJQLzBYczUrUGl3YmFpc1B2eFdEOWVDMk1rdCtJV2pkRlRNcU4wSHNicHlX?= =?utf-8?B?OTNVc21wQUI2aVZUOEZWaDZMY0JSSXEvbGhubE9EUHdYM3hXOEJCcWdsZjBK?= =?utf-8?B?K3VnZ2hOOVFNNkNvNDVsUVdGZzM2dE5tRlgvNUtpbkRDdWNxMnhDTURDZmVv?= =?utf-8?B?Y3NyRk54SEVvcUhFNUt4WGw5MTVvZkZIc0wvVmFUZWZCQW5XVTVOUHBIS1J2?= =?utf-8?B?dmM1VlZqaGx5cHN0TkF2RjQrNjhhNmxDR1lKRlI2cHp1TlVFc0Z3VTlpODdR?= =?utf-8?B?em1NY1VlLzZ1SWgyMEJDUGs3cnZkM0hWeGZORXBFcnF2NUE0U2Y2bFFCbStT?= =?utf-8?B?bWh6czJHRGJzK01ndG5rakFZbER0azEzQTNDNUlvckp6U3NCOVNUOFEwVlQy?= =?utf-8?B?eHJRQ2haRklDcFNtNUVFWE5jakNJdDVTUDR4RDRyTmx6cjRFNzJSQVd3eU5j?= =?utf-8?B?TUdzdnZhS1YzTTNnREJldXp1cXE0M2JvU2h3UDRlaHF5aGJUQzllK3Fkeklv?= =?utf-8?B?OEZzaG9jMVp2SnBvY3Ezajg1cWVXbk9PYUNmMllqbnk1MEJ4eVZtM24vcnJC?= =?utf-8?B?TWNodnp3TEZ6RHpyTDYvZmN3cGt1S1BETHlwOUtKMkloM09SV2tacFpXcm1t?= =?utf-8?B?WXpkcnYvRWZIZExWV2dFSXp3cTFxaGllTS9VWTd4S0pVa29QMXdhK0x5NjFV?= =?utf-8?B?SXJrQm9XM0hMY0crcUdqT2Jac1h6MXJ0ZU9jZng1TFJzaE1UQ3hmTFN5N2RT?= =?utf-8?B?MWZ3Z0RUTWpjbWVkS29XSzJnbmphd1E4d3pZL0loVGZSTlErdXQ2TnVqZWhN?= =?utf-8?B?U3dwdTlMRHRqQ1ZqcmFkYzRXeW5obU15SWhRd1p4NjdJeTdhSTVHVVlhRXhU?= =?utf-8?B?djVMQURNYmR0V1NEdkNqeG9uT3Q1ejNrVS82d0lOeFlwZ2sxSHRzSWYvcXFM?= =?utf-8?B?Y0VFNnZzQjhnV201MUYvVFB6d3lrU3JTQjZyRjRXVzlKTVpkZ0dqTVI5bkh3?= =?utf-8?B?T01jRDh3eGFjVUJON0JzbmV3eWZWYmhYc1VZSEdpajdib1RxRWpwVStOYkpY?= =?utf-8?B?VXY3L1hmVnF1ajJkVXd1aTZhWHpkb0pSYW5YTFZrZlA0eU1oTk5uQXFlWXF3?= =?utf-8?B?STBQbXBJQnZxejFybW94NTBIY2tCNTVGbWRHVHhRMVRNZzVsSy9MWEpydGk0?= =?utf-8?B?eWtRNG83STZvL1FKeFhrRVdzRmtLbVdzNHZWYk9zRXE5RHd4c21xZUpPbG1l?= =?utf-8?B?WThwSWxmc2VxVjdtZ0pSRlVwNkVQYWN2SXVkdWlWdUx0RkdUdnZnTzBDU2tU?= =?utf-8?B?dEdLMWRWUTc3N25ha2VWODNMSm84UFBoaFJPWkxlTnAvNU14dUtZM2l1MGdJ?= =?utf-8?B?cm96c2xORWdnelhHM0pCR01LUFhvZFA4Q0w4QVZycXJERkhTeUhWUjFHc25T?= =?utf-8?B?QnRNam56bzFnYzRDQmR2anVYSUVxd0RWcDlHVHZUQmtMZ1oxc0tLR1Y1UUFV?= =?utf-8?B?Q1pZR0hxYjFhNW1BSForUmJoY3o1UDZEbFR3SHgycnlmV0RSRDdwM0UzMWpS?= =?utf-8?B?NXl6NWtDeUh6czBkVHEzZXNIaEhlOTl6QzhEdEQ4M0xhT3JjdERTSXp4Tloz?= =?utf-8?B?YjBOZ3ZzRU0rcis2RGNER2dmdjFvNmEyVWJGNE43MmtVZGR5ZUd5VStPZ1h0?= =?utf-8?B?K0FTZVJMbVFKY2pXbUdrWmF5ZEo1V0Y1ektoZlVGL3ptam92cHVPczBBd3Yw?= =?utf-8?B?eks4Y1lWcUd1WmYrM0hlN3VKWmdHY0k4ZE1ZVkY4UkFrSTR4djFDZHJ6WUxM?= =?utf-8?B?c08rZHNPd3Z1NXdEVDMvaURFTW1tMFNHTHFsSVNjMDF6VnpwYXV2Wjh0SXZW?= =?utf-8?Q?67F0Xz6gMaLnTaww=3D?= X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: c4c73369-28ce-487c-3495-08df2ec786ba X-MS-Exchange-CrossTenant-AuthSource: VI0PR08MB10656.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Oct 2026 16:31:00.4608 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GBGK0VvPgXXcT81ka9UrfWW3Pdk96Nn/KEuyTmxd71JCK6eOR5REQDztAHF7TmLf4Z7mQKJgiwUQFjPFDX6bGMZuyew6HT/mAh22zDvYInU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS4PR08MB7553 After cpr-exec, new QEMU mmaps the device BARs again on the preserved device fd. Some drivers don't allow that: the NVIDIA vGPU VFIO driver refuses to map a VF BAR that is still recorded as mapped by old QEMU, and that record is dropped only when the device fd is closed. So BAR1 and BAR3 of the vGPU can't be mapped, and the guest loses the device. Keep the mappings instead. Save each BAR mapping in the CPR ramblock list, so cpr_exec_cb() marks it with MADV_DOEXEC together with guest RAM, and new QEMU finds it there and reuses it instead of calling mmap(). Same find-or-create pattern as for other preserved objects, e.g. vector eventfds in vfio_notifier_init(). No saved mapping means old QEMU didn't map the region, e.g. its mmap failed or the region is mapped lazily, so new QEMU maps it afresh. The ID is " region mmaps[]". It doesn't depend on the device path, which is gone when the display region is finalized, so the entry is dropped together with the mapping. The saved addresses only make sense for cpr-exec, which keeps the address space. A cpr-transfer target is another process and would have to map the regions anew, which the drivers above refuse while the old mappings exist. So block cpr-transfer for VFIO devices. This needs a kernel which supports MADV_DOEXEC on shared PFN mappings of device memory. Without it cpr-exec of a VM with VFIO devices fails. Signed-off-by: Andrey Drobyshev --- hw/vfio/device.c | 24 ++++++++++++++++++++++-- hw/vfio/region.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-cpr.h | 1 + 3 files changed, 68 insertions(+), 2 deletions(-) diff --git a/hw/vfio/device.c b/hw/vfio/device.c index 5a14a7d5c0a..2a7073ccec8 100644 --- a/hw/vfio/device.c +++ b/hw/vfio/device.c @@ -464,12 +464,32 @@ bool vfio_device_attach(char *name, VFIODevice *vbasedev, TYPE_VFIO_IOMMU_IOMMUFD : TYPE_VFIO_IOMMU_LEGACY; - return vfio_device_attach_by_iommu_type(iommu_type, name, vbasedev, - as, errp); + /* + * Only cpr-exec keeps the address space and so the region mappings, + * see vfio_region_mmap(). A cpr-transfer target would have to map the + * regions anew, which some drivers (NVIDIA vGPU) refuse. + */ + error_setg(&vbasedev->cpr.mmap_blocker, + "cpr-transfer is not supported with VFIO devices; " + "use the cpr-exec mode"); + if (migrate_add_blocker_modes(&vbasedev->cpr.mmap_blocker, errp, + MIG_MODE_CPR_TRANSFER, -1)) { + return false; + } + + if (!vfio_device_attach_by_iommu_type(iommu_type, name, vbasedev, + as, errp)) { + migrate_del_blocker(&vbasedev->cpr.mmap_blocker); + return false; + } + + return true; } void vfio_device_detach(VFIODevice *vbasedev) { + migrate_del_blocker(&vbasedev->cpr.mmap_blocker); + if (!vbasedev->bcontainer) { return; } diff --git a/hw/vfio/region.c b/hw/vfio/region.c index b0332f27fcd..4d58794b3cf 100644 --- a/hw/vfio/region.c +++ b/hw/vfio/region.c @@ -30,6 +30,7 @@ #include "qemu/units.h" #include "monitor/monitor.h" #include "system/ramblock.h" +#include "migration/cpr.h" #include "vfio-helpers.h" /* @@ -227,8 +228,43 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, return 0; } +static char *vfio_region_cpr_name(VFIORegion *region, int index) +{ + return g_strdup_printf("%s region %d mmaps[%d]", region->vbasedev->name, + region->nr, index); +} + +/* + * A saved mapping is one preserved by cpr-exec, reuse it. None means old + * QEMU didn't map the region, e.g. its mmap failed or it's mapped lazily, + * so map it afresh. + */ +static void *vfio_region_cpr_find(VFIORegion *region, int index) +{ + g_autofree char *name = vfio_region_cpr_name(region, index); + cpr_ramblock_t ram = cpr_find_ramblock(name); + + if (ram.addr) { + assert(ram.length == region->mmaps[index].size); + } + + return (void *)(uintptr_t)ram.addr; +} + +static void vfio_region_cpr_save(VFIORegion *region, int index) +{ + g_autofree char *name = vfio_region_cpr_name(region, index); + void *addr = region->mmaps[index].mmap; + size_t size = region->mmaps[index].size; + + cpr_save_ramblock(name, (cpr_ramblock_t) { (uintptr_t)addr, size }); +} + static void vfio_subregion_unmap(VFIORegion *region, int index) { + g_autofree char *cpr_name = vfio_region_cpr_name(region, index); + + cpr_delete_ramblock(cpr_name); trace_vfio_region_unmap(memory_region_name(®ion->mmaps[index].mem), region->mmaps[index].offset, region->mmaps[index].offset + @@ -314,6 +350,11 @@ int vfio_region_mmap(VFIORegion *region) size_t align = MIN(1ULL << ctz64(region->mmaps[i].size), 1 * GiB); void *map_base, *map_align; + region->mmaps[i].mmap = vfio_region_cpr_find(region, i); + if (region->mmaps[i].mmap) { + goto mapped; + } + /* * Align the mmap for more efficient mapping in the kernel. Ideally * we'd know the PMD and PUD mapping sizes to use as discrete alignment @@ -347,7 +388,9 @@ int vfio_region_mmap(VFIORegion *region) ret = -errno; goto no_mmap; } + vfio_region_cpr_save(region, i); +mapped: name = g_strdup_printf("%s mmaps[%d]", memory_region_name(region->mem), i); memory_region_init_ram_device_ptr(®ion->mmaps[i].mem, @@ -427,6 +470,8 @@ void vfio_region_finalize(VFIORegion *region) for (i = 0; i < region->nr_mmaps; i++) { if (region->mmaps[i].mmap) { + g_autofree char *cpr_name = vfio_region_cpr_name(region, i); + cpr_delete_ramblock(cpr_name); munmap(region->mmaps[i].mmap, region->mmaps[i].size); } } diff --git a/include/hw/vfio/vfio-cpr.h b/include/hw/vfio/vfio-cpr.h index 4606da500a7..fa1699857ca 100644 --- a/include/hw/vfio/vfio-cpr.h +++ b/include/hw/vfio/vfio-cpr.h @@ -34,6 +34,7 @@ typedef struct VFIOContainerCPR { typedef struct VFIODeviceCPR { Error *mdev_blocker; Error *id_blocker; + Error *mmap_blocker; uint32_t hwpt_id; uint32_t ioas_id; } VFIODeviceCPR; -- 2.47.1