From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (unknown [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id EB6A18045D for ; Fri, 16 Oct 2026 21:37:01 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 69GLZm4T021526; Sat, 17 Oct 2026 00:35:50 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 69GLZm4T021526 Authentication-Results: mail.openvz.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=virtuozzo.com header.i=@virtuozzo.com header.b="dGZ/zsjh" Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 69GLZj4V021509 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Sat, 17 Oct 2026 00:35:45 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 69GLZj4V021509 Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-48dea677afcso256524f8f.1 for ; Fri, 16 Oct 2026 14:35:45 -0700 (PDT) X-Gm-Message-State: AFq9FYLCDLiDK9yMuxConIyNwbXwiQ0wuEFKy9hJuBkrQs1uF2Mt9wLr ikf/7iWVoPtwDvzAOpJY/pagP8+I6UyoYBES+vxKdd/3fgWPuyAJQuSrs3YiUor0sMFjacYHLO0 snBgLTmo1RG+xanWrnKGKiu0m0up5wKtJVsFPyDc5729zLgXzpQsreA== X-Gm-Gg: AYBFou3+qrWe6Oq3koFOyTU9xTIbM2ZAvaN+7TXQSe49ouiXq/kZNEn+4w8jQcWaiN4 WBi66ZLORfuzPhrqFFmwpMKCm8QEHV5R1ub/EO2jjZrGDQDeWz1E2x/YG0tSNsRN9rT+0yamezd LW2eS3u/p8M5J2qSTMy3d43Rwy6Z4AFL5LIE/AaAIzxlWuABLbzPX/+IPJIxF2t/ArT57CJfBtu nCKuHlQLXkCSZEI/T3di7w6UU2BI17KHwgM1NL7CADoU35NJDBAk3faPdZSZIf9iy1h3/CsLsxt oTGUwveYtmaViyzXdpxk8kxPo4ZszMMVSU54oNf8PQ7BKW7Q20on7IH/h9q5YJW8xJbiHXUP65o C04qJVh1eY7L2qudURQ== X-Received: by 2002:a5d:6f0b:0:b0:48e:6f:19b2 with SMTP id ffacd0b85a97d-48e006f1b27mr6953570f8f.25.1792186545263; Fri, 16 Oct 2026 14:35:45 -0700 (PDT) X-Received: by 2002:a5d:6f0b:0:b0:48e:6f:19b2 with SMTP id ffacd0b85a97d-48e006f1b27mr6953530f8f.25.1792186544705; Fri, 16 Oct 2026 14:35:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1792186544; cv=none; d=google.com; s=arc-20260327; b=nCG03v2Y/gs/iTNoxFoDcDVEZf/yM0Wqv61af2vlwEvjvbTDTDLBIhlaHvUpBDlFKv sddHY6ImoRh83hC77HxhWyHFw8l/nna0tcaDMDRuHb32nWX9RGc/AQauzDqCr77JrGj5 z01tfy37AIrEe8cCFBZAneOjFnNgYOsBZZbB9K7p0XND+CI/qACORrEo+HXuGpVfA437 a8S31NW2GI9JUI7sNwJm+rYE5at6QmjaytwOKCW29L7nM3kJkhprYUyXmgS1LZMGrk7C CRPYRlT+h0RFh3vLIdreS14ZYr9Q913qSyzCfBUC5rGH8h87UTNGIF+AtVQFXz1YUepQ v49w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:dkim-signature; bh=Zz/5xoZ0uTMPnZpYZx+uVEi3+8sWHeC4FbH1a+76DFg=; fh=ORqO450YHYjnAvQVnHrQRoSXc/Ljts4WdUx12SqEe+w=; b=S4162wodMTriDwUoz4V2ZcAIbsNKRdZTdv7vhYpsyqLLOBXt0DLZiwmJ2UX9p+0Zge Iu7Jw5BQBnvdg8DvJrWT/JFMzsuD/p0QKv81sh97zggMPh8lkjGF+sjb3KpEJOioQYoa tgY0q8bxi399WoDNrHMXQ86PEuXJmDJJkeJfMJ8oQmg15bJ9AKRt8uHGK/pGfyoaMHUS uUfkgS22wpzhDQQxiYwlewTBDtH9HuPKEisaH9+MPqxamB2HhQ72uLiAhjT8cXFd/Ovu mEy2h3dWT2vSaq3ZKZA5U3Vn6CdkHu1mMCTl0tc9fwnspxwecAF6jxpAUgzU9oRq+rc2 uGdQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b="dGZ/zsjh"; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com Received: from relay.virtuozzo.com (relay.virtuozzo.com. [130.117.225.111]) by mx.google.com with ESMTPS id ffacd0b85a97d-48dfd525d08si6963403f8f.395.2026.10.16.14.35.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Oct 2026 14:35:44 -0700 (PDT) Received-SPF: pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) client-ip=130.117.225.111; Authentication-Results: mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b="dGZ/zsjh"; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=Message-Id:Content-Type:MIME-Version:Subject:Date :From; bh=Zz/5xoZ0uTMPnZpYZx+uVEi3+8sWHeC4FbH1a+76DFg=; b=dGZ/zsjhCH2ndUm1+LW 7CIByJbMVRkzKAp1mXpbD9pqFZdQENov3SN9zlkOW2NhFY2xRLaQmeF88ct42fD2azQipqv+UXBLs Wwa0Lz59IXcUAX7LQZTTf4pKvDc7Edl/LZlWWMTKWu7rKo+lstHI8b/6RjSqu+oYXbKbK1hRURU+L daRCXOGUEGemon5AexGS89GLJ7HNXzWowPVu25nnMTJquwiwMN0i/pVP4D/hVALvmhnJWZP1HZN/S qOxaOR0QgnbeIFOyFqfvfesKGwsF7glDCCN8r418nLUHZdUp+l/byC2wh4Bt30TABxEDJFNxeWp9K 2haikg/VMZcGTcw==; Received: from ch-demo-asa.virtuozzo.com ([130.117.225.8] helo=f0.sw.ru) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1xHpX5-00GeHO-30; Fri, 16 Oct 2026 23:35:40 +0200 From: Konstantin Khorenko Date: Fri, 16 Oct 2026 23:35:39 +0200 MIME-Version: 1.0 Message-Id: <20261016-b4-prep-khorenko-vstor-103860-4-doexec-vfio-bars-v2-1-2237f8fbb38e@virtuozzo.com> References: <20261016-b4-prep-khorenko-vstor-103860-4-doexec-vfio-bars-v2-0-2237f8fbb38e@virtuozzo.com> In-Reply-To: <20261016-b4-prep-khorenko-vstor-103860-4-doexec-vfio-bars-v2-0-2237f8fbb38e@virtuozzo.com> To: Andrey Drobyshev , Pavel Tikhomirov , Vasileios Almpanis X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=11977; i=khorenko@virtuozzo.com; h=from:subject:message-id; bh=qlvcTmwg2PizPGGT/I1C95ljKEYkNjxKYRmEHfDSrVo=; b=owEBbQKS/ZANAwAKAVGWCkf5YVwqAcsmYgBq0pivhHaTLdLL6Gim3s3IAT8ewUuNEwBxSfWvS EV/MPUp6ymJAjMEAAEKAB0WIQRWD23IPcXT0GO9CCxRlgpH+WFcKgUCatKYrwAKCRBRlgpH+WFc KpJcD/4muxUd7IxtEAFBiO7x2AUkZhmcB0NSs8k+j70SycQalIk9XG4cUJB3og8ZZZWHk54+q4x AlCT8kO/eegboieMUM/dE6F7q8WgIiBNiMs+9sUkD3tmzPg19ujVKnYAydYIf5hY006D72GRATL saxLwyQNV2mXlbwbOoy9D6Z5CBRd8ty9Z9ZtyNRTxv06gwuwmI1zelmx3262Qj3JKn11fgZn6H7 /fhv4onw2pR4448l39n7Qp8ftgnzmVbMkMi9+ce35q0WZHYRScyBTliJDC8Vd5moUKcuRQ4UUiU yf4ypGQvTEz5WA6lyxm4U7YCbPcYO/6O4mbRGGXszOGk2dpeuMcZCTiYt8OR4mWY7YZEM0ysQhr Ri8D1rAK9jPGvJE5ttQnGqEX6YRw9XcwMPmmZHI8vsp9olnlb3CmTqGKek3gOhvGFN/B70bX/sj G1w423CSFiWrY6BbiHQ2GnjKZE+9WNfLbs1FLGqD8G8tJn+PN+3m5CJxuiGcHBgaGjtF6IOho24 AwwKJLUXBNm97xYkKtV+v6EXkJyopAKMFSuO3Xq3fmNivj4F/ajHk/MkwMBa8coNkrqXlMaX34m 5xxnPtDGBwiDETSzdGZk/rnCmVBMXcjk7AfCexpkMVJUQbQvyqsb0IBW3Gb5ozPOe14XhQMGoLm qMPJcmpjbcnPFKw== X-Developer-Key: i=khorenko@virtuozzo.com; a=openpgp; fpr=560F6DC83DC5D3D063BD082C51960A47F9615C2A X-OZ-Fwd: true Cc: devel@openvz.org Subject: [Devel] [PATCH hci-8.1 v2 1/2] mm: allow MADV_DOEXEC on shared PFN mappings of device memory X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org From: Andrey Drobyshev QEMU live update (cpr-exec) keeps guest RAM across exec with MADV_DOEXEC, but VFIO device BARs mapped into QEMU are lost on exec, and the new QEMU has to mmap them again on the same device fd. Some drivers don't allow that. The NVIDIA vGPU VFIO driver records each BAR mapping and refuses an overlapping one with -EINVAL. For VF BARs the record is dropped only when the device fd is closed, so after exec the new QEMU can't map BAR1/BAR3 and the vGPU becomes unusable. Let MADV_DOEXEC also preserve shared VM_PFNMAP mappings of a VFIO device file, the BARs. vma_dup() duplicates them the way dup_mmap() duplicates a shared file mapping: clear the inherited VM_PAT so that track_pfn_copy() reserves the memtype for the copy, fork the anon_vma if the old VMA ever got one, link the VMA into the file's i_mmap (so unmap_mapping_range() on device reset still reaches it), call vm_ops->open() and copy the special PTEs with copy_page_range(). vma_link() places the copy after the original in i_mmap, since an equal key goes right in the interval tree, which is the order dup_mmap() gets from vma_interval_tree_insert_after(). Admit only a VFIO file, checked with vfio_file_is_valid() as KVM does; of those only a device file can be mapped. Other drivers keep per-mm state in such mappings and rely on VM_DONTCOPY to keep fork away: perf charges a ring buffer to the mm's pinned_vm, so a copy would drive the counter of the new mm negative and lift RLIMIT_MEMLOCK, and Xen privcmd frees per-mapping state in ->close() without an ->open(). VM_DONTCOPY itself cannot be the test, because QEMU marks every RAM block MADV_DONTFORK, VFIO BARs included. A VFIO mapping whose vm_ops has ->close() but no ->open() is admitted too: the NVIDIA vGPU driver maps the BAR0 pieces of a vGPU that way. The driver then gets ->close() for the original VMA at the exec, while the copy is mapped, and again for the copy later, with no ->open() in between. A fork that copied the mapping would do the same, but QEMU's MADV_DONTFORK keeps fork away from it, so this sequence is new to the driver. qemu-update with a vGPU was tested this way. No in-tree VFIO driver keeps per-VMA state, and the vfio-pci family revokes BAR mappings through the device file's mapping, which reaches the copy. An out-of-tree driver that tracks its mappings has to count the copy in ->open(). One that maps with remap_pfn_range(), which sets VM_PAT, and revokes with unmap_mapping_range() inherits the windows fork has between track_pfn_copy() and a concurrent zap, and the one munmap has between the zap of the old VMA and a device reset; no in-tree VFIO driver does both. Dropping the mark gives VM_MAYEXEC back only to anonymous memory, where do_mmap() decides it. A marked PFN map does not get it back, since there the driver had the last word. Once the copy is in i_mmap, a zap from a device reset may clear VM_PAT of either VMA holding only i_mmap_rwsem. So clear VM_EXEC_KEEP of the copy before linking it, and touch VM_ACCOUNT of the old VMA only when it is set. The source VMA is write-locked explicitly, as in dup_mmap(), instead of through that vm_flags_clear(). Such a VMA may have NULL vm_ops, in which case vma_is_anonymous() returns true for it. So the new vma_is_shared_pfnmap() helper is checked first, also in zap_vma_mt(), whose threaded path for anonymous memory would skip untrack_pfn(). https://virtuozzo.atlassian.net/browse/VSTOR-150233 Signed-off-by: Andrey Drobyshev Feature: oracle/mm: MADV_DOEXEC madvise() flag (cherry picked from commit f1903df83b065cd59ebe30cfcf5fde6391fcc4e3) Changes while porting: added the untrack_pfn_clear() call, the unconditional anon_vma_fork(), the VFIO-only admission, the VM_MAYEXEC handling and the flag ordering above; the file reference now comes from an earlier fix that takes it for every file-backed VMA. Assisted-by: LLM Signed-off-by: Konstantin Khorenko --- include/linux/mm.h | 12 ++++++++++-- mm/madvise.c | 28 +++++++++++++++++++++++----- mm/memory.c | 2 +- mm/mmap.c | 53 ++++++++++++++++++++++++++++++++++++++--------------- 4 files changed, 72 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 91f305fd5c025..496cbdfc7141a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1018,6 +1018,16 @@ static inline bool vma_is_anonymous(struct vm_area_struct *vma) return !vma->vm_ops; } +/* + * Shared PFN mapping of a file, such as a device BAR mapped through VFIO. + * Its vm_ops may be NULL, so it must be tested before vma_is_anonymous(). + */ +static inline bool vma_is_shared_pfnmap(struct vm_area_struct *vma) +{ + return vma->vm_file && + (vma->vm_flags & (VM_SHARED | VM_PFNMAP)) == (VM_SHARED | VM_PFNMAP); +} + /* * Indicate if the VMA is a heap for the given task; for * /proc/PID/maps that is the heap of the main task. @@ -2481,10 +2491,8 @@ void free_pgd_range(struct mmu_gather *tlb, unsigned long addr, unsigned long end, unsigned long floor, unsigned long ceiling); int copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -#ifdef CONFIG_PADATA int copy_page_range_mt(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma); -#endif int generic_access_phys(struct vm_area_struct *vma, unsigned long addr, void *buf, int len, int write); diff --git a/mm/madvise.c b/mm/madvise.c index e6972bcfc9749..cf3e580894b17 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -31,6 +31,7 @@ #include #include #include +#include #include @@ -1249,6 +1250,20 @@ static long madvise_guard_remove(struct vm_area_struct *vma, &guard_remove_walk_ops, NULL); } +/* Only VFIO device mappings are known to cope with a copy in another mm. */ +static bool vma_doexec_pfnmap_ok(struct vm_area_struct *vma) +{ + bool (*is_vfio)(struct file *file); + bool ret; + + is_vfio = symbol_get(vfio_file_is_valid); + if (!is_vfio) + return false; + ret = is_vfio(vma->vm_file); + symbol_put(vfio_file_is_valid); + return ret; +} + /* * Apply an madvise behavior to a region of a vma. madvise_update_vma * will handle splitting a vm area into separate areas, each area with its own @@ -1309,12 +1324,15 @@ static int madvise_vma_behavior(struct vm_area_struct *vma, break; case MADV_DOEXEC: /* - * MADV_DOEXEC is only supported on private, non-executable, - * non-stack, non-special anonymous memory and if the - * VM_EXEC_KEEP flag is available. + * MADV_DOEXEC is supported on private, non-executable, + * non-stack, non-special anonymous memory, and on shared + * PFN mappings of VFIO devices, if the VM_EXEC_KEEP flag + * is available. */ - if (!VM_EXEC_KEEP || !vma_is_anonymous(vma) || - vma->vm_flags & (VM_EXEC | VM_SHARED | VM_STACK | VM_SPECIAL)) { + if (!VM_EXEC_KEEP || vma->vm_flags & (VM_EXEC | VM_STACK) || + (vma_is_shared_pfnmap(vma) ? !vma_doexec_pfnmap_ok(vma) : + !vma_is_anonymous(vma) || + vma->vm_flags & (VM_SHARED | VM_SPECIAL))) { error = -EINVAL; goto out; } diff --git a/mm/memory.c b/mm/memory.c index 2566f02fb7d5e..2f15bc06ccf90 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2239,7 +2239,7 @@ void zap_vma_mt(struct vm_area_struct *vma) struct mmu_notifier_range range; mmap_assert_write_locked(vma->vm_mm); - if (vma_is_anonymous(vma)) { + if (vma_is_anonymous(vma) && !vma_is_shared_pfnmap(vma)) { lru_add_drain(); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma->vm_mm, vma->vm_start, vma->vm_end); diff --git a/mm/mmap.c b/mm/mmap.c index 69dd1514d750e..959810d8047a8 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1752,31 +1752,41 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) /* * The VM_MAYEXEC that MADV_DOEXEC took from @vma, to give back when the mark - * goes: what do_mmap() grants such a mapping. + * goes: what do_mmap() grants such a mapping. A driver may have withheld it + * from a PFN map, so leave that one without it. */ vm_flags_t vma_doexec_mayexec(struct vm_area_struct *vma) { - if (vma->vm_file && path_noexec(&vma->vm_file->f_path)) + if (vma_is_shared_pfnmap(vma) || + (vma->vm_file && path_noexec(&vma->vm_file->f_path))) return 0; return VM_MAYEXEC; } /* - * Duplicate an anonymous VMA into a new mm as part of preserving - * it across exec. + * Duplicate a VMA into a new mm as part of preserving it across exec. + * + * Anonymous VMAs get their pages shared with the new mm fork-style. + * Shared PFN mappings (e.g. VFIO BARs) are duplicated the way fork + * duplicates a shared file mapping: same file, same special PTEs, + * nothing to copy-on-write. */ int vma_dup(struct vm_area_struct *old_vma, struct mm_struct *mm) { struct vm_area_struct *vma; unsigned long npages; + bool pfnmap = vma_is_shared_pfnmap(old_vma); int ret = -ENOMEM; - if (WARN_ON(!vma_is_anonymous(old_vma))) + if (WARN_ON(!pfnmap && !vma_is_anonymous(old_vma))) return -EINVAL; if (find_vma_intersection(mm, old_vma->vm_start, old_vma->vm_end)) return -EEXIST; + /* As in dup_mmap(), keep per-VMA lock users out of the source. */ + vma_start_write(old_vma); + npages = vma_pages(old_vma); vm_stat_account(mm, old_vma->vm_flags, npages); @@ -1784,6 +1794,10 @@ int vma_dup(struct vm_area_struct *old_vma, struct mm_struct *mm) if (!vma) goto fail_nomem; + /* track_pfn_copy() will later take care of copying internal state. */ + if (pfnmap) + untrack_pfn_clear(vma); + ret = vma_dup_policy(old_vma, vma); if (ret) goto fail_nomem_policy; @@ -1797,17 +1811,24 @@ int vma_dup(struct vm_area_struct *old_vma, struct mm_struct *mm) /* * Clear functionality that should not carry over to the new - * process. Note that VM_EXEC_KEEP is cleared later to allow - * code called by copy_page_range to infer that the copying is - * for preserving over exec and not for process forking. + * process. Note that VM_EXEC_KEEP of an anonymous copy is cleared + * later to allow code called by copy_page_range to infer that the + * copying is for preserving over exec and not for process forking. */ vm_flags_clear(vma, VM_LOCKED|VM_LOCKONFAULT|VM_UFFD_MISSING|VM_UFFD_WP|VM_UFFD_MINOR); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; + /* Once linked, unmap_mapping_range() may modify the flags concurrently. */ + if (pfnmap) + vm_flags_clear(vma, VM_EXEC_KEEP); + /* Also links a file VMA into the file's i_mmap, for unmap_mapping_range(). */ ret = vma_link(mm, vma); if (ret) goto fail_nomem_vma_link; + if (pfnmap && vma->vm_ops && vma->vm_ops->open) + vma->vm_ops->open(vma); + /* * Now that the dup vma is inserted into the mm, clear VM_ACCOUNT * from old_vma. Since vma_dup() is only called during exec to @@ -1815,14 +1836,16 @@ int vma_dup(struct vm_area_struct *old_vma, struct mm_struct *mm) * process, this effectively transfers the accounting from the old * vma to the new one. */ - vm_flags_clear(old_vma, VM_ACCOUNT); + if (old_vma->vm_flags & VM_ACCOUNT) + vm_flags_clear(old_vma, VM_ACCOUNT); -#ifdef CONFIG_PADATA - ret = copy_page_range_mt(vma, old_vma); -#else - ret = copy_page_range(vma, old_vma); -#endif - vm_flags_mod(vma, vma_doexec_mayexec(vma), VM_EXEC_KEEP); + /* For PFN maps this copies the device PTEs and the PAT memtype */ + if (pfnmap || !IS_ENABLED(CONFIG_PADATA)) + ret = copy_page_range(vma, old_vma); + else + ret = copy_page_range_mt(vma, old_vma); + if (!pfnmap) + vm_flags_mod(vma, vma_doexec_mayexec(vma), VM_EXEC_KEEP); return ret; -- 2.43.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel