[RFC HCI-8.1 PATCH 0/5] qemu: allow qemu-update with VFIO hostdevs on shared memfd RAM #VSTOR-103860

[Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>]

qemu-update (cpr-exec) keeps anonymous guest RAM across exec with
MADV_DOEXEC, which shares the pages with the new process copy-on-write.
For a VM with a VFIO device that breaks the device: VFIO pins all guest
RAM for DMA, so the first guest write after exec copies the page, while
the device keeps doing DMA to the old pinned one.  Shared memfd RAM is
handed over to the new QEMU as is and has no such problem.

This series lifts the blanket hostdev restriction for qemu-update and
makes it safe:

 - only PCI (VFIO) and USB hostdevs are allowed, mdev/scsi/scsi_host as
   well as NVMe disks and vDPA interfaces are still refused;
 - new VMs with devices pinning guest RAM (PCI/mdev hostdevs, NVMe disks,
   vDPA) get shared memfd RAM, unless memory backing is set explicitly;
 - their auxiliary RAM (VRAM, ROMs) is shared too via aux-ram-share=on;
 - qemu-update is refused for a PCI hostdev on non-shared RAM, e.g. a VM
   started by an older libvirt; a cold restart switches it to memfd.

KSM is not lost: it never merges pinned pages anyway.  THP works for memfd
RAM with the QEMU fix aligning fd-backed RAM to 2M and
/sys/kernel/mm/transparent_hugepage/shmem_enabled set to "advise".

Tested on whole GPU passthrough (RTX 6000 Ada): 32 qemu-updates in a
row, the GPU keeps working and all 16G of guest RAM stays on 2M pages.
With the QEMU VFIO CPR fixes also on an NVIDIA vGPU: qemu-update keeps
the vGPU working, a GPU<->guest RAM DMA checker shows no mismatches.

This is an alternative to the vzkernel series moving MADV_DOEXEC memory
into the new mm, which needs no libvirt changes.

Andrey Drobyshev (5):
  qemu update: allow qemu-update for VMs with PCI and USB hostdevs
    #VSTOR-103860 #VSTOR-137800
  qemu: capabilities: probe aux-ram-share machine property #VSTOR-103860
  qemu: back guest RAM with shared memfd for VMs with DMA-pinning
    devices #VSTOR-103860
  qemu: share auxiliary RAM of VMs with DMA-pinning devices
    #VSTOR-103860
  qemu update: refuse PCI hostdevs on non-shared guest RAM #VSTOR-103860

 src/qemu/qemu_capabilities.c                  |  2 +
 src/qemu/qemu_capabilities.h                  |  1 +
 src/qemu/qemu_command.c                       |  3 +
 src/qemu/qemu_domain.c                        | 36 ++++++++
 src/qemu/qemu_domain.h                        |  6 ++
 src/qemu/qemu_driver.c                        | 44 ++++++++-
 src/qemu/qemu_process.c                       | 91 +++++++++++++++++++
 .../caps_10.0.0_aarch64.xml                   |  1 +
 .../caps_10.0.0_ppc64.xml                     |  1 +
 .../caps_10.0.0_s390x.xml                     |  1 +
 .../caps_10.0.0_x86_64+amdsev.xml             |  1 +
 .../caps_10.0.0_x86_64.xml                    |  1 +
 .../caps_10.1.0_x86_64+inteltdx.xml           |  1 +
 tests/qemustatusxml2xmldata/modern-in.xml     |  1 +
 .../disk-nvme.x86_64-latest.args              |  4 +-
 .../disk-slices.x86_64-latest.args            |  4 +-
 ...tdev-mdev-display-ramfb.x86_64-latest.args |  4 +-
 ...play-spice-egl-headless.x86_64-latest.args |  4 +-
 ...ev-display-spice-opengl.x86_64-latest.args |  4 +-
 ...isplay-vnc-egl-headless.x86_64-latest.args |  4 +-
 ...ostdev-mdev-display-vnc.x86_64-latest.args |  4 +-
 .../hostdev-mdev-display.x86_64-latest.args   |  4 +-
 ...hostdev-mdev-precreated.x86_64-latest.args |  4 +-
 ...tdev-pci-address-device.x86_64-latest.args |  4 +-
 ...-pci-address-unassigned.x86_64-latest.args |  4 +-
 .../hostdev-pci-address.x86_64-latest.args    |  4 +-
 ...stdev-pci-display-ramfb.x86_64-latest.args |  4 +-
 ...stdev-pci-multifunction.x86_64-latest.args |  4 +-
 ...tdev-subsys-mdev-vfio-ap.s390x-latest.args |  4 +-
 ...ubsys-mdev-vfio-ccw-boot.s390x-latest.args |  4 +-
 ...dev-subsys-mdev-vfio-ccw.s390x-latest.args |  4 +-
 ...ostdev-vfio-multidomain.x86_64-latest.args |  4 +-
 ...o-zpci-autogenerate-fids.s390x-latest.args |  4 +-
 ...o-zpci-autogenerate-uids.s390x-latest.args |  4 +-
 ...v-vfio-zpci-autogenerate.s390x-latest.args |  4 +-
 ...dev-vfio-zpci-boundaries.s390x-latest.args |  4 +-
 ...vfio-zpci-ccw-memballoon.s390x-latest.args |  4 +-
 ...io-zpci-multidomain-many.s390x-latest.args |  4 +-
 .../hostdev-vfio-zpci.s390x-latest.args       |  4 +-
 .../hostdev-vfio.x86_64-latest.args           |  4 +-
 .../machine-loadparm-hostdev.s390x-9.1.0.args |  2 +-
 ...machine-loadparm-hostdev.s390x-latest.args |  4 +-
 .../net-hostdev-bootorder.x86_64-latest.args  |  4 +-
 ...net-hostdev-multidomain.x86_64-latest.args |  4 +-
 ...ostdev-vfio-multidomain.x86_64-latest.args |  4 +-
 .../net-hostdev-vfio.x86_64-latest.args       |  4 +-
 .../net-hostdev.x86_64-latest.args            |  4 +-
 .../net-vdpa-multiqueue.x86_64-latest.args    |  4 +-
 .../net-vdpa.x86_64-latest.args               |  4 +-
 ...-virtio-teaming-hostdev.x86_64-latest.args |  4 +-
 ...-virtio-teaming-network.x86_64-latest.args |  4 +-
 .../net-virtio-teaming.x86_64-latest.args     |  4 +-
 .../pci-rom.x86_64-latest.args                |  4 +-
 .../pseries-hostdevs-1.ppc64-latest.args      |  4 +-
 .../pseries-hostdevs-2.ppc64-latest.args      |  4 +-
 .../pseries-hostdevs-3.ppc64-latest.args      |  4 +-
 56 files changed, 271 insertions(+), 85 deletions(-)

-- 
2.47.1