[PATCH v3 hci-8.0/hci-8.1 0/6] i386: keep CPU features of guests from older nodes on vz machine types #VSTOR-152264

["Denis V. Lunev" <den@openvz.org>]

A guest started on an older node can have a CPU feature that the
destination host no longer offers: TSX disabled by microcode, a newer
kernel that stops offering spec-ctrl or bus-lock-detect on AMD, or SVM
leaf features that QEMU 2.12 on Vz7 kept with svm off. QEMU filters the
feature on the destination, and libvirt refuses the live migration,
restore or snapshot revert:

  guest CPU doesn't match specification: missing features: hle,rtm

The guest read its CPUID at boot, so the bit has to stay until the guest
restarts.

Until now libvirt relaxed the check, in three places that behave
differently. One of them crashes libvirtd on the destination of any
migration of a host-passthrough VM with a vz7 machine type
(VSTOR-152247). This series keeps the bits in QEMU instead: one
x-...-compat X86CPU property per feature, all turned on by a new
pc_vz_9_2_compat on vz9.2.0 and every older vz machine type. Release
7.x ships up to vz9.1.0, and vz9.2.0 is the last type based on a RHEL 9
machine type. vz10.x types and -cpu host expansion stay strict. A
property keeps a bit only when the CPU model or the command line asks
for it and the condition in the commit message holds; KVM ignores
CPUID bits it does not support.

Patch 1 lets a row of the vz machine type tables name a fourth compat
array, which the i440fx vz7.16.0 row needs. Patches 2 to 6 add one
property each:

  2  hle, rtm                      PSBM-149402 (field: VSTOR-75689,
                                   VSTOR-105458)
  3  spec-ctrl, AMD                VSTOR-151770
  4  bus-lock-detect, AMD          VSTOR-152291
  5  v-vmsave-vmload, AMD          code review, nested guests on Zen 4
                                   client parts, no field report
  6  SVM leaf features, svm off    PSBM-144630

The same patches apply to hci-8.0 and hci-8.1 without changes; the
base given below is hci-dev. On 8.0 they also fix the migration on
their own: hci-8.0 libvirt keeps its relaxations, and they no longer
see a filtered bit. That covers bus-lock-detect (VSTOR-152291), which
hci-8.0 libvirt does not relax. The vz7 CPU fixup there still runs on
migration (with the NULL check from VSTOR-152247) and forces npt and
nrip-save off, which the guest does not see, because leaf 0x8000000a
is hidden without svm. Only the destination needs the new qemu-kvm:
the compat acts when the incoming QEMU process starts.

On hci-dev the libvirt side drops its relaxations and the qemu.d hooks
in libvirt.ovz pr/den-VSTOR-152247-drop-vz-cpu-relaxations. It needs
this series in qemu-kvm first.

Testing:
- dev-cl04, EPYC 7443P: a saved guest whose migration cookie CPU was
  rewritten to the shape of a Vz7 host-model guest
  (svm=off,npt=on,nrip-save=on, plus spec-ctrl) fails to restore
  without the series with "missing features: npt,nrip-save" and
  "missing features: spec-ctrl". With it the guest restores, the
  features stay on and filtered-features is empty. Stock 8.1 libvirt,
  which carries the same relaxations as hci-8.0, restores it with no
  warning as well.
- sqvm2-1015, Broadwell without TSX: the VSTOR-75689 guest CPU fails
  with "missing features: hle,rtm" without the series and keeps hle
  and rtm with it.
- sqvm2-1013 <-> sqvm2-1015, with the libvirt PR: live migration in
  both directions of a pc-i440fx-vz7.12.0 host-passthrough guest and
  of the VSTOR-75689 Broadwell-IBRS guest. No crash, no warning,
  hle/rtm kept, the CPU stays match='exact'. Stock libvirt crashes on
  the first of them.
- hci-8.0 and hci-8.1 builds: -cpu EPYC-Milan,svm=off,npt=on,
  nrip-save=on,enforce and -cpu EPYC-Milan,spec-ctrl=on,enforce start
  on vz7.12.0, vz9.0.0 and vz9.2.0 and are refused on vz10.x. A fresh
  Broadwell-IBRS start with check='full' is refused on vz10.1.0.
- both builds, TCG: -cpu EPYC-Milan,svm=off warns about the same
  features on vz9.2.0 as on vz10.0.0, none from leaf 0x8000000a.
- both builds, qtest-x86_64: ivshmem-test, test-hmp and qom-test fail
  as on the base (qemu_ram_free assertion, isapc aborting at start).
- both builds, spec-ctrl on vz9.2.0 and vz7.16.0: EPYC-Milan with
  spec-ctrl=on,enforce starts, also with only one of ibrs, amd-stibp
  and amd-ssbd left; EPYC-IBPB, or EPYC-Milan without ibpb or without
  all three, is refused. Everything is refused on vz10.1.0.
- bus-lock-detect and v-vmsave-vmload are build-tested only: no Zen 5
  or Zen 4 client host was available.

Changes v2 -> v3:
- 3/6: keep spec-ctrl only when the guest also has the amd-* bits that
  give it the MSRs, not only the host (Andrey)
- 2/6: pc_vz_9_2_compat first in the vz7.16.0 row (Andrey); TSX_CTRL
  makes XBEGIN abort, not #UD, and libvirt check='partial' still
  refuses a fresh start
- 5/6, 6/6: commit messages clarified
- R-b on 1/6, A-b on 2/6, 4/6, 5/6 and 6/6

Changes v1 -> v2:
- 6/6: keep SVM leaf features past the svm dependency under KVM only,
  TCG warned about them
- 2/6, 6/6, cover: commit messages clarified after the Sashiko review

v2: http://lore.virtuozzo.com/qemu/20261029113436.93521-1-den@openvz.org/
v1: http://lore.virtuozzo.com/kernel/20261029095645.4173762-1-den@openvz.org/

Signed-off-by: Denis V. Lunev <den@openvz.org>

Denis V. Lunev (6):
  i386/pc: raise the limit of compat arrays per Virtuozzo machine type
    to four #VSTOR-152264
  i386: keep hle and rtm on Virtuozzo machine types up to vz9.2.0
    #PSBM-149402
  i386: keep spec-ctrl for AMD guests on vz machine types up to vz9.2.0
    #VSTOR-151770
  i386: keep bus-lock-detect for AMD guests on vz machine types up to
    vz9.2.0 #VSTOR-152291
  i386: keep v-vmsave-vmload for nested AMD guests on vz types up to
    vz9.2.0 #VSTOR-152264
  i386: keep SVM leaf features without svm on vz types up to vz9.2.0
    #PSBM-144630

 hw/i386/pc.c         |  9 ++++++
 hw/i386/pc_piix.c    |  5 ++--
 hw/i386/pc_q35.c     |  4 +--
 include/hw/i386/pc.h | 10 +++++--
 target/i386/cpu.c    | 68 +++++++++++++++++++++++++++++++++++++++++++-
 target/i386/cpu.h    | 11 +++++++
 6 files changed, 100 insertions(+), 7 deletions(-)


base-commit: 0c1497c79831c1ec8efd91d5887236963adf0f18
-- 
2.53.0