[Devel] [PATCH hci-8.1 v3 0/1] vfio: kvm reference leak on live update

[Devel] [PATCH hci-8.1 v3 0/1] vfio: kvm reference leak on live update

[Andrey Drobyshev]

A live update keeps the passthrough device open while the VM is
re-created. The device holds a reference to the kvm it was opened
with, so the first VM is never destroyed until the device closes. Take
that reference only for the drivers that read device->kvm.

Tested with this patch alone: 30 live updates of a vGPU VM (RTX 6000
Ada SR-IOV VF) under guest DMA load, then 4 live migrations between
two such nodes: one KVM instance after every update, no "duplicate
directory", and no DMA'd buffer differing from the pattern written
into it.

v2 -> v3:

- Patch 2 (debugfs directory) dropped: with patch 1 nothing holds the
  old VM past exec. Traced: the VM fd release and kvm_destroy_vm() run
  at exec, 180 ms before the new QEMU creates its VM, so the directory
  is gone before the name can clash; the test above ran without it

- Patch 1: "in open_device()" and "device->kvm stays NULL without the
  flag" in the comment and the message, how nvidia-vgpu-vfio was
  checked, Assisted-by moved here, blank line before Feature:

v2: http://lore.virtuozzo.com/20261105121733.643754-1-andrey.drobyshev@virtuozzo.com
v1: http://lore.virtuozzo.com/20261030144933.944360-1-andrey.drobyshev@virtuozzo.com

Assisted-by: LLM


Andrey Drobyshev (1):
  vfio: take the kvm reference only for drivers that read device->kvm

 drivers/gpu/drm/i915/gvt/kvmgt.c  | 1 +
 drivers/s390/crypto/vfio_ap_ops.c | 1 +
 drivers/vfio/pci/vfio_pci_core.c  | 1 +
 drivers/vfio/vfio_main.c          | 6 ++++--
 include/linux/vfio.h              | 7 +++++++
 5 files changed, 14 insertions(+), 2 deletions(-)

-- 
2.47.1

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v3 1/1] vfio: take the kvm reference only for drivers that read device->kvm

[Andrey Drobyshev]

A device takes a reference to its group's kvm at the first open and
drops it at the last close, so that device->kvm stays valid for the
driver in between. QEMU's cpr-exec and cpr-transfer keep the device
open while the VM is re-created, so the first VM stays referenced for
as long as the device is: its vcpus, mm and in-kernel PIT thread with
it. Seen as a second KVM instance in debugfs after a live update of a
vGPU VM.

Only three drivers read device->kvm: KVMGT, vfio_ap and vfio-pci's
zdev support, all in open_device(), the first and the last at close
too. Let them say so with needs_kvm_ref. Without it device->kvm stays
NULL and no reference is taken, so the device can't keep a VM alive.
Tested with nvidia-vgpu-vfio 595.71.03 on RTX 6000 Ada SR-IOV VFs, a
vfio-pci variant driver that doesn't read the pointer: a grep for kvm
over its sources (nvidia-595.71.03/nvidia-vgpu-vfio) finds only the
kvm_host.h include and a comment.

Fixes: 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock")
https://virtuozzo.atlassian.net/browse/VSTOR-150597

Feature: fix ms/vfio
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 drivers/gpu/drm/i915/gvt/kvmgt.c  | 1 +
 drivers/s390/crypto/vfio_ap_ops.c | 1 +
 drivers/vfio/pci/vfio_pci_core.c  | 1 +
 drivers/vfio/vfio_main.c          | 6 ++++--
 include/linux/vfio.h              | 7 +++++++
 5 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/gvt/kvmgt.c b/drivers/gpu/drm/i915/gvt/kvmgt.c
index 96d23717684f..a0d2f9f4b720 100644
--- a/drivers/gpu/drm/i915/gvt/kvmgt.c
+++ b/drivers/gpu/drm/i915/gvt/kvmgt.c
@@ -1418,6 +1418,7 @@ static int intel_vgpu_init_dev(struct vfio_device *vfio_dev)
 	int ret;
 
 	vgpu->gvt = kdev_to_i915(mdev->type->parent->dev)->gvt;
+	vfio_dev->needs_kvm_ref = true;
 	ret = intel_gvt_create_vgpu(vgpu, type->conf);
 	if (ret)
 		return ret;
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 766557547f83..78e8566a9ced 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -779,6 +779,7 @@ static int vfio_ap_mdev_init_dev(struct vfio_device *vdev)
 		container_of(vdev, struct ap_matrix_mdev, vdev);
 
 	matrix_mdev->mdev = to_mdev_device(vdev->dev);
+	vdev->needs_kvm_ref = true;
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
 	matrix_mdev->pqap_hook = handle_pqap;
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);
diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c
index 7e360f6f89eb..1b02c2fb6c20 100644
--- a/drivers/vfio/pci/vfio_pci_core.c
+++ b/drivers/vfio/pci/vfio_pci_core.c
@@ -2094,6 +2094,7 @@ int vfio_pci_core_init_dev(struct vfio_device *core_vdev)
 
 	vdev->pdev = to_pci_dev(core_vdev->dev);
 	vdev->irq_type = VFIO_PCI_NUM_IRQS;
+	core_vdev->needs_kvm_ref = IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM);
 	mutex_init(&vdev->igate);
 	spin_lock_init(&vdev->irqlock);
 	mutex_init(&vdev->ioeventfds_lock);
diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index bf01fc64ee64..dfeee692c2a4 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -441,7 +441,8 @@ void vfio_device_get_kvm_safe(struct vfio_device *device, struct kvm *kvm)
 
 	lockdep_assert_held(&device->dev_set->lock);
 
-	if (!kvm)
+	/* Only a driver that reads device->kvm needs the KVM referenced */
+	if (!kvm || !device->needs_kvm_ref)
 		return;
 
 	pfn = symbol_get(kvm_put_kvm);
@@ -1505,7 +1506,8 @@ static void vfio_device_file_set_kvm(struct file *file, struct kvm *kvm)
  * @kvm: KVM to link
  *
  * When a VFIO device is first opened the KVM will be available in
- * device->kvm if one was associated with the file.
+ * device->kvm if one was associated with the file and the driver set
+ * needs_kvm_ref.
  */
 void vfio_file_set_kvm(struct file *file, struct kvm *kvm)
 {
diff --git a/include/linux/vfio.h b/include/linux/vfio.h
index e90859956514..9236058fc4b8 100644
--- a/include/linux/vfio.h
+++ b/include/linux/vfio.h
@@ -52,6 +52,13 @@ struct vfio_device {
 	struct vfio_device_set *dev_set;
 	struct list_head dev_set_list;
 	unsigned int migration_flags;
+	/*
+	 * Set before registration by a driver that reads device->kvm in
+	 * open_device(): the core then keeps the KVM the device was opened
+	 * with referenced until close_device(). Without it device->kvm
+	 * stays NULL.
+	 */
+	bool needs_kvm_ref;
 	struct kvm *kvm;
 
 	/* Members below here are private, not for driver use */
-- 
2.47.1

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v3 1/1] vfio: take the kvm reference only for drivers that read device->kvm

[no-reply]

> A device takes a reference to its group's kvm at the first open and
> drops it at the last close, so that device->kvm stays valid for the
> driver in between. QEMU's cpr-exec and cpr-transfer keep the device
> open while the VM is re-created, so the first VM stays referenced for
> as long as the device is: its vcpus, mm and in-kernel PIT thread with
> it. Seen as a second KVM instance in debugfs after a live update of a
> vGPU VM.
> 
> Only three drivers read device->kvm: KVMGT, vfio_ap and vfio-pci's
> zdev support, all in open_device(), the first and the last at close
> too. Let them say so with needs_kvm_ref. Without it device->kvm stays
> NULL and no reference is taken, so the device can't keep a VM alive.
> Tested with nvidia-vgpu-vfio 595.71.03 on RTX 6000 Ada SR-IOV VFs, a
> vfio-pci variant driver that doesn't read the pointer: a grep for kvm
> over its sources (nvidia-595.71.03/nvidia-vgpu-vfio) finds only the
> kvm_host.h include and a comment.
> 
> Fixes: 2b48f52f2bff ("vfio: fix deadlock between group lock and kvm lock")
> https://virtuozzo.atlassian.net/browse/VSTOR-150597
> 
> Feature: fix ms/vfio
> Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261105141759.750676-1-andrey.drobyshev@virtuozzo.com?part=1


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v3 0/1] vfio: kvm reference leak on live update

[Konstantin Khorenko]

Applying with these fixes on my side:
- commit message: KVMGT reads device->kvm while the guest runs too,
  not only in open_device() and at close;
- struct vfio_device comment: the flag is for a driver that uses
  device->kvm while the device is open, not only in open_device();
- the comments before the kvm get in vfio_df_group_open() and
  vfio_df_ioctl_bind_iommufd(): the pointer is saved only when the
  driver set needs_kvm_ref.

On Thu, 5 Nov 2026 16:17:58 +0200, Andrey Drobyshev wrote:
> Andrey Drobyshev (1):
>   vfio: take the kvm reference only for drivers that read device->kvm

Thanks, the code is right, and the test with this patch alone answers
the patch 2 question.

The first two fixes undo a mistake of mine: item 1 of my v2 reply said
"in open_device(), not after it". That holds for vfio_ap and zdev, but
KVMGT also calls kvm_write_track_add_gfn() on device->kvm while the
guest runs. Details are in the reply to the patch.

For the mainline version (VSTOR-153437): all three drivers already
take their own reference on the kvm in open_device(). So upstream may
prefer the core to hold its reference only across open_device() and
drop it right after, with no flag at all. Worth raising there.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v3 1/1] vfio: take the kvm reference only for drivers that read device->kvm

[Konstantin Khorenko]

Applying with these fixes on my side:
- commit message: "all in open_device(), the first and the last at
  close too" becomes "vfio_ap reads it only in open_device(), zdev
  also at close, KVMGT also at close and while the guest runs";
- struct vfio_device comment: "reads device->kvm in open_device()"
  becomes "uses device->kvm while the device is open";
- the comments before the kvm get in vfio_df_group_open() and
  vfio_df_ioctl_bind_iommufd(): "Save the pointer in the device for
  use by drivers." is followed by "Both only if the driver set
  needs_kvm_ref.", since the flag gates the reference too.

On Thu, 5 Nov 2026 16:17:59 +0200, Andrey Drobyshev wrote:
> Only three drivers read device->kvm: KVMGT, vfio_ap and vfio-pci's
> zdev support, all in open_device(), the first and the last at close
> too. Let them say so with needs_kvm_ref. Without it device->kvm stays
> NULL and no reference is taken, so the device can't keep a VM alive.

KVMGT write-protects guest pages through device->kvm while the guest
runs, in intel_gvt_page_track_add() and intel_gvt_page_track_remove().
The list of readers is right, only "in open_device()" is too narrow.
That came from item 1 of my v2 reply, which was wrong for KVMGT.

> +	 * Set before registration by a driver that reads device->kvm in
> +	 * open_device(): the core then keeps the KVM the device was opened
> +	 * with referenced until close_device(). Without it device->kvm
> +	 * stays NULL.

The same for the comment: a future driver that reads the pointer only
from an ioctl would follow "in open_device()", leave the flag unset and
get NULL.

The two open paths still say the pointer is always saved for drivers,
which is no longer true without the flag.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v3 0/1] vfio: kvm reference leak on live update

[Konstantin Khorenko]

On Thu, 05 Nov 2026 16:17:58 +0200, Andrey Drobyshev wrote:
> vfio: kvm reference leak on live update
> 
> A live update keeps the passthrough device open while the VM is
> re-created. The device holds a reference to the kvm it was opened
> with, so the first VM is never destroyed until the device closes. Take
> that reference only for the drivers that read device->kvm.
> 
> [...]

Applied to branch-rh10-6.12.0-211.52.1.22.x.vz10-ovz, will be released in rh10-6.12.0-211.52.1.22.19.vz10, thanks!

[1/1] vfio: take the kvm reference only for drivers that read device->kvm
      https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/d83d59385873

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel