Virtuozzo QEMU development (svt-core@virtuozzo.com)
 help / color / mirror / Atom feed
* [PATCH hci-8.1 v5 0/5] vhost-blk: switch from VHOST_SET_NWORKERS to the mainstream worker ioctls
@ 2026-10-31 15:35 Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 1/5] vhost-backend: add vhost_set_fork_from_owner op Alexander Ivanov
                   ` (4 more replies)
  0 siblings, 5 replies; 7+ messages in thread
From: Alexander Ivanov @ 2026-10-31 15:35 UTC (permalink / raw)
  To: andrey.zhadchenko; +Cc: andrey.drobyshev, svt-core

The kernel drops our VHOST_SET_NWORKERS ioctl in favour of the
mainstream VHOST_NEW_WORKER/VHOST_ATTACH_VRING_WORKER, which vhost-blk
now supports.

Patch 1 wraps VHOST_SET_FORK_FROM_OWNER in a vhost backend op. Patch 2
refuses num-queues above VHOST_BLK_MAX_QUEUES at realize. Patch 3 is a
cleanup taking &s->dev into a local pointer. Patch 4 makes vhost-blk
create its num-threads workers with the mainstream ioctls, keeping the
same worker layout, and fail to start if the kernel cannot provide
them. Patch 5 drops VHOST_SET_NWORKERS from linux-headers.

v5:
- patch 4: fail to start vhost-blk on any worker ioctl failure instead
  of warning and running on fewer threads than requested
- patch 4: switch to vhost_task mode for num-threads=1 as well, so the
  default thread is of the same kind whatever the number of workers

v4:
- new patch 3: split the &s->dev -> hdev conversion out of patch 4
- patch 2: quote the error realize fails with, add Fixes:
- patch 4: do not key on the errno; on any worker ioctl failure warn
  and continue with the threads created so far instead of failing
  to start vhost-blk
- patch 4: drop the comments, explain in the commit message instead

v3:
- new patch 1: call VHOST_SET_FORK_FROM_OWNER, VHOST_NEW_WORKER and
  VHOST_ATTACH_VRING_WORKER through vhost_ops instead of raw ioctls
- new patch 2: refuse num-queues above VHOST_BLK_MAX_QUEUES at realize
  instead of clamping num-threads in patch 3
- patch 3: create each worker right before attaching its first vq

v2:
- patch 1: clamp num-threads to VHOST_BLK_MAX_QUEUES
- patch 1: note in the commit message that our kernel, unlike
  mainline, defaults to kthreads

https://virtuozzo.atlassian.net/browse/VSTOR-107537

Alexander Ivanov (5):
  vhost-backend: add vhost_set_fork_from_owner op
  vhost-blk: reject num-queues above VHOST_BLK_MAX_QUEUES
  vhost-blk: use a local vhost_dev pointer in start and stop
  vhost-blk: create kernel threads with the mainstream worker ioctls
  linux-headers: drop VHOST_SET_NWORKERS

 hw/block/vhost-blk.c              | 95 ++++++++++++++++++++++++-------
 hw/virtio/vhost-backend.c         |  7 +++
 include/hw/virtio/vhost-backend.h |  3 +
 linux-headers/linux/vhost.h       |  9 ---
 4 files changed, 86 insertions(+), 28 deletions(-)

-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH hci-8.1 v5 1/5] vhost-backend: add vhost_set_fork_from_owner op
  2026-10-31 15:35 [PATCH hci-8.1 v5 0/5] vhost-blk: switch from VHOST_SET_NWORKERS to the mainstream worker ioctls Alexander Ivanov
@ 2026-10-31 15:35 ` Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 2/5] vhost-blk: reject num-queues above VHOST_BLK_MAX_QUEUES Alexander Ivanov
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Alexander Ivanov @ 2026-10-31 15:35 UTC (permalink / raw)
  To: andrey.zhadchenko; +Cc: andrey.drobyshev, svt-core

Wrap VHOST_SET_FORK_FROM_OWNER for the kernel backend next to the
worker ops, so devices switch the vhost worker mode the same way they
create and attach workers.

https://virtuozzo.atlassian.net/browse/VSTOR-107537

Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
Reviewed-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/virtio/vhost-backend.c         | 7 +++++++
 include/hw/virtio/vhost-backend.h | 3 +++
 2 files changed, 10 insertions(+)

diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
index 1447d129630..7f8c731a607 100644
--- a/hw/virtio/vhost-backend.c
+++ b/hw/virtio/vhost-backend.c
@@ -187,6 +187,12 @@ static int vhost_kernel_get_vring_worker(struct vhost_dev *dev,
     return vhost_kernel_call(dev, VHOST_GET_VRING_WORKER, worker);
 }
 
+static int vhost_kernel_set_fork_from_owner(struct vhost_dev *dev,
+                                            uint8_t fork_owner)
+{
+    return vhost_kernel_call(dev, VHOST_SET_FORK_FROM_OWNER, &fork_owner);
+}
+
 static int vhost_kernel_set_backend_cap(struct vhost_dev *dev)
 {
     uint64_t features;
@@ -386,6 +392,7 @@ const VhostOps kernel_ops = {
         .vhost_attach_vring_worker = vhost_kernel_attach_vring_worker,
         .vhost_new_worker = vhost_kernel_new_worker,
         .vhost_free_worker = vhost_kernel_free_worker,
+        .vhost_set_fork_from_owner = vhost_kernel_set_fork_from_owner,
         .vhost_set_features_ex = vhost_kernel_set_features,
         .vhost_get_features_ex = vhost_kernel_get_features,
         .vhost_set_backend_cap = vhost_kernel_set_backend_cap,
diff --git a/include/hw/virtio/vhost-backend.h b/include/hw/virtio/vhost-backend.h
index 18ce5ea9a07..78cd66378ef 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -95,6 +95,8 @@ typedef int (*vhost_new_worker_op)(struct vhost_dev *dev,
                                    struct vhost_worker_state *worker);
 typedef int (*vhost_free_worker_op)(struct vhost_dev *dev,
                                     struct vhost_worker_state *worker);
+typedef int (*vhost_set_fork_from_owner_op)(struct vhost_dev *dev,
+                                            uint8_t fork_owner);
 typedef int (*vhost_set_features_ex_op)(struct vhost_dev *dev,
                                         const uint64_t *features);
 typedef int (*vhost_get_features_ex_op)(struct vhost_dev *dev,
@@ -190,6 +192,7 @@ typedef struct VhostOps {
     vhost_free_worker_op vhost_free_worker;
     vhost_get_vring_worker_op vhost_get_vring_worker;
     vhost_attach_vring_worker_op vhost_attach_vring_worker;
+    vhost_set_fork_from_owner_op vhost_set_fork_from_owner;
     vhost_set_features_ex_op vhost_set_features_ex;
     vhost_get_features_ex_op vhost_get_features_ex;
     vhost_set_features_op vhost_set_features;
-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH hci-8.1 v5 2/5] vhost-blk: reject num-queues above VHOST_BLK_MAX_QUEUES
  2026-10-31 15:35 [PATCH hci-8.1 v5 0/5] vhost-blk: switch from VHOST_SET_NWORKERS to the mainstream worker ioctls Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 1/5] vhost-backend: add vhost_set_fork_from_owner op Alexander Ivanov
@ 2026-10-31 15:35 ` Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 3/5] vhost-blk: use a local vhost_dev pointer in start and stop Alexander Ivanov
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 7+ messages in thread
From: Alexander Ivanov @ 2026-10-31 15:35 UTC (permalink / raw)
  To: andrey.zhadchenko; +Cc: andrey.drobyshev, svt-core

Only the automatic num-queues value is capped at VHOST_BLK_MAX_QUEUES,
an explicit one is taken as is. The kernel vhost-blk device has that
many virtqueues and rejects any vq beyond them, so realize fails with
an error that does not point at num-queues:

  qemu-system-x86_64: vhost_set_vring_call failed: No buffer space available (105)
  qemu-system-x86_64: vhost-blk: vhost initialization failed: No buffer space available

Check num-queues against the limit up front instead.

https://virtuozzo.atlassian.net/browse/VSTOR-107537

Fixes: 0ff3cb5ff28f ("block: add vhost-blk backend")
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
Reviewed-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 hw/block/vhost-blk.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/hw/block/vhost-blk.c b/hw/block/vhost-blk.c
index abddff02c6d..8a015d54fa7 100644
--- a/hw/block/vhost-blk.c
+++ b/hw/block/vhost-blk.c
@@ -598,8 +598,9 @@ static void vhost_blk_device_realize(DeviceState *dev, Error **errp)
                                VHOST_BLK_MAX_QUEUES);
     }
 
-    if (!conf->num_queues) {
-        error_setg(errp, "vhost-blk: num-queues property must be larger than 0");
+    if (!conf->num_queues || conf->num_queues > VHOST_BLK_MAX_QUEUES) {
+        error_setg(errp, "vhost-blk: invalid num-queues property (%" PRIu16 "), "
+                   "must be 1-%d", conf->num_queues, VHOST_BLK_MAX_QUEUES);
         return;
     }
 
-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH hci-8.1 v5 3/5] vhost-blk: use a local vhost_dev pointer in start and stop
  2026-10-31 15:35 [PATCH hci-8.1 v5 0/5] vhost-blk: switch from VHOST_SET_NWORKERS to the mainstream worker ioctls Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 1/5] vhost-backend: add vhost_set_fork_from_owner op Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 2/5] vhost-blk: reject num-queues above VHOST_BLK_MAX_QUEUES Alexander Ivanov
@ 2026-10-31 15:35 ` Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 4/5] vhost-blk: create kernel threads with the mainstream worker ioctls Alexander Ivanov
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 5/5] linux-headers: drop VHOST_SET_NWORKERS Alexander Ivanov
  4 siblings, 0 replies; 7+ messages in thread
From: Alexander Ivanov @ 2026-10-31 15:35 UTC (permalink / raw)
  To: andrey.zhadchenko; +Cc: andrey.drobyshev, svt-core

Take &s->dev once into hdev in vhost_blk_start() and vhost_blk_stop()
instead of spelling it out at every use. No functional change.

https://virtuozzo.atlassian.net/browse/VSTOR-107537

Fixes: 0ff3cb5ff28f ("block: add vhost-blk backend")
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
---
 hw/block/vhost-blk.c | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/hw/block/vhost-blk.c b/hw/block/vhost-blk.c
index 8a015d54fa7..b3eb181ceae 100644
--- a/hw/block/vhost-blk.c
+++ b/hw/block/vhost-blk.c
@@ -36,6 +36,7 @@
 static int vhost_blk_start(VirtIODevice *vdev)
 {
     VHostBlk *s = VHOST_BLK(vdev);
+    struct vhost_dev *hdev = &s->dev;
     struct vhost_vring_file backend;
     int ret, i, nworkers;
     BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
@@ -70,22 +71,22 @@ static int vhost_blk_start(VirtIODevice *vdev)
         }
     }
 
-    ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, true);
+    ret = k->set_guest_notifiers(qbus->parent, hdev->nvqs, true);
     if (ret < 0) {
         error_report("vhost-blk: unable to bind guest notifiers");
         return ret;
     }
 
-    ret = vhost_dev_enable_notifiers(&s->dev, vdev);
+    ret = vhost_dev_enable_notifiers(hdev, vdev);
     if (ret < 0) {
         error_report("vhost-blk: unable to enable dev notifiers: %s",
                      strerror(errno));
         goto err_set;
     }
 
-    s->dev.acked_features = vdev->guest_features & s->dev.features;
+    hdev->acked_features = vdev->guest_features & hdev->features;
 
-    ret = vhost_dev_start(&s->dev, vdev, false);
+    ret = vhost_dev_start(hdev, vdev, false);
     if (ret < 0) {
         error_report("vhost-blk: unable to start vhost dev");
         goto err_disable;
@@ -100,11 +101,11 @@ static int vhost_blk_start(VirtIODevice *vdev)
         goto err_stop;
     }
 
-    for (i = 0; i < s->dev.nvqs; i++) {
-        vhost_virtqueue_mask(&s->dev, vdev, i, false);
+    for (i = 0; i < hdev->nvqs; i++) {
+        vhost_virtqueue_mask(hdev, vdev, i, false);
     }
 
-    for (i = 0; i < s->dev.nvqs; i++) {
+    for (i = 0; i < hdev->nvqs; i++) {
         event_notifier_set(virtio_queue_get_host_notifier(virtio_get_queue(vdev, i)));
     }
 
@@ -113,11 +114,11 @@ static int vhost_blk_start(VirtIODevice *vdev)
     return 0;
 
 err_stop:
-    vhost_dev_stop(&s->dev, vdev, false);
+    vhost_dev_stop(hdev, vdev, false);
 err_disable:
-    vhost_dev_disable_notifiers(&s->dev, vdev);
+    vhost_dev_disable_notifiers(hdev, vdev);
 err_set:
-    ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
+    ret = k->set_guest_notifiers(qbus->parent, hdev->nvqs, false);
     if (ret < 0) {
         error_report("vhost-blk: unable to unbind guest notifiers");
     }
@@ -131,6 +132,7 @@ static void vhost_blk_stop(VirtIODevice *vdev)
     BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
     VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
     VHostBlk *s = VHOST_BLK(vdev);
+    struct vhost_dev *hdev = &s->dev;
     struct vhost_vring_file backend;
     int ret;
 
@@ -138,11 +140,11 @@ static void vhost_blk_stop(VirtIODevice *vdev)
         return;
     }
 
-    ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false);
+    ret = k->set_guest_notifiers(qbus->parent, hdev->nvqs, false);
     if (ret < 0) {
         error_report("vhost-blk: unable to unbind guest notifiers");
     }
-    vhost_dev_disable_notifiers(&s->dev, vdev);
+    vhost_dev_disable_notifiers(hdev, vdev);
 
     /*
      * Detach the backend before vhost_dev_stop() samples the vring base,
@@ -156,7 +158,7 @@ static void vhost_blk_stop(VirtIODevice *vdev)
     ret = ioctl(s->vhostfd, VHOST_BLK_SET_BACKEND, &backend);
     assert(ret == 0);
 
-    vhost_dev_stop(&s->dev, vdev, false);
+    vhost_dev_stop(hdev, vdev, false);
 
     s->vhost_started = false;
 }
-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH hci-8.1 v5 4/5] vhost-blk: create kernel threads with the mainstream worker ioctls
  2026-10-31 15:35 [PATCH hci-8.1 v5 0/5] vhost-blk: switch from VHOST_SET_NWORKERS to the mainstream worker ioctls Alexander Ivanov
                   ` (2 preceding siblings ...)
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 3/5] vhost-blk: use a local vhost_dev pointer in start and stop Alexander Ivanov
@ 2026-10-31 15:35 ` Alexander Ivanov
  2026-11-02 16:26   ` Andrey Drobyshev
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 5/5] linux-headers: drop VHOST_SET_NWORKERS Alexander Ivanov
  4 siblings, 1 reply; 7+ messages in thread
From: Alexander Ivanov @ 2026-10-31 15:35 UTC (permalink / raw)
  To: andrey.zhadchenko; +Cc: andrey.drobyshev, svt-core

num-threads was passed to our own VHOST_SET_NWORKERS kernel ioctl. The
kernel drops it with the revert of "drivers/vhost: add ioctl to
increase the number of workers" in favour of the mainstream worker
ioctls, which vhost-blk now supports. Create num-threads - 1 workers
with VHOST_NEW_WORKER and attach them to the virtqueues round-robin
with VHOST_ATTACH_VRING_WORKER, the layout VHOST_SET_NWORKERS produced.

Our kernel, unlike mainline, defaults to kthread mode, where
VHOST_NEW_WORKER fails with -EFAULT rather than -ENOTTY. So switch the
device to vhost_task mode with VHOST_SET_FORK_FROM_OWNER before
VHOST_SET_OWNER. Do it for num-threads=1 as well, so the default
thread is of the same kind whatever the number of workers.

Fail to start vhost-blk if any of these ioctls fails rather than
silently run on fewer threads than requested. Released vz10 kernels
have VHOST_SET_FORK_FROM_OWNER, but their vhost-blk does not handle
the worker ioctls, so num-threads above 1 now requires a kernel that
does.

https://virtuozzo.atlassian.net/browse/VSTOR-107537

Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
---
 hw/block/vhost-blk.c | 62 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 58 insertions(+), 4 deletions(-)

diff --git a/hw/block/vhost-blk.c b/hw/block/vhost-blk.c
index b3eb181ceae..3cde12051fb 100644
--- a/hw/block/vhost-blk.c
+++ b/hw/block/vhost-blk.c
@@ -33,6 +33,50 @@
 #include "migration/cpr.h"
 #include "system/runstate.h"
 
+/*
+ * Spread the virtqueues round-robin over nworkers kernel threads. Thread 0 is
+ * the default worker VHOST_SET_OWNER created; vq 0 and every nworkers-th vq
+ * after it stay on it.
+ */
+static int vhost_blk_set_workers(struct vhost_dev *hdev, int nworkers)
+{
+    uint32_t ids[VHOST_BLK_MAX_QUEUES];
+    int i, ret;
+
+    assert(nworkers <= ARRAY_SIZE(ids));
+
+    for (i = 1; i < hdev->nvqs; i++) {
+        int w = i % nworkers;
+        struct vhost_vring_worker vq_worker = { .index = i };
+
+        if (w == 0) {
+            continue;
+        }
+
+        if (i < nworkers) {
+            struct vhost_worker_state worker = { };
+
+            ret = hdev->vhost_ops->vhost_new_worker(hdev, &worker);
+            if (ret < 0) {
+                error_report("vhost-blk: unable to create kernel thread: %s",
+                             strerror(-ret));
+                return ret;
+            }
+            ids[w] = worker.worker_id;
+        }
+
+        vq_worker.worker_id = ids[w];
+        ret = hdev->vhost_ops->vhost_attach_vring_worker(hdev, &vq_worker);
+        if (ret < 0) {
+            error_report("vhost-blk: unable to attach kernel thread to "
+                         "queue %d: %s", i, strerror(-ret));
+            return ret;
+        }
+    }
+
+    return 0;
+}
+
 static int vhost_blk_start(VirtIODevice *vdev)
 {
     VHostBlk *s = VHOST_BLK(vdev);
@@ -52,15 +96,25 @@ static int vhost_blk_start(VirtIODevice *vdev)
         return 0;
     }
 
+    ret = hdev->vhost_ops->vhost_set_fork_from_owner(hdev,
+                                                     VHOST_FORK_OWNER_TASK);
+    if (ret < 0) {
+        error_report("vhost-blk: unable to set vhost_task mode: %s",
+                     strerror(-ret));
+        return ret;
+    }
+
     if (ioctl(s->vhostfd, VHOST_SET_OWNER, NULL)) {
         error_report("vhost-blk: unable to set owner");
         return -ENOSYS;
     }
 
-    nworkers = s->conf.num_threads;
-    if (nworkers != 1 && ioctl(s->vhostfd, VHOST_SET_NWORKERS, &nworkers)) {
-        error_report("vhost-blk: unable to set number of kernel threads");
-        return -EINVAL;
+    nworkers = MIN(s->conf.num_threads, hdev->nvqs);
+    if (nworkers > 1) {
+        ret = vhost_blk_set_workers(hdev, nworkers);
+        if (ret < 0) {
+            return ret;
+        }
     }
 
     if (s->conf.serial) {
-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* [PATCH hci-8.1 v5 5/5] linux-headers: drop VHOST_SET_NWORKERS
  2026-10-31 15:35 [PATCH hci-8.1 v5 0/5] vhost-blk: switch from VHOST_SET_NWORKERS to the mainstream worker ioctls Alexander Ivanov
                   ` (3 preceding siblings ...)
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 4/5] vhost-blk: create kernel threads with the mainstream worker ioctls Alexander Ivanov
@ 2026-10-31 15:35 ` Alexander Ivanov
  4 siblings, 0 replies; 7+ messages in thread
From: Alexander Ivanov @ 2026-10-31 15:35 UTC (permalink / raw)
  To: andrey.zhadchenko; +Cc: andrey.drobyshev, svt-core

vhost-blk no longer uses it, and the kernel drops the ioctl.

https://virtuozzo.atlassian.net/browse/VSTOR-107537

Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
Reviewed-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
---
 linux-headers/linux/vhost.h | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/linux-headers/linux/vhost.h b/linux-headers/linux/vhost.h
index baad06412d4..b6eb628d973 100644
--- a/linux-headers/linux/vhost.h
+++ b/linux-headers/linux/vhost.h
@@ -102,15 +102,6 @@
 #define VHOST_GET_VRING_WORKER _IOWR(VHOST_VIRTIO, 0x16,		\
 				     struct vhost_vring_worker)
 
-/* Set number of vhost workers
- * Currently number of vhost workers can only be increased.
- * All workers are freed upon reset.
- * If the value is too big it is silently truncated to the maximum number of
- * supported vhost workers
- * Even if the error is returned it is possible that some workers were created
- */
-#define VHOST_SET_NWORKERS _IOW(VHOST_VIRTIO, 0x1F, int)
-
 /* The following ioctls use eventfd file descriptors to signal and poll
  * for events. */
 
-- 
2.53.0


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [PATCH hci-8.1 v5 4/5] vhost-blk: create kernel threads with the mainstream worker ioctls
  2026-10-31 15:35 ` [PATCH hci-8.1 v5 4/5] vhost-blk: create kernel threads with the mainstream worker ioctls Alexander Ivanov
@ 2026-11-02 16:26   ` Andrey Drobyshev
  0 siblings, 0 replies; 7+ messages in thread
From: Andrey Drobyshev @ 2026-11-02 16:26 UTC (permalink / raw)
  To: Alexander Ivanov, andrey.zhadchenko; +Cc: svt-core

On 10/31/26 5:35 PM, Alexander Ivanov wrote:
> num-threads was passed to our own VHOST_SET_NWORKERS kernel ioctl. The
> kernel drops it with the revert of "drivers/vhost: add ioctl to
> increase the number of workers" in favour of the mainstream worker
> ioctls, which vhost-blk now supports. Create num-threads - 1 workers
> with VHOST_NEW_WORKER and attach them to the virtqueues round-robin
> with VHOST_ATTACH_VRING_WORKER, the layout VHOST_SET_NWORKERS produced.
> 
> Our kernel, unlike mainline, defaults to kthread mode, where
> VHOST_NEW_WORKER fails with -EFAULT rather than -ENOTTY. So switch the
> device to vhost_task mode with VHOST_SET_FORK_FROM_OWNER before
> VHOST_SET_OWNER. Do it for num-threads=1 as well, so the default
> thread is of the same kind whatever the number of workers.
> 
> Fail to start vhost-blk if any of these ioctls fails rather than
> silently run on fewer threads than requested. Released vz10 kernels
> have VHOST_SET_FORK_FROM_OWNER, but their vhost-blk does not handle
> the worker ioctls, so num-threads above 1 now requires a kernel that
> does.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-107537
> 
> Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
> ---
>  hw/block/vhost-blk.c | 62 +++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 58 insertions(+), 4 deletions(-)
> 
> diff --git a/hw/block/vhost-blk.c b/hw/block/vhost-blk.c
> index b3eb181ceae..3cde12051fb 100644
> --- a/hw/block/vhost-blk.c
> +++ b/hw/block/vhost-blk.c
> @@ -33,6 +33,50 @@
>  #include "migration/cpr.h"
>  #include "system/runstate.h"
>  
> +/*
> + * Spread the virtqueues round-robin over nworkers kernel threads. Thread 0 is
> + * the default worker VHOST_SET_OWNER created; vq 0 and every nworkers-th vq
> + * after it stay on it.
> + */
> +static int vhost_blk_set_workers(struct vhost_dev *hdev, int nworkers)
> +{
> +    uint32_t ids[VHOST_BLK_MAX_QUEUES];
> +    int i, ret;
> +
> +    assert(nworkers <= ARRAY_SIZE(ids));
> +
> +    for (i = 1; i < hdev->nvqs; i++) {
> +        int w = i % nworkers;
> +        struct vhost_vring_worker vq_worker = { .index = i };
> +
> +        if (w == 0) {
> +            continue;
> +        }
> +
> +        if (i < nworkers) {
> +            struct vhost_worker_state worker = { };
> +
> +            ret = hdev->vhost_ops->vhost_new_worker(hdev, &worker);
> +            if (ret < 0) {
> +                error_report("vhost-blk: unable to create kernel thread: %s",
> +                             strerror(-ret));
> +                return ret;
> +            }
> +            ids[w] = worker.worker_id;
> +        }
> +
> +        vq_worker.worker_id = ids[w];
> +        ret = hdev->vhost_ops->vhost_attach_vring_worker(hdev, &vq_worker);
> +        if (ret < 0) {
> +            error_report("vhost-blk: unable to attach kernel thread to "
> +                         "queue %d: %s", i, strerror(-ret));
> +            return ret;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
>  static int vhost_blk_start(VirtIODevice *vdev)
>  {
>      VHostBlk *s = VHOST_BLK(vdev);
> @@ -52,15 +96,25 @@ static int vhost_blk_start(VirtIODevice *vdev)
>          return 0;
>      }
>  
> +    ret = hdev->vhost_ops->vhost_set_fork_from_owner(hdev,
> +                                                     VHOST_FORK_OWNER_TASK);
> +    if (ret < 0) {
> +        error_report("vhost-blk: unable to set vhost_task mode: %s",
> +                     strerror(-ret));
> +        return ret;
> +    }

Note, with this newly introduced strict retval checking:

> long vhost_dev_ioctl()
> {
>     if (ioctl == VHOST_SET_OWNER) { ... }
> 
> #ifdef CONFIG_VHOST_ENABLE_FORK_OWNER_CONTROL
>     if (ioctl == VHOST_SET_FORK_FROM_OWNER) { ... }

That means any vhost-blk disk will fail to start on an (older) kernel
which lacks VHOST_SET_FORK_FROM_OWNER.

This shouldn't be an issue for us, but still worth keeping in mind.
E.g. our vzkernel 6.12.0-55.* didn't have this setting AFAICT.
> +
>      if (ioctl(s->vhostfd, VHOST_SET_OWNER, NULL)) {
>          error_report("vhost-blk: unable to set owner");
>          return -ENOSYS;
>      }
>  
> -    nworkers = s->conf.num_threads;
> -    if (nworkers != 1 && ioctl(s->vhostfd, VHOST_SET_NWORKERS, &nworkers)) {
> -        error_report("vhost-blk: unable to set number of kernel threads");
> -        return -EINVAL;
> +    nworkers = MIN(s->conf.num_threads, hdev->nvqs);
> +    if (nworkers > 1) {
> +        ret = vhost_blk_set_workers(hdev, nworkers);
> +        if (ret < 0) {
> +            return ret;

Pre-existing, but each return after VHOST_SET_OWNER above leaves the
device in a stale state.  This might deserve another fix, separately
from this patch.
> +        }
>      }
>  
>      if (s->conf.serial) {

LGTM overall.

Reviewed-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2026-11-02 16:26 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-31 15:35 [PATCH hci-8.1 v5 0/5] vhost-blk: switch from VHOST_SET_NWORKERS to the mainstream worker ioctls Alexander Ivanov
2026-10-31 15:35 ` [PATCH hci-8.1 v5 1/5] vhost-backend: add vhost_set_fork_from_owner op Alexander Ivanov
2026-10-31 15:35 ` [PATCH hci-8.1 v5 2/5] vhost-blk: reject num-queues above VHOST_BLK_MAX_QUEUES Alexander Ivanov
2026-10-31 15:35 ` [PATCH hci-8.1 v5 3/5] vhost-blk: use a local vhost_dev pointer in start and stop Alexander Ivanov
2026-10-31 15:35 ` [PATCH hci-8.1 v5 4/5] vhost-blk: create kernel threads with the mainstream worker ioctls Alexander Ivanov
2026-11-02 16:26   ` Andrey Drobyshev
2026-10-31 15:35 ` [PATCH hci-8.1 v5 5/5] linux-headers: drop VHOST_SET_NWORKERS Alexander Ivanov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox