From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from AM0PR02CU008.outbound.protection.outlook.com (mail-westeuropeazon11023089.outbound.protection.outlook.com [52.101.72.89]) by lore.virtuozzo.com (Postfix) with ESMTPS id A3B8B80EAD for ; Wed, 28 Oct 2026 12:33:41 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kHSwNAobQkoXpBMCfXvOaae2OxvhvHZAg5ISeObZxbS8EH9H8U3s1pUKVBwfbpXoUtL03RlVvh8So4Fp7JDpT9f1eM/fu1uZXyndJx5ZFVqOEYcWHPtlB5JtJc7fCYjpRbqCcUo77dHP9s5Xpyxlo9E2phMxNgFYAoGx/x1GKIn7zgJL6IYQ/5JrMmRhaa5OCJgJ6xoqM1GhmToiU16FchvYnyQZwNRSNNhMiPFex730m45c8A0+dk9FJoi3EVW4OTg4vcA/eOveBUhiwG3YA1wk8CFnSDINpF4R6lJIB0yPuKmsgX9iG7RbxytvT21NISsV5Hl4wLAQaocwYA3TBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=74ugv/wGpry4PTSl+tLE8R9nOA3Q5Fei4e7mIImA5z8=; b=LGM+VkZn8/PhCzqPx+lBdPjg1h7g/riA2+tbbv4cILZiX0O1QycvshsKjDrFxoqD4my+ybbTRMJGql+LeD1IfMPbOZv4Qw0QsJ2ny2BWc8yCw/fwblM0kbdtwe8QViwWvFCGo361Nqd3nRhLuPYYCt241Mgb+ZfsnXO+AOSOUsCg/V0QEJ8kJjyJxpKYPfa7JBZtLZjQzkhgAwwlVW3ubX3VoOi1W8lXdWEtvAcEQebOYe5psSQAySsfNFIoOhBgnOkf/YnLX9oJhJOhn1KuxKA096iSz8S0amc+cvgV3nymNL46BjsUdTxngXXYtl1kbgFkKrLeJJhYBK1aYuDNFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 130.117.225.111) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=virtuozzo.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=virtuozzo.com; dkim=pass (signature was verified) header.d=virtuozzo.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=74ugv/wGpry4PTSl+tLE8R9nOA3Q5Fei4e7mIImA5z8=; b=DySWzx48/Om7NhMYUed67rYsQ3z+bjNFeCOtX/8KmPtIkfpgHr1lLaXN84J41BCYJyK4g2rihn9gKb8tU5XFeY9PcrR3DBcaIHBJtpQWKkrl5ahbT+NgUE6TdatkzqbRDKK67MulligsUjEfTfs39zUlA9zwqk1ApWaM8FdrEDNr/d2ueAzipiZ7P/N9Qhob6JT91y7MBTu+TH0u/8d8c2ub1rcOc2Kric3d00S9l+9lgyxaX5tW+ifntIW/ZB1m6r8BLuLxyKhRbJ5UZ5rxJCn0KQc6cOCvGU5PsDyI6DRlk7f3Opmukg+piHEscF41TTQMd/0vbaH4IGam0RDvyw== Received: from AS4P251CA0019.EURP251.PROD.OUTLOOK.COM (2603:10a6:20b:5d3::7) by BR2PR08MB999907.eurprd08.prod.outlook.com (2603:10a6:2c8:1d::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.546.22; Wed, 28 Oct 2026 12:33:37 +0000 Received: from AMS1EPF0000009D.eurprd02.prod.outlook.com (2603:10a6:20b:5d3:cafe::60) by AS4P251CA0019.outlook.office365.com (2603:10a6:20b:5d3::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.546.22 via Frontend Transport; Wed, 28 Oct 2026 12:33:37 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 130.117.225.111) smtp.mailfrom=virtuozzo.com; dkim=pass (signature was verified) header.d=virtuozzo.com;dmarc=pass action=none header.from=virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of virtuozzo.com designates 130.117.225.111 as permitted sender) receiver=protection.outlook.com; client-ip=130.117.225.111; helo=relay.virtuozzo.com; pr=C Received: from relay.virtuozzo.com (130.117.225.111) by AMS1EPF0000009D.mail.protection.outlook.com (10.167.242.74) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.569.4 via Frontend Transport; Wed, 28 Oct 2026 12:33:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-ID:Date:Subject:From: Content-Type; bh=74ugv/wGpry4PTSl+tLE8R9nOA3Q5Fei4e7mIImA5z8=; b=oIfEuQL33IPo zFrmUKALMMFyeOKa4/GpPLMd8W9A9aDfMWL+zxgGAZ3M/EwECypxUqBgms+SA5uhwbB48U6H8jjey 9KOPqSXmtH+tvSA/105JpxE1dYnaf0yuUovI0efJNzVesQcozjeWkfxkQINGlvwIzZIwydWniV1EC ma1G4NeYuEG2V5TuamTD7S3MKFfd8z3D1xHVaZojew+dSGeFp30UUerw8QNfrjQu0J5iK12/zGlWS wJfg8SRZLj5V5MStIDIto60TBsOIb2GQxckUzYLZ+CP4p0RtzD3r5R+Mg/i6FFGK/AEc4PiVtacuE FvSVwvfI16Y2ABxUMcNiSQ==; Received: from ch-vpn.virtuozzo.com ([130.117.225.6] helo=sky) by relay.virtuozzo.com with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1xM2ml-007Thg-0B; Wed, 28 Oct 2026 13:33:36 +0100 From: Alexander Ivanov To: andrey.zhadchenko@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com, svt-core@virtuozzo.com Subject: [PATCH hci-8.1 v3 3/4] vhost-blk: create kernel threads with the mainstream worker ioctls Date: Wed, 28 Oct 2026 13:33:32 +0100 Message-ID: <20261028123333.3145676-4-alexander.ivanov@virtuozzo.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20261028123333.3145676-1-alexander.ivanov@virtuozzo.com> References: <20261028123333.3145676-1-alexander.ivanov@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS1EPF0000009D:EE_|BR2PR08MB999907:EE_ X-MS-Office365-Filtering-Correlation-Id: 12c2e193-cbe7-4954-3569-08df34efb0a5 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|36860700016|376014|82310400026|1800799024|18002099003|5113699003|22082099003|18092099006|261026140227099003|261021123027099003|261021121427099003|261009221227099003|56012099006|5023799004|261009223027099003|10067099003|261022105027099003|261022075727099003|261026121527099003; X-Microsoft-Antispam-Message-Info: VtLo5VMoeHu5coUlKaS0mc9128xZpqx5JDZpJAcp72lITt9Yti7Ka5TtwtmkBSLVOvlZ1K7OZuhvl/WQkXhUZxfwZCvdi/PkG50hc+iM3WnCjeura0a+VjfMFO4zTwMOSZbZz1paQy+dV6xk1KmaGG/Of8zCFjKhcnDTJ7mZX57jysYta4ES8IjmzoJ5OPIWlxltvuLT6oOm0305O2szEMdAxoQVgh5s2AM/w1KEyEe46gKmgmVDBX1qNWqh80A/LxXmzycAqNhTvk3TURbBu8xiKUQOiONhFHkckVauUfBDrlfj7emxTfix9820pG4UCB7gNdvVW5f1Gkw3vILdazpVZzmhuANdRIqXq3bf8bl4ieJHMI0hXUjGxUqO5hBo2rd5vGJ3y9ehAeNEE8HEK7pXkWUt2jURCmMnIR2kjSxWfhm0SZwmQuEYwPzPEsswRGctzP0snBVQCSj5rEbDD/MiliU1njcBsUGaFxvdADU8aLz5BhUeFuEfhrFOQNnefmeDUs1+wZKdyf3IsOlccGbwLShD6wb004j+c9nWmykX53uwxZraSpMBxATeSINHq3AL8HVuPR2N41Zrvxp4GwOgFhIFr02kvcAFEI+vpmAdObgJ47zOpYoPAVwffqp3azWf+d74ktBz7TjeM6MZQQ== X-Forefront-Antispam-Report: CIP:130.117.225.111;CTRY:CH;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:relay.virtuozzo.com;PTR:relay.virtuozzo.com;CAT:NONE;SFS:(13230040)(23010399003)(36860700016)(376014)(82310400026)(1800799024)(18002099003)(5113699003)(22082099003)(18092099006)(261026140227099003)(261021123027099003)(261021121427099003)(261009221227099003)(56012099006)(5023799004)(261009223027099003)(10067099003)(261022105027099003)(261022075727099003)(261026121527099003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: di6RR1FHFoFlcU+im2U1xw8wSufnMGnXwTw7WiAKIlYt+ter5pheLj/n8uCkOa93NVynUrlGr1wmdzwFQE1A12fphgLwAFxp5dffJjBDuIMDbcvUnQQGI9ksZMh8vzKshmGwR9Qjeft2uOcC8Rz/SvKxGpR8A97QJf8DZeU1lh6GmdsgqxLUXtIOxHrH3BV1ZQxD5/DEsEKy/pNsmkzFOsVCm8vJTAOWRP2cUxpGmTpkKYNPXHdkMaxIPvd4itYedNsyYrWo5W4EwHWfiY4x8dwEzWEJ/3oPx66/GpYxYU6j2UlreGlbLg6ltRTW24zL1cWjzgrLeYpucZWN7mDXRLIWZ/+oWk9/6AfHXG2/BoGtI8P+dqWD7ufl59n29TQpLOQyglXGNwoJfiQAVapLj5hX4ORcz8XleNqDaxHfyNiQ5ahE8xC+J+E0pLkp+IbA X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Oct 2026 12:33:37.5301 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 12c2e193-cbe7-4954-3569-08df34efb0a5 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=0bc7f26d-0264-416e-a6fc-8352af79c58f;Ip=[130.117.225.111];Helo=[relay.virtuozzo.com] X-MS-Exchange-CrossTenant-AuthSource: AMS1EPF0000009D.eurprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BR2PR08MB999907 num-threads was passed to our own VHOST_SET_NWORKERS kernel ioctl. The kernel drops it with the revert of "drivers/vhost: add ioctl to increase the number of workers" in favour of the mainstream worker ioctls, which vhost-blk now supports. Create num-threads - 1 workers with VHOST_NEW_WORKER and attach them to the virtqueues round-robin with VHOST_ATTACH_VRING_WORKER, the layout VHOST_SET_NWORKERS produced. Our kernel, unlike mainline, defaults to kthread mode, where VHOST_NEW_WORKER fails with -EFAULT rather than -ENOTTY. So switch the device to vhost_task mode with VHOST_SET_FORK_FROM_OWNER before VHOST_SET_OWNER. With num-threads=1 nothing changes: no worker ioctls, kthread mode. Released vz10 kernels have VHOST_SET_FORK_FROM_OWNER, but their vhost-blk does not handle the worker ioctls and fails VHOST_NEW_WORKER with -ENOTTY. On such a kernel, or one without VHOST_SET_FORK_FROM_OWNER, warn and run a single thread rather than fail to start vhost-blk. https://virtuozzo.atlassian.net/browse/VSTOR-107537 Signed-off-by: Alexander Ivanov --- hw/block/vhost-blk.c | 100 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 86 insertions(+), 14 deletions(-) diff --git a/hw/block/vhost-blk.c b/hw/block/vhost-blk.c index 8a015d54fa7..cc05eb057df 100644 --- a/hw/block/vhost-blk.c +++ b/hw/block/vhost-blk.c @@ -33,9 +33,60 @@ #include "migration/cpr.h" #include "system/runstate.h" +/* + * Spread the virtqueues round-robin over nworkers kernel threads. Thread 0 is + * the default worker VHOST_SET_OWNER created; vq 0 and every nworkers-th vq + * after it stay on it. + */ +static int vhost_blk_set_workers(struct vhost_dev *hdev, int nworkers) +{ + uint32_t ids[VHOST_BLK_MAX_QUEUES]; + int i, ret; + + assert(nworkers <= ARRAY_SIZE(ids)); + + for (i = 1; i < hdev->nvqs; i++) { + int w = i % nworkers; + struct vhost_vring_worker vq_worker = { .index = i }; + + if (w == 0) { + continue; + } + + /* vq i < nworkers is the first user of thread i: create it here */ + if (i < nworkers) { + struct vhost_worker_state worker = { }; + + ret = hdev->vhost_ops->vhost_new_worker(hdev, &worker); + if (ret == -ENOTTY) { + /* vhost-blk predating worker ioctls, fails on vq 1 only */ + warn_report_once("vhost-blk: kernel does not support worker " + "ioctls, using a single kernel thread"); + return 0; + } else if (ret < 0) { + error_report("vhost-blk: unable to create kernel thread: %s", + strerror(-ret)); + return ret; + } + ids[w] = worker.worker_id; + } + + vq_worker.worker_id = ids[w]; + ret = hdev->vhost_ops->vhost_attach_vring_worker(hdev, &vq_worker); + if (ret < 0) { + error_report("vhost-blk: unable to attach kernel thread to " + "queue %d: %s", i, strerror(-ret)); + return ret; + } + } + + return 0; +} + static int vhost_blk_start(VirtIODevice *vdev) { VHostBlk *s = VHOST_BLK(vdev); + struct vhost_dev *hdev = &s->dev; struct vhost_vring_file backend; int ret, i, nworkers; BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev))); @@ -51,15 +102,36 @@ static int vhost_blk_start(VirtIODevice *vdev) return 0; } + nworkers = MIN(s->conf.num_threads, hdev->nvqs); + if (nworkers > 1) { + /* + * VHOST_NEW_WORKER needs vhost_task mode; our kernel defaults to + * kthreads and VHOST_RESET_OWNER brings that default back, so ask + * on every start. Must precede VHOST_SET_OWNER. + */ + ret = hdev->vhost_ops->vhost_set_fork_from_owner(hdev, + VHOST_FORK_OWNER_TASK); + if (ret == -ENOTTY) { + warn_report_once("vhost-blk: kernel does not support vhost tasks, " + "using a single kernel thread"); + nworkers = 1; + } else if (ret < 0) { + error_report("vhost-blk: unable to set vhost task mode: %s", + strerror(-ret)); + return ret; + } + } + if (ioctl(s->vhostfd, VHOST_SET_OWNER, NULL)) { error_report("vhost-blk: unable to set owner"); return -ENOSYS; } - nworkers = s->conf.num_threads; - if (nworkers != 1 && ioctl(s->vhostfd, VHOST_SET_NWORKERS, &nworkers)) { - error_report("vhost-blk: unable to set number of kernel threads"); - return -EINVAL; + if (nworkers > 1) { + ret = vhost_blk_set_workers(hdev, nworkers); + if (ret < 0) { + return ret; + } } if (s->conf.serial) { @@ -70,22 +142,22 @@ static int vhost_blk_start(VirtIODevice *vdev) } } - ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, true); + ret = k->set_guest_notifiers(qbus->parent, hdev->nvqs, true); if (ret < 0) { error_report("vhost-blk: unable to bind guest notifiers"); return ret; } - ret = vhost_dev_enable_notifiers(&s->dev, vdev); + ret = vhost_dev_enable_notifiers(hdev, vdev); if (ret < 0) { error_report("vhost-blk: unable to enable dev notifiers: %s", strerror(errno)); goto err_set; } - s->dev.acked_features = vdev->guest_features & s->dev.features; + hdev->acked_features = vdev->guest_features & hdev->features; - ret = vhost_dev_start(&s->dev, vdev, false); + ret = vhost_dev_start(hdev, vdev, false); if (ret < 0) { error_report("vhost-blk: unable to start vhost dev"); goto err_disable; @@ -100,11 +172,11 @@ static int vhost_blk_start(VirtIODevice *vdev) goto err_stop; } - for (i = 0; i < s->dev.nvqs; i++) { - vhost_virtqueue_mask(&s->dev, vdev, i, false); + for (i = 0; i < hdev->nvqs; i++) { + vhost_virtqueue_mask(hdev, vdev, i, false); } - for (i = 0; i < s->dev.nvqs; i++) { + for (i = 0; i < hdev->nvqs; i++) { event_notifier_set(virtio_queue_get_host_notifier(virtio_get_queue(vdev, i))); } @@ -113,11 +185,11 @@ static int vhost_blk_start(VirtIODevice *vdev) return 0; err_stop: - vhost_dev_stop(&s->dev, vdev, false); + vhost_dev_stop(hdev, vdev, false); err_disable: - vhost_dev_disable_notifiers(&s->dev, vdev); + vhost_dev_disable_notifiers(hdev, vdev); err_set: - ret = k->set_guest_notifiers(qbus->parent, s->dev.nvqs, false); + ret = k->set_guest_notifiers(qbus->parent, hdev->nvqs, false); if (ret < 0) { error_report("vhost-blk: unable to unbind guest notifiers"); } -- 2.53.0