From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from PA4PR04CU001.outbound.protection.outlook.com (mail-francecentralazon11023137.outbound.protection.outlook.com [40.107.162.137]) by lore.virtuozzo.com (Postfix) with ESMTPS id 1B410803F6 for ; Tue, 17 Nov 2026 10:50:14 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KZyMjr2vh+DFkGpPlrj8xnVxbGRiF1U+VzMGjTP3xr+a3vL1JXGM2MUPMPfZOqju5oh2G8XxVkdw80om8XNhfK0qu2I0PKbzpLAZ8UknyZfwS7wUOf6TCIT0+tmAhqhUqVf47kdnH6gTaYJIZYn+wbV8CKZyAH2A4MApCvRBNsDlYc2ISvpsVlDvTvE5QKpt4qM3NTUAZWh5IxzG7L4ClnsWjXnyXD292q1FPDqdx83uPM+tMLtobRl84dpF1BV5thov7AqlFtmrxErCCaHP7hqFjGMsRK7DETxWzo2jb5DQ+jEaIkutPYC47xc+hZrT8RTmBIj2i2e+5sfRen8/vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gkhUdeLrSPOO9aPECQUFtvPmIXNO2N72K5PbvS8kqj4=; b=S4SPgQutt5yGXqZa2foJo3HSNjKJWsevxMYuP5Mqfr8hlCZOFycGR8QbydKPrpdhH12OkN40Jcnl0r2dr5YFE7khwx73l2aCHnQ6uSNjxxJNPujWGolFRfZ++Adq/3WoVR+RTHQx63guwwL/rcy6HSBmvnD5gvZQroUknjHZfvZTpnaWPNUUo6T9G0w3F3CjnpU+uK/1u6vVjzIWUknAn/KPFwmsMbOIukGWwpXSoLvjmDveL3JCtCQHnq03WpY/pVVzMhdTswJL8/0kQ3ILR+gUazbxMdlt/Slz3P3Jroi1xx8XVHj3oA5ZxzsdGqfsWK7CukEmGubvAtb0Bz2bxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 130.117.225.111) smtp.rcpttodomain=virtuozzo.com smtp.mailfrom=virtuozzo.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=virtuozzo.com; dkim=pass (signature was verified) header.d=virtuozzo.com; arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gkhUdeLrSPOO9aPECQUFtvPmIXNO2N72K5PbvS8kqj4=; b=n1VUvqbEPhM8O0oiDcF40uehYQolO8T9hbiqaqtoTqP5OHU8a+qf2VL5DimXz863akq5jv9V0rgpDnEdbiW6UI2Lnz+7bWp0VOnc/e/n2vATIl9+yxRFhb5mZ12oSQDcmLsj/bjQrTSHVkHgeagUhIvd4wrjGwtIlJHtjTbzAM7kFAVqmhSSFCw0u62rlxh/xyUr44wQd9Cu/uwrvSRXNIr0oBtBRirJGVbXsMOVAmPeTVfe2SqKE9R60gGAAOgGgsfi0NvN39Vv/wSg6Exi3Z1/DqHnz2CuuizB3N5aL9Ql6PiMHGfFntmEddz/nA0lxtojQqRO5Cyf9hUssu4ptw== Received: from OS6P279CA0166.NORP279.PROD.OUTLOOK.COM (2603:10a6:e10:38::6) by DU5PR08MB10437.eurprd08.prod.outlook.com (2603:10a6:10:524::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.615.14; Tue, 17 Nov 2026 10:50:07 +0000 Received: from CPH1EPF0000051A.eurprd04.prod.outlook.com (2603:10a6:e10:38:cafe::9a) by OS6P279CA0166.outlook.office365.com (2603:10a6:e10:38::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.615.14 via Frontend Transport; Tue, 17 Nov 2026 10:50:06 +0000 X-MS-Exchange-Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 130.117.225.111) smtp.mailfrom=virtuozzo.com; dkim=pass (signature was verified) header.d=virtuozzo.com;dmarc=pass action=none header.from=virtuozzo.com; Received-SPF: Pass (protection.outlook.com: domain of virtuozzo.com designates 130.117.225.111 as permitted sender) receiver=protection.outlook.com; client-ip=130.117.225.111; helo=relay.virtuozzo.com; pr=C Received: from relay.virtuozzo.com (130.117.225.111) by CPH1EPF0000051A.mail.protection.outlook.com (10.167.241.229) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.21.639.5 via Frontend Transport; Tue, 17 Nov 2026 10:50:06 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=MIME-Version:Message-ID:Date:Subject:From: Content-Type; bh=gkhUdeLrSPOO9aPECQUFtvPmIXNO2N72K5PbvS8kqj4=; b=bnN/w9V4g5zR sgbVa52JOac5o9KwIBrdC04fngxyYIXXjD0HVEEFQRlFOyr/7GnTnXm3eISqKuvZJnnU0qD77ynir dCLrvwKCfQUGQts2pDtnnshUfdEHBS0N9K6kBs1qJp3ri7zUTmQpBpcAeZyjrj4uR6NHA83FTyxhV IXRyJCpkaCQkg7xREwAx+6FM6N2CbNaEc6QP1ubGr8LxSsdTmicy1jjoe+DGQrPKRRF3eKpSev/YF bKWL/MpGtjoUxb7uIW4wG2XARvaUvWZ9jYPiTFO4xwCHoiUi16zFcDuV9x5L6LPCJCAfTd1zRsK0/ 7EUfVjjyzyOLxFhHCIZ80w==; Received: from ch-vpn.virtuozzo.com ([130.117.225.6] helo=sky) by relay.virtuozzo.com with esmtps (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1xTGh4-00A3yD-0D; Tue, 17 Nov 2026 11:49:56 +0100 From: Alexander Ivanov To: andrey.zhadchenko@virtuozzo.com Cc: andrey.drobyshev@virtuozzo.com, svt-core@virtuozzo.com Subject: [PATCH hci-8.1 2/2] vhost-blk: pause the VM on an IO_ERROR uevent from dm-qcow2 #VSTOR-152175 Date: Tue, 17 Nov 2026 11:50:03 +0100 Message-ID: <20261117105003.3082044-3-alexander.ivanov@virtuozzo.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20261117105003.3082044-1-alexander.ivanov@virtuozzo.com> References: <20261117105003.3082044-1-alexander.ivanov@virtuozzo.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CPH1EPF0000051A:EE_|DU5PR08MB10437:EE_ X-MS-Office365-Filtering-Correlation-Id: 60cd5ef0-ab03-4408-92ac-08df44988b03 X-LD-Processed: 0bc7f26d-0264-416e-a6fc-8352af79c58f,ExtAddr List-Id: svt-core@virtuozzo.com X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|1800799024|82310400026|36860700016|10067099003|56012099006|261026121527099003|261026140227099003|261028144727099003|261021135927099003|261022075727099003|261021123027099003|261021121427099003|261022105027099003|261102123227099003|261009223027099003|22082099003|18002099003|5113699003|6133799003|13003099007; X-Microsoft-Antispam-Message-Info: AwmyGxW/f5K1yJOSpgTaylEFZEbXjniln+EbpHMoP+cKdChm+oyZXFx9vFU/VcS5zEL2NUpB0CU15VKXRXY3gkkj+x6FIRuEFEy8kkxCSIFrO+57zYtoGMutua7uzPqWgbTyt7SuQcCqkCMSW+WKnG0ZY3BuxfoUwd5Jw9fnqdZ4x6ctqNE3EvB5T3B3gs0N+5G6j3euntZr/KgxTSWK71O0Uhx6ohBYc+NVFtzDGlqabchBCIxiqpJxB9rKERqkv15rwyKzqcCG/PLtYCgqSkdAHpABEMO9BQxn+fiKQKffP3ozwnH/d8ETBCH6Bam7JSCtWTEluzMAdyAry6NNRwthQ/fRUt+xxac/NWgRQlv2uNJEj8voIf8WzuGeHQkVGqGYJVie99MyNSZANO1LzjDnfPUqUWYl126GJv7ZuDiRB3j1WeDrtGDcmtm3xVGDqeeO8TdSySwMf+XH5UdKi+kXdmRMkNi77b3POmPtrCta8fL2tLVfg7GdaVjTG6rXFf8z8TB4WxxR0nyuFcH/Ao9QMw5JuIwfZ3M30yCnEJHGf55f7Y7BqygiL5KoftY+uTCl+Al3af4LfAxOnMi6SICUxJXul0A/ah2noMU4iDzH6ydp8BrS9/GvZA/IZYM8iZvCpWDupQlMlVeRZ9CLyQ== X-Forefront-Antispam-Report: CIP:130.117.225.111;CTRY:CH;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:relay.virtuozzo.com;PTR:relay.virtuozzo.com;CAT:NONE;SFS:(13230040)(23010399003)(376014)(1800799024)(82310400026)(36860700016)(10067099003)(56012099006)(261026121527099003)(261026140227099003)(261028144727099003)(261021135927099003)(261022075727099003)(261021123027099003)(261021121427099003)(261022105027099003)(261102123227099003)(261009223027099003)(22082099003)(18002099003)(5113699003)(6133799003)(13003099007);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: KS2e6qvFyZVfoh3EkDyQmL62Dcu7upb+qq/yzdjgo2ZYDRWPEIDKFBo4p3mtvocXkw2m3np6hZWsZxlqFBSksS/ZTdpsJNrX+pl0qfkH3vCUBukIIOFFPoY/c77J3Mxy8gMW41xmqgElD4MNZt04m3exTCqY+iz9Fj94MKWCTLwOePh3utkWFsNUdoOuxegQIN4y2yMcPyjd0ewPoaAhHCNlwirjROMj64ezLUOuOAiMi10qgVXGoUB4hCgaQKrt9kz1bkHfK9e7/N+ePd9BD1HzUwALYum/IrAuzzr3yca0M4RKwgtIUIM+cZU2u6WuvZcYCNZ6p33HbSuiucv/td9b1gaoXHdHHa6vtLGvWc0sPg4FEpYFhoBVhLMZUErs66o1ShE/kNxA5AqfeG2eMC028JVYYNzLhhR6+g0wwf5W9hhuS71zqsH8dQXswIzO X-Auto-Response-Suppress: DR, OOF, AutoReply X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Nov 2026 10:50:06.7809 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 60cd5ef0-ab03-4408-92ac-08df44988b03 X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=0bc7f26d-0264-416e-a6fc-8352af79c58f;Ip=[130.117.225.111];Helo=[relay.virtuozzo.com] X-MS-Exchange-CrossTenant-AuthSource: CPH1EPF0000051A.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU5PR08MB10437 vhost-blk completes requests to the guest in the kernel, so werror and rerror had no effect: dm-qcow2 held the failed write under the enospc or stop policy, but the VM kept running with the request stuck. dm-qcow2 reports a held request with a change uevent carrying IO_ERROR=1, IO_ERROR_OP=read|write and IO_ERROR_ERRNO=. Parse it on the uevent socket and, if the policy of the device says so, stop the VM with the io-error reason and emit BLOCK_IO_ERROR, as the block layer does for other devices. The device field is the id of the proxy, which is the disk alias libvirt looks the disk up by. The VM stops with RUN_STATE_IO_ERROR, on which vhost-blk keeps running, so the held requests survive the pause. libvirt resubmits them before cont. dm-qcow2 repeats the event on every failed retry, so ignore it unless the VM is running. Migration and savevm from IO_ERROR do not notify the VM state handlers, so vhost-blk would keep running and its vring state would be saved stale. The held requests cannot be migrated either, so block migration until the VM runs again, and fail it in pre_save for a migration that was already running when the error came. Reject the stop policy without the ueventfd property: without the socket QEMU never learns about the held request. https://virtuozzo.atlassian.net/browse/VSTOR-152175 Signed-off-by: Alexander Ivanov --- hw/block/vhost-blk.c | 93 +++++++++++++++++++++++++++++++++-- include/hw/virtio/vhost-blk.h | 2 + 2 files changed, 92 insertions(+), 3 deletions(-) diff --git a/hw/block/vhost-blk.c b/hw/block/vhost-blk.c index 354111a1b41..6775a561eab 100644 --- a/hw/block/vhost-blk.c +++ b/hw/block/vhost-blk.c @@ -32,6 +32,7 @@ #include "migration/blocker.h" #include "migration/cpr.h" #include "system/runstate.h" +#include "qapi/qapi-events-block-core.h" static int vhost_blk_start(VirtIODevice *vdev) { @@ -205,6 +206,10 @@ static void vhost_blk_vm_state(void *opaque, bool running, RunState state) vhost_blk_stop(vdev); vhost_blk_reset(vdev); } else if (state == RUN_STATE_RUNNING) { + VHostBlk *s = VHOST_BLK(vdev); + + s->io_error_held = false; + migrate_del_blocker(&s->io_error_blocker); vhost_blk_start(vdev); } } @@ -293,6 +298,45 @@ static void vhost_blk_block_cpr(VHostBlk *s) } } +static void vhost_blk_io_error(VHostBlk *s, bool is_read, int error) +{ + DeviceState *dev = DEVICE(s); + DeviceState *proxy = dev->parent_bus ? dev->parent_bus->parent : NULL; + BlockdevOnError on_err = is_read ? s->conf.conf.rerror + : s->conf.conf.werror; + g_autofree char *path = NULL; + + if (on_err != BLOCKDEV_ON_ERROR_STOP && + (on_err != BLOCKDEV_ON_ERROR_ENOSPC || error != ENOSPC)) { + return; + } + + /* dm-qcow2 repeats the event on every failed retry of a held request */ + if (!runstate_is_running()) { + return; + } + + path = object_get_canonical_path(OBJECT(dev)); + + s->io_error_held = true; + if (!s->io_error_blocker) { + error_setg(&s->io_error_blocker, + "vhost-blk: '%s' has requests held in the kernel after " + "an I/O error", path); + migrate_add_blocker_internal(&s->io_error_blocker, NULL); + } + + qemu_system_vmstop_request_prepare(); + qapi_event_send_block_io_error(path, + proxy && proxy->id ? proxy->id : "", + NULL, + is_read ? IO_OPERATION_TYPE_READ + : IO_OPERATION_TYPE_WRITE, + BLOCK_ERROR_ACTION_STOP, true, + error == ENOSPC, strerror(error), 0, 0); + qemu_system_vmstop_request(RUN_STATE_IO_ERROR); +} + /* * The uevent socket is created, bound and filtered by the management * layer and passed to us via the "ueventfd" property. @@ -310,6 +354,9 @@ static void vhost_blk_uevent_read(void *opaque) bool action_change = false; bool subsystem_block = false; bool resize = false; + bool io_error = false; + bool io_error_read = false; + int io_error_errno = EIO; char *field; char *end; ssize_t len; @@ -351,6 +398,17 @@ static void vhost_blk_uevent_read(void *opaque) subsystem_block = true; } else if (!strcmp(field, "RESIZE=1")) { resize = true; + } else if (!strcmp(field, "IO_ERROR=1")) { + io_error = true; + } else if (!strcmp(field, "IO_ERROR_OP=read")) { + io_error_read = true; + } else if (g_str_has_prefix(field, "IO_ERROR_ERRNO=")) { + int value; + + if (!qemu_strtoi(field + strlen("IO_ERROR_ERRNO="), NULL, 10, + &value) && value > 0) { + io_error_errno = value; + } } else if (g_str_has_prefix(field, "MAJOR=")) { uint64_t value; @@ -373,11 +431,18 @@ static void vhost_blk_uevent_read(void *opaque) field += field_len + 1; } - if (action_change && subsystem_block && resize && - event_major == major(s->backend_rdev) && - event_minor == minor(s->backend_rdev)) { + if (!action_change || !subsystem_block || + event_major != major(s->backend_rdev) || + event_minor != minor(s->backend_rdev)) { + continue; + } + + if (resize) { qemu_bh_schedule(s->resize_bh); } + if (io_error) { + vhost_blk_io_error(s, io_error_read, io_error_errno); + } } } @@ -635,6 +700,14 @@ static void vhost_blk_device_realize(DeviceState *dev, Error **errp) return; } + if (!conf->ueventfd && + (conf->conf.werror == BLOCKDEV_ON_ERROR_STOP || + conf->conf.rerror == BLOCKDEV_ON_ERROR_STOP)) { + error_setg(errp, "vhost-blk: 'stop' error policy requires the " + "'ueventfd' property"); + return; + } + if (!vhost_blk_open_backend(s, errp)) { return; } @@ -709,6 +782,7 @@ static void vhost_blk_device_unrealize(DeviceState *dev) VHostBlk *s = VHOST_BLK(dev); vhost_blk_uevent_detach(s); + migrate_del_blocker(&s->io_error_blocker); qemu_del_vm_change_state_handler(s->mighand); vhost_blk_set_status(vdev, 0); vhost_dev_cleanup(&s->dev); @@ -816,10 +890,23 @@ static const Property vhost_blk_properties[] = { DEFINE_BLOCK_ERROR_PROPERTIES(VHostBlk, conf.conf), }; +static int vhost_blk_pre_save(void *opaque) +{ + VHostBlk *s = opaque; + + if (s->io_error_held) { + error_report("vhost-blk: requests are held in the kernel after an " + "I/O error, resubmit or fail them first"); + return -EBUSY; + } + return 0; +} + static const VMStateDescription vmstate_vhost_blk = { .name = "vhost-blk", .minimum_version_id = 1, .version_id = 1, + .pre_save = vhost_blk_pre_save, .fields = (VMStateField[]) { VMSTATE_VIRTIO_DEVICE, VMSTATE_END_OF_LIST() diff --git a/include/hw/virtio/vhost-blk.h b/include/hw/virtio/vhost-blk.h index 059f1ebc701..935b35f6d6b 100644 --- a/include/hw/virtio/vhost-blk.h +++ b/include/hw/virtio/vhost-blk.h @@ -48,6 +48,8 @@ typedef struct VHostBlk { uint64_t backend_rdev; QEMUBH *resize_bh; Error *cpr_blocker; + Error *io_error_blocker; + bool io_error_held; } VHostBlk; #endif -- 2.53.0