From: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
To: Alexander Ivanov <alexander.ivanov@virtuozzo.com>,
andrey.zhadchenko@virtuozzo.com
Cc: svt-core@virtuozzo.com
Subject: Re: [PATCH hci-8.1 2/2] vhost-blk: pause the VM on an IO_ERROR uevent from dm-qcow2 #VSTOR-152175
Date: Tue, 17 Nov 2026 20:12:20 +0200 [thread overview]
Message-ID: <04daead2-b57d-4fdd-88aa-ea58598f1f6a@virtuozzo.com> (raw)
In-Reply-To: <20261117105003.3082044-3-alexander.ivanov@virtuozzo.com>
On 11/17/26 12:50 PM, Alexander Ivanov wrote:
> vhost-blk completes requests to the guest in the kernel, so werror and
> rerror had no effect: dm-qcow2 held the failed write under the enospc
> or stop policy, but the VM kept running with the request stuck.
>
> dm-qcow2 reports a held request with a change uevent carrying
> IO_ERROR=1, IO_ERROR_OP=read|write and IO_ERROR_ERRNO=<errno>. Parse it
> on the uevent socket and, if the policy of the device says so, stop
> the VM with the io-error reason and emit BLOCK_IO_ERROR, as the block
> layer does for other devices. The device field is the id of the proxy,
> which is the disk alias libvirt looks the disk up by.
>
> The VM stops with RUN_STATE_IO_ERROR, on which vhost-blk keeps running,
> so the held requests survive the pause. libvirt resubmits them before
> cont. dm-qcow2 repeats the event on every failed retry, so ignore it
> unless the VM is running.
>
> Migration and savevm from IO_ERROR do not notify the VM state handlers,
> so vhost-blk would keep running and its vring state would be saved
> stale. The held requests cannot be migrated either, so block migration
> until the VM runs again, and fail it in pre_save for a migration that
> was already running when the error came.
>
> Reject the stop policy without the ueventfd property: without the
> socket QEMU never learns about the held request.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-152175
>
> Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
> ---
> hw/block/vhost-blk.c | 93 +++++++++++++++++++++++++++++++++--
> include/hw/virtio/vhost-blk.h | 2 +
> 2 files changed, 92 insertions(+), 3 deletions(-)
>
> diff --git a/hw/block/vhost-blk.c b/hw/block/vhost-blk.c
> index 354111a1b41..6775a561eab 100644
> --- a/hw/block/vhost-blk.c
> +++ b/hw/block/vhost-blk.c
> @@ -32,6 +32,7 @@
> #include "migration/blocker.h"
> #include "migration/cpr.h"
> #include "system/runstate.h"
> +#include "qapi/qapi-events-block-core.h"
>
> static int vhost_blk_start(VirtIODevice *vdev)
> {
> @@ -205,6 +206,10 @@ static void vhost_blk_vm_state(void *opaque, bool running, RunState state)
> vhost_blk_stop(vdev);
> vhost_blk_reset(vdev);
> } else if (state == RUN_STATE_RUNNING) {
> + VHostBlk *s = VHOST_BLK(vdev);
> +
> + s->io_error_held = false;
> + migrate_del_blocker(&s->io_error_blocker);
> vhost_blk_start(vdev);
> }
> }
> @@ -293,6 +298,45 @@ static void vhost_blk_block_cpr(VHostBlk *s)
> }
> }
>
> +static void vhost_blk_io_error(VHostBlk *s, bool is_read, int error)
> +{
> + DeviceState *dev = DEVICE(s);
> + DeviceState *proxy = dev->parent_bus ? dev->parent_bus->parent : NULL;
> + BlockdevOnError on_err = is_read ? s->conf.conf.rerror
> + : s->conf.conf.werror;
> + g_autofree char *path = NULL;
> +
> + if (on_err != BLOCKDEV_ON_ERROR_STOP &&
> + (on_err != BLOCKDEV_ON_ERROR_ENOSPC || error != ENOSPC)) {
> + return;
> + }
IIUC libvirt gets rerror/werror policies from XML, then sets them in
dm-qcow2 table and provides to QEMU via conf settings. If so, they
can't mismatch. Therefore:
1. Returning on this mismatch is wrong. This way requests stay parked in
the kernel, nobody's told and libvirt never sends resubmit.
2. This mismatch shouldn't happen in the 1st place, so let's at least
add warn_report() on it.
> +
> + /* dm-qcow2 repeats the event on every failed retry of a held request */
> + if (!runstate_is_running()) {
> + return;
> + }
What if a VM is stopped for another reason (e.g. IO error on another
disk)? We return here and don't set the migration blocker. Maybe
instead return on:
> if (!s->vhost_started || runstate_check(RUN_STATE_SUSPENDED))
> +
> + path = object_get_canonical_path(OBJECT(dev));
> +
> + s->io_error_held = true;
> + if (!s->io_error_blocker) {
> + error_setg(&s->io_error_blocker,
> + "vhost-blk: '%s' has requests held in the kernel after "
> + "an I/O error", path);
> + migrate_add_blocker_internal(&s->io_error_blocker, NULL);
> + }
And then here after the mig blocker is already set, once again check:
> if (!runstate_is_running()) {
> return;
> }
So that we don't request stop of an already stopped VM.
Andrey
> + qemu_system_vmstop_request_prepare();
> + qapi_event_send_block_io_error(path,
> + proxy && proxy->id ? proxy->id : "",
> + NULL,
> + is_read ? IO_OPERATION_TYPE_READ
> + : IO_OPERATION_TYPE_WRITE,
> + BLOCK_ERROR_ACTION_STOP, true,
> + error == ENOSPC, strerror(error), 0, 0);
> + qemu_system_vmstop_request(RUN_STATE_IO_ERROR);
> +}
> +
> /*
> * The uevent socket is created, bound and filtered by the management
> * layer and passed to us via the "ueventfd" property.
> @@ -310,6 +354,9 @@ static void vhost_blk_uevent_read(void *opaque)
> bool action_change = false;
> bool subsystem_block = false;
> bool resize = false;
> + bool io_error = false;
> + bool io_error_read = false;
> + int io_error_errno = EIO;
> char *field;
> char *end;
> ssize_t len;
> @@ -351,6 +398,17 @@ static void vhost_blk_uevent_read(void *opaque)
> subsystem_block = true;
> } else if (!strcmp(field, "RESIZE=1")) {
> resize = true;
> + } else if (!strcmp(field, "IO_ERROR=1")) {
> + io_error = true;
> + } else if (!strcmp(field, "IO_ERROR_OP=read")) {
> + io_error_read = true;
> + } else if (g_str_has_prefix(field, "IO_ERROR_ERRNO=")) {
> + int value;
> +
> + if (!qemu_strtoi(field + strlen("IO_ERROR_ERRNO="), NULL, 10,
> + &value) && value > 0) {
> + io_error_errno = value;
> + }
> } else if (g_str_has_prefix(field, "MAJOR=")) {
> uint64_t value;
>
> @@ -373,11 +431,18 @@ static void vhost_blk_uevent_read(void *opaque)
> field += field_len + 1;
> }
>
> - if (action_change && subsystem_block && resize &&
> - event_major == major(s->backend_rdev) &&
> - event_minor == minor(s->backend_rdev)) {
> + if (!action_change || !subsystem_block ||
> + event_major != major(s->backend_rdev) ||
> + event_minor != minor(s->backend_rdev)) {
> + continue;
> + }
> +
> + if (resize) {
> qemu_bh_schedule(s->resize_bh);
> }
> + if (io_error) {
> + vhost_blk_io_error(s, io_error_read, io_error_errno);
> + }
> }
> }
>
> @@ -635,6 +700,14 @@ static void vhost_blk_device_realize(DeviceState *dev, Error **errp)
> return;
> }
>
> + if (!conf->ueventfd &&
> + (conf->conf.werror == BLOCKDEV_ON_ERROR_STOP ||
> + conf->conf.rerror == BLOCKDEV_ON_ERROR_STOP)) {
> + error_setg(errp, "vhost-blk: 'stop' error policy requires the "
> + "'ueventfd' property");
> + return;
> + }
> +
> if (!vhost_blk_open_backend(s, errp)) {
> return;
> }
> @@ -709,6 +782,7 @@ static void vhost_blk_device_unrealize(DeviceState *dev)
> VHostBlk *s = VHOST_BLK(dev);
>
> vhost_blk_uevent_detach(s);
> + migrate_del_blocker(&s->io_error_blocker);
> qemu_del_vm_change_state_handler(s->mighand);
> vhost_blk_set_status(vdev, 0);
> vhost_dev_cleanup(&s->dev);
> @@ -816,10 +890,23 @@ static const Property vhost_blk_properties[] = {
> DEFINE_BLOCK_ERROR_PROPERTIES(VHostBlk, conf.conf),
> };
>
> +static int vhost_blk_pre_save(void *opaque)
> +{
> + VHostBlk *s = opaque;
> +
> + if (s->io_error_held) {
> + error_report("vhost-blk: requests are held in the kernel after an "
> + "I/O error, resubmit or fail them first");
> + return -EBUSY;
> + }
> + return 0;
> +}
> +
> static const VMStateDescription vmstate_vhost_blk = {
> .name = "vhost-blk",
> .minimum_version_id = 1,
> .version_id = 1,
> + .pre_save = vhost_blk_pre_save,
> .fields = (VMStateField[]) {
> VMSTATE_VIRTIO_DEVICE,
> VMSTATE_END_OF_LIST()
> diff --git a/include/hw/virtio/vhost-blk.h b/include/hw/virtio/vhost-blk.h
> index 059f1ebc701..935b35f6d6b 100644
> --- a/include/hw/virtio/vhost-blk.h
> +++ b/include/hw/virtio/vhost-blk.h
> @@ -48,6 +48,8 @@ typedef struct VHostBlk {
> uint64_t backend_rdev;
> QEMUBH *resize_bh;
> Error *cpr_blocker;
> + Error *io_error_blocker;
> + bool io_error_held;
> } VHostBlk;
>
> #endif
next prev parent reply other threads:[~2026-11-17 18:12 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 10:50 [PATCH hci-8.1 0/2] vhost-blk: support the werror/rerror policies Alexander Ivanov
2026-11-17 10:50 ` [PATCH hci-8.1 1/2] vhost-blk: add the werror and rerror properties #VSTOR-152175 Alexander Ivanov
2026-11-17 11:03 ` no-reply
2026-11-17 18:12 ` Andrey Drobyshev
2026-11-17 10:50 ` [PATCH hci-8.1 2/2] vhost-blk: pause the VM on an IO_ERROR uevent from dm-qcow2 #VSTOR-152175 Alexander Ivanov
2026-11-17 11:05 ` no-reply
2026-11-17 18:12 ` Andrey Drobyshev [this message]
2026-11-17 10:59 ` [PATCH hci-8.1 0/2] vhost-blk: support the werror/rerror policies Andrey Drobyshev
2026-11-17 18:14 ` Denis V. Lunev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=04daead2-b57d-4fdd-88aa-ea58598f1f6a@virtuozzo.com \
--to=andrey.drobyshev@virtuozzo.com \
--cc=alexander.ivanov@virtuozzo.com \
--cc=andrey.zhadchenko@virtuozzo.com \
--cc=svt-core@virtuozzo.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.