* [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path
@ 2026-11-13 11:11 Andrey Zhadchenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel Andrey Zhadchenko
` (8 more replies)
0 siblings, 9 replies; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
dm-qcow2 processes every request sequentially in a single kwork.
Unfortunately this introduces latency and cpu-bound limits. This
patchset adds a fast path: right after splitting, qios which need no
metadata changes are processed outside the main kwork, in parallel
with it and with each other. These are:
- reads and writes of fully allocated clusters;
- reads of unallocated clusters, which are zero-filled or forwarded
to the lower delta;
- partial discards, which only punch a hole in the image file.
Everything else (allocations, COW, compressed clusters, discards
changing metadata, backward merge) still goes to the main kwork.
Fast path qios are batched in per-cpu llists and processed by one
work per submitting CPU, to avoid queue_work() cost on every I/O. The
target also declares DM_TARGET_NOWAIT now, so io_uring submits I/O
inline instead of punting every request to io-wq.
Patch 1 fixes a race with md page insertion which already exists
between qcow2_llseek_hole() and the main kwork, and which the fast
path makes more likely. Patch 3 counts a fast path write or partial
discard in md writers under the lock which checks its L2 entry, so a
discard unmapping the cluster waits for it before the cluster is
freed.
Known limitation: fast path reads are not counted, and nothing pins
their cluster between the L2 parse and the data submit. Main kwork
may discard the cluster, free it and reuse it for another allocation
in that window, and a fast path read then returns the data of another
guest offset. A follow-up series fixes this:
https://virtuozzo.atlassian.net/browse/VSTOR-154659
The fast path can be turned off at runtime with the fast_path module
parameter.
v2:
- patch 2 waits for L2 writeback, so a fast path write is not acked
before its L2 entry is on disk
- batching goes before DM_TARGET_NOWAIT, and its worker no longer
loops unbounded
- measurements moved to the thread
- subject prefix changed to dm-qcow2
v3:
- rebased on the new hci-dev top, which includes "dm-qcow2: let
discards of one L2 page wait for in-flight writes together"
- "wait for L2 writeback in perform_rw_mapped()" is replaced with
"count fast path writes in parse_metadata()": a write to an L2 index
in flight goes to the main kwork at parse, and a counted fast path
write or partial discard is waited for by a discard unmapping its
cluster, instead of rechecking the L2 entry in perform_rw_mapped()
- new patch: fast_path module parameter
- md page lookup is repeated once only
- the known limitation of fast path reads is described
Details are in the patches.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Andrey Zhadchenko (8):
dm-qcow2: repeat md page lookup if it was inserted in parallel
dm-qcow2: prepare process_one_qio() for fastpath
dm-qcow2: count fast path writes in parse_metadata()
dm-qcow2: add per-qio work infrastructure
dm-qcow2: implement per-qio fast processing
dm-qcow2: add fast_path module parameter
dm-qcow2: process fast path qios in per-cpu batches on a workqueue
dm-qcow2: declare DM_TARGET_NOWAIT
drivers/md/dm-qcow2-map.c | 279 +++++++++++++++++++++++++++++------
drivers/md/dm-qcow2-target.c | 12 +-
drivers/md/dm-qcow2.h | 16 +-
3 files changed, 257 insertions(+), 50 deletions(-)
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:22 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 2/8] dm-qcow2: prepare process_one_qio() for fastpath Andrey Zhadchenko
` (7 subsequent siblings)
8 siblings, 2 replies; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
parse_metadata() is called not only from the main kwork:
qcow2_llseek_hole() calls it in process context, in parallel with
the kwork. When both miss the same md page, both drop md_pages_lock
in submit_read_md_page() and try to insert a new md page. The one
that loses the insert gets -EEXIST from qcow2_alloc_and_insert_md_page()
and returns it as is: a guest I/O on the kwork side ends with EIO,
or lseek(SEEK_DATA/SEEK_HOLE) fails with -EEXIST.
Teach __handle_md_page() to handle -EEXIST by repeating the lookup:
the page inserted in parallel is found then, and the qio is either
served from it or postponed until its read completes.
Repeat it once only. The second lookup misses the page only if its
read failed and erased it in between, and then the qio fails with
-EIO as the qios waiting for that read did.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Fixes: 412d5a4f7bf1 ("dm/dm-qcow2: add llseek_hole")
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v2: say in the message what fails without the patch: the race of
qcow2_llseek_hole() with main kwork; added Fixes: tag
v3: repeat the lookup once only, a second miss fails the qio with -EIO
drivers/md/dm-qcow2-map.c | 23 ++++++++++++++++++-----
1 file changed, 18 insertions(+), 5 deletions(-)
diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
index 32373114580e9..58b95d7a955cf 100644
--- a/drivers/md/dm-qcow2-map.c
+++ b/drivers/md/dm-qcow2-map.c
@@ -1676,20 +1676,33 @@ static int submit_read_md_page(struct qcow2 *qcow2, struct qio **qio,
/*
* This may be called with @qio == NULL, in case of we are
* interesting in searching cached in memory md only.
- * This is aimed to be called not only from main kwork
- * for L1/L2 pages, so all callers looking for L1/L2
- * must care about submit_read_md_page() may return EEXIST.
+ *
+ * submit_read_md_page() drops md_pages_lock, so the page may be
+ * inserted in parallel. Then it is looked up once more. It is not
+ * found again only if its read failed in the meantime, and then
+ * @qio fails too.
*/
static int __handle_md_page(struct qcow2 *qcow2, u64 page_id,
struct qio **qio, struct md_page **ret_md)
{
+ bool retried = false;
struct md_page *md;
+ int ret;
lockdep_assert_held(&qcow2->md_pages_lock);
+again:
md = qcow2_md_page_find_or_postpone(qcow2, page_id, qio);
if (!md) {
- if (qio && *qio)
- return submit_read_md_page(qcow2, qio, page_id);
+ if (qio && *qio) {
+ ret = submit_read_md_page(qcow2, qio, page_id);
+ if (unlikely(ret == -EEXIST)) {
+ if (retried)
+ return -EIO;
+ retried = true;
+ goto again;
+ }
+ return ret;
+ }
return 0;
}
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 2/8] dm-qcow2: prepare process_one_qio() for fastpath
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:26 ` no-reply
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata() Andrey Zhadchenko
` (6 subsequent siblings)
8 siblings, 1 reply; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
dm-qcow2 heavily relies on main kwork do_qcow2_work() being
sequential. All metadata change happens there and cannot
run in parallel. Unfortunately that caps dm-qcow2 performance.
But if the request does not require metadata changes we should
be able to immediately dispatch it.
process_one_qio() and handle_metadata() already know what to
do: just add a fast flag to process_one_qio(), passed as
avoid_metadata_changes to handle_metadata(), and return codes.
While at it, just return in handle_metadata() instead of
setting ret to 0 and jumping to if (ret < 0) label.
Also pass the flag to parse_metadata() and return to the slow path
before it looks up R1/R2 pages. They are needed only by COW writes
and cluster unmapping discards, which go to the slow path anyway.
Besides the useless lookup, R1 is unstable outside main kwork: an R1
page read in parallel with relocate_refcount_table() may stay cached
under a stale page id.
Return to the slow path from parse_metadata() also for a write to
an index changed in a dirty or under writeback L2 page, i.e. to a
cluster whose allocation is not on disk yet. Such a write must be
attached to md->wbd, or wait for the writeback end, so it is not
acked before its L2 entry is written. Leave this to main kwork, so
the fast path never refers to md->wbd.
Also exclude compressed clusters as there are some complications
with wpc_readers.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v2: return to the slow path before R1/R2 lookup in parse_metadata();
QIO_NEEDS_MAIN_KWORK instead of magic 2; fixed the
"avoid_metadata_changes() flag" typo
v3: a write to an index changed in a dirty or under writeback L2 page
goes to main kwork from parse_metadata(); process_one_qio()
parameter renamed to fast; dropped redundant else
drivers/md/dm-qcow2-map.c | 63 +++++++++++++++++++++++++++++++--------
1 file changed, 50 insertions(+), 13 deletions(-)
diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
index 58b95d7a955cf..5e254c11ae7b8 100644
--- a/drivers/md/dm-qcow2-map.c
+++ b/drivers/md/dm-qcow2-map.c
@@ -2094,6 +2094,12 @@ static int qcow2_handle_r1r2_maps(struct qcow2 *qcow2, loff_t pos, struct qio **
return ret;
}
+/*
+ * Returned by parse_metadata() and handle_metadata() with
+ * @avoid_metadata_changes, when @qio must be processed by main kwork.
+ */
+#define QIO_NEEDS_MAIN_KWORK 2
+
/*
* This caches pages of allocated on disk md levels, which are
* required for submission @qio, and checks they are stable.
@@ -2101,11 +2107,17 @@ static int qcow2_handle_r1r2_maps(struct qcow2 *qcow2, loff_t pos, struct qio **
* Returns: negative in case of error, or 0 if success.
* Special case: return 0 with zeroed @qio means @qio was deferred
* till some event: reading of md page, end of writeback, etc.
+ * With @avoid_metadata_changes, return QIO_NEEDS_MAIN_KWORK if @qio
+ * needs R1/R2: outside main kwork R1 is unstable, see
+ * __handle_r1r2_maps(). The same for a write to an index changed in
+ * a dirty or under writeback L2 page: it must be attached to md->wbd
+ * or wait for the writeback, which is left to main kwork.
*/
static int parse_metadata(struct qcow2 *qcow2, struct qio **qio,
- struct qcow2_map *map)
+ struct qcow2_map *map, bool avoid_metadata_changes)
{
bool write = op_is_write((*qio)->bi_op);
+ bool l2_in_flight = false;
struct md_page *md;
u64 pos;
s64 ret;
@@ -2148,10 +2160,15 @@ static int parse_metadata(struct qcow2 *qcow2, struct qio **qio,
/* Find DATA cluster (from L2 page) */
pos = ret = parse_l2(qcow2, map, qio, write);
+ if (ret > 0 && write && avoid_metadata_changes)
+ l2_in_flight = dirty_or_writeback(qcow2, md,
+ map->l2.index_in_page + !!(qcow2->ext_l2));
unlock:
spin_unlock_irq(&qcow2->md_pages_lock);
if (ret <= 0) /* Err, delayed, DATA is not allocated, or zero read */
return ret;
+ if (l2_in_flight)
+ return QIO_NEEDS_MAIN_KWORK;
map->data_clu_pos = pos;
if (!write)
@@ -2167,6 +2184,9 @@ static int parse_metadata(struct qcow2 *qcow2, struct qio **qio,
if (!qio_may_modify_image(qcow2, *qio))
return 0;
+ if (avoid_metadata_changes)
+ return QIO_NEEDS_MAIN_KWORK;
+
/* Now refcounters table/block */
ret = qcow2_handle_r1r2_maps(qcow2, pos, qio, &map->r1,
&map->r2, map->compressed);
@@ -3716,24 +3736,31 @@ static int prepare_cluster_discard(struct qcow2 *qcow2, struct qio **qio,
}
static int handle_metadata(struct qcow2 *qcow2, struct qio **qio,
- struct qcow2_map *map)
+ struct qcow2_map *map, bool avoid_metadata_changes)
{
bool write = op_is_write((*qio)->bi_op);
int ret;
- ret = parse_metadata(qcow2, qio, map);
+ ret = parse_metadata(qcow2, qio, map, avoid_metadata_changes);
if (ret < 0 || !*qio) /* Error or postponed */
goto check_err;
+ if (ret == QIO_NEEDS_MAIN_KWORK)
+ return QIO_NEEDS_MAIN_KWORK;
+
+ if (avoid_metadata_changes && map->compressed)
+ return QIO_NEEDS_MAIN_KWORK;
ret = 1;
if (unlikely(qcow2->backward_merge_in_process)) {
+ if (avoid_metadata_changes)
+ return QIO_NEEDS_MAIN_KWORK;
/* Keep in mind the below may replace *qio */
ret = prepare_backward_merge(qcow2, qio, map, write);
} else if (unlikely(fake_l1cow_qio(*qio)) &&
(!map->clu_is_cow || (map->level & L2_LEVEL))) {
/* Nothing to COW or L1 is mapped exactly once */
qio_endio(*qio);
- ret = 0;
+ return 0;
} else if (unlikely(op_is_discard((*qio)->bi_op))) {
/*
* Discard carries no data and is advisory. If it does not
@@ -3741,13 +3768,17 @@ static int handle_metadata(struct qcow2 *qcow2, struct qio **qio,
* is not present, end early.
*/
if (qio_discard_updates_metadata(qcow2, *qio, map)) {
+ if (avoid_metadata_changes)
+ return QIO_NEEDS_MAIN_KWORK;
ret = prepare_cluster_discard(qcow2, qio, map);
} else if (!map->data_clu_alloced || map->clu_is_cow) {
qio_endio(*qio);
- ret = 0;
+ return 0;
}
} else if (write &&
(!qio_is_fully_alloced(qcow2, *qio, map) || map->clu_is_cow)) {
+ if (avoid_metadata_changes)
+ return QIO_NEEDS_MAIN_KWORK;
if (map->clu_is_cow) {
/* COW to compressed or shared with snapshot cluster */
ret = prepare_l1l2_replace(qcow2, *qio, map);
@@ -3859,13 +3890,17 @@ static void process_read_qio(struct qcow2 *qcow2, struct qio *qio,
qio_endio(qio);
}
-static void process_one_qio(struct qcow2 *qcow2, struct qio *qio)
+static int process_one_qio(struct qcow2 *qcow2, struct qio *qio, bool fast)
{
struct qcow2_map map = { .qcow2 = qcow2, };
bool write;
+ int ret;
- if (!handle_metadata(qcow2, &qio, &map))
- return;
+ ret = handle_metadata(qcow2, &qio, &map, fast);
+ if (!ret)
+ return 0;
+ if (ret == QIO_NEEDS_MAIN_KWORK)
+ return 1;
/*
* Merge machinery makes out of order merges for present
@@ -3877,7 +3912,7 @@ static void process_one_qio(struct qcow2 *qcow2, struct qio *qio)
if (unlikely(qcow2->backward_merge_in_process) &&
(fake_merge_qio(qio) || qcow2_file_is_writable(qcow2))) {
submit_top_delta_read(&map, qio);
- return;
+ return 0;
}
write = op_is_write(qio->bi_op);
@@ -3885,13 +3920,13 @@ static void process_one_qio(struct qcow2 *qcow2, struct qio *qio)
/* Process discard early. Don't do COW just to discard */
if (unlikely(op_is_discard(qio->bi_op))) {
issue_discard(&map, qio);
- return;
+ return 0;
}
if (unlikely(map.compressed)) {
/* Compressed qio never uses sub-clus */
submit_read_compressed(&map, qio, write);
- return;
+ return 0;
}
if (!write) {
@@ -3904,6 +3939,8 @@ static void process_one_qio(struct qcow2 *qcow2, struct qio *qio)
else
perform_rw_mapped(&map, qio);
}
+
+ return 0;
}
static struct bio_vec *create_bvec_from_rq(struct request *rq)
@@ -4006,7 +4043,7 @@ static void process_deferred_qios(struct qcow2 *qcow2, struct list_head *qios)
continue;
}
- process_one_qio(qcow2, qio);
+ process_one_qio(qcow2, qio, false);
}
}
@@ -4946,7 +4983,7 @@ static int qcow2_llseek_hole_qio(struct qio *qio, int whence, loff_t *result)
map.qcow2 = qio->qcow2;
qptr = qio;
- ret = parse_metadata(qio->qcow2, &qptr, &map);
+ ret = parse_metadata(qio->qcow2, &qptr, &map, false);
if (ret < 0)
break;
if (qptr == NULL) {
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata()
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel Andrey Zhadchenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 2/8] dm-qcow2: prepare process_one_qio() for fastpath Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:27 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 4/8] dm-qcow2: add per-qio work infrastructure Andrey Zhadchenko
` (5 subsequent siblings)
8 siblings, 2 replies; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
perform_rw_mapped() reads md->status and md->wbd without
md_pages_lock and counts the write in md writers, relying on
dm-qcow2 sequential nature: main kwork parses a qio and submits it
in one pass. Outside main kwork an L1/L2 page changes only when its
writeback ends, and parse_l2() already delays writes to the changed
indexes of a page under writeback.
This does not hold for the per-qio fast path, which parses L2 and
submits in parallel with main kwork. The fast path takes only writes
to fully allocated clusters mapped exactly once, whose indexes are
neither locked nor changed in a dirty or under writeback L2 page. In
between, main kwork may still discard or zero the same range, which
is a valid race with the write, or allocate other subclusters of the
cluster, which does not affect the write. But a discard unmapping the
cluster lets it be reused, and a write landing after that corrupts
the new owner. The discard waits for writes counted in md writers,
but the counter is valid only if it is increased from main kwork or
under md_pages_lock.
So count a fast path write in parse_metadata() under the same
md_pages_lock hold, which checks the L2 entry. A discard switches the
writer generation after it clears the entry under the lock, so
either the fast path sees the entry cleared, or the discard waits for
the write. Drop the count, if the qio goes to main kwork afterwards.
A partial discard on the fast path punches a hole in the cluster, so
it is counted the same way and drops the count after the punch.
A discard of a shared cluster punches nothing and is not counted.
Add @fast parameter to perform_rw_mapped(). With it, skip the md->wbd
attach, which the fast path never needs, and the writers increase.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v3: new patch, replaces "wait for L2 writeback in perform_rw_mapped()":
a fast path write or partial discard is counted in md writers
under the lock which checks its L2 entry, so a discard unmapping
the cluster waits for it before the cluster is freed
drivers/md/dm-qcow2-map.c | 37 ++++++++++++++++++++++++-------------
1 file changed, 24 insertions(+), 13 deletions(-)
diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
index 5e254c11ae7b8..b9daa258285e6 100644
--- a/drivers/md/dm-qcow2-map.c
+++ b/drivers/md/dm-qcow2-map.c
@@ -2160,9 +2160,14 @@ static int parse_metadata(struct qcow2 *qcow2, struct qio **qio,
/* Find DATA cluster (from L2 page) */
pos = ret = parse_l2(qcow2, map, qio, write);
- if (ret > 0 && write && avoid_metadata_changes)
+ if (ret > 0 && write && avoid_metadata_changes) {
l2_in_flight = dirty_or_writeback(qcow2, md,
map->l2.index_in_page + !!(qcow2->ext_l2));
+ /* A discard is counted only if it is going to punch a hole */
+ if (!l2_in_flight &&
+ (!op_is_discard((*qio)->bi_op) || !map->clu_is_cow))
+ inc_writers(md, *qio);
+ }
unlock:
spin_unlock_irq(&qcow2->md_pages_lock);
if (ret <= 0) /* Err, delayed, DATA is not allocated, or zero read */
@@ -3089,7 +3094,7 @@ static void data_rw_complete(struct qio *qio)
}
}
-static void perform_rw_mapped(struct qcow2_map *map, struct qio *qio)
+static void perform_rw_mapped(struct qcow2_map *map, struct qio *qio, bool fast)
{
struct qcow2 *qcow2 = map->qcow2;
struct md_page *md = map->l2.md;
@@ -3112,15 +3117,16 @@ static void perform_rw_mapped(struct qcow2_map *map, struct qio *qio)
* so lock is not needed for MD_DIRTY/changed_indexes visibility.
*/
index_in_page = map->l2.index_in_page + !!(qcow2->ext_l2);
- if (rw == WRITE && (md->status & MD_DIRTY) &&
- test_bit(index_in_page, md->wbd->changed_indexes)) {
- spin_lock_irqsave(&qcow2->md_pages_lock, flags);
- md->wbd->nr_submitted++;
- qio->data = md->wbd;
- spin_unlock_irqrestore(&qcow2->md_pages_lock, flags);
- }
- if (rw == WRITE)
+ if (rw == WRITE && !fast) {
+ if ((md->status & MD_DIRTY) &&
+ test_bit(index_in_page, md->wbd->changed_indexes)) {
+ spin_lock_irqsave(&qcow2->md_pages_lock, flags);
+ md->wbd->nr_submitted++;
+ qio->data = md->wbd;
+ spin_unlock_irqrestore(&qcow2->md_pages_lock, flags);
+ }
inc_writers(md, qio);
+ }
map_and_submit_rw(qcow2, map->data_clu_pos, qio);
}
@@ -3608,6 +3614,8 @@ static void issue_discard(struct qcow2_map *map, struct qio *qio)
if (!map->clu_is_cow) {
pos = bio_sector_to_file_pos(qcow2, qio, map);
ret = qcow2_punch_hole(qcow2->file, pos, qio->bi_iter.bi_size);
+ if (qio->writer_md)
+ dec_writers(qio);
if (ret) {
qio->bi_status = errno_to_blk_status(ret);
@@ -3878,7 +3886,7 @@ static void process_read_qio(struct qcow2 *qcow2, struct qio *qio,
qcow2_dispatch_qios(qio->qcow2, qio, NULL);
} else {
/* Mapped */
- perform_rw_mapped(map, qio);
+ perform_rw_mapped(map, qio, false);
}
qio = split;
@@ -3899,8 +3907,11 @@ static int process_one_qio(struct qcow2 *qcow2, struct qio *qio, bool fast)
ret = handle_metadata(qcow2, &qio, &map, fast);
if (!ret)
return 0;
- if (ret == QIO_NEEDS_MAIN_KWORK)
+ if (ret == QIO_NEEDS_MAIN_KWORK) {
+ if (qio->writer_md)
+ dec_writers(qio);
return 1;
+ }
/*
* Merge machinery makes out of order merges for present
@@ -3937,7 +3948,7 @@ static int process_one_qio(struct qcow2 *qcow2, struct qio *qio, bool fast)
else if (unlikely(map.clu_is_cow || map.backing_file_cow))
submit_read_sliced_cow_clu(&map, qio);
else
- perform_rw_mapped(&map, qio);
+ perform_rw_mapped(&map, qio, fast);
}
return 0;
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 4/8] dm-qcow2: add per-qio work infrastructure
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
` (2 preceding siblings ...)
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata() Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:30 ` no-reply
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing Andrey Zhadchenko
` (4 subsequent siblings)
8 siblings, 1 reply; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
dm-qcow2 was created to run non-concurrently, so qio never had
per work struct field. Let's add one and use it to queue first
dm-qcow2 pipeline step: qio preparation. It requires no metadata,
so everything is simple here.
We would later move more pipeline steps into qio.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v2: updated QLIST_EMBEDDED comment
v3: no changes
drivers/md/dm-qcow2-map.c | 60 ++++++++++++++++++++++++++++-----------
drivers/md/dm-qcow2.h | 12 ++++++--
2 files changed, 54 insertions(+), 18 deletions(-)
diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
index b9daa258285e6..496fbcc613488 100644
--- a/drivers/md/dm-qcow2-map.c
+++ b/drivers/md/dm-qcow2-map.c
@@ -362,6 +362,8 @@ struct qio *qcow2_alloc_qio(mempool_t *pool, bool zero)
}
ALLOW_ERROR_INJECTION(qcow2_alloc_qio, NULL);
+static void do_qio_work(struct work_struct *ws);
+
void qcow2_init_qio(struct qio *qio, unsigned int bi_op, struct qcow2 *qcow2)
{
qio->qcow2 = qcow2;
@@ -374,7 +376,9 @@ void qcow2_init_qio(struct qio *qio, unsigned int bi_op, struct qcow2 *qcow2)
qio->flags = 0;
qio->ref_index = REF_INDEX_INVALID;
qio->writer_gen = 0;
+ qio->state = QIO_STATE_START;
atomic_set(&qio->remaining, 1);
+ INIT_WORK(&qio->worker, do_qio_work);
/*
* Initially set into BLK_STS_OK, while aio complete,
@@ -3985,10 +3989,10 @@ static void qio_init_iter_from_rq(struct qio *qio, struct request *rq)
qio->bi_iter.bi_bvec_done = 0;
}
-static void prepare_one_embedded_qio(struct qcow2 *qcow2, struct qio *qio,
- struct list_head *deferred_qios)
+static void prepare_one_embedded_qio(struct qio *qio, struct list_head *deferred_qios)
{
struct qcow2_rq *qrq = qio->endio_cb_data;
+ struct qcow2 *qcow2 = qio->qcow2;
struct request *rq = qrq->rq;
struct bio_vec *bvec = NULL;
LIST_HEAD(list);
@@ -4033,15 +4037,6 @@ static void prepare_one_embedded_qio(struct qcow2 *qcow2, struct qio *qio,
qio_endio(qio);
}
-static void process_embedded_qios(struct qcow2 *qcow2, struct list_head *qios,
- struct list_head *deferred_qios)
-{
- struct qio *qio;
-
- while ((qio = qio_list_pop(qios)))
- prepare_one_embedded_qio(qcow2, qio, deferred_qios);
-}
-
static void process_deferred_qios(struct qcow2 *qcow2, struct list_head *qios)
{
struct qio *qio;
@@ -4561,7 +4556,6 @@ static void process_seek_qios(struct qcow2 *qcow, struct list_head *qios)
void do_qcow2_work(struct work_struct *ws)
{
struct qcow2 *qcow2 = container_of(ws, struct qcow2, worker);
- LIST_HEAD(embedded_qios);
LIST_HEAD(deferred_qios);
LIST_HEAD(zread_qios);
LIST_HEAD(bwrite_qios);
@@ -4574,7 +4568,6 @@ void do_qcow2_work(struct work_struct *ws)
current->flags |= PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO;
spin_lock_irq(&qcow2->deferred_lock);
- list_splice_init(&qcow2->qios[QLIST_EMBEDDED], &embedded_qios);
list_splice_init(&qcow2->qios[QLIST_DEFERRED], &deferred_qios);
list_splice_init(&qcow2->qios[QLIST_ZREAD], &zread_qios);
list_splice_init(&qcow2->qios[QLIST_BMERGE_WRITE], &bwrite_qios);
@@ -4585,7 +4578,6 @@ void do_qcow2_work(struct work_struct *ws)
list_splice_init(&qcow2->qios[QLIST_SEEK], &seek_qios);
spin_unlock_irq(&qcow2->deferred_lock);
- process_embedded_qios(qcow2, &embedded_qios, &deferred_qios);
process_deferred_qios(qcow2, &deferred_qios);
process_compressed_read(&zread_qios, &cow_data_qios);
process_backward_merge_write(qcow2, &bwrite_qios);
@@ -4710,7 +4702,7 @@ static void qcow2_submit_embedded_qio(struct qcow2_target *tgt, struct qio *qio)
if (blk_rq_bytes(rq)) {
queue_list_id = QLIST_EMBEDDED;
- worker = &qcow2->worker;
+ worker = &qio->worker;
} else {
WARN_ON_ONCE(qio->bi_op != REQ_OP_FLUSH);
queue_list_id = QLIST_FLUSH;
@@ -4726,7 +4718,10 @@ static void qcow2_submit_embedded_qio(struct qcow2_target *tgt, struct qio *qio)
qio->qcow2 = qcow2;
qio->queue_list_id = queue_list_id;
qio->ref_index = ref_index;
- list_add_tail(&qio->link, &qcow2->qios[qio->queue_list_id]);
+ if (queue_list_id == QLIST_EMBEDDED)
+ qio->state = QIO_STATE_START;
+ else
+ list_add_tail(&qio->link, &qcow2->qios[qio->queue_list_id]);
}
spin_unlock_irqrestore(&qcow2->deferred_lock, flags);
@@ -4822,6 +4817,39 @@ static void handle_cleanup_mask(struct qio *qio)
}
}
+static void do_qio_work(struct work_struct *ws)
+{
+ struct qio *qio = container_of(ws, struct qio, worker);
+ struct qcow2 *qcow2 = qio->qcow2;
+ struct qcow2_target *tgt = qcow2->tgt;
+ unsigned int pflags = current->flags;
+
+ current->flags |= PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO;
+
+ switch (qio->state) {
+ case QIO_STATE_START: {
+ LIST_HEAD(deferred_qios);
+
+ prepare_one_embedded_qio(qio, &deferred_qios);
+ if (list_empty(&deferred_qios))
+ break;
+
+ spin_lock_irq(&qcow2->deferred_lock);
+ list_splice_tail(&deferred_qios, &qcow2->qios[QLIST_DEFERRED]);
+ spin_unlock_irq(&qcow2->deferred_lock);
+ queue_work(tgt->wq, &qcow2->worker);
+ break;
+ }
+
+ default:
+ qio->bi_status = BLK_STS_IOERR;
+ qio_endio(qio);
+ break;
+ }
+
+ current_restore_flags(pflags, PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO);
+}
+
struct qio_data_llseek_hole {
struct completion compl;
struct qio *higher;
diff --git a/drivers/md/dm-qcow2.h b/drivers/md/dm-qcow2.h
index 3f1d5b5eef26f..ff46cce8587e2 100644
--- a/drivers/md/dm-qcow2.h
+++ b/drivers/md/dm-qcow2.h
@@ -227,8 +227,10 @@ struct qcow2_target {
enum {
QLIST_EMBEDDED = 0, /*
- * List for initial setup embedded qios
- * related to prq (in process context).
+ * Only a queue_list_id, qios are never
+ * linked to this list: it marks embedded
+ * qios related to prq, which are set up
+ * outside main kwork.
* This is used only for top qcow2 image.
*/
QLIST_DEFERRED,
@@ -245,6 +247,10 @@ enum {
QLIST_INVALID = QLIST_COUNT,
};
+enum {
+ QIO_STATE_START = 0,
+};
+
struct qcow2 {
struct qcow2_target *tgt;
struct file *file;
@@ -386,6 +392,8 @@ struct qio {
struct qcow2 *qcow2;
qcow2_endio_t endio_cb;
void *endio_cb_data;
+ struct work_struct worker;
+ int state;
};
#define CLU_OFF(qcow2, pos) (pos & (qcow2->clu_size - 1))
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
` (3 preceding siblings ...)
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 4/8] dm-qcow2: add per-qio work infrastructure Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:31 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 6/8] dm-qcow2: add fast_path module parameter Andrey Zhadchenko
` (3 subsequent siblings)
8 siblings, 2 replies; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
Each qio is handled by its own work and process_one_qio() can now
work with qios not requiring metadata. Try to process qios right
after splitting. Insert all leftovers into respective lists for
do_qcow2_work(), keeping the order of the splits.
Now process_one_qio() runs in parallel with do_qcow2_work(), so main
kwork may change an L2 entry between the fast path parse and the data
submit. A fast path write or partial discard is counted in md writers
under the lock that checks its L2 entry, so a discard unmapping the
cluster waits for it before the cluster is freed.
Known limitation: reads are not counted, and nothing pins their
cluster between the parse and the submit. Main kwork may discard the
cluster, free it and reuse it for another allocation in that window,
and a fast path read may then return the data of another guest
offset. A follow-up series fixes this in VSTOR-154659.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v2: list_add_tail() keeps the order of deferred splits; commit message
mentions the race with md writeback; fixed "it's" typo
v3: commit message describes the window between the fast path parse
and the data submit and the known limitation for reads, which a
follow-up series fixes; reworded the line starting with "From"
drivers/md/dm-qcow2-map.c | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
index 496fbcc613488..fb2a1c7c42111 100644
--- a/drivers/md/dm-qcow2-map.c
+++ b/drivers/md/dm-qcow2-map.c
@@ -4817,9 +4817,18 @@ static void handle_cleanup_mask(struct qio *qio)
}
}
+static void qio_try_fast(struct qio *qio, struct list_head *deferred_qios)
+{
+ list_del(&qio->link);
+
+ if (process_one_qio(qio->qcow2, qio, true))
+ list_add_tail(&qio->link, deferred_qios);
+}
+
static void do_qio_work(struct work_struct *ws)
{
struct qio *qio = container_of(ws, struct qio, worker);
+ struct qio *i, *next;
struct qcow2 *qcow2 = qio->qcow2;
struct qcow2_target *tgt = qcow2->tgt;
unsigned int pflags = current->flags;
@@ -4829,8 +4838,12 @@ static void do_qio_work(struct work_struct *ws)
switch (qio->state) {
case QIO_STATE_START: {
LIST_HEAD(deferred_qios);
+ LIST_HEAD(split_qios);
+
+ prepare_one_embedded_qio(qio, &split_qios);
+ list_for_each_entry_safe(i, next, &split_qios, link)
+ qio_try_fast(i, &deferred_qios);
- prepare_one_embedded_qio(qio, &deferred_qios);
if (list_empty(&deferred_qios))
break;
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 6/8] dm-qcow2: add fast_path module parameter
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
` (4 preceding siblings ...)
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:31 ` no-reply
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue Andrey Zhadchenko
` (2 subsequent siblings)
8 siblings, 1 reply; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
Add a writable fast_path module parameter, on by default. When it is
off, every split of a request goes to the main kwork deferred list,
as before the fast path, so the fast path can be turned off at
runtime if it misbehaves, and the two modes can be compared on the
same device.
It is checked for each split in qio_try_fast(), so a change applies
to the splits prepared after it. Splits already on the fast path
complete there.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v3: new patch
drivers/md/dm-qcow2-map.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
index fb2a1c7c42111..5d70ab5d68f4c 100644
--- a/drivers/md/dm-qcow2-map.c
+++ b/drivers/md/dm-qcow2-map.c
@@ -15,6 +15,11 @@
#include "dm-rq.h"
#include "dm-qcow2.h"
+static bool fast_path = true;
+module_param(fast_path, bool, 0644);
+MODULE_PARM_DESC(fast_path,
+ "Process qios not changing metadata outside main kwork");
+
/* "Exactly one bit" has the same number in L1 and L2 */
#define LX_REFCOUNT_EXACTLY_ONE (1ULL << 63)
#define L1_RESERVED_ZERO_MASK 0x7F000000000001FFULL
@@ -4821,7 +4826,7 @@ static void qio_try_fast(struct qio *qio, struct list_head *deferred_qios)
{
list_del(&qio->link);
- if (process_one_qio(qio->qcow2, qio, true))
+ if (!READ_ONCE(fast_path) || process_one_qio(qio->qcow2, qio, true))
list_add_tail(&qio->link, deferred_qios);
}
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
` (5 preceding siblings ...)
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 6/8] dm-qcow2: add fast_path module parameter Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:33 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 8/8] dm-qcow2: declare DM_TARGET_NOWAIT Andrey Zhadchenko
2026-11-13 19:48 ` [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Konstantin Khorenko
8 siblings, 2 replies; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
queue_work() is pretty costly under high load. It takes pool->lock;
unbound pools do not track running workers (WORKER_UNBOUND is part
of WORKER_NOT_RUNNING), so an idle kworker is woken (idle exit + IPI +
context switch) even if a running worker would pick the work up right
away. Every qio has its own work, so this is paid for every I/O. With
several submitters this ends up in several context switches per I/O
and in contention on pool->lock.
Batch requests with percpu llists. Additionally track if the worker
is busy and queue_work() only if it is not. The worker processes one
batch per run and requeues itself if more qios came meanwhile, so
flush_workqueue(), e.g. on backward merge completion with I/O still
running, does not wait for a worker which is refilled all the time.
The fast path is now serial per submitting CPU: qios queued from one
CPU are processed one by one by its work, while qios from different
CPUs are still processed in parallel. The data I/O itself does not
wait in the worker: O_DIRECT is enforced for every delta, so a fast
path qio is submitted as direct AIO. Still, the worker may sleep, and
the qios queued after it on that CPU wait then:
- preparation allocates under memory pressure: qcow2_alloc_qio() for
the splits does mempool_alloc(GFP_NOIO), and create_bvec_from_rq()
for a multi-bio request does kvmalloc_array(GFP_NOIO);
- direct AIO submission may sleep on the image inode lock, held
exclusively by punch hole, fallocate or truncate of the image;
- a partial discard punches the hole synchronously in the worker.
None of this is new: before the fast path main kwork did the same
allocations and submissions for every request, and a sleep there
stalled the whole device, not one CPU. The punch hole case is the
notable one, as it waits for all direct I/O in flight on the image.
Most of the qios queued behind it would wait anyway: on ext4 and xfs
punch hole takes the inode lock exclusively, and direct I/O
submission to the same image sleeps on it.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v2: moved before "declare DM_TARGET_NOWAIT"; process one batch per run
and requeue, so flush_workqueue() does not wait for a work which is
refilled all the time; commit message: fixed the claim about
re-queued qios, describes that the fast path is serial per
submitting CPU and that partial discards punch the hole in the
worker, measurements moved to the thread
v3: dropped smp_mb() after llist_add(), which is fully ordered; commit
message lists all the steps where the worker may sleep
drivers/md/dm-qcow2-map.c | 102 +++++++++++++++++++++++++++++++----
drivers/md/dm-qcow2-target.c | 10 +++-
drivers/md/dm-qcow2.h | 6 ++-
3 files changed, 105 insertions(+), 13 deletions(-)
diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
index 5d70ab5d68f4c..eaeb7e73c4023 100644
--- a/drivers/md/dm-qcow2-map.c
+++ b/drivers/md/dm-qcow2-map.c
@@ -92,6 +92,7 @@ static void process_read_qio(struct qcow2 *qcow2, struct qio *qio,
struct qcow2_map *map);
static void init_qrq_and_embedded_qio(struct qcow2_target *tgt, struct request *rq,
struct qcow2_rq *qrq, struct qio *qio);
+static void qcow2_fast_submit(struct qcow2_target *tgt, struct qio *qio);
static loff_t bytes_off_in_cluster(struct qcow2 *qcow2, struct qio *qio)
{
@@ -367,8 +368,6 @@ struct qio *qcow2_alloc_qio(mempool_t *pool, bool zero)
}
ALLOW_ERROR_INJECTION(qcow2_alloc_qio, NULL);
-static void do_qio_work(struct work_struct *ws);
-
void qcow2_init_qio(struct qio *qio, unsigned int bi_op, struct qcow2 *qcow2)
{
qio->qcow2 = qcow2;
@@ -383,7 +382,6 @@ void qcow2_init_qio(struct qio *qio, unsigned int bi_op, struct qcow2 *qcow2)
qio->writer_gen = 0;
qio->state = QIO_STATE_START;
atomic_set(&qio->remaining, 1);
- INIT_WORK(&qio->worker, do_qio_work);
/*
* Initially set into BLK_STS_OK, while aio complete,
@@ -4707,7 +4705,7 @@ static void qcow2_submit_embedded_qio(struct qcow2_target *tgt, struct qio *qio)
if (blk_rq_bytes(rq)) {
queue_list_id = QLIST_EMBEDDED;
- worker = &qio->worker;
+ worker = NULL;
} else {
WARN_ON_ONCE(qio->bi_op != REQ_OP_FLUSH);
queue_list_id = QLIST_FLUSH;
@@ -4730,8 +4728,13 @@ static void qcow2_submit_embedded_qio(struct qcow2_target *tgt, struct qio *qio)
}
spin_unlock_irqrestore(&qcow2->deferred_lock, flags);
- if (queue)
+ if (!queue)
+ return;
+
+ if (worker)
queue_work(tgt->wq, worker);
+ else
+ qcow2_fast_submit(tgt, qio);
}
void qcow2_submit_embedded_qios(struct qcow2_target *tgt, struct list_head *list)
@@ -4830,15 +4833,11 @@ static void qio_try_fast(struct qio *qio, struct list_head *deferred_qios)
list_add_tail(&qio->link, deferred_qios);
}
-static void do_qio_work(struct work_struct *ws)
+static void do_qio_work(struct qio *qio)
{
- struct qio *qio = container_of(ws, struct qio, worker);
- struct qio *i, *next;
struct qcow2 *qcow2 = qio->qcow2;
struct qcow2_target *tgt = qcow2->tgt;
- unsigned int pflags = current->flags;
-
- current->flags |= PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO;
+ struct qio *i, *next;
switch (qio->state) {
case QIO_STATE_START: {
@@ -4864,8 +4863,89 @@ static void do_qio_work(struct work_struct *ws)
qio_endio(qio);
break;
}
+}
+
+struct qcow2_fast_cpu {
+ struct llist_head qios;
+ struct work_struct work;
+ struct qcow2_target *tgt;
+ atomic_t busy;
+};
+
+static void qcow2_fast_submit(struct qcow2_target *tgt, struct qio *qio)
+{
+ struct qcow2_fast_cpu *fc = get_cpu_ptr(tgt->fast);
+
+ /*
+ * llist_add() is fully ordered (try_cmpxchg()),
+ * pairs with smp_mb() in qcow2_fast_work()
+ */
+ llist_add(&qio->fast_link, &fc->qios);
+ if (!atomic_read(&fc->busy) && atomic_cmpxchg(&fc->busy, 0, 1) == 0)
+ queue_work(tgt->wq, &fc->work);
+ put_cpu_ptr(tgt->fast);
+}
+
+static void qcow2_fast_work(struct work_struct *work)
+{
+ struct qcow2_fast_cpu *fc = container_of(work, struct qcow2_fast_cpu, work);
+ unsigned int pflags = current->flags;
+ struct llist_node *node;
+ struct qio *qio, *next;
+
+ current->flags |= PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO;
+
+ node = llist_del_all(&fc->qios);
+ node = llist_reverse_order(node);
+ llist_for_each_entry_safe(qio, next, node, fast_link)
+ do_qio_work(qio);
current_restore_flags(pflags, PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO);
+
+ if (llist_empty(&fc->qios)) {
+ atomic_set(&fc->busy, 0);
+ /* Pairs with llist_add() in qcow2_fast_submit() */
+ smp_mb();
+ if (llist_empty(&fc->qios))
+ return;
+ if (atomic_cmpxchg(&fc->busy, 0, 1) != 0)
+ return;
+ }
+ queue_work(fc->tgt->wq, &fc->work);
+}
+
+void qcow2_fast_exit(struct qcow2_target *tgt)
+{
+ struct qcow2_fast_cpu *fc;
+ int cpu;
+
+ if (!tgt->fast)
+ return;
+ for_each_possible_cpu(cpu) {
+ fc = per_cpu_ptr(tgt->fast, cpu);
+ WARN_ON_ONCE(!llist_empty(&fc->qios));
+ }
+ free_percpu(tgt->fast);
+ tgt->fast = NULL;
+}
+
+int qcow2_fast_init(struct qcow2_target *tgt)
+{
+ struct qcow2_fast_cpu *fc;
+ int cpu;
+
+ tgt->fast = alloc_percpu(struct qcow2_fast_cpu);
+ if (!tgt->fast)
+ return -ENOMEM;
+ for_each_possible_cpu(cpu) {
+ fc = per_cpu_ptr(tgt->fast, cpu);
+ init_llist_head(&fc->qios);
+ INIT_WORK(&fc->work, qcow2_fast_work);
+ fc->tgt = tgt;
+ atomic_set(&fc->busy, 0);
+ }
+
+ return 0;
}
struct qio_data_llseek_hole {
diff --git a/drivers/md/dm-qcow2-target.c b/drivers/md/dm-qcow2-target.c
index f4004f6095a72..891a232ad5616 100644
--- a/drivers/md/dm-qcow2-target.c
+++ b/drivers/md/dm-qcow2-target.c
@@ -253,6 +253,7 @@ static void qcow2_tgt_destroy(struct qcow2_target *tgt)
/* Now kill the queue */
destroy_workqueue(tgt->wq);
}
+ qcow2_fast_exit(tgt);
qcow2_merge_backward_cancel(tgt);
mempool_destroy(tgt->qio_pool);
@@ -523,6 +524,11 @@ static struct qcow2_target *alloc_qcow2_target(struct dm_target *ti)
goto out_pool;
}
+ if (qcow2_fast_init(tgt)) {
+ ti->error = "Can't alloc fast path slots";
+ goto out_wq;
+ }
+
for (i = 0; i < 2; i++) {
release = i ? inflight_ref_exit1 : inflight_ref_exit0;
if (percpu_ref_init(&tgt->inflight_ref[i], release,
@@ -530,7 +536,7 @@ static struct qcow2_target *alloc_qcow2_target(struct dm_target *ti)
if (i)
percpu_ref_exit(&tgt->inflight_ref[0]);
ti->error = "could not alloc percpu_ref";
- goto out_wq;
+ goto out_fast;
}
}
@@ -550,6 +556,8 @@ static struct qcow2_target *alloc_qcow2_target(struct dm_target *ti)
qcow2_set_service_operations(ti, false);
return tgt;
+out_fast:
+ qcow2_fast_exit(tgt);
out_wq:
destroy_workqueue(tgt->wq);
out_pool:
diff --git a/drivers/md/dm-qcow2.h b/drivers/md/dm-qcow2.h
index ff46cce8587e2..2b73c5f7c6597 100644
--- a/drivers/md/dm-qcow2.h
+++ b/drivers/md/dm-qcow2.h
@@ -4,6 +4,7 @@
#include <linux/percpu-refcount.h>
#include <linux/device-mapper.h>
+#include <linux/llist.h>
#include <linux/fs.h>
#include <linux/eventfd.h>
@@ -199,6 +200,7 @@ struct qcow2_target {
*/
struct qcow2 *top;
struct workqueue_struct *wq;
+ struct qcow2_fast_cpu __percpu *fast;
struct completion inflight_ref_comp;
struct percpu_ref inflight_ref[2];
@@ -392,7 +394,7 @@ struct qio {
struct qcow2 *qcow2;
qcow2_endio_t endio_cb;
void *endio_cb_data;
- struct work_struct worker;
+ struct llist_node fast_link;
int state;
};
@@ -409,6 +411,8 @@ loff_t qcow2_llseek_hole(struct dm_target *ti, loff_t offset, int whence);
void do_qcow2_work(struct work_struct *ws);
void do_qcow2_fsync_work(struct work_struct *ws);
+int qcow2_fast_init(struct qcow2_target *tgt);
+void qcow2_fast_exit(struct qcow2_target *tgt);
int qcow2_alloc_and_insert_md_page(struct qcow2 *qcow2, u64 index,
struct md_page **md);
int qcow2_md_page_read_sync(struct qcow2 *qcow2, u64 id,
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* [Devel] [PATCH hci-8.1 v3 8/8] dm-qcow2: declare DM_TARGET_NOWAIT
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
` (6 preceding siblings ...)
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue Andrey Zhadchenko
@ 2026-11-13 11:11 ` Andrey Zhadchenko
2026-11-13 11:33 ` no-reply
2026-11-13 19:48 ` [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Konstantin Khorenko
8 siblings, 1 reply; 22+ messages in thread
From: Andrey Zhadchenko @ 2026-11-13 11:11 UTC (permalink / raw)
To: khorenko, alexander.ivanov; +Cc: devel
dm-qcow2 is a request-based target: REQ_NOWAIT handling is done by blk-mq
(tag allocation) before a request reaches us, and clone_and_map_rq()
never blocks. Still, the target does not declare DM_TARGET_NOWAIT, so
dm_table_set_restrictions() clears BLK_FEAT_NOWAIT on the mapped device
and the block device file gets no FMODE_NOWAIT.
As a result io_uring cannot issue I/O in nowait mode and punts every
request to an io-wq worker thread. Declare DM_TARGET_NOWAIT, so io_uring
submits inline and saves a thread handoff per I/O.
https://virtuozzo.atlassian.net/browse/VSTOR-95494
Feature: dm-qcow2: block device over QCOW2 files driver
Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
---
v2: moved after "process fast path qios in per-cpu batches on a
workqueue"; measurements moved to the thread
v3: no changes
drivers/md/dm-qcow2-target.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/md/dm-qcow2-target.c b/drivers/md/dm-qcow2-target.c
index 891a232ad5616..a4c7fe1dcc72d 100644
--- a/drivers/md/dm-qcow2-target.c
+++ b/drivers/md/dm-qcow2-target.c
@@ -1240,7 +1240,7 @@ static void qcow2_resume(struct dm_target *ti)
static struct target_type qcow2_target = {
.name = "qcow2",
.version = {1, 0, 0},
- .features = DM_TARGET_SINGLETON,
+ .features = DM_TARGET_SINGLETON | DM_TARGET_NOWAIT,
.module = THIS_MODULE,
.ctr = qcow2_ctr,
.dtr = qcow2_dtr,
--
2.43.5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel Andrey Zhadchenko
@ 2026-11-13 11:22 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:22 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] dm-qcow2: message and comment misdescribe -EIO on second lookup
miss
--
commit ac0961cf11bb91c58bc95e6d03ca39499fd6547e
Author: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
dm-qcow2: repeat md page lookup if it was inserted in parallel
This teaches __handle_md_page() to repeat the md page lookup once when submit_read_md_page() returns -EEXIST. -EEXIST means qcow2_llseek_hole() and the main kwork raced to insert the same md page.
> diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
> index 32373114580e..58b95d7a955c 100644
> --- a/drivers/md/dm-qcow2-map.c
> +++ b/drivers/md/dm-qcow2-map.c
> @@ -1676,20 +1676,33 @@ static int submit_read_md_page(struct qcow2 *qcow2, struct qio **qio,
> /*
> * This may be called with @qio == NULL, in case of we are
> * interesting in searching cached in memory md only.
> - * This is aimed to be called not only from main kwork
> - * for L1/L2 pages, so all callers looking for L1/L2
> - * must care about submit_read_md_page() may return EEXIST.
> + *
> + * submit_read_md_page() drops md_pages_lock, so the page may be
> + * inserted in parallel. Then it is looked up once more. It is not
> + * found again only if its read failed in the meantime, and then
> + * @qio fails too.
> */
> static int __handle_md_page(struct qcow2 *qcow2, u64 page_id,
> struct qio **qio, struct md_page **ret_md)
> {
> + bool retried = false;
> struct md_page *md;
> + int ret;
>
> lockdep_assert_held(&qcow2->md_pages_lock);
> +again:
> md = qcow2_md_page_find_or_postpone(qcow2, page_id, qio);
> if (!md) {
> - if (qio && *qio)
> - return submit_read_md_page(qcow2, qio, page_id);
> + if (qio && *qio) {
> + ret = submit_read_md_page(qcow2, qio, page_id);
> + if (unlikely(ret == -EEXIST)) {
> + if (retried)
> + return -EIO;
> + retried = true;
> + goto again;
> + }
> + return ret;
> + }
> return 0;
> }
>
[Severity: Low]
This isn't a bug, but does the code match what the comment and commit message say about a second lookup miss?
They say the second lookup misses only if the read failed and erased the page, and that the qio then fails with -EIO. The code does something else. After the retry lookup misses with *qio still set, it calls submit_read_md_page() again. That call allocates and inserts a fresh md page, queues the qio on it and issues a new read. So the qio is retried, not failed.
-EIO is returned only when the second submit_read_md_page() also returns -EEXIST, which means two insert races in a row. Could the comment and commit message describe that instead? Alternatively, the retry path could fail the qio directly when the second lookup misses.
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=1
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 2/8] dm-qcow2: prepare process_one_qio() for fastpath
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 2/8] dm-qcow2: prepare process_one_qio() for fastpath Andrey Zhadchenko
@ 2026-11-13 11:26 ` no-reply
0 siblings, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:26 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
> dm-qcow2 heavily relies on main kwork do_qcow2_work() being
> sequential. All metadata change happens there and cannot
> run in parallel. Unfortunately that caps dm-qcow2 performance.
> But if the request does not require metadata changes we should
> be able to immediately dispatch it.
> process_one_qio() and handle_metadata() already know what to
> do: just add a fast flag to process_one_qio(), passed as
> avoid_metadata_changes to handle_metadata(), and return codes.
> While at it, just return in handle_metadata() instead of
> setting ret to 0 and jumping to if (ret < 0) label.
>
> Also pass the flag to parse_metadata() and return to the slow path
> before it looks up R1/R2 pages. They are needed only by COW writes
> and cluster unmapping discards, which go to the slow path anyway.
> Besides the useless lookup, R1 is unstable outside main kwork: an R1
> [ ... ]
> with wpc_readers.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-95494
> Feature: dm-qcow2: block device over QCOW2 files driver
> Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=2
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata()
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata() Andrey Zhadchenko
@ 2026-11-13 11:27 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:27 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
> perform_rw_mapped() reads md->status and md->wbd without
> md_pages_lock and counts the write in md writers, relying on
> dm-qcow2 sequential nature: main kwork parses a qio and submits it
> in one pass. Outside main kwork an L1/L2 page changes only when its
> writeback ends, and parse_l2() already delays writes to the changed
> indexes of a page under writeback.
>
> This does not hold for the per-qio fast path, which parses L2 and
> submits in parallel with main kwork. The fast path takes only writes
> to fully allocated clusters mapped exactly once, whose indexes are
> neither locked nor changed in a dirty or under writeback L2 page. In
> between, main kwork may still discard or zero the same range, which
> is a valid race with the write, or allocate other subclusters of the
> cluster, which does not affect the write. But a discard unmapping the
> cluster lets it be reused, and a write landing after that corrupts
> [ ... ]
> attach, which the fast path never needs, and the writers increase.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-95494
> Feature: dm-qcow2: block device over QCOW2 files driver
> Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=3
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 4/8] dm-qcow2: add per-qio work infrastructure
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 4/8] dm-qcow2: add per-qio work infrastructure Andrey Zhadchenko
@ 2026-11-13 11:30 ` no-reply
0 siblings, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:30 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] dm-qcow2: concurrent qio preparers hold partial splits while
sleeping in mempool_alloc()
- [Low] dm-qcow2: redundant qio->state init in
qcow2_submit_embedded_qio()
--
commit f0e3abee17282a35ee80199fc8894df280f02698
Author: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
dm-qcow2: add per-qio work infrastructure
This commit adds a work_struct and a state field to struct qio. It uses them to queue qio preparation (prepare_one_embedded_qio()) as a per-request work item. Previously this ran inside the main qcow2->worker.
> diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
> index b9daa258285e..496fbcc61348 100644
> --- a/drivers/md/dm-qcow2-map.c
> +++ b/drivers/md/dm-qcow2-map.c
> @@ -4710,7 +4702,7 @@ static void qcow2_submit_embedded_qio(struct qcow2_target *tgt, struct qio *qio)
>
> if (blk_rq_bytes(rq)) {
> queue_list_id = QLIST_EMBEDDED;
> - worker = &qcow2->worker;
> + worker = &qio->worker;
> } else {
[Severity: Medium]
As reported by no-reply@sashiko.virtuozzo.com on v2 [1], can concurrent
preparers deadlock on qio_pool under memory pressure?
Every data request now gets its own work item on tgt->wq, which is
WQ_UNBOUND, so many of them can run at once. Each one runs
do_qio_work() -> prepare_one_embedded_qio() -> qcow2_split_qio_to_list()
-> split_and_chain_qio(), which calls qcow2_alloc_qio() on the shared
512-element qio_pool. That is a blocking mempool_alloc(GFP_NOIO).
The splits are kept on a local list and are released only after the whole
request has been split. Several preparers can each hold part of the
reserve while sleeping in mempool_alloc() for more. None of them can make
progress, and the device hangs. Before this patch only one preparer ran
at a time, inside qcow2->worker.
Would a non-blocking first pass, with a serialized sleeping fallback as
dm-crypt does, avoid this?
> @@ -4726,7 +4718,10 @@ static void qcow2_submit_embedded_qio(struct qcow2_target *tgt, struct qio *qio)
> qio->qcow2 = qcow2;
> qio->queue_list_id = queue_list_id;
> qio->ref_index = ref_index;
> - list_add_tail(&qio->link, &qcow2->qios[qio->queue_list_id]);
> + if (queue_list_id == QLIST_EMBEDDED)
> + qio->state = QIO_STATE_START;
> + else
> + list_add_tail(&qio->link, &qcow2->qios[qio->queue_list_id]);
> }
> spin_unlock_irqrestore(&qcow2->deferred_lock, flags);
[Severity: Low]
This isn't a bug, but is this qio->state assignment needed?
qcow2_init_qio() already sets qio->state = QIO_STATE_START before the qio
is submitted, and QIO_STATE_START is the only state. Dropping it would
also remove the if/else inside the deferred_lock section, leaving just
the list_add_tail() for the non-embedded case.
[1] https://lore.kernel.org/r/010701a1c556541f-a3b013d6-70e1-45b6-bfc1-37c018cdf88f-000000@eu-central-1.amazonses.com
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=4
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 6/8] dm-qcow2: add fast_path module parameter
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 6/8] dm-qcow2: add fast_path module parameter Andrey Zhadchenko
@ 2026-11-13 11:31 ` no-reply
0 siblings, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:31 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
> Add a writable fast_path module parameter, on by default. When it is
> off, every split of a request goes to the main kwork deferred list,
> as before the fast path, so the fast path can be turned off at
> runtime if it misbehaves, and the two modes can be compared on the
> same device.
>
> It is checked for each split in qio_try_fast(), so a change applies
> to the splits prepared after it. Splits already on the fast path
> complete there.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-95494
> Feature: dm-qcow2: block device over QCOW2 files driver
> Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=6
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing Andrey Zhadchenko
@ 2026-11-13 11:31 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:31 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] dm-qcow2: unpinned fast-path reads can hit a freed and reused
cluster
--
commit d4a94f4cb8fd390d1180cc72ba95c9cab302fcdc
Author: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
dm-qcow2: implement per-qio fast processing
This commit tries to process each qio right after it is split, in the per-qio work. Qios that still need metadata work go back onto the deferred list for the main kwork.
> diff --git a/drivers/md/dm-qcow2-map.c b/drivers/md/dm-qcow2-map.c
> index 496fbcc61348..fb2a1c7c4211 100644
> --- a/drivers/md/dm-qcow2-map.c
> +++ b/drivers/md/dm-qcow2-map.c
> @@ -4817,9 +4817,18 @@ static void handle_cleanup_mask(struct qio *qio)
> }
> }
>
> +static void qio_try_fast(struct qio *qio, struct list_head *deferred_qios)
> +{
> + list_del(&qio->link);
> +
> + if (process_one_qio(qio->qcow2, qio, true))
> + list_add_tail(&qio->link, deferred_qios);
> +}
> +
[Severity: Medium]
As reported by Konstantin Khorenko on v2 [1], can a fast-path read hit a cluster that has been freed and reused?
qio_try_fast() runs process_one_qio() in parallel with the main kwork. For a read, parse_metadata() takes the data cluster position from the L2 entry and then drops md_pages_lock. It does not pin the cluster, because inc_writers() is only called for writes and partial discards. process_read_qio() then calls perform_rw_mapped(..., false) against that stale position, with no revalidation.
If the fast worker stalls between the parse and the submit, the main kwork can discard the cluster, write back the L2 entry and free the cluster. It can then hand the cluster to another allocation. The read then returns data belonging to a different guest offset. If another consumer reuses the cluster, this also leaks that consumer's data.
The commit message documents this and defers the fix to VSTOR-154659. The race still ships with this patch, and nothing later in the series adds read pinning.
[1] https://lore.kernel.org/r/179433339615.3573054.12995587619826112869.b4-review@b4
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=5
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue Andrey Zhadchenko
@ 2026-11-13 11:33 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:33 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
> queue_work() is pretty costly under high load. It takes pool->lock;
> unbound pools do not track running workers (WORKER_UNBOUND is part
> of WORKER_NOT_RUNNING), so an idle kworker is woken (idle exit + IPI +
> context switch) even if a running worker would pick the work up right
> away. Every qio has its own work, so this is paid for every I/O. With
> several submitters this ends up in several context switches per I/O
> and in contention on pool->lock.
>
> Batch requests with percpu llists. Additionally track if the worker
> is busy and queue_work() only if it is not. The worker processes one
> batch per run and requeues itself if more qios came meanwhile, so
> flush_workqueue(), e.g. on backward merge completion with I/O still
> running, does not wait for a worker which is refilled all the time.
>
> The fast path is now serial per submitting CPU: qios queued from one
> [ ... ]
> submission to the same image sleeps on it.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-95494
> Feature: dm-qcow2: block device over QCOW2 files driver
> Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=7
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 8/8] dm-qcow2: declare DM_TARGET_NOWAIT
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 8/8] dm-qcow2: declare DM_TARGET_NOWAIT Andrey Zhadchenko
@ 2026-11-13 11:33 ` no-reply
0 siblings, 0 replies; 22+ messages in thread
From: no-reply @ 2026-11-13 11:33 UTC (permalink / raw)
To: Andrey Zhadchenko, khorenko, alexander.ivanov; +Cc: devel
> dm-qcow2 is a request-based target: REQ_NOWAIT handling is done by blk-mq
> (tag allocation) before a request reaches us, and clone_and_map_rq()
> never blocks. Still, the target does not declare DM_TARGET_NOWAIT, so
> dm_table_set_restrictions() clears BLK_FEAT_NOWAIT on the mapped device
> and the block device file gets no FMODE_NOWAIT.
>
> As a result io_uring cannot issue I/O in nowait mode and punts every
> request to an io-wq worker thread. Declare DM_TARGET_NOWAIT, so io_uring
> submits inline and saves a thread handoff per I/O.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-95494
> Feature: dm-qcow2: block device over QCOW2 files driver
> Signed-off-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113111126.793039-1-andrey.zhadchenko@virtuozzo.com?part=8
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
` (7 preceding siblings ...)
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 8/8] dm-qcow2: declare DM_TARGET_NOWAIT Andrey Zhadchenko
@ 2026-11-13 19:48 ` Konstantin Khorenko
8 siblings, 0 replies; 22+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:48 UTC (permalink / raw)
To: Andrey Zhadchenko; +Cc: devel
I have reviewed v3: no objections to the code, but a few texts need
changing, see below.
I can change them myself when applying the patches, or you send v4 -
your call, Andrey. Either way I am waiting for Alexander Ivanov's
review.
On Fri, 13 Nov 2026 14:11:18 +0300, Andrey Zhadchenko wrote:
> Known limitation: fast path reads are not counted, and nothing pins
> their cluster between the L2 parse and the data submit. Main kwork
> may discard the cluster, free it and reuse it for another allocation
> in that window, and a fast path read then returns the data of another
> guest offset. A follow-up series fixes this:
> https://virtuozzo.atlassian.net/browse/VSTOR-154659
All v2 items are done. The writer count closes the failed L2 writeback
and the discard that frees a cluster under a counted write, and I
found no code defect in v3. What is left is text:
1. Patch 1, commit message and the comment above __handle_md_page():
a second miss does not fail the qio. submit_read_md_page() inserts
the page again and issues a new read; -EIO comes only when that
insert loses a race again.
2. Patch 3, commit message: "either the fast path sees the entry
cleared" becomes "either the fast path sees the index locked".
3. Patch 5 and the cover letter: VSTOR-154659 is resolved, it is the
write counting already in the tree. The read race is VSTOR-154939,
closed as Won't Fix. Say that it is accepted, and name VSTOR-154939
instead of a follow-up series.
4. Patch 7, commit message: add the md page read on a cache miss in
parse_metadata() and the read splits in process_read_qio() to the
list of sleeps, and say that punch hole waits for direct I/O on ext4
and xfs only.
Details are in the replies to the patches.
--
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel Andrey Zhadchenko
2026-11-13 11:22 ` no-reply
@ 2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:48 UTC (permalink / raw)
To: Andrey Zhadchenko; +Cc: devel
No objections to the code, a text change below: I fix it when
applying or you send v4, see the cover letter reply.
On Fri, 13 Nov 2026 14:11:19 +0300, Andrey Zhadchenko wrote:
> Repeat it once only. The second lookup misses the page only if its
> read failed and erased it in between, and then the qio fails with
> -EIO as the qios waiting for that read did.
Item 1 of the cover letter reply: describe what the code does on a
second miss, here and in the comment below.
> + * submit_read_md_page() drops md_pages_lock, so the page may be
> + * inserted in parallel. Then it is looked up once more. It is not
> + * found again only if its read failed in the meantime, and then
> + * @qio fails too.
Why: Sashiko is right on v3. When the second lookup misses,
__handle_md_page() calls submit_read_md_page() again, which inserts a
new page, queues the qio on it and reads it. The qio is retried, not
failed. -EIO is returned only if that second insert gets -EEXIST too.
Both outcomes are fine, only the text is wrong.
--
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata()
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata() Andrey Zhadchenko
2026-11-13 11:27 ` no-reply
@ 2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:48 UTC (permalink / raw)
To: Andrey Zhadchenko; +Cc: devel
No objections to the code, a text change below: I fix it when
applying or you send v4, see the cover letter reply.
On Fri, 13 Nov 2026 14:11:21 +0300, Andrey Zhadchenko wrote:
> md_pages_lock hold, which checks the L2 entry. A discard switches the
> writer generation after it clears the entry under the lock, so
> either the fast path sees the entry cleared, or the discard waits for
> the write. Drop the count, if the qio goes to main kwork afterwards.
Item 2 of the cover letter reply: "sees the entry cleared" becomes
"sees the index locked".
Why: prepare_cluster_discard() locks the index under md_pages_lock,
and from then on parse_l2() parks the fast qio in delay_if_locked().
The entry itself is cleared only later, in process_indexes_write().
The mechanism is right, the sentence names the wrong guard.
--
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing Andrey Zhadchenko
2026-11-13 11:31 ` no-reply
@ 2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:48 UTC (permalink / raw)
To: Andrey Zhadchenko; +Cc: devel
No objections to the code, a text change below: I fix it when
applying or you send v4, see the cover letter reply.
On Fri, 13 Nov 2026 14:11:23 +0300, Andrey Zhadchenko wrote:
> and a fast path read may then return the data of another guest
> offset. A follow-up series fixes this in VSTOR-154659.
Item 3 of the cover letter reply: say the limitation is accepted and
name VSTOR-154939.
Why: VSTOR-154659 is the write counting that is already in the tree, and
it is resolved. The read race was filed as VSTOR-154939 and closed as
Won't Fix: it needs a read and a discard of the same LBA in flight
together, which takes raw access to a disk its owner can read and write
anyway, and counting every read would cost the hot read path. No
follow-up series will come, so the message must not promise one.
--
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue Andrey Zhadchenko
2026-11-13 11:33 ` no-reply
@ 2026-11-13 19:48 ` Konstantin Khorenko
1 sibling, 0 replies; 22+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:48 UTC (permalink / raw)
To: Andrey Zhadchenko; +Cc: devel
No objections to the code, a text change below: I fix it when
applying or you send v4, see the cover letter reply.
On Fri, 13 Nov 2026 14:11:25 +0300, Andrey Zhadchenko wrote:
> - a partial discard punches the hole synchronously in the worker.
> None of this is new: before the fast path main kwork did the same
> allocations and submissions for every request, and a sleep there
> stalled the whole device, not one CPU. The punch hole case is the
> notable one, as it waits for all direct I/O in flight on the image.
Item 4 of the cover letter reply:
- add to the list: an md page cache miss in parse_metadata()
allocates the page and submits its read from the worker, and
process_read_qio() splits reads with mempool_alloc();
- "it waits for all direct I/O in flight on the image" holds on ext4
and xfs only.
Why: the v3 changelog says this list is complete, and these two sleep
in the worker too. fuse_file_fallocate() does not call
inode_dio_wait(), and images on vstorage are on FUSE, so there the
punch does not wait for direct I/O.
--
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2026-11-13 19:51 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-13 11:11 [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Andrey Zhadchenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 1/8] dm-qcow2: repeat md page lookup if it was inserted in parallel Andrey Zhadchenko
2026-11-13 11:22 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 2/8] dm-qcow2: prepare process_one_qio() for fastpath Andrey Zhadchenko
2026-11-13 11:26 ` no-reply
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 3/8] dm-qcow2: count fast path writes in parse_metadata() Andrey Zhadchenko
2026-11-13 11:27 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 4/8] dm-qcow2: add per-qio work infrastructure Andrey Zhadchenko
2026-11-13 11:30 ` no-reply
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 5/8] dm-qcow2: implement per-qio fast processing Andrey Zhadchenko
2026-11-13 11:31 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 6/8] dm-qcow2: add fast_path module parameter Andrey Zhadchenko
2026-11-13 11:31 ` no-reply
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 7/8] dm-qcow2: process fast path qios in per-cpu batches on a workqueue Andrey Zhadchenko
2026-11-13 11:33 ` no-reply
2026-11-13 19:48 ` Konstantin Khorenko
2026-11-13 11:11 ` [Devel] [PATCH hci-8.1 v3 8/8] dm-qcow2: declare DM_TARGET_NOWAIT Andrey Zhadchenko
2026-11-13 11:33 ` no-reply
2026-11-13 19:48 ` [Devel] [PATCH hci-8.1 v3 0/8] dm-qcow2: per-qio fast path Konstantin Khorenko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.