[Devel] [PATCH hci-8.1 0/4] dm-ploop: fix the pio submission stop around cmds

[Devel] [PATCH hci-8.1 0/4] dm-ploop: fix the pio submission stop around cmds

[Denis V. Lunev]

A cmd stops pio submission and waits for the inflight pios before it
rewrites the BAT levels or deltas[], and expects nothing to touch them
until it resumes. Konstantin found three ways this does not hold while
reviewing the VSTOR-151979 series (VSTOR-152985, VSTOR-152984,
VSTOR-152981):

- a submitter racing the stop either runs without the cmd waiting for
  it, or is parked after resume and never completes (patch 1);

- the ploop_allocator() thread is not stopped at all. It truncates the
  top delta while merge or flip moves it in deltas[], cutting the live
  image to the preallocation size (patch 3). The thread can also miss a
  wakeup, which stalls the device by itself and would hang the new wait,
  so patch 2 fixes that first;

- a grow failing to copy a cluster resumes submission with that
  cluster's BAT page still busy, and writes to it hang (patch 4).

Patch 3 is the one losing data: a snapshot delete under write load is
enough.

Tested on dev-cl04, stock 6.12.0-211.52.1.22.11.vz10, dm-ploop built
out of tree:
- each bug reproduced on the stock module, the races with a test-only
  kprobe module widening the window: a pio running past the suspend of
  a cmd; a pio never completing after resume; the top image truncated
  from 512M to 128M by a merge and to 256M by a flip; after a grow
  failing with ENOSPC on a full host fs, a writer stuck in D state, an
  unsuspendable device and a later cmd failing with ETIMEDOUT. None of
  them reproduce with the series;
- fio 4k random reads and writes, O_DIRECT, 1 to 48 jobs, on a fully
  allocated image over brd: IOPS and system CPU stay within the
  run-to-run spread of the stock module. This setup tops out at about
  420K IOPS elsewhere in dm-ploop and would not show a lock taken on
  submission either, so patch 1 keeps that path lockless;
- the ploop pytest suite, ploop and qcow2 formats, passes. A test for
  patch 4 goes to the ploop tests separately.

Denis V. Lunev (4):
  dm-ploop: fix pio submission racing with the suspend and resume of a
    cmd
  dm-ploop: fix a lost wakeup of the ploop_allocator() thread
  dm-ploop: wait for the ploop_allocator() thread before changing
    deltas[]
  dm-ploop: fix a failed cluster copy on grow leaving the BAT page busy

 drivers/md/dm-ploop-cmd.c    | 18 ++++++++++++++++--
 drivers/md/dm-ploop-map.c    | 22 +++++++++++++++-------
 drivers/md/dm-ploop-target.c |  4 ++++
 drivers/md/dm-ploop.h        |  1 +
 4 files changed, 36 insertions(+), 9 deletions(-)


base-commit: 09eb9a82fe872048d082391415d1cdeda2ac6879
-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 0/4] dm-ploop: fix the pio submission stop around cmds

[Denis V. Lunev]

A cmd stops pio submission and waits for the inflight pios before it
rewrites the BAT levels or deltas[], and expects nothing to touch them
until it resumes. Konstantin found three ways this does not hold while
reviewing the VSTOR-151979 series (VSTOR-152985, VSTOR-152984,
VSTOR-152981):

- a submitter racing the stop either runs without the cmd waiting for
  it, or is parked after resume and never completes (patch 1);

- the ploop_allocator() thread is not stopped at all. It truncates the
  top delta while merge or flip moves it in deltas[], cutting the live
  image to the preallocation size (patch 3). The thread can also miss a
  wakeup, which stalls the device by itself and would hang the new wait,
  so patch 2 fixes that first;

- a grow failing to copy a cluster resumes submission with that
  cluster's BAT page still busy, and writes to it hang (patch 4).

Patch 3 is the one losing data: a snapshot delete under write load is
enough.

Tested on dev-cl04, stock 6.12.0-211.52.1.22.11.vz10, dm-ploop built
out of tree:
- each bug reproduced on the stock module, the races with a test-only
  kprobe module widening the window: a pio running past the suspend of
  a cmd; a pio never completing after resume; the top image truncated
  from 512M to 128M by a merge and to 256M by a flip; after a grow
  failing with ENOSPC on a full host fs, a writer stuck in D state, an
  unsuspendable device and a later cmd failing with ETIMEDOUT. None of
  them reproduce with the series;
- fio 4k random reads and writes, O_DIRECT, 1 to 48 jobs, on a fully
  allocated image over brd: IOPS and system CPU stay within the
  run-to-run spread of the stock module. This setup tops out at about
  420K IOPS elsewhere in dm-ploop and would not show a lock taken on
  submission either, so patch 1 keeps that path lockless;
- the ploop pytest suite, ploop and qcow2 formats, passes. A test for
  patch 4 goes to the ploop tests separately.

Denis V. Lunev (4):
  dm-ploop: fix pio submission racing with the suspend and resume of a
    cmd
  dm-ploop: fix a lost wakeup of the ploop_allocator() thread
  dm-ploop: wait for the ploop_allocator() thread before changing
    deltas[]
  dm-ploop: fix a failed cluster copy on grow leaving the BAT page busy

 drivers/md/dm-ploop-cmd.c    | 18 ++++++++++++++++--
 drivers/md/dm-ploop-map.c    | 22 +++++++++++++++-------
 drivers/md/dm-ploop-target.c |  4 ++++
 drivers/md/dm-ploop.h        |  1 +
 4 files changed, 36 insertions(+), 9 deletions(-)


base-commit: 09eb9a82fe872048d082391415d1cdeda2ac6879
-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 0/4] dm-ploop: fix the pio submission stop around cmds

[Andrey Zhadchenko]

Reviewed-by: Andrey Zhadchenko <andrey.zhadchenko@virtuozzo.com>

On 11/3/26 09:36, Denis V. Lunev wrote:
> A cmd stops pio submission and waits for the inflight pios before it
> rewrites the BAT levels or deltas[], and expects nothing to touch them
> until it resumes. Konstantin found three ways this does not hold while
> reviewing the VSTOR-151979 series (VSTOR-152985, VSTOR-152984,
> VSTOR-152981):
> 
> - a submitter racing the stop either runs without the cmd waiting for
>    it, or is parked after resume and never completes (patch 1);
> 
> - the ploop_allocator() thread is not stopped at all. It truncates the
>    top delta while merge or flip moves it in deltas[], cutting the live
>    image to the preallocation size (patch 3). The thread can also miss a
>    wakeup, which stalls the device by itself and would hang the new wait,
>    so patch 2 fixes that first;
> 
> - a grow failing to copy a cluster resumes submission with that
>    cluster's BAT page still busy, and writes to it hang (patch 4).
> 
> Patch 3 is the one losing data: a snapshot delete under write load is
> enough.
> 
> Tested on dev-cl04, stock 6.12.0-211.52.1.22.11.vz10, dm-ploop built
> out of tree:
> - each bug reproduced on the stock module, the races with a test-only
>    kprobe module widening the window: a pio running past the suspend of
>    a cmd; a pio never completing after resume; the top image truncated
>    from 512M to 128M by a merge and to 256M by a flip; after a grow
>    failing with ENOSPC on a full host fs, a writer stuck in D state, an
>    unsuspendable device and a later cmd failing with ETIMEDOUT. None of
>    them reproduce with the series;
> - fio 4k random reads and writes, O_DIRECT, 1 to 48 jobs, on a fully
>    allocated image over brd: IOPS and system CPU stay within the
>    run-to-run spread of the stock module. This setup tops out at about
>    420K IOPS elsewhere in dm-ploop and would not show a lock taken on
>    submission either, so patch 1 keeps that path lockless;
> - the ploop pytest suite, ploop and qcow2 formats, passes. A test for
>    patch 4 goes to the ploop tests separately.
> 
> Denis V. Lunev (4):
>    dm-ploop: fix pio submission racing with the suspend and resume of a
>      cmd
>    dm-ploop: fix a lost wakeup of the ploop_allocator() thread
>    dm-ploop: wait for the ploop_allocator() thread before changing
>      deltas[]
>    dm-ploop: fix a failed cluster copy on grow leaving the BAT page busy
> 
>   drivers/md/dm-ploop-cmd.c    | 18 ++++++++++++++++--
>   drivers/md/dm-ploop-map.c    | 22 +++++++++++++++-------
>   drivers/md/dm-ploop-target.c |  4 ++++
>   drivers/md/dm-ploop.h        |  1 +
>   4 files changed, 36 insertions(+), 9 deletions(-)
> 
> 
> base-commit: 09eb9a82fe872048d082391415d1cdeda2ac6879

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 0/4] dm-ploop: fix the pio submission stop around cmds

[Konstantin Khorenko]

I won't apply this version, please send v2 with the changes below.

On Tue, 03 Nov 2026 09:36:38 +0100, Denis V. Lunev wrote:
> Denis V. Lunev (4):
>   dm-ploop: fix pio submission racing with the suspend and resume of a
>     cmd
>   dm-ploop: fix a lost wakeup of the ploop_allocator() thread
>   dm-ploop: wait for the ploop_allocator() thread before changing
>     deltas[]
>   dm-ploop: fix a failed cluster copy on grow leaving the BAT page busy

What to change for v2:

1. Patch 1, commit message only, small: in the first race add
   notify_merged to "a merge, resize or delta index update". The code
   stays as is.

2. Patch 2, commit message only, small:
   - drop "until some later request wakes the thread", the stall does
     not end by itself;
   - add one sentence that the same reorder also fixes a lost
     kthread_stop() wakeup.
   The code stays as is.

3. Patch 3, code and commit message:
   - code: also wait for ploop_no_pending_prealloc() in
     ploop_flip_upper_deltas() before process_flip_upper_deltas(), keep
     the wait in ploop_postsuspend();
   - message: "merge and notify_merged_backward" becomes "merge and
     notify_merged", both directions move the top delta.

4. Patch 4: no changes, resend it as is.

5. One more old bug found, not part of this series: the forward
   notify_merged stores the new file_size into the wrong deltas[] slot
   (details below). Please file a Jira task for it and fix it with a
   separate patch, not in v2.

6. Cover letter of v2:
   - "cutting the live image to the preallocation size" becomes
     "cutting the live image";
   - reword "Patch 3 is the one losing data".

Details on each item are in the replies to the patches. Below are the
details on items 5 and 6.

Item 5, the old bug. In ploop_delta_clusters_merged() the forward case
stores file_size, file_preallocated_area_start and nr_be into
deltas[level + 1] after notify_delta_merged() has already shifted the
slots down, so the delta that took the clusters is at deltas[level] by
then. With level == top - 2 that overwrites the top delta's file_size,
and the next preallocation can truncate the live top image. It has been
there since the target was added.

> - the ploop_allocator() thread is not stopped at all. It truncates the
>   top delta while merge or flip moves it in deltas[], cutting the live
>   image to the preallocation size (patch 3). The thread can also miss a

Item 6. Only a merge cuts the image to the preallocation size. A flip
cuts the old top image to the size of the other delta plus the
preallocation.

> Patch 3 is the one losing data: a snapshot delete under write load is
> enough.

Item 6. Race A of patch 1 loses data too: a pio can read and write
through a delta that merge has already released. Patch 3 is the easiest
one to hit, not the only one losing data.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel