All Virtuozzo development lists (kernel + QEMU)
 help / color / mirror / Atom feed
* [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried
@ 2026-11-09 19:06 Denis V. Lunev
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy() Denis V. Lunev
                   ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-09 19:06 UTC (permalink / raw)
  To: khorenko, andrey.zhadchenko; +Cc: devel

A write that fails with ENOSPC on the image is held and retried by
enospc_timer, so that I/O resumes once space is freed. The timer only
sets a flag that the dispatcher consumes, and nothing wakes the
dispatcher, so the held writes are retried only when other I/O reaches
the device. On a device whose only I/O is the held writes, the writers
stay in D state for good after the space is freed (patch 2,
VSTOR-154182).

Patch 1 makes the destructor shut the timer down first. Nothing stops
it there today, and with patch 2 the callback dereferences the
dispatcher thread as well as struct ploop.

Found in a vmcore: 16 writers in D state, submit_enospc set, all 16
pios on enospc_pios, the timer expired 12 s before the crash.

Tested on 6.12.0-211.52.1.22.19.vz10, dm-ploop built out of tree:
- 16 O_DIRECT writers held on a full host ext4, then the space freed
  with no other I/O on the device. Stock: still stuck after 60 s, one
  unrelated read releases them. With the series: done 8 s after the
  space is freed, on the next timer tick;
- ploop test suite, make -j48 test on dev-cl04, green for ploop and
  qcow2 with this series together with the VSTOR-154096 one.

Denis V. Lunev (2):
  dm-ploop: stop enospc_timer in ploop_destroy()
  dm-ploop: fix the ENOSPC retry timer not waking the dispatcher

 drivers/md/dm-ploop-map.c    | 9 +++++----
 drivers/md/dm-ploop-target.c | 1 +
 2 files changed, 6 insertions(+), 4 deletions(-)

-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy()
  2026-11-09 19:06 [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried Denis V. Lunev
@ 2026-11-09 19:06 ` Denis V. Lunev
  2026-11-09 19:17   ` no-reply
  2026-11-10 17:03   ` Konstantin Khorenko
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher Denis V. Lunev
  2026-11-10 17:03 ` [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried Konstantin Khorenko
  2 siblings, 2 replies; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-09 19:06 UTC (permalink / raw)
  To: khorenko, andrey.zhadchenko; +Cc: devel

From: Denis V. Lunev <den@openvz.org>

The destructor never stops enospc_timer. ploop_presuspend() deletes it
with timer_delete_sync_try(), which does not wait for a callback that
runs on another CPU, and ploop_try_delay_enospc() tests wants_suspend
without deferred_lock, so a write that fails with ENOSPC around the
presuspend can arm the timer again. The callback dereferences the
struct ploop that ploop_dtr() frees.

Shut the timer down at the start of ploop_destroy(), which waits for a
running callback and keeps the timer from being armed again. The timer
is set up before kt_worker is created, so it is valid wherever
ploop_destroy() finds kt_worker set.

https://virtuozzo.atlassian.net/browse/VSTOR-154182

Fixes: 6ddad5a4d226 ("dm-ploop: Add ploop target driver")
Feature: dm-ploop: ploop target driver
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 drivers/md/dm-ploop-target.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/md/dm-ploop-target.c b/drivers/md/dm-ploop-target.c
index d8279c79ef56..5ab25cbd5c32 100644
--- a/drivers/md/dm-ploop-target.c
+++ b/drivers/md/dm-ploop-target.c
@@ -164,6 +164,7 @@ static void ploop_destroy(struct ploop *ploop)
 	int i;
 
 	if (ploop->kt_worker) {
+		timer_shutdown_sync(&ploop->enospc_timer);
 		ploop->force_md_writeback = 1;
 		wake_up_process(ploop->kt_worker->task);
 		/* try to send all pending - if we have partial io and enospc end bellow */
-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher
  2026-11-09 19:06 [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried Denis V. Lunev
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy() Denis V. Lunev
@ 2026-11-09 19:06 ` Denis V. Lunev
  2026-11-09 19:18   ` no-reply
  2026-11-10 17:03   ` Konstantin Khorenko
  2026-11-10 17:03 ` [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried Konstantin Khorenko
  2 siblings, 2 replies; 8+ messages in thread
From: Denis V. Lunev @ 2026-11-09 19:06 UTC (permalink / raw)
  To: khorenko, andrey.zhadchenko; +Cc: devel

From: Denis V. Lunev <den@openvz.org>

A write that fails with ENOSPC on the image is held on enospc_pios and
retried when enospc_timer fires, so that I/O resumes once space is
freed. ploop_enospc_timer() only sets submit_enospc, which only
do_ploop_run_work() consumes, and nothing wakes the dispatcher for it.
ploop_worker() also goes to sleep with the flag set, as its sleep
condition checks the pio lists only. The held writes are retried only
when other I/O reaches the device: on a device whose only I/O is the
held writes, the writers stay in D state after the space is freed.

Wake the dispatcher from the timer, and keep ploop_worker() awake while
submit_enospc is set, so that a wakeup that comes while the dispatcher
is busy is not lost. Test and clear the flag with xchg() in
do_ploop_run_work(), as the timer now sets it from another CPU.

https://virtuozzo.atlassian.net/browse/VSTOR-154182

Fixes: afc60480b037 ("dm-ploop: resubmit enospc pios from dispatcher thread")
Feature: dm-ploop: ploop target driver
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 drivers/md/dm-ploop-map.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/md/dm-ploop-map.c b/drivers/md/dm-ploop-map.c
index ae3e287604b7..a4e17aa090a6 100644
--- a/drivers/md/dm-ploop-map.c
+++ b/drivers/md/dm-ploop-map.c
@@ -165,7 +165,9 @@ static void ploop_init_prq_and_embedded_pio(struct ploop *ploop,
 void ploop_enospc_timer(struct timer_list *timer)
 {
 	struct ploop *ploop = timer_container_of(ploop, timer, enospc_timer);
-	ploop->submit_enospc = true;
+
+	WRITE_ONCE(ploop->submit_enospc, true);
+	wake_up_process(ploop->kt_worker->task);
 }
 
 void do_ploop_event_work(struct work_struct *ws)
@@ -2241,10 +2243,8 @@ static void do_ploop_run_work(struct ploop *ploop)
 
 	current->flags |= PLOOP_PFLAGS;
 
-	if (ploop->submit_enospc) {
-		ploop->submit_enospc = false;
+	if (xchg(&ploop->submit_enospc, false))
 		ploop_resubmit_enospc_pios(ploop);
-	}
 
 	llembedded_pios = llist_del_all(&ploop->pios[PLOOP_LIST_PREPARE]);
 
@@ -2447,6 +2447,7 @@ int ploop_worker(void *data)
 		    llist_empty(&ploop->pios[PLOOP_LIST_DISCARD]) &&
 		    llist_empty(&ploop->pios[PLOOP_LIST_COW]) &&
 		    llist_empty(&ploop->llresubmit_pios) &&
+		    !READ_ONCE(ploop->submit_enospc) &&
 		    !ploop->force_md_writeback) {
 			if (kthread_should_stop()) {
 				wait_event_interruptible(
-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy()
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy() Denis V. Lunev
@ 2026-11-09 19:17   ` no-reply
  2026-11-10 17:03   ` Konstantin Khorenko
  1 sibling, 0 replies; 8+ messages in thread
From: no-reply @ 2026-11-09 19:17 UTC (permalink / raw)
  To: Denis V. Lunev, andrey.zhadchenko, khorenko; +Cc: devel

> From: Denis V. Lunev <den@openvz.org>
> 
> The destructor never stops enospc_timer. ploop_presuspend() deletes it
> with timer_delete_sync_try(), which does not wait for a callback that
> runs on another CPU, and ploop_try_delay_enospc() tests wants_suspend
> without deferred_lock, so a write that fails with ENOSPC around the
> presuspend can arm the timer again. The callback dereferences the
> struct ploop that ploop_dtr() frees.
> 
> Shut the timer down at the start of ploop_destroy(), which waits for a
> running callback and keeps the timer from being armed again. The timer
> is set up before kt_worker is created, so it is valid wherever
> ploop_destroy() finds kt_worker set.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-154182
> 
> Fixes: 6ddad5a4d226 ("dm-ploop: Add ploop target driver")
> Feature: dm-ploop: ploop target driver
> Signed-off-by: Denis V. Lunev <den@openvz.org>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261109190657.1851487-1-den@openvz.org?part=1


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher Denis V. Lunev
@ 2026-11-09 19:18   ` no-reply
  2026-11-10 17:03   ` Konstantin Khorenko
  1 sibling, 0 replies; 8+ messages in thread
From: no-reply @ 2026-11-09 19:18 UTC (permalink / raw)
  To: Denis V. Lunev, andrey.zhadchenko, khorenko; +Cc: devel

> From: Denis V. Lunev <den@openvz.org>
> 
> A write that fails with ENOSPC on the image is held on enospc_pios and
> retried when enospc_timer fires, so that I/O resumes once space is
> freed. ploop_enospc_timer() only sets submit_enospc, which only
> do_ploop_run_work() consumes, and nothing wakes the dispatcher for it.
> ploop_worker() also goes to sleep with the flag set, as its sleep
> condition checks the pio lists only. The held writes are retried only
> when other I/O reaches the device: on a device whose only I/O is the
> held writes, the writers stay in D state after the space is freed.
> 
> Wake the dispatcher from the timer, and keep ploop_worker() awake while
> submit_enospc is set, so that a wakeup that comes while the dispatcher
> is busy is not lost. Test and clear the flag with xchg() in
> do_ploop_run_work(), as the timer now sets it from another CPU.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-154182
> 
> Fixes: afc60480b037 ("dm-ploop: resubmit enospc pios from dispatcher thread")
> Feature: dm-ploop: ploop target driver
> Signed-off-by: Denis V. Lunev <den@openvz.org>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261109190657.1851487-1-den@openvz.org?part=2


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried
  2026-11-09 19:06 [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried Denis V. Lunev
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy() Denis V. Lunev
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher Denis V. Lunev
@ 2026-11-10 17:03 ` Konstantin Khorenko
  2 siblings, 0 replies; 8+ messages in thread
From: Konstantin Khorenko @ 2026-11-10 17:03 UTC (permalink / raw)
  To: Denis V. Lunev; +Cc: devel

I won't apply this version, please send v2 with the changes below.

On Mon, 9 Nov 2026 20:06:55 +0100, Denis V. Lunev wrote:
> dm-ploop: fix writes held on ENOSPC never being retried

The code of both patches is right. For v2:

1. Patch 1: cancel event_work in ploop_destroy() next to the timer.
2. Patch 1 message: name mod_timer() after the wants_suspend test as
   the race, not the lockless test, and say what the user gets.
3. Patch 2 message: drop the wrong reason given for xchg().
4. Patch 2 message: start a new paragraph at "The held writes are
   retried only...".
5. Cover letter: drop the vmcore paragraph and the measured numbers.

> Found in a vmcore: 16 writers in D state, submit_enospc set, all 16
> pios on enospc_pios, the timer expired 12 s before the crash.

Item 5. The cover letter goes into each commit as the "Patchset
description", so this ends up in the git log. How the bug was found
and the timings belong in VSTOR-154182, not in a commit message.

> Tested on 6.12.0-211.52.1.22.19.vz10, dm-ploop built out of tree:
> - 16 O_DIRECT writers held on a full host ext4, then the space freed
>   with no other I/O on the device. Stock: still stuck after 60 s, one
>   unrelated read releases them. With the series: done 8 s after the
>   space is freed, on the next timer tick;
> - ploop test suite, make -j48 test on dev-cl04, green for ploop and
>   qcow2 with this series together with the VSTOR-154096 one.

Same here: say what was tested (writers held on a full image, space
freed with no other I/O, they finish when enospc_timer next expires;
the ploop test suite passes), without the numbers and the host name.

Separate patch, not for this series: dm-qcow2 has the same hole.
qcow2_tgt_destroy() stops neither enospc_timer nor event_work, the
mod_timer() in the ENOSPC path runs after event_lock is dropped, and
presuspend uses timer_delete_sync_try() as well.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher Denis V. Lunev
  2026-11-09 19:18   ` no-reply
@ 2026-11-10 17:03   ` Konstantin Khorenko
  1 sibling, 0 replies; 8+ messages in thread
From: Konstantin Khorenko @ 2026-11-10 17:03 UTC (permalink / raw)
  To: Denis V. Lunev; +Cc: devel

I won't apply this version, please send v2 with the changes below.

On Mon, 9 Nov 2026 20:06:57 +0100, Denis V. Lunev wrote:
> ploop_worker() also goes to sleep with the flag set, as its sleep
> condition checks the pio lists only. The held writes are retried only
> when other I/O reaches the device: on a device whose only I/O is the
> held writes, the writers stay in D state after the space is freed.

The code is fine, only the message needs two changes.

- Item 4 of the cover letter reply: start a new paragraph at "The held
  writes are retried only...".

The first paragraph then says what the code does and why that is
wrong, the second what the user sees.

> Wake the dispatcher from the timer, and keep ploop_worker() awake while
> submit_enospc is set, so that a wakeup that comes while the dispatcher
> is busy is not lost. Test and clear the flag with xchg() in
> do_ploop_run_work(), as the timer now sets it from another CPU.

- Item 3 of the cover letter reply: "now" is not right, drop the
  reason or give a true one.

The timer has set the flag from softirq on any CPU since afc60480b037,
this patch does not change that. The old test-then-clear lost nothing
either, as ploop_resubmit_enospc_pios() takes the whole list after the
clear. xchg() is fine as a cleanup, just without that clause.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy()
  2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy() Denis V. Lunev
  2026-11-09 19:17   ` no-reply
@ 2026-11-10 17:03   ` Konstantin Khorenko
  1 sibling, 0 replies; 8+ messages in thread
From: Konstantin Khorenko @ 2026-11-10 17:03 UTC (permalink / raw)
  To: Denis V. Lunev; +Cc: devel

I won't apply this version, please send v2 with the changes below.

On Mon, 9 Nov 2026 20:06:56 +0100, Denis V. Lunev wrote:
> The destructor never stops enospc_timer. ploop_presuspend() deletes it
> with timer_delete_sync_try(), which does not wait for a callback that
> runs on another CPU, and ploop_try_delay_enospc() tests wants_suspend
> without deferred_lock, so a write that fails with ENOSPC around the
> presuspend can arm the timer again. The callback dereferences the
> struct ploop that ploop_dtr() frees.

- Item 2 of the cover letter reply: the race is mod_timer() after the
  test, and the message should say what the user gets.

The lockless test is not the cause. In 6ddad5a4d226 the test was under
deferred_lock, but mod_timer() ran after the unlock, so the timer could
be armed after presuspend's timer_delete_sync_try() from the start.
That is also why the Fixes tag is right.

"does not wait for a callback that runs on another CPU" can go: a
callback that runs during presuspend is done long before dtr.

Say what happens then: the timer expires on freed memory, and with
patch 2 it also calls wake_up_process() on a freed task.

The paragraph is one sentence chained with commas; something like:

    The destructor never stops enospc_timer. ploop_try_delay_enospc()
    arms it after testing wants_suspend, and nothing orders that against
    timer_delete_sync_try() in ploop_presuspend(). A write that fails
    with ENOSPC around the presuspend leaves the timer pending after the
    suspend, and it expires on the struct ploop that ploop_dtr() freed.

>  	if (ploop->kt_worker) {
> +		timer_shutdown_sync(&ploop->enospc_timer);
>  		ploop->force_md_writeback = 1;

- Item 1 of the cover letter reply: cancel event_work here too.

ploop_try_delay_enospc() calls schedule_work(&ploop->event_work) right
after mod_timer(), so the same race queues event_work after
presuspend's flush_work(). Nothing in ploop_destroy() waits for it,
and do_ploop_event_work() reads ploop->ti after the free. The window
is much shorter than the timer's, but it is the same race:

    timer_shutdown_sync(&ploop->enospc_timer);
    cancel_work_sync(&ploop->event_work);

INIT_WORK() runs before kt_worker is created, so it is valid under the
same check.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-10 17:05 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-09 19:06 [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried Denis V. Lunev
2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 1/2] dm-ploop: stop enospc_timer in ploop_destroy() Denis V. Lunev
2026-11-09 19:17   ` no-reply
2026-11-10 17:03   ` Konstantin Khorenko
2026-11-09 19:06 ` [Devel] [PATCH hci-8.1 2/2] dm-ploop: fix the ENOSPC retry timer not waking the dispatcher Denis V. Lunev
2026-11-09 19:18   ` no-reply
2026-11-10 17:03   ` Konstantin Khorenko
2026-11-10 17:03 ` [Devel] [PATCH hci-8.1 0/2] dm-ploop: fix writes held on ENOSPC never being retried Konstantin Khorenko

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.