* [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file
@ 2026-09-18 10:25 Liu Kui
2026-09-19 11:14 ` Alexey Kuznetsov
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Liu Kui @ 2026-09-18 10:25 UTC (permalink / raw)
To: Konstantin Khorenko; +Cc: azaitsev, devel, Liu Kui
The shoot-on-sight path in fuse_writepages_send() for files being
invalidated violates the writectr protocol: the increment is not
gated on writectr >=0, and the matching decrement lives in a separate
fi->lock critical section. A concurrent fuse_set_nowrite() and
fuse_release_nowrite() pair lands in that window observes writectr ==
FUSE_NOWRITE + 1 and triggers BUG_ON(fi->writectr != FUSE_NOWRITE) in
__fuse_release_nowrite():
kernel BUG at fs/fuse/dir.c:1842!
RIP: fuse_release_nowrite+0x45/0x50 [fuse]
fuse_release+0xd0/0x300 [fuse]
__fput+0xe1/0x2a0
__x64_sys_close+0x3c/0x80
Fix it by always routing writeback through fi->queued_writes so the
writectr gating applies, and failing requests on invalidated files
in fuse_send_writepage() instead, where the existing out_free path
keeps the writectr increment and decrement balanced within a single
fi->lock critical section. Note that if a NOWRITE section is active,
queue the request instead of failing it immediately is harmless for
the file invalidation operation.
https://virtuozzo.atlassian.net/browse/VSTOR-145614
Signed-off-by: Liu Kui <kui.liu@virtuozzo.com>
---
Fix a writectr protocol violation in the shoot-on-sight writeback path
for invalidated files, which races with fuse_set_nowrite()/
fuse_release_nowrite() and triggers BUG_ON(fi->writectr != FUSE_NOWRITE)
in __fuse_release_nowrite() (vmcore analysis in VSTOR-145614).
https://virtuozzo.atlassian.net/browse/VSTOR-145614
---
fs/fuse/file.c | 20 +++++++-------------
1 file changed, 7 insertions(+), 13 deletions(-)
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index c96a631b9529..180f60fc539a 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -2237,6 +2237,11 @@ __acquires(fi->lock)
int err;
fi->writectr++;
+ if (fuse_file_fail_immediately(wpa->ia.ff)) {
+ /* File is under invalidation: fail request without sending */
+ mapping_set_error(wpa->inode->i_mapping, -EIO);
+ goto out_free;
+ }
if (inarg->offset + data_size <= size) {
inarg->size = data_size;
} else if (inarg->offset < size) {
@@ -2619,28 +2624,17 @@ static void fuse_writepages_send(struct fuse_fill_wb_data *data)
{
struct fuse_writepage_args *wpa = data->wpa;
struct inode *inode = data->inode;
- struct fuse_mount *fm = get_fuse_mount(inode);
struct fuse_inode *fi = get_fuse_inode(inode);
int num_pages = wpa->ia.ap.num_pages;
- struct fuse_args *args = &wpa->ia.ap.args;
- int shoot_on_sight = 0;
int i;
spin_lock(&fi->lock);
- if (fuse_file_fail_immediately(data->ff)) {
- shoot_on_sight = 1;
- fi->writectr++;
- } else {
- list_add_tail(&wpa->queue_entry, &fi->queued_writes);
- fuse_flush_writepages(inode);
- }
+ list_add_tail(&wpa->queue_entry, &fi->queued_writes);
+ fuse_flush_writepages(inode);
spin_unlock(&fi->lock);
for (i = 0; i < num_pages; i++)
end_page_writeback(data->orig_pages[i]);
-
- if (shoot_on_sight)
- args->end(fm, args, -EIO);
}
/*
---
base-commit: 397370a45ebb33b582fe6e5c0b2240cc4edb7bc9
change-id: 20260918-vstor-145614-writectr-17baf632e5b7
Best regards,
--
Liu Kui <kui.liu@virtuozzo.com>
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file
2026-09-18 10:25 [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file Liu Kui
@ 2026-09-19 11:14 ` Alexey Kuznetsov
2026-09-21 15:09 ` Konstantin Khorenko
2026-09-21 15:58 ` Konstantin Khorenko
2 siblings, 0 replies; 5+ messages in thread
From: Alexey Kuznetsov @ 2026-09-19 11:14 UTC (permalink / raw)
To: Liu Kui; +Cc: azaitsev, devel
Ack
On Fri, Sep 18, 2026 at 6:41 PM Liu Kui <kui.liu@virtuozzo.com> wrote:
>
> The shoot-on-sight path in fuse_writepages_send() for files being
> invalidated violates the writectr protocol: the increment is not
> gated on writectr >=0, and the matching decrement lives in a separate
> fi->lock critical section. A concurrent fuse_set_nowrite() and
> fuse_release_nowrite() pair lands in that window observes writectr ==
> FUSE_NOWRITE + 1 and triggers BUG_ON(fi->writectr != FUSE_NOWRITE) in
> __fuse_release_nowrite():
>
> kernel BUG at fs/fuse/dir.c:1842!
> RIP: fuse_release_nowrite+0x45/0x50 [fuse]
> fuse_release+0xd0/0x300 [fuse]
> __fput+0xe1/0x2a0
> __x64_sys_close+0x3c/0x80
>
> Fix it by always routing writeback through fi->queued_writes so the
> writectr gating applies, and failing requests on invalidated files
> in fuse_send_writepage() instead, where the existing out_free path
> keeps the writectr increment and decrement balanced within a single
> fi->lock critical section. Note that if a NOWRITE section is active,
> queue the request instead of failing it immediately is harmless for
> the file invalidation operation.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-145614
>
> Signed-off-by: Liu Kui <kui.liu@virtuozzo.com>
> ---
> Fix a writectr protocol violation in the shoot-on-sight writeback path
> for invalidated files, which races with fuse_set_nowrite()/
> fuse_release_nowrite() and triggers BUG_ON(fi->writectr != FUSE_NOWRITE)
> in __fuse_release_nowrite() (vmcore analysis in VSTOR-145614).
>
> https://virtuozzo.atlassian.net/browse/VSTOR-145614
> ---
> fs/fuse/file.c | 20 +++++++-------------
> 1 file changed, 7 insertions(+), 13 deletions(-)
>
> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> index c96a631b9529..180f60fc539a 100644
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -2237,6 +2237,11 @@ __acquires(fi->lock)
> int err;
>
> fi->writectr++;
> + if (fuse_file_fail_immediately(wpa->ia.ff)) {
> + /* File is under invalidation: fail request without sending */
> + mapping_set_error(wpa->inode->i_mapping, -EIO);
> + goto out_free;
> + }
> if (inarg->offset + data_size <= size) {
> inarg->size = data_size;
> } else if (inarg->offset < size) {
> @@ -2619,28 +2624,17 @@ static void fuse_writepages_send(struct fuse_fill_wb_data *data)
> {
> struct fuse_writepage_args *wpa = data->wpa;
> struct inode *inode = data->inode;
> - struct fuse_mount *fm = get_fuse_mount(inode);
> struct fuse_inode *fi = get_fuse_inode(inode);
> int num_pages = wpa->ia.ap.num_pages;
> - struct fuse_args *args = &wpa->ia.ap.args;
> - int shoot_on_sight = 0;
> int i;
>
> spin_lock(&fi->lock);
> - if (fuse_file_fail_immediately(data->ff)) {
> - shoot_on_sight = 1;
> - fi->writectr++;
> - } else {
> - list_add_tail(&wpa->queue_entry, &fi->queued_writes);
> - fuse_flush_writepages(inode);
> - }
> + list_add_tail(&wpa->queue_entry, &fi->queued_writes);
> + fuse_flush_writepages(inode);
> spin_unlock(&fi->lock);
>
> for (i = 0; i < num_pages; i++)
> end_page_writeback(data->orig_pages[i]);
> -
> - if (shoot_on_sight)
> - args->end(fm, args, -EIO);
> }
>
> /*
>
> ---
> base-commit: 397370a45ebb33b582fe6e5c0b2240cc4edb7bc9
> change-id: 20260918-vstor-145614-writectr-17baf632e5b7
>
> Best regards,
> --
> Liu Kui <kui.liu@virtuozzo.com>
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file
2026-09-18 10:25 [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file Liu Kui
2026-09-19 11:14 ` Alexey Kuznetsov
@ 2026-09-21 15:09 ` Konstantin Khorenko
2026-09-21 15:55 ` Konstantin Khorenko
2026-09-21 15:58 ` Konstantin Khorenko
2 siblings, 1 reply; 5+ messages in thread
From: Konstantin Khorenko @ 2026-09-21 15:09 UTC (permalink / raw)
To: Liu Kui; +Cc: devel, azaitsev
> The shoot-on-sight path in fuse_writepages_send() for files being
> invalidated violates the writectr protocol: the increment is not
> gated on writectr >=0, and the matching decrement lives in a separate
> fi->lock critical section. A concurrent fuse_set_nowrite() and
> fuse_release_nowrite() pair lands in that window observes writectr ==
> FUSE_NOWRITE + 1 and triggers BUG_ON(fi->writectr != FUSE_NOWRITE) in
> __fuse_release_nowrite():
>
> kernel BUG at fs/fuse/dir.c:1842!
> RIP: fuse_release_nowrite+0x45/0x50 [fuse]
> fuse_release+0xd0/0x300 [fuse]
> __fput+0xe1/0x2a0
> __x64_sys_close+0x3c/0x80
>
> Fix it by always routing writeback through fi->queued_writes so the
> writectr gating applies, and failing requests on invalidated files
> in fuse_send_writepage() instead, where the existing out_free path
> keeps the writectr increment and decrement balanced within a single
> fi->lock critical section. Note that if a NOWRITE section is active,
> queue the request instead of failing it immediately is harmless for
> the file invalidation operation.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-145614
The ungated fi->writectr++ under FUSE_S_FAIL_IMMEDIATELY was added to
fuse_writepages_send() by 2d23aac3630e, so:
Fixes: 2d23aac3630e ("fuse: fix writeback/release race")
?
>
> Signed-off-by: Liu Kui <kui.liu@virtuozzo.com>
>
> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> index c96a631b95292..180f60fc539a5 100644
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -2237,6 +2237,11 @@ __acquires(fi->lock)
> int err;
>
> fi->writectr++;
> + if (fuse_file_fail_immediately(wpa->ia.ff)) {
> + /* File is under invalidation: fail request without sending */
> + mapping_set_error(wpa->inode->i_mapping, -EIO);
> + goto out_free;
> + }
[Severity: Medium]
Does the out_free path drop the fuse_file references held by the auxiliary
requests on a close_wait mount?
Every wpa created in fuse_writepages_fill() takes its own reference:
fs/fuse/file.c:fuse_writepages_fill() {
...
wpa = fuse_writepage_args_setup(folio, data->ff);
...
fuse_file_get(wpa->ia.ff);
...
}
and when fuse_writepage_add() finds the page already under writeback the
new wpa is chained onto old_wpa->next and keeps that reference. The two
release points are split by fc->close_wait:
fs/fuse/file.c:fuse_writepage_finish() {
...
if (wpa->ia.ff && fc->close_wait)
__fuse_file_put(wpa->ia.ff);
...
}
fs/fuse/file.c:fuse_writepage_free() {
...
if (wpa->ia.ff && !fc->close_wait)
fuse_file_put(wpa->ia.ff, false);
...
}
The out_free path this change now jumps to calls fuse_writepage_finish()
for the primary only. The auxiliary requests get fuse_writepage_finish_stat()
and fuse_writepage_free():
fs/fuse/file.c:fuse_send_writepage() {
...
out_free:
fi->writectr--;
rb_erase(&wpa->writepages_entry, &fi->writepages);
fuse_writepage_finish(wpa);
spin_unlock(&fi->lock);
/* After rb_erase() aux request list is private */
for (aux = wpa->next; aux; aux = next) {
next = aux->next;
aux->next = NULL;
fuse_writepage_finish_stat(aux->inode, aux->ia.ap.pages[0]);
fuse_writepage_free(aux);
}
...
}
With fc->close_wait set, neither call puts aux->ia.ff, so each auxiliary
request leaves one reference on the fuse_file behind.
Before this change a request for a FUSE_S_FAIL_IMMEDIATELY file never sat in
fi->queued_writes, so an auxiliary request could only attach to a primary
that was already in flight, and that primary was completed through
fuse_writepage_end(), which resends each aux with fuse_send_writepage() and
so ends up in fuse_writepage_finish() for it.
Now the request waits in fi->queued_writes while fuse_set_nowrite() is
active (truncate, fsync, or open with truncate on another file). If the page
is redirtied in that window, which fuse_writepage_need_send() documents as
possible for a direct read into an mmapped page, the next writeback pass
chains an aux onto the queued primary. When __fuse_release_nowrite() finally
sends it, this new check takes out_free and the aux references are lost.
On a close_wait mount that reference is what fuse_release() waits for:
fs/fuse/file.c:fuse_release() {
...
if (fc->close_wait)
wait_event(fi->page_waitq, refcount_read(&ff->count) == 1);
...
}
There is no timeout and nothing else drops the count, so close() on that
file would never return.
The same gap already exists for the "Got truncated off completely" and
broken-connection exits of out_free, so this is not new code, but this
change makes out_free the regular completion path for every invalidated
file. Should the aux loop in out_free call fuse_writepage_finish() (or
otherwise do the close_wait __fuse_file_put()) for each aux, either in this
patch or in a fix preceding it?
> if (inarg->offset + data_size <= size) {
> inarg->size = data_size;
> } else if (inarg->offset < size) {
[ ... ]
--
Konstantin Khorenko <khorenko@virtuozzo.com>
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file
2026-09-21 15:09 ` Konstantin Khorenko
@ 2026-09-21 15:55 ` Konstantin Khorenko
0 siblings, 0 replies; 5+ messages in thread
From: Konstantin Khorenko @ 2026-09-21 15:55 UTC (permalink / raw)
To: Liu Kui, Alexey Kuznetsov; +Cc: devel, azaitsev
Actors: T1 opens and later closes the file through ff B; T2 writes and truncates through ff C; T3 does an O_DIRECT read into C's mmap; flusher is background writeback; fused is the userspace daemon; kworker runs fuse_inval_files_work(). Inode I,
page P, vstorage mount (close_wait=1, writeback_cache=1).
Part 1. Setup: a stale open makes B the head of write_files
step who what happens state after the step
---- ------- ---------------------------------------------------------- ------------------------------------
1 T1 open(f) -> fuse_send_open(): FUSE_OPEN for B sent to fused, B: open_seq = N, no reply yet
waiting for the reply
2 fused writes FUSE_NOTIFY_INVAL_FILES(I) to /dev/fuse fi->inval_files_seq = N+1
-> fuse_invalidate_files(): every ff on fi->rw_files (old ones):
set_bit(FAIL_IMMEDIATELY) on every ff in fi->rw_files FAIL_IMMEDIATELY
set_bit(FUSE_I_INVAL_FILES, &fi->state) INVAL bit set
queue_delayed_work(inval_files_work)
3 kworker fuse_inval_files_work(): page cache of I is empty
truncate_pagecache_range(I, 0, -1) INVAL bit cleared
clear_bit(FUSE_I_INVAL_FILES)
4 T2 open(f) -> FUSE_OPEN for C, fused replies C: open_seq = N+2 > N+1, not stale
fuse_finish_open() -> fuse_link_write_file(): list_add fi->write_files = [C, ...]
fuse_link_rw_file(): open_seq >= inval_files_seq, fine
5 fused replies to FUSE_OPEN for B (step 1)
T1 fuse_finish_open() -> fuse_link_write_file(): list_add fi->write_files = [B, C, ...]
fuse_link_rw_file(): B->open_seq N < inval_files_seq N+1 B: FAIL_IMMEDIATELY
-> set_bit(FUSE_S_FAIL_IMMEDIATELY, &B->ff_state)
B->count = 1 (the open only)
From here on __fuse_write_file_get() for I returns B: it is first in the list and it is "dead". This is old behaviour, unrelated to the patch.
Part 2. Concurrent part: primary waits in the queue, aux attaches to it, out_free
One column per thread, time runs top to bottom. Rightmost column is shared state.
T2 (ff C) T3 (ff C, mmap) flusher state
------------------------------ ---------------------------- ----------------------------------- ----------------------------
6. mmap(C); writes into P P dirty, PTE writable
page_mkwrite() ok, C not FAIL
7. read(X, addr(P), O_DIRECT)
GUP: FOLL_WRITE, P already
dirty and writable, pinned
request sent to disk, waiting
8. ftruncate(C)
fuse_do_setattr():
inode_lock(I)
fuse_set_nowrite():
writectr += FUSE_NOWRITE writectr = FUSE_NOWRITE
wait writectr == NOWRITE (nothing in flight, returns)
FUSE_SETATTR -> fused,
waiting for the reply
9. fuse_writepages(I)
fill(P):
INVAL bit clear -> proceed
data->ff = write_file_get() = B B->count = 2
clear_page_dirty_for_io(P) PTE of P write-protected
wpa1 = args_setup(P, B)
fuse_file_get(B) B->count = 3
tree_insert(wpa1) P in fi->writepages
fuse_writepages_send():
list_add_tail(queued_writes) wpa1 queued
fuse_flush_writepages():
writectr < 0 -> nothing
end_page_writeback(P)
fuse_release_ff(B) B->count = 2
(before the patch: shoot on
sight, wpa1 completed at
once, never queued)
10. disk replied, DIO read
completes:
unpin_user_pages_dirty_lock()
-> set_page_dirty(P) P dirty again,
no page_mkwrite, hence no fuse_page_is_writeback(P)
fuse_wait_on_page_writeback = true (wpa1 in the tree)
11. fuse_writepages(I), next pass
fill(P):
data->ff = B B->count = 3
wpa2 = args_setup(P, B)
fuse_file_get(B) B->count = 4
fuse_writepage_add(wpa2, P):
insert_writeback finds wpa1
no P among wpa1->next
wpa1->next = wpa2 wpa2 = aux of wpa1
return false
folio_end_writeback(P)
fuse_release_ff(B) B->count = 3
(1 open + wpa1 + wpa2)
12. fused replied to SETATTR
fuse_do_setattr():
spin_lock(fi->lock)
__fuse_release_nowrite():
writectr = 0
fuse_flush_writepages():
fuse_send_writepage(wpa1):
writectr++ writectr = 1
fail_immediately(B) -> true <== new check from the patch
mapping_set_error(-EIO)
goto out_free:
writectr-- writectr = 0
rb_erase(wpa1)
fuse_writepage_finish(wpa1):
close_wait -> __fuse_file_put(B) B->count = 2
spin_unlock
for aux = wpa1->next (wpa2):
fuse_writepage_finish_stat(wpa2)
fuse_writepage_free(wpa2):
close_wait -> put is NOT done <== the leak B->count = 2, should be 1
kfree(wpa2)
fuse_writepage_free(wpa1): no put (close_wait), kfree
spin_unlock(fi->lock)
inode_unlock(I)
Part 3. Outcome
step who what happens result
---- ---- -------------------------------------------------------------- ---------------------------------
13 T1 close(B) -> fuse_release():
write_inode_now(I, 1) no dirty pages left
list_del_init(&B->write_entry)
wait_event(fi->page_waitq,
refcount_read(&B->count) == 1) B->count = 2 forever,
no timeout, nobody to wake it:
close() never returns
i think the leak is not that probable and i can commit the current patch as is
and we can go production without fixing that leak.
What do you think?
--
Best regards,
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file
2026-09-18 10:25 [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file Liu Kui
2026-09-19 11:14 ` Alexey Kuznetsov
2026-09-21 15:09 ` Konstantin Khorenko
@ 2026-09-21 15:58 ` Konstantin Khorenko
2 siblings, 0 replies; 5+ messages in thread
From: Konstantin Khorenko @ 2026-09-21 15:58 UTC (permalink / raw)
To: Liu Kui; +Cc: devel, azaitsev
On Fri, 18 Sep 2026 18:25:41 +0800, Liu Kui wrote:
> fs/fuse: fix writectr imbalance when failing writeback of invalidated file
Applied to branch-rh10-6.12.0-211.52.1.20.x.vz10-ovz, will be released in rh10-6.12.0-211.52.1.20.10.vz10, thanks!
[1/1] fs/fuse: fix writectr imbalance when failing writeback of invalidated file
https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/c99164232e9d
--
Konstantin Khorenko <khorenko@virtuozzo.com>
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-09-21 15:59 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-09-18 10:25 [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file Liu Kui
2026-09-19 11:14 ` Alexey Kuznetsov
2026-09-21 15:09 ` Konstantin Khorenko
2026-09-21 15:55 ` Konstantin Khorenko
2026-09-21 15:58 ` Konstantin Khorenko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox