[Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file

[Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file

[Liu Kui]

The shoot-on-sight path in fuse_writepages_send() for files being
invalidated violates the writectr protocol: the increment is not
gated on writectr >=0, and the matching decrement lives in a separate
fi->lock critical section. A concurrent fuse_set_nowrite() and
fuse_release_nowrite() pair lands in that window observes writectr ==
FUSE_NOWRITE + 1 and triggers BUG_ON(fi->writectr != FUSE_NOWRITE) in
__fuse_release_nowrite():

  kernel BUG at fs/fuse/dir.c:1842!
  RIP: fuse_release_nowrite+0x45/0x50 [fuse]
   fuse_release+0xd0/0x300 [fuse]
   __fput+0xe1/0x2a0
   __x64_sys_close+0x3c/0x80

Fix it by always routing writeback through fi->queued_writes so the
writectr gating applies, and failing requests on invalidated files
in fuse_send_writepage() instead, where the existing out_free path
keeps the writectr increment and decrement balanced within a single
fi->lock critical section. Note that if a NOWRITE section is active,
queue the request instead of failing it immediately is harmless for
the file invalidation operation.

https://virtuozzo.atlassian.net/browse/VSTOR-145614

Signed-off-by: Liu Kui <kui.liu@virtuozzo.com>
---
Fix a writectr protocol violation in the shoot-on-sight writeback path
for invalidated files, which races with fuse_set_nowrite()/
fuse_release_nowrite() and triggers BUG_ON(fi->writectr != FUSE_NOWRITE)
in __fuse_release_nowrite() (vmcore analysis in VSTOR-145614).

https://virtuozzo.atlassian.net/browse/VSTOR-145614
---
 fs/fuse/file.c | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index c96a631b9529..180f60fc539a 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -2237,6 +2237,11 @@ __acquires(fi->lock)
 	int err;
 
 	fi->writectr++;
+	if (fuse_file_fail_immediately(wpa->ia.ff)) {
+		/* File is under invalidation: fail request without sending */
+		mapping_set_error(wpa->inode->i_mapping, -EIO);
+		goto out_free;
+	}
 	if (inarg->offset + data_size <= size) {
 		inarg->size = data_size;
 	} else if (inarg->offset < size) {
@@ -2619,28 +2624,17 @@ static void fuse_writepages_send(struct fuse_fill_wb_data *data)
 {
 	struct fuse_writepage_args *wpa = data->wpa;
 	struct inode *inode = data->inode;
-	struct fuse_mount *fm = get_fuse_mount(inode);
 	struct fuse_inode *fi = get_fuse_inode(inode);
 	int num_pages = wpa->ia.ap.num_pages;
-	struct fuse_args *args = &wpa->ia.ap.args;
-	int shoot_on_sight = 0;
 	int i;
 
 	spin_lock(&fi->lock);
-	if (fuse_file_fail_immediately(data->ff)) {
-		shoot_on_sight = 1;
-		fi->writectr++;
-	} else {
-		list_add_tail(&wpa->queue_entry, &fi->queued_writes);
-		fuse_flush_writepages(inode);
-	}
+	list_add_tail(&wpa->queue_entry, &fi->queued_writes);
+	fuse_flush_writepages(inode);
 	spin_unlock(&fi->lock);
 
 	for (i = 0; i < num_pages; i++)
 		end_page_writeback(data->orig_pages[i]);
-
-	if (shoot_on_sight)
-		args->end(fm, args, -EIO);
 }
 
 /*

---
base-commit: 397370a45ebb33b582fe6e5c0b2240cc4edb7bc9
change-id: 20260918-vstor-145614-writectr-17baf632e5b7

Best regards,
--  
Liu Kui <kui.liu@virtuozzo.com>

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file

[Alexey Kuznetsov]

Ack

On Fri, Sep 18, 2026 at 6:41 PM Liu Kui <kui.liu@virtuozzo.com> wrote:
>
> The shoot-on-sight path in fuse_writepages_send() for files being
> invalidated violates the writectr protocol: the increment is not
> gated on writectr >=0, and the matching decrement lives in a separate
> fi->lock critical section. A concurrent fuse_set_nowrite() and
> fuse_release_nowrite() pair lands in that window observes writectr ==
> FUSE_NOWRITE + 1 and triggers BUG_ON(fi->writectr != FUSE_NOWRITE) in
> __fuse_release_nowrite():
>
>   kernel BUG at fs/fuse/dir.c:1842!
>   RIP: fuse_release_nowrite+0x45/0x50 [fuse]
>    fuse_release+0xd0/0x300 [fuse]
>    __fput+0xe1/0x2a0
>    __x64_sys_close+0x3c/0x80
>
> Fix it by always routing writeback through fi->queued_writes so the
> writectr gating applies, and failing requests on invalidated files
> in fuse_send_writepage() instead, where the existing out_free path
> keeps the writectr increment and decrement balanced within a single
> fi->lock critical section. Note that if a NOWRITE section is active,
> queue the request instead of failing it immediately is harmless for
> the file invalidation operation.
>
> https://virtuozzo.atlassian.net/browse/VSTOR-145614
>
> Signed-off-by: Liu Kui <kui.liu@virtuozzo.com>
> ---
> Fix a writectr protocol violation in the shoot-on-sight writeback path
> for invalidated files, which races with fuse_set_nowrite()/
> fuse_release_nowrite() and triggers BUG_ON(fi->writectr != FUSE_NOWRITE)
> in __fuse_release_nowrite() (vmcore analysis in VSTOR-145614).
>
> https://virtuozzo.atlassian.net/browse/VSTOR-145614
> ---
>  fs/fuse/file.c | 20 +++++++-------------
>  1 file changed, 7 insertions(+), 13 deletions(-)
>
> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> index c96a631b9529..180f60fc539a 100644
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -2237,6 +2237,11 @@ __acquires(fi->lock)
>         int err;
>
>         fi->writectr++;
> +       if (fuse_file_fail_immediately(wpa->ia.ff)) {
> +               /* File is under invalidation: fail request without sending */
> +               mapping_set_error(wpa->inode->i_mapping, -EIO);
> +               goto out_free;
> +       }
>         if (inarg->offset + data_size <= size) {
>                 inarg->size = data_size;
>         } else if (inarg->offset < size) {
> @@ -2619,28 +2624,17 @@ static void fuse_writepages_send(struct fuse_fill_wb_data *data)
>  {
>         struct fuse_writepage_args *wpa = data->wpa;
>         struct inode *inode = data->inode;
> -       struct fuse_mount *fm = get_fuse_mount(inode);
>         struct fuse_inode *fi = get_fuse_inode(inode);
>         int num_pages = wpa->ia.ap.num_pages;
> -       struct fuse_args *args = &wpa->ia.ap.args;
> -       int shoot_on_sight = 0;
>         int i;
>
>         spin_lock(&fi->lock);
> -       if (fuse_file_fail_immediately(data->ff)) {
> -               shoot_on_sight = 1;
> -               fi->writectr++;
> -       } else {
> -               list_add_tail(&wpa->queue_entry, &fi->queued_writes);
> -               fuse_flush_writepages(inode);
> -       }
> +       list_add_tail(&wpa->queue_entry, &fi->queued_writes);
> +       fuse_flush_writepages(inode);
>         spin_unlock(&fi->lock);
>
>         for (i = 0; i < num_pages; i++)
>                 end_page_writeback(data->orig_pages[i]);
> -
> -       if (shoot_on_sight)
> -               args->end(fm, args, -EIO);
>  }
>
>  /*
>
> ---
> base-commit: 397370a45ebb33b582fe6e5c0b2240cc4edb7bc9
> change-id: 20260918-vstor-145614-writectr-17baf632e5b7
>
> Best regards,
> --
> Liu Kui <kui.liu@virtuozzo.com>

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file

[Konstantin Khorenko]

> The shoot-on-sight path in fuse_writepages_send() for files being
> invalidated violates the writectr protocol: the increment is not
> gated on writectr >=0, and the matching decrement lives in a separate
> fi->lock critical section. A concurrent fuse_set_nowrite() and
> fuse_release_nowrite() pair lands in that window observes writectr ==
> FUSE_NOWRITE + 1 and triggers BUG_ON(fi->writectr != FUSE_NOWRITE) in
> __fuse_release_nowrite():
> 
>   kernel BUG at fs/fuse/dir.c:1842!
>   RIP: fuse_release_nowrite+0x45/0x50 [fuse]
>    fuse_release+0xd0/0x300 [fuse]
>    __fput+0xe1/0x2a0
>    __x64_sys_close+0x3c/0x80
> 
> Fix it by always routing writeback through fi->queued_writes so the
> writectr gating applies, and failing requests on invalidated files
> in fuse_send_writepage() instead, where the existing out_free path
> keeps the writectr increment and decrement balanced within a single
> fi->lock critical section. Note that if a NOWRITE section is active,
> queue the request instead of failing it immediately is harmless for
> the file invalidation operation.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-145614

The ungated fi->writectr++ under FUSE_S_FAIL_IMMEDIATELY was added to
fuse_writepages_send() by 2d23aac3630e, so:

Fixes: 2d23aac3630e ("fuse: fix writeback/release race")

?
> 
> Signed-off-by: Liu Kui <kui.liu@virtuozzo.com>
>
> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> index c96a631b95292..180f60fc539a5 100644
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -2237,6 +2237,11 @@ __acquires(fi->lock)
>  	int err;
>  
>  	fi->writectr++;
> +	if (fuse_file_fail_immediately(wpa->ia.ff)) {
> +		/* File is under invalidation: fail request without sending */
> +		mapping_set_error(wpa->inode->i_mapping, -EIO);
> +		goto out_free;
> +	}

[Severity: Medium]
Does the out_free path drop the fuse_file references held by the auxiliary
requests on a close_wait mount?

Every wpa created in fuse_writepages_fill() takes its own reference:

fs/fuse/file.c:fuse_writepages_fill() {
    ...
		wpa = fuse_writepage_args_setup(folio, data->ff);
		...
		fuse_file_get(wpa->ia.ff);
    ...
}

and when fuse_writepage_add() finds the page already under writeback the
new wpa is chained onto old_wpa->next and keeps that reference. The two
release points are split by fc->close_wait:

fs/fuse/file.c:fuse_writepage_finish() {
    ...
	if (wpa->ia.ff && fc->close_wait)
		__fuse_file_put(wpa->ia.ff);
    ...
}

fs/fuse/file.c:fuse_writepage_free() {
    ...
	if (wpa->ia.ff && !fc->close_wait)
		fuse_file_put(wpa->ia.ff, false);
    ...
}

The out_free path this change now jumps to calls fuse_writepage_finish()
for the primary only. The auxiliary requests get fuse_writepage_finish_stat()
and fuse_writepage_free():

fs/fuse/file.c:fuse_send_writepage() {
    ...
 out_free:
	fi->writectr--;
	rb_erase(&wpa->writepages_entry, &fi->writepages);
	fuse_writepage_finish(wpa);
	spin_unlock(&fi->lock);

	/* After rb_erase() aux request list is private */
	for (aux = wpa->next; aux; aux = next) {
		next = aux->next;
		aux->next = NULL;
		fuse_writepage_finish_stat(aux->inode, aux->ia.ap.pages[0]);
		fuse_writepage_free(aux);
	}
    ...
}

With fc->close_wait set, neither call puts aux->ia.ff, so each auxiliary
request leaves one reference on the fuse_file behind.

Before this change a request for a FUSE_S_FAIL_IMMEDIATELY file never sat in
fi->queued_writes, so an auxiliary request could only attach to a primary
that was already in flight, and that primary was completed through
fuse_writepage_end(), which resends each aux with fuse_send_writepage() and
so ends up in fuse_writepage_finish() for it.

Now the request waits in fi->queued_writes while fuse_set_nowrite() is
active (truncate, fsync, or open with truncate on another file). If the page
is redirtied in that window, which fuse_writepage_need_send() documents as
possible for a direct read into an mmapped page, the next writeback pass
chains an aux onto the queued primary. When __fuse_release_nowrite() finally
sends it, this new check takes out_free and the aux references are lost.

On a close_wait mount that reference is what fuse_release() waits for:

fs/fuse/file.c:fuse_release() {
    ...
		if (fc->close_wait)
			wait_event(fi->page_waitq, refcount_read(&ff->count) == 1);
    ...
}

There is no timeout and nothing else drops the count, so close() on that
file would never return.

The same gap already exists for the "Got truncated off completely" and
broken-connection exits of out_free, so this is not new code, but this
change makes out_free the regular completion path for every invalidated
file. Should the aux loop in out_free call fuse_writepage_finish() (or
otherwise do the close_wait __fuse_file_put()) for each aux, either in this
patch or in a fix preceding it?

>  	if (inarg->offset + data_size <= size) {
>  		inarg->size = data_size;
>  	} else if (inarg->offset < size) {

[ ... ]

-- 
Konstantin Khorenko <khorenko@virtuozzo.com>
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file

[Konstantin Khorenko]

  Actors: T1 opens and later closes the file through ff B; T2 writes and truncates through ff C; T3 does an O_DIRECT read into C's mmap; flusher is background writeback; fused is the userspace daemon; kworker runs fuse_inval_files_work(). Inode I,
  page P, vstorage mount (close_wait=1, writeback_cache=1).

  Part 1. Setup: a stale open makes B the head of write_files

  step who      what happens                                                state after the step
  ---- -------  ----------------------------------------------------------  ------------------------------------
   1   T1       open(f) -> fuse_send_open(): FUSE_OPEN for B sent to fused,  B: open_seq = N, no reply yet
                waiting for the reply

   2   fused    writes FUSE_NOTIFY_INVAL_FILES(I) to /dev/fuse               fi->inval_files_seq = N+1
                -> fuse_invalidate_files():                                  every ff on fi->rw_files (old ones):
                   set_bit(FAIL_IMMEDIATELY) on every ff in fi->rw_files         FAIL_IMMEDIATELY
                   set_bit(FUSE_I_INVAL_FILES, &fi->state)                   INVAL bit set
                   queue_delayed_work(inval_files_work)

   3   kworker  fuse_inval_files_work():                                     page cache of I is empty
                   truncate_pagecache_range(I, 0, -1)                        INVAL bit cleared
                   clear_bit(FUSE_I_INVAL_FILES)

   4   T2       open(f) -> FUSE_OPEN for C, fused replies                    C: open_seq = N+2 > N+1, not stale
                fuse_finish_open() -> fuse_link_write_file(): list_add       fi->write_files = [C, ...]
                fuse_link_rw_file(): open_seq >= inval_files_seq, fine

   5   fused    replies to FUSE_OPEN for B (step 1)
       T1       fuse_finish_open() -> fuse_link_write_file(): list_add       fi->write_files = [B, C, ...]
                fuse_link_rw_file(): B->open_seq N < inval_files_seq N+1     B: FAIL_IMMEDIATELY
                   -> set_bit(FUSE_S_FAIL_IMMEDIATELY, &B->ff_state)
                                                                             B->count = 1 (the open only)

  From here on __fuse_write_file_get() for I returns B: it is first in the list and it is "dead". This is old behaviour, unrelated to the patch.

  Part 2. Concurrent part: primary waits in the queue, aux attaches to it, out_free

  One column per thread, time runs top to bottom. Rightmost column is shared state.

  T2 (ff C)                       T3 (ff C, mmap)               flusher                              state
  ------------------------------  ----------------------------  -----------------------------------  ----------------------------
  6. mmap(C); writes into P                                                                          P dirty, PTE writable
     page_mkwrite() ok, C not FAIL

                                  7. read(X, addr(P), O_DIRECT)
                                     GUP: FOLL_WRITE, P already
                                     dirty and writable, pinned
                                     request sent to disk, waiting

  8. ftruncate(C)
     fuse_do_setattr():
       inode_lock(I)
       fuse_set_nowrite():
         writectr += FUSE_NOWRITE                                                                      writectr = FUSE_NOWRITE
         wait writectr == NOWRITE   (nothing in flight, returns)
       FUSE_SETATTR -> fused,
       waiting for the reply
                                                                9. fuse_writepages(I)
                                                                   fill(P):
                                                                     INVAL bit clear -> proceed
                                                                     data->ff = write_file_get() = B  B->count = 2
                                                                     clear_page_dirty_for_io(P)      PTE of P write-protected
                                                                     wpa1 = args_setup(P, B)
                                                                     fuse_file_get(B)                 B->count = 3
                                                                     tree_insert(wpa1)                P in fi->writepages
                                                                   fuse_writepages_send():
                                                                     list_add_tail(queued_writes)     wpa1 queued
                                                                     fuse_flush_writepages():
                                                                       writectr < 0 -> nothing
                                                                     end_page_writeback(P)
                                                                   fuse_release_ff(B)                  B->count = 2
                                                                                                      (before the patch: shoot on
                                                                                                       sight, wpa1 completed at
                                                                                                       once, never queued)

                                  10. disk replied, DIO read
                                      completes:
                                      unpin_user_pages_dirty_lock()
                                      -> set_page_dirty(P)                                            P dirty again,
                                      no page_mkwrite, hence no                                       fuse_page_is_writeback(P)
                                      fuse_wait_on_page_writeback                                     = true (wpa1 in the tree)

                                                                11. fuse_writepages(I), next pass
                                                                    fill(P):
                                                                      data->ff = B                    B->count = 3
                                                                      wpa2 = args_setup(P, B)
                                                                      fuse_file_get(B)                B->count = 4
                                                                      fuse_writepage_add(wpa2, P):
                                                                        insert_writeback finds wpa1
                                                                        no P among wpa1->next
                                                                        wpa1->next = wpa2             wpa2 = aux of wpa1
                                                                        return false
                                                                      folio_end_writeback(P)
                                                                    fuse_release_ff(B)                B->count = 3
                                                                                                      (1 open + wpa1 + wpa2)

  12. fused replied to SETATTR
      fuse_do_setattr():
        spin_lock(fi->lock)
        __fuse_release_nowrite():
          writectr = 0
          fuse_flush_writepages():
            fuse_send_writepage(wpa1):
              writectr++                                                                              writectr = 1
              fail_immediately(B) -> true   <== new check from the patch
              mapping_set_error(-EIO)
              goto out_free:
                writectr--                                                                            writectr = 0
                rb_erase(wpa1)
                fuse_writepage_finish(wpa1):
                  close_wait -> __fuse_file_put(B)                                                    B->count = 2
                spin_unlock
                for aux = wpa1->next (wpa2):
                  fuse_writepage_finish_stat(wpa2)
                  fuse_writepage_free(wpa2):
                    close_wait -> put is NOT done   <== the leak                                      B->count = 2, should be 1
                    kfree(wpa2)
                fuse_writepage_free(wpa1): no put (close_wait), kfree
        spin_unlock(fi->lock)
      inode_unlock(I)

  Part 3. Outcome

  step who   what happens                                                    result
  ---- ----  --------------------------------------------------------------  ---------------------------------
  13   T1    close(B) -> fuse_release():
               write_inode_now(I, 1)                  no dirty pages left
               list_del_init(&B->write_entry)
               wait_event(fi->page_waitq,
                          refcount_read(&B->count) == 1)                     B->count = 2 forever,
                                                                             no timeout, nobody to wake it:
                                                                             close() never returns

i think the leak is not that probable and i can commit the current patch as is
and we can go production without fixing that leak.

What do you think?

--
Best regards,

Konstantin Khorenko,
Virtuozzo Linux Kernel Team


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.0] fs/fuse: fix writectr imbalance when failing writeback of invalidated file

[Konstantin Khorenko]

On Fri, 18 Sep 2026 18:25:41 +0800, Liu Kui wrote:
> fs/fuse: fix writectr imbalance when failing writeback of invalidated file

Applied to branch-rh10-6.12.0-211.52.1.20.x.vz10-ovz, will be released in rh10-6.12.0-211.52.1.20.10.vz10, thanks!

[1/1] fs/fuse: fix writectr imbalance when failing writeback of invalidated file
      https://bitbucket.org/virtuozzocore/vzkernel.vzs/commits/c99164232e9d

-- 
Konstantin Khorenko <khorenko@virtuozzo.com>

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel