[Devel] [PATCH hci-8.1 v6 0/7] block/blk-cbt: don't lose writes held across a dm suspend

[Devel] [PATCH hci-8.1 v6 0/7] block/blk-cbt: don't lose writes held across a dm suspend

[Pavel Tikhomirov]

Fix CBT missing writes that request-based dm holds across a suspend,
which made incremental backups taken with ploop-backup silently lose
data.

Claude tested full and incremental ploop-backup under write load (a
random writer, and fio with 4 jobs x iodepth 64): the incremental backup
lost blocks without the fix, and was consistent in 30/30 and 10/10 runs
with it. CBT start/stop and device resize under fio ran without
warnings. With writes throttled by io.max on a loop device, CBT recorded
24 of 64 written blocks without the throttle patch, and all 64 with it.
A RWF_NOWAIT write into a missing bitmap page now fails with EAGAIN
instead of sleeping, and succeeds when retried without RWF_NOWAIT.

https://virtuozzo.atlassian.net/browse/VSTOR-149268

Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
Changes in v6:
- New patch: load a map slot once with READ_ONCE() in CBT_PAGE(), as a
  racing CBT_PAGE_MISSED store could make it return (struct page *)0x1,
  and install a map page with smp_store_release(), so that a lockless
  reader never sees it before its zeroing (sashiko review).
- Store CBT_PAGE_MISSED with cmpxchg() instead of under cbt->lock, since
  taking a spinlock_t in the flush IPI is not allowed: it may sleep on
  PREEMPT_RT (sashiko review).
- Skip queueing a page allocation that is already queued, instead of
  taking cbt->lock for every write while the page is missing (sashiko
  review).
- Link to v5: http://lore.virtuozzo.com/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v5-0-88dccc74b48b@virtuozzo.com

Changes in v5:
- New patch: store CBT_PAGE_MISSED under cbt->lock, as the cache flush
  could overwrite a page installed meanwhile; the allocation queued at
  submission in v4 made this much more likely (sashiko review).
- New patch: walk cbt_list under SRCU, so that a sleeping bitmap page
  allocation no longer drops the RCU lock in the middle of the walk, which
  let a concurrent CBT stop or resize free the next cbt under it, a use
  after free (the pre-existing problem noted in v4).
- New patch: don't sleep in CBT for a REQ_NOWAIT write, fail it with
  BLK_STS_AGAIN if a bitmap page is missing (also noted in v4).
- Link to v4: http://lore.virtuozzo.com/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v4-0-b4d92264f6a4@virtuozzo.com

Changes in v4:
- Only queue the bitmap page allocation at submission instead of
  allocating there: the sleeping allocation dropped the RCU lock in the
  middle of the cbt_list walk, and could put an extent into the cache of
  a cbt that a resize had just replaced (sashiko review).
- Link to v3: http://lore.virtuozzo.com/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v3-0-e1bc61a829a0@virtuozzo.com

Changes in v3:
- New patch: account a write delayed by blk-throttle in CBT, as the
  throttle dispatch work skipped CBT (sashiko review).
- New patch: shift the CBT block size as unsigned, and use the unsigned
  shift in the new cbt_set_present() too (sashiko review).
- Link to v2: http://lore.virtuozzo.com/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v2-1-c64ee6178730@virtuozzo.com

Changes in v2:
- No GFP_ATOMIC allocation at dispatch: bitmap pages are now allocated
  at bio submission, and the dispatch hook only sets bits in present pages.
- A page missing at dispatch (a write submitted before BLKCBTSTART) is
  allocated with GFP_NOIO from the kblockd workqueue, and dm-rq returns
  BLK_STS_RESOURCE so that blk-mq dispatches the request again later,
  instead of setting CBT_ERROR when an atomic allocation fails.
- Move the hook from dm_start_request() to dm_mq_queue_rq(), before the
  request is started, so that it can return BLK_STS_RESOURCE.
- Link to v1: http://lore.virtuozzo.com/20261016-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v1-1-1f20b151431d@virtuozzo.com

---
Pavel Tikhomirov (7):
      block/blk-cbt: shift the block size as unsigned
      block/blk-throttle: account a throttled write in CBT
      block/blk-cbt: read a map slot once, publish a map page with release
      block/blk-cbt: don't overwrite a map page with CBT_PAGE_MISSED
      block/blk-cbt: walk cbt_list under SRCU
      block/blk-cbt: don't sleep in CBT for a REQ_NOWAIT write
      block/blk-cbt: account a write request also when dm starts it

 block/blk-cbt.c        | 303 ++++++++++++++++++++++++++++++++++++++++++-------
 block/blk-core.c       |   5 +-
 block/blk-throttle.c   |   5 +-
 drivers/md/dm-rq.c     |   3 +
 include/linux/blkdev.h |  12 +-
 5 files changed, 285 insertions(+), 43 deletions(-)
---
base-commit: f5d1fdcfffe101fe2bafa7df09d95d2e811c5c5e
change-id: 20261016-ptikhomirov-vstor-149268-cbt-account-at-rq-start-a592cf2010aa

Best regards,
--  
Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v6 1/7] block/blk-cbt: shift the block size as unsigned

[Pavel Tikhomirov]

cbt_ioc_init() accepts any power of two block size, so block_bits can be
31, and then 1 << block_bits overflows int: the sign extended block size
makes the rounded up block counts wrong. Shift an unsigned one instead.

https://virtuozzo.atlassian.net/browse/VSTOR-149268
Feature: cbt: changed block tracking (for backup)
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 block/blk-cbt.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/block/blk-cbt.c b/block/blk-cbt.c
index 90219f58f1ae5..1066b6a33b6b5 100644
--- a/block/blk-cbt.c
+++ b/block/blk-cbt.c
@@ -229,7 +229,7 @@ static void blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len)
 
 	if (unlikely(test_bit(CBT_ERROR, &cbt->flags)))
 		return;
-	end = DIV_ROUND_UP(start + len, 1 << cbt->block_bits);
+	end = DIV_ROUND_UP(start + len, 1UL << cbt->block_bits);
 	start >>= cbt->block_bits;
 	len = end - start;
 	if (unlikely(test_bit(CBT_NOCACHE, &cbt->flags))) {
@@ -575,7 +575,7 @@ static void blk_cbt_update_cbt_size(struct cbt_info *cbt, loff_t new_sz)
 	unsigned bsz;
 	int in_use = 0;
 
-	bsz = 1 << cbt->block_bits;
+	bsz = 1U << cbt->block_bits;
 	if (DIV_ROUND_UP(new_sz, bsz) <= cbt->block_max)
 		return;
 
@@ -964,7 +964,7 @@ static int cbt_ioc_set(struct block_device *bdev, struct blk_user_cbt_info __use
 		struct cbt_extent ex;
 
 		ex.start  = cur_ex->ce_physical >> cbt->block_bits;
-		ex.len  = DIV_ROUND_UP(cur_ex->ce_length, 1 << cbt->block_bits);
+		ex.len  = DIV_ROUND_UP(cur_ex->ce_length, 1UL << cbt->block_bits);
 		if (ex.start > cbt->block_max ||
 		    ex.start + ex.len > cbt->block_max ||
 		    ex.len == 0) {

-- 
2.55.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v6 1/7] block/blk-cbt: shift the block size as unsigned

[no-reply]

> cbt_ioc_init() accepts any power of two block size, so block_bits can be
> 31, and then 1 << block_bits overflows int: the sign extended block size
> makes the rounded up block counts wrong. Shift an unsigned one instead.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-149268
> Feature: cbt: changed block tracking (for backup)
> Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v6-0-f9870cf5c7eb@virtuozzo.com?part=1


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v6 2/7] block/blk-throttle: account a throttled write in CBT

[Pavel Tikhomirov]

submit_bio_noacct() accounts a write in CBT only when blk-throttle does
not hold it: a throttled bio is later issued from the throttle dispatch
work with submit_bio_noacct_nocheck(), which skips CBT. So writes delayed
by an IO limit never reach CBT, and incremental backups miss them.

Account the bio in CBT when the dispatch work issues it. The work runs
in process context with no locks held, so CBT may allocate there as it
does at submission.

https://virtuozzo.atlassian.net/browse/VSTOR-149268
Feature: cbt: changed block tracking (for backup)
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 block/blk-throttle.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 397b6a410f9e5..e1e008a30bf2c 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1223,8 +1223,11 @@ static void blk_throtl_dispatch_work_fn(struct work_struct *work)
 
 	if (!bio_list_empty(&bio_list_on_stack)) {
 		blk_start_plug(&plug);
-		while ((bio = bio_list_pop(&bio_list_on_stack)))
+		while ((bio = bio_list_pop(&bio_list_on_stack))) {
+			/* submit_bio_noacct() did not account it in CBT */
+			blk_cbt_bio_queue(q, bio);
 			submit_bio_noacct_nocheck(bio);
+		}
 		blk_finish_plug(&plug);
 	}
 }

-- 
2.55.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v6 2/7] block/blk-throttle: account a throttled write in CBT

[no-reply]

> submit_bio_noacct() accounts a write in CBT only when blk-throttle does
> not hold it: a throttled bio is later issued from the throttle dispatch
> work with submit_bio_noacct_nocheck(), which skips CBT. So writes delayed
> by an IO limit never reach CBT, and incremental backups miss them.
> 
> Account the bio in CBT when the dispatch work issues it. The work runs
> in process context with no locks held, so CBT may allocate there as it
> does at submission.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-149268
> Feature: cbt: changed block tracking (for backup)
> Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v6-0-f9870cf5c7eb@virtuozzo.com?part=2


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v6 3/7] block/blk-cbt: read a map slot once, publish a map page with release

[Pavel Tikhomirov]

CBT_PAGE() loads a map slot twice. A lockless reader racing with the
cache flush, which marks an empty slot CBT_PAGE_MISSED, can get NULL from
the first load and the marker from the second, and then use
(struct page *)0x1 as a page.

A page is also installed with a plain store under cbt->lock, but readers
don't take that lock: on a weakly ordered CPU a reader may see the
pointer before the page's zeroing, and the late zeroing can then clear a
bit the reader has just set.

Load the slot once with READ_ONCE(), whose address dependency orders the
reader's accesses to the page, and install a page with
smp_store_release().

https://virtuozzo.atlassian.net/browse/VSTOR-149268
Feature: cbt: changed block tracking (for backup)
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 block/blk-cbt.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/block/blk-cbt.c b/block/blk-cbt.c
index 1066b6a33b6b5..73df03c6c7e74 100644
--- a/block/blk-cbt.c
+++ b/block/blk-cbt.c
@@ -27,8 +27,11 @@
 #define BITS_PER_PAGE		(1UL << (PAGE_SHIFT + 3))
 
 #define CBT_PAGE_MISSED (struct page *)(0x1)
-#define CBT_PAGE(cbt, idx) (cbt->map[idx] == CBT_PAGE_MISSED ? \
-			    NULL : cbt->map[idx])
+/* Load the slot once: the cache flush may mark it missing meanwhile */
+#define CBT_PAGE(cbt, idx) ({						\
+	struct page *__page = READ_ONCE((cbt)->map[idx]);		\
+	__page == CBT_PAGE_MISSED ? NULL : __page;			\
+})
 
 static __cacheline_aligned_in_smp DEFINE_MUTEX(cbt_mutex);
 
@@ -148,10 +151,12 @@ static int cbt_page_alloc(struct cbt_info  **cbt_pp, unsigned long idx,
 		return -ENOMEM;
 	}
 
-	if (likely(CBT_PAGE(cbt, idx) == NULL))
-		cbt->map[idx] = page;
-	else
+	if (likely(CBT_PAGE(cbt, idx) == NULL)) {
+		/* Release: lockless readers use the page once they see it */
+		smp_store_release(&cbt->map[idx], page);
+	} else {
 		__free_page(page);
+	}
 
 	page = NULL;
 	spin_unlock_irq(&cbt->lock);

-- 
2.55.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v6 3/7] block/blk-cbt: read a map slot once, publish a map page with release

[no-reply]

> CBT_PAGE() loads a map slot twice. A lockless reader racing with the
> cache flush, which marks an empty slot CBT_PAGE_MISSED, can get NULL from
> the first load and the marker from the second, and then use
> (struct page *)0x1 as a page.
> 
> A page is also installed with a plain store under cbt->lock, but readers
> don't take that lock: on a weakly ordered CPU a reader may see the
> pointer before the page's zeroing, and the late zeroing can then clear a
> bit the reader has just set.
> 
> Load the slot once with READ_ONCE(), whose address dependency orders the
> reader's accesses to the page, and install a page with
> smp_store_release().
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-149268
> Feature: cbt: changed block tracking (for backup)
> Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v6-0-f9870cf5c7eb@virtuozzo.com?part=3


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v6 4/7] block/blk-cbt: don't overwrite a map page with CBT_PAGE_MISSED

[Pavel Tikhomirov]

The cache flush marks a missing map page with CBT_PAGE_MISSED from an
IPI, with a plain store. A page installed in between, by
cbt_page_alloc(), is then overwritten: it leaks, and the bits already
set in it are lost.

Store the marker with cmpxchg() only into an empty slot, and set the
bits in the page if one appeared meanwhile.

https://virtuozzo.atlassian.net/browse/VSTOR-149268
Feature: cbt: changed block tracking (for backup)
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 block/blk-cbt.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/block/blk-cbt.c b/block/blk-cbt.c
index 73df03c6c7e74..5486f0bea7ee0 100644
--- a/block/blk-cbt.c
+++ b/block/blk-cbt.c
@@ -199,10 +199,15 @@ static int __blk_cbt_set(struct cbt_info  *cbt, blkcnt_t block,
 			block += len;
 			continue;
 		} else if (pages_missed) {
+			struct page *old;
+
+			/* Don't overwrite a page installed meanwhile */
+			old = cmpxchg(&cbt->map[idx], NULL, CBT_PAGE_MISSED);
+			if (old && old != CBT_PAGE_MISSED)
+				continue;
 			(*pages_missed)++;
 			if (!*idx_first)
 				*idx_first = idx;
-			cbt->map[idx] = CBT_PAGE_MISSED;
 			count -= len;
 			block += len;
 			continue;

-- 
2.55.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v6 4/7] block/blk-cbt: don't overwrite a map page with CBT_PAGE_MISSED

[no-reply]

> The cache flush marks a missing map page with CBT_PAGE_MISSED from an
> IPI, with a plain store. A page installed in between, by
> cbt_page_alloc(), is then overwritten: it leaks, and the bits already
> set in it are lost.
> 
> Store the marker with cmpxchg() only into an empty slot, and set the
> bits in the page if one appeared meanwhile.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-149268
> Feature: cbt: changed block tracking (for backup)
> Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v6-0-f9870cf5c7eb@virtuozzo.com?part=4


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v6 5/7] block/blk-cbt: walk cbt_list under SRCU

[Pavel Tikhomirov]

blk_cbt_bio_queue() walks q->cbt_list under rcu_read_lock(), but a cbt
may need a bitmap page there: cbt_page_alloc() then drops the RCU lock
around the sleeping allocation, pinning only the current cbt. A CBT stop
or resize meanwhile can unlink it and free the next one, which the walk
then reaches through the stale ->next.

Walk cbt_list and release cbts under SRCU instead, whose readers may
sleep, so the allocation no longer drops the lock.

https://virtuozzo.atlassian.net/browse/VSTOR-149268
Feature: cbt: changed block tracking (for backup)
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 block/blk-cbt.c | 50 ++++++++++++++++++++++++++------------------------
 1 file changed, 26 insertions(+), 24 deletions(-)

diff --git a/block/blk-cbt.c b/block/blk-cbt.c
index 5486f0bea7ee0..824f12a5a7324 100644
--- a/block/blk-cbt.c
+++ b/block/blk-cbt.c
@@ -15,6 +15,7 @@
 #include <linux/cpuset.h>
 #include <linux/list.h>
 #include <linux/spinlock.h>
+#include <linux/srcu.h>
 #include <linux/pagemap.h>
 #include <linux/vmalloc.h>
 #include <linux/log2.h>
@@ -34,6 +35,8 @@
 })
 
 static __cacheline_aligned_in_smp DEFINE_MUTEX(cbt_mutex);
+/* Protects the q->cbt_list walks, which may sleep to allocate */
+DEFINE_STATIC_SRCU(cbt_srcu);
 
 struct cbt_extent{
 	blkcnt_t start;
@@ -105,8 +108,7 @@ static void set_bits(void *bm, int cur, int len, bool is_set)
  * -EBADF if cbt is dead,
  * -ENOMEM if alloc_page failed.
  */
-static int cbt_page_alloc(struct cbt_info  **cbt_pp, unsigned long idx,
-			  int in_rcu)
+static int cbt_page_alloc(struct cbt_info  **cbt_pp, unsigned long idx)
 {
 	struct cbt_info	 *cbt = *cbt_pp;
 	struct page *page;
@@ -118,7 +120,9 @@ static int cbt_page_alloc(struct cbt_info  **cbt_pp, unsigned long idx,
 	} else {
 		struct cbt_info *i, *new = NULL;
 
-		list_for_each_entry_rcu(i, &cbt->queue->cbt_list, list)
+		list_for_each_entry_srcu(i, &cbt->queue->cbt_list, list,
+					 srcu_read_lock_held(&cbt_srcu) ||
+					 lockdep_is_held(&cbt_mutex))
 			if (!memcmp(i->name, cbt->name, sizeof(cbt->name)))
 				new = i;
 
@@ -132,15 +136,11 @@ static int cbt_page_alloc(struct cbt_info  **cbt_pp, unsigned long idx,
 		}
 	}
 	spin_unlock_irq(&cbt->lock);
-	if (in_rcu)
-		rcu_read_unlock();
 	page = alloc_page(GFP_NOIO|__GFP_ZERO);
-	if (in_rcu)
-		rcu_read_lock();
 	spin_lock_irq(&cbt->lock);
 	if (unlikely(!--(cbt->count) && test_bit(CBT_DEAD, &cbt->flags))) {
 		spin_unlock_irq(&cbt->lock);
-		call_rcu(&cbt->rcu, &cbt_release_callback);
+		call_srcu(&cbt_srcu, &cbt->rcu, &cbt_release_callback);
 		if (page)
 			__free_page(page);
 		return -EBADF;
@@ -165,7 +165,7 @@ static int cbt_page_alloc(struct cbt_info  **cbt_pp, unsigned long idx,
 }
 
 static int __blk_cbt_set(struct cbt_info  *cbt, blkcnt_t block,
-			 blkcnt_t count, bool in_rcu, bool set,
+			 blkcnt_t count, bool set,
 			 unsigned long *pages_missed,
 			 unsigned long *idx_first)
 {
@@ -220,7 +220,7 @@ static int __blk_cbt_set(struct cbt_info  *cbt, blkcnt_t block,
 			}
 		}
 
-		ret = cbt_page_alloc(&cbt, idx, in_rcu);
+		ret = cbt_page_alloc(&cbt, idx);
 		if (ret == -EAGAIN) /* new cbt */
 			continue;
 		else if (ret == -EBADF) /* dead cbt */
@@ -243,7 +243,7 @@ static void blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len)
 	start >>= cbt->block_bits;
 	len = end - start;
 	if (unlikely(test_bit(CBT_NOCACHE, &cbt->flags))) {
-		__blk_cbt_set(cbt, start, len, 1, 1, NULL, NULL);
+		__blk_cbt_set(cbt, start, len, 1, NULL, NULL);
 		return;
 	}
 	ex = get_cpu_ptr(cbt->cache);
@@ -258,22 +258,24 @@ static void blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len)
 	put_cpu_ptr(cbt->cache);
 
 	if (likely(old.len))
-		__blk_cbt_set(cbt, old.start, old.len, 1, 1, NULL, NULL);
+		__blk_cbt_set(cbt, old.start, old.len, 1, NULL, NULL);
 }
 
 void blk_cbt_bio_queue(struct request_queue *q, struct bio *bio)
 {
 	struct cbt_info *cbt;
+	int idx;
 
-	rcu_read_lock();
+	idx = srcu_read_lock(&cbt_srcu);
 	if (list_empty(&q->cbt_list) || bio_data_dir(bio) == READ || !bio->bi_iter.bi_size)
-		goto out_rcu;
+		goto out_srcu;
 
-	list_for_each_entry_rcu(cbt, &q->cbt_list, list)
+	list_for_each_entry_srcu(cbt, &q->cbt_list, list,
+				 srcu_read_lock_held(&cbt_srcu))
 		blk_cbt_add(cbt, bio->bi_iter.bi_sector << 9, bio->bi_iter.bi_size);
 
-out_rcu:
-	rcu_read_unlock();
+out_srcu:
+	srcu_read_unlock(&cbt_srcu, idx);
 }
 
 static struct cbt_info *do_cbt_alloc(struct request_queue *q, __u8 *name,
@@ -555,7 +557,7 @@ static int blk_cbt_snap_merge_back(struct request_queue *q, __u8 *name)
 
 		if (!page_main) {
 			ret = -ENOMEM;
-			if (cbt_page_alloc(&cbt, i, 0))
+			if (cbt_page_alloc(&cbt, i))
 				goto out;
 
 			page_main = cbt->map[i];
@@ -608,7 +610,7 @@ static void blk_cbt_update_cbt_size(struct cbt_info *cbt, loff_t new_sz)
 	in_use = cbt->count;
 	spin_unlock_irq(&cbt->lock);
 	if (!in_use)
-		call_rcu(&cbt->rcu, &cbt_release_callback);
+		call_srcu(&cbt_srcu, &cbt->rcu, &cbt_release_callback);
 }
 
 void blk_cbt_update_size(struct block_device *bdev)
@@ -696,7 +698,7 @@ static void blk_cbt_del(struct cbt_info *cbt)
 	in_use = cbt->count;
 	spin_unlock_irqrestore(&cbt->lock, flags);
 	if (!in_use)
-		call_rcu(&cbt->rcu, &cbt_release_callback);
+		call_srcu(&cbt_srcu, &cbt->rcu, &cbt_release_callback);
 }
 
 void blk_cbt_release(struct request_queue *q)
@@ -744,7 +746,7 @@ static inline void __cbt_flush_cpu_cache(void *ptr)
 	struct cbt_extent *ex = this_cpu_ptr(cbt->cache);
 
 	if (ex->len) {
-		int ret = __blk_cbt_set(cbt, ex->start, ex->len, 0, 1,
+		int ret = __blk_cbt_set(cbt, ex->start, ex->len, 1,
 					&ctx->pages_missed,
 					&ctx->idx_first);
 		if (!ret) {
@@ -775,7 +777,7 @@ static void cbt_flush_cache(struct cbt_info *cbt)
 			if (cbt->map[i] != CBT_PAGE_MISSED)
 				continue;
 
-			ret = cbt_page_alloc(&cbt, i, 0);
+			ret = cbt_page_alloc(&cbt, i);
 			if (ret == -EAGAIN) /* new cbt */
 				goto try_again;
 			else if (ret) /* dead cbt or alloc_page failed */
@@ -897,7 +899,7 @@ static int cbt_ioc_get(struct block_device *bdev, struct blk_user_cbt_info __use
 		cur_ex->ce_length = ex.len << cbt->block_bits;
 
 		if (ci.ci_flags & CI_FLAG_ONCE)
-			__blk_cbt_set(cbt, ex.start, ex.len, 0, 0, NULL, NULL);
+			__blk_cbt_set(cbt, ex.start, ex.len, 0, NULL, NULL);
 		cur_ex++;
 		ci.ci_mapped_extents++;
 		block = ex.start + ex.len;
@@ -981,7 +983,7 @@ static int cbt_ioc_set(struct block_device *bdev, struct blk_user_cbt_info __use
 			ret = -EINVAL;
 			break;
 		}
-		ret = __blk_cbt_set(cbt, ex.start, ex.len, 0, set, NULL, NULL);
+		ret = __blk_cbt_set(cbt, ex.start, ex.len, set, NULL, NULL);
 		if (ret)
 			break;
 		cur_ex++;

-- 
2.55.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v6 5/7] block/blk-cbt: walk cbt_list under SRCU

[no-reply]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] block/blk-cbt: blk_cbt_update_size() iterator reads ->next of a cbt already queued for call_srcu() free
--

commit b854cacb9d00c0de28510b3f76417083233caa38
Author: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

block/blk-cbt: walk cbt_list under SRCU

This patch converts the q->cbt_list walk in blk_cbt_bio_queue() from
plain RCU to SRCU, so that cbt_page_alloc() no longer drops the read
lock around its sleeping allocation. It also switches the deferred
frees of a cbt from call_rcu() to call_srcu().

Link: https://virtuozzo.atlassian.net/browse/VSTOR-149268

> diff --git a/block/blk-cbt.c b/block/blk-cbt.c
> --- a/block/blk-cbt.c
> +++ b/block/blk-cbt.c

[ ... ]

> @@ -608,7 +610,7 @@ static void blk_cbt_update_cbt_size(struct cbt_info *cbt, loff_t new_sz)
>  	in_use = cbt->count;
>  	spin_unlock_irq(&cbt->lock);
>  	if (!in_use)
> -		call_rcu(&cbt->rcu, &cbt_release_callback);
> +		call_srcu(&cbt_srcu, &cbt->rcu, &cbt_release_callback);
>  }
>  
>  void blk_cbt_update_size(struct block_device *bdev)

[Severity: Medium]
This problem wasn't introduced by this patch, but the patch touches the
call site involved. Can blk_cbt_update_size() read the ->next pointer of
a cbt that has already been freed?

blk_cbt_update_size() walks the list with a plain iterator, holding only
cbt_mutex:

    mutex_lock(&cbt_mutex);
    list_for_each_entry(cbt, &q->cbt_list, list)
        blk_cbt_update_cbt_size(cbt, bdev_nr_bytes(bdev));

For each cbt, blk_cbt_update_cbt_size() can replace it in the list and
queue the old one for freeing:

    list_replace_rcu(&cbt->list, &new->list);
    in_use = cbt->count;
    ...
    if (!in_use)
        call_srcu(&cbt_srcu, &cbt->rcu, &cbt_release_callback);

When the callee returns, the loop advances through the old cbt's
->list.next. The caller holds no SRCU read lock and no reference on that
cbt. cbt_mutex does not keep an SRCU grace period from completing.

If the task is preempted between call_srcu() and the advance for a full
grace period, cbt_release_callback() can free the cbt before the read.
The window is narrow, but it looks reachable on a preemptible kernel.

Would list_for_each_entry_safe() be appropriate here, since it loads the
next pointer before the callee runs? Alternatively, could the loop be
wrapped in an SRCU read-side section?

[ ... ]

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v6-0-f9870cf5c7eb@virtuozzo.com?part=5

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v6 6/7] block/blk-cbt: don't sleep in CBT for a REQ_NOWAIT write

[Pavel Tikhomirov]

Accounting a write in CBT may allocate a bitmap page with GFP_NOIO,
either for the write itself or for the extent it evicts from the per-cpu
cache, so a REQ_NOWAIT bio can sleep in submit_bio_noacct().

For a REQ_NOWAIT write, skip the cache and set the bits directly, which
doesn't allocate when all the pages are there. If one is missing, fail
the bio with BLK_STS_AGAIN, for the submitter to retry it in a context
that may block. The throttle dispatch work may sleep, so it doesn't use
this.

https://virtuozzo.atlassian.net/browse/VSTOR-149268
Feature: cbt: changed block tracking (for backup)
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 block/blk-cbt.c        | 52 +++++++++++++++++++++++++++++++++++++++++++-------
 block/blk-core.c       |  5 ++++-
 block/blk-throttle.c   |  2 +-
 include/linux/blkdev.h |  7 +++++--
 4 files changed, 55 insertions(+), 11 deletions(-)

diff --git a/block/blk-cbt.c b/block/blk-cbt.c
index 824f12a5a7324..5535480847405 100644
--- a/block/blk-cbt.c
+++ b/block/blk-cbt.c
@@ -231,26 +231,52 @@ static int __blk_cbt_set(struct cbt_info  *cbt, blkcnt_t block,
 	return (pages_missed && *pages_missed) ? -EAGAIN : 0;
 }
 
-static void blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len)
+/* Return true if all the map pages for [block, block + count) are there */
+static bool cbt_pages_present(struct cbt_info *cbt, blkcnt_t block,
+			      blkcnt_t count)
+{
+	unsigned long idx, last;
+
+	if (unlikely(!count || block + count > cbt->block_max))
+		return true;
+
+	last = (block + count - 1) / BITS_PER_PAGE;
+	for (idx = block / BITS_PER_PAGE; idx <= last; idx++)
+		if (!CBT_PAGE(cbt, idx))
+			return false;
+
+	return true;
+}
+
+/* Return true if nowait and a bitmap page is missing */
+static bool blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len,
+			bool nowait)
 {
 	struct cbt_extent *ex;
 	struct cbt_extent old;
 	blkcnt_t end;
 
 	if (unlikely(test_bit(CBT_ERROR, &cbt->flags)))
-		return;
+		return false;
 	end = DIV_ROUND_UP(start + len, 1UL << cbt->block_bits);
 	start >>= cbt->block_bits;
 	len = end - start;
+	if (unlikely(nowait)) {
+		/* Don't sleep: set the bits directly, only with all pages there */
+		if (!cbt_pages_present(cbt, start, len))
+			return true;
+		__blk_cbt_set(cbt, start, len, 1, NULL, NULL);
+		return false;
+	}
 	if (unlikely(test_bit(CBT_NOCACHE, &cbt->flags))) {
 		__blk_cbt_set(cbt, start, len, 1, NULL, NULL);
-		return;
+		return false;
 	}
 	ex = get_cpu_ptr(cbt->cache);
 	if (ex->start + ex->len == start) {
 		ex->len += len;
 		put_cpu_ptr(cbt->cache);
-		return;
+		return false;
 	}
 	old = *ex;
 	ex->start = start;
@@ -259,11 +285,17 @@ static void blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len)
 
 	if (likely(old.len))
 		__blk_cbt_set(cbt, old.start, old.len, 1, NULL, NULL);
+	return false;
 }
 
-void blk_cbt_bio_queue(struct request_queue *q, struct bio *bio)
+/*
+ * Account a write in CBT. With nowait, don't sleep: return true if a bitmap
+ * page is missing, for the caller to fail the bio with BLK_STS_AGAIN.
+ */
+bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio, bool nowait)
 {
 	struct cbt_info *cbt;
+	bool wouldblock = false;
 	int idx;
 
 	idx = srcu_read_lock(&cbt_srcu);
@@ -271,11 +303,17 @@ void blk_cbt_bio_queue(struct request_queue *q, struct bio *bio)
 		goto out_srcu;
 
 	list_for_each_entry_srcu(cbt, &q->cbt_list, list,
-				 srcu_read_lock_held(&cbt_srcu))
-		blk_cbt_add(cbt, bio->bi_iter.bi_sector << 9, bio->bi_iter.bi_size);
+				 srcu_read_lock_held(&cbt_srcu)) {
+		if (blk_cbt_add(cbt, bio->bi_iter.bi_sector << 9,
+				bio->bi_iter.bi_size, nowait)) {
+			wouldblock = true;
+			break;
+		}
+	}
 
 out_srcu:
 	srcu_read_unlock(&cbt_srcu, idx);
+	return wouldblock;
 }
 
 static struct cbt_info *do_cbt_alloc(struct request_queue *q, __u8 *name,
diff --git a/block/blk-core.c b/block/blk-core.c
index bcc3060de670c..6bc78f7c23d38 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -871,7 +871,10 @@ void submit_bio_noacct(struct bio *bio)
 
 	if (blk_throtl_bio(bio))
 		return;
-	blk_cbt_bio_queue(q, bio);
+	if (blk_cbt_bio_queue(q, bio, bio->bi_opf & REQ_NOWAIT)) {
+		bio_wouldblock_error(bio);
+		return;
+	}
 	submit_bio_noacct_nocheck(bio);
 	return;
 
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index e1e008a30bf2c..00f0db0a57f8d 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1225,7 +1225,7 @@ static void blk_throtl_dispatch_work_fn(struct work_struct *work)
 		blk_start_plug(&plug);
 		while ((bio = bio_list_pop(&bio_list_on_stack))) {
 			/* submit_bio_noacct() did not account it in CBT */
-			blk_cbt_bio_queue(q, bio);
+			blk_cbt_bio_queue(q, bio, false);
 			submit_bio_noacct_nocheck(bio);
 		}
 		blk_finish_plug(&plug);
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 1284bb0e4ebc0..885555f480462 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1626,7 +1626,8 @@ static inline bool blk_crypto_register(struct blk_crypto_profile *profile,
 #if defined (CONFIG_BLK_DEV_CBT)
 extern void blk_cbt_update_size(struct block_device *bdev);
 extern void blk_cbt_release(struct request_queue *q);
-extern void blk_cbt_bio_queue(struct request_queue *q, struct bio *bio);
+extern bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio,
+			      bool nowait);
 extern int blk_cbt_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg);
 #else /* CONFIG_BLK_DEV_CBT */
 static inline void blk_cbt_update_size(struct block_device *bdev)
@@ -1635,8 +1636,10 @@ static inline void blk_cbt_update_size(struct block_device *bdev)
 static inline void blk_cbt_release(struct request_queue *q)
 {
 }
-static inline void blk_cbt_bio_queue(struct request_queue *q, struct bio *bio)
+static inline bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio,
+				     bool nowait)
 {
+	return false;
 }
 static inline int blk_cbt_ioctl(struct block_device *bdev, unsigned cmd,
 				char __user *arg)

-- 
2.55.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v6 6/7] block/blk-cbt: don't sleep in CBT for a REQ_NOWAIT write

[no-reply]

> Accounting a write in CBT may allocate a bitmap page with GFP_NOIO,
> either for the write itself or for the extent it evicts from the per-cpu
> cache, so a REQ_NOWAIT bio can sleep in submit_bio_noacct().
> 
> For a REQ_NOWAIT write, skip the cache and set the bits directly, which
> doesn't allocate when all the pages are there. If one is missing, fail
> the bio with BLK_STS_AGAIN, for the submitter to retry it in a context
> that may block. The throttle dispatch work may sleep, so it doesn't use
> this.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-149268
> Feature: cbt: changed block tracking (for backup)
> Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261027-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v6-0-f9870cf5c7eb@virtuozzo.com?part=6


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v6 7/7] block/blk-cbt: account a write request also when dm starts it

[Pavel Tikhomirov]

CBT accounts a write at bio submission. dm suspend of request based dm
waits only for started requests, so a request submitted before the
suspend but dispatched after resume is accounted before a backup point
taken under the suspend (BLKCBTSTART, CBT_SNAP_CREATE) while its data
lands after it. push_backup then saves the old data, and no incremental
backup copies the new one.

Account a write again when dm starts it. This runs under the dispatch
rcu_read_lock(), so it can't allocate: it sets the bits directly in the
bitmap pages, and the allocation of missing pages is now queued to the
kblockd workqueue already at submission. If a page is still missing when
dm starts the request (the work did not run yet, or the write was
submitted before BLKCBTSTART), dm returns BLK_STS_RESOURCE, so that
blk-mq dispatches the request again later.

https://virtuozzo.atlassian.net/browse/VSTOR-149268
Feature: cbt: changed block tracking (for backup)
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 block/blk-cbt.c        | 175 +++++++++++++++++++++++++++++++++++++++++++++++++
 drivers/md/dm-rq.c     |   3 +
 include/linux/blkdev.h |   5 ++
 3 files changed, 183 insertions(+)

diff --git a/block/blk-cbt.c b/block/blk-cbt.c
index 5535480847405..c114fb63cc134 100644
--- a/block/blk-cbt.c
+++ b/block/blk-cbt.c
@@ -11,6 +11,7 @@
 #include <linux/string.h>
 #include <linux/bio.h>
 #include <linux/blkdev.h>
+#include <linux/blk-mq.h>
 #include <linux/cpu.h>
 #include <linux/cpuset.h>
 #include <linux/list.h>
@@ -58,6 +59,10 @@ struct cbt_info {
 	struct page **snp_map;
 	blkcnt_t snp_block_max;
 
+	/* Map pages to allocate, see cbt_alloc_work_fn() */
+	unsigned long *missing;
+	struct work_struct alloc_work;
+
 	spinlock_t lock;
 	struct list_head list;
 };
@@ -231,6 +236,85 @@ static int __blk_cbt_set(struct cbt_info  *cbt, blkcnt_t block,
 	return (pages_missed && *pages_missed) ? -EAGAIN : 0;
 }
 
+/* Drop a cbt->count reference, release the cbt if it is dead */
+static void cbt_put(struct cbt_info *cbt)
+{
+	unsigned long flags;
+	bool release;
+
+	spin_lock_irqsave(&cbt->lock, flags);
+	release = !--cbt->count && test_bit(CBT_DEAD, &cbt->flags);
+	spin_unlock_irqrestore(&cbt->lock, flags);
+	if (release)
+		call_srcu(&cbt_srcu, &cbt->rcu, &cbt_release_callback);
+}
+
+/* Allocate the map pages queued in cbt->missing */
+static void cbt_alloc_work_fn(struct work_struct *work)
+{
+	struct cbt_info *cbt = container_of(work, struct cbt_info, alloc_work);
+	unsigned long idx;
+
+	for_each_set_bit(idx, cbt->missing, NR_PAGES(cbt->block_max)) {
+		struct page *page;
+
+		clear_bit(idx, cbt->missing);
+		if (CBT_PAGE(cbt, idx))
+			continue;
+
+		page = alloc_page(GFP_NOIO | __GFP_ZERO);
+		spin_lock_irq(&cbt->lock);
+		if (!CBT_PAGE(cbt, idx) && !test_bit(CBT_DEAD, &cbt->flags)) {
+			if (page) {
+				/* Release: lockless readers use the page */
+				smp_store_release(&cbt->map[idx], page);
+				page = NULL;
+			} else {
+				set_bit(CBT_ERROR, &cbt->flags);
+			}
+		}
+		spin_unlock_irq(&cbt->lock);
+		if (page)
+			__free_page(page);
+	}
+	cbt_put(cbt);
+}
+
+static void cbt_queue_alloc(struct cbt_info *cbt, unsigned long idx)
+{
+	unsigned long flags;
+
+	/* Already queued: the work allocates it */
+	if (test_and_set_bit(idx, cbt->missing))
+		return;
+	spin_lock_irqsave(&cbt->lock, flags);
+	if (test_bit(CBT_DEAD, &cbt->flags)) {
+		spin_unlock_irqrestore(&cbt->lock, flags);
+		return;
+	}
+	cbt->count++;
+	spin_unlock_irqrestore(&cbt->lock, flags);
+
+	/* The queued work drops the reference, or drop it if already queued */
+	if (!kblockd_schedule_work(&cbt->alloc_work))
+		cbt_put(cbt);
+}
+
+/* Queue the allocation of the missing map pages for [block, block + count) */
+static void cbt_queue_alloc_pages(struct cbt_info *cbt, blkcnt_t block,
+				  blkcnt_t count)
+{
+	unsigned long idx, last;
+
+	if (unlikely(!count || block + count > cbt->block_max))
+		return;
+
+	last = (block + count - 1) / BITS_PER_PAGE;
+	for (idx = block / BITS_PER_PAGE; idx <= last; idx++)
+		if (!CBT_PAGE(cbt, idx))
+			cbt_queue_alloc(cbt, idx);
+}
+
 /* Return true if all the map pages for [block, block + count) are there */
 static bool cbt_pages_present(struct cbt_info *cbt, blkcnt_t block,
 			      blkcnt_t count)
@@ -272,6 +356,8 @@ static bool blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len,
 		__blk_cbt_set(cbt, start, len, 1, NULL, NULL);
 		return false;
 	}
+	/* Queue the allocation early, so blk_cbt_rq_start() finds the pages */
+	cbt_queue_alloc_pages(cbt, start, len);
 	ex = get_cpu_ptr(cbt->cache);
 	if (ex->start + ex->len == start) {
 		ex->len += len;
@@ -316,6 +402,86 @@ bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio, bool nowait)
 	return wouldblock;
 }
 
+/*
+ * Set the bits of [start, start + len) in the present pages, without
+ * allocating. Return true if a page is missing, its allocation is queued.
+ */
+static bool cbt_set_present(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len)
+{
+	blkcnt_t block, end;
+	unsigned long flags;
+	bool missing = false;
+
+	end = DIV_ROUND_UP(start + len, 1UL << cbt->block_bits);
+	block = start >> cbt->block_bits;
+	if (unlikely(end > cbt->block_max)) {
+		set_bit(CBT_ERROR, &cbt->flags);
+		return false;
+	}
+
+	while (block < end) {
+		unsigned long idx = block / BITS_PER_PAGE;
+		unsigned long off = block & (BITS_PER_PAGE - 1);
+		unsigned long n = min_t(unsigned long, BITS_PER_PAGE - off,
+					end - block);
+		struct page *page = CBT_PAGE(cbt, idx);
+
+		if (page) {
+			local_irq_save(flags);
+			spin_lock_page(page);
+			set_bits(page_address(page), off, n, 1);
+			unlock_page(page);
+			local_irq_restore(flags);
+		} else {
+			cbt_queue_alloc(cbt, idx);
+			missing = true;
+		}
+		block += n;
+	}
+
+	return missing;
+}
+
+/*
+ * Account a write again when request based dm starts it: dm suspend waits
+ * only for started requests, so one accounted at submission before a
+ * backup point taken under suspend may still reach the disk after it.
+ *
+ * Runs under the dispatch rcu_read_lock() (blk_mq_run_dispatch_ops()), so
+ * it must not sleep: if a bitmap page is missing, its allocation is queued
+ * and true is returned, for dm to retry the request later.
+ */
+bool blk_cbt_rq_start(struct request *rq)
+{
+	struct request_queue *q = rq->q;
+	struct cbt_info *cbt;
+	struct bio *bio;
+	bool retry = false;
+	int idx;
+
+	if (likely(list_empty(&q->cbt_list)))
+		return false;
+	if (rq_data_dir(rq) == READ || blk_rq_is_passthrough(rq))
+		return false;
+
+	idx = srcu_read_lock(&cbt_srcu);
+	list_for_each_entry_srcu(cbt, &q->cbt_list, list,
+				 srcu_read_lock_held(&cbt_srcu)) {
+		if (unlikely(test_bit(CBT_ERROR, &cbt->flags)))
+			continue;
+		__rq_for_each_bio(bio, rq) {
+			if (bio->bi_iter.bi_size &&
+			    cbt_set_present(cbt, bio->bi_iter.bi_sector << 9,
+					    bio->bi_iter.bi_size))
+				retry = true;
+		}
+	}
+	srcu_read_unlock(&cbt_srcu, idx);
+
+	return retry;
+}
+EXPORT_SYMBOL_GPL(blk_cbt_rq_start);
+
 static struct cbt_info *do_cbt_alloc(struct request_queue *q, __u8 *name,
 				     loff_t size, loff_t blocksize)
 {
@@ -338,8 +504,16 @@ static struct cbt_info *do_cbt_alloc(struct request_queue *q, __u8 *name,
 		goto err_pcpu;
 
 	memset(cbt->map, 0, NR_PAGES(cbt->block_max) * sizeof(void*));
+
+	cbt->missing = bitmap_zalloc(NR_PAGES(cbt->block_max), GFP_KERNEL);
+	if (!cbt->missing)
+		goto err_map;
+	INIT_WORK(&cbt->alloc_work, cbt_alloc_work_fn);
+
 	cbt->queue = q;
 	return cbt;
+err_map:
+	vfree(cbt->map);
 err_pcpu:
 	free_percpu(cbt->cache);
 err_cbt:
@@ -721,6 +895,7 @@ static void cbt_release_callback(struct rcu_head *head)
 		free_map(cbt->snp_map, nr_pages);
 	}
 
+	bitmap_free(cbt->missing);
 	free_percpu(cbt->cache);
 	kfree(cbt);
 }
diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c
index 20f495bf5399d..5a8cd59ec257f 100644
--- a/drivers/md/dm-rq.c
+++ b/drivers/md/dm-rq.c
@@ -522,6 +522,9 @@ static blk_status_t dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx,
 	if (ti->type->busy && ti->type->busy(ti))
 		return BLK_STS_RESOURCE;
 
+	if (blk_cbt_rq_start(rq))
+		return BLK_STS_RESOURCE;
+
 	dm_start_request(md, rq);
 
 	/* Init tio using md established in .init_request */
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index 885555f480462..9698a582356a9 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1628,6 +1628,7 @@ extern void blk_cbt_update_size(struct block_device *bdev);
 extern void blk_cbt_release(struct request_queue *q);
 extern bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio,
 			      bool nowait);
+extern bool blk_cbt_rq_start(struct request *rq);
 extern int blk_cbt_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg);
 #else /* CONFIG_BLK_DEV_CBT */
 static inline void blk_cbt_update_size(struct block_device *bdev)
@@ -1641,6 +1642,10 @@ static inline bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio,
 {
 	return false;
 }
+static inline bool blk_cbt_rq_start(struct request *rq)
+{
+	return false;
+}
 static inline int blk_cbt_ioctl(struct block_device *bdev, unsigned cmd,
 				char __user *arg)
 {

-- 
2.55.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel