From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (mail.openvz.org [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id B912D8017D for ; Wed, 4 Nov 2026 12:12:22 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 6A4CB1J1023919; Wed, 4 Nov 2026 15:11:02 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 6A4CB1J1023919 Authentication-Results: mail.openvz.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=virtuozzo.com header.i=@virtuozzo.com header.b="ZqXvinF/" Received: from mail-yw1-f200.google.com (mail-yw1-f200.google.com [209.85.128.200]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 6A4CB0C2023915 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Wed, 4 Nov 2026 15:11:01 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 6A4CB0C2023915 Received: by mail-yw1-f200.google.com with SMTP id 00721157ae682-8b2fd108b28so9870287b3.1 for ; Wed, 04 Nov 2026 04:11:00 -0800 (PST) X-Gm-Message-State: APuU4Z/niY/rwGGzJS42C4nLtlPFFk7SKyuPK75IN8WQzTAwXAFJbSqA k0kA0hhSn9fOvxH9yWbFM5BOFk/FxPVZOg/Kl3NsfLWPs0AUDG0oxj7u/g8VBn6xBnwHju2M2wP wQnCfWZus0+RObfMML9bqfcSf9i1G3wN2gYSNljFAwfmJieEeMqdj+g== X-Gm-Gg: AYBFou1LeosQsb6IssQ4//8oCo73/PR0/sBkNDzch0RV/t9BqkfhnT5q/tnJMrFoHhV +Xak7TeAUntEXrB9quTnTMvzY81vVpPunXZFj3T3p413CEerK2Ei52zbRoZJI4QoVN1/1VKT7Yz pFYnP2+6q9JsRECf6dIpXy29H3Nn0k64fMW3bPehJ/fyR6lfQzvuo2T977qikBOClOpNG0d31Ut /GJu+mTle4Jz1k4tny1GrDb/BBSm1tlafy50v6HJ49vTLzDaOCKcmOC3MCK+fAWX2PonQkNylIj dN0HS1XAMOIgwQ2oPg2HVKG62n0dfODpwhFHV46lf8mAwtdPnr3s6WrUQshgII4AC51TxrtYZHp e+Yg3vdh4fwsOkhXbKhl4pkmbkcMgrDVdo1fXAy67ROKi1SCa/i/I5tlZkaeEHchGEDxuVDV9X9 XEGSgR5efIH6AH5xmc/oc9+FaJ X-Received: by 2002:a05:690c:4186:b0:8b1:1dd8:1d74 with SMTP id 00721157ae682-8b2b676594emr72211837b3.65.1793794259889; Wed, 04 Nov 2026 04:10:59 -0800 (PST) X-Received: by 2002:a05:690c:4186:b0:8b1:1dd8:1d74 with SMTP id 00721157ae682-8b2b676594emr72211587b3.65.1793794259328; Wed, 04 Nov 2026 04:10:59 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1793794259; cv=pass; d=google.com; s=arc-20260327; b=ACcV3JxKzauP7nsI/mcil+R2xCWVNH0dZmJeiISonYvQ9sku1CIBtWb5qYQbrwEPvM mIqoFz3hEYueMt6Be+iypEY5BVFTL937VGFdP78W7/ECF5rgMpqPOQA987yyGiyZk6Zn XFXr7VNIQbWa2k4Z7+FowZoKXpJaklX0X80rk98xb1BAyuaCz+o6euhqkticFkFhgiYt dL+e9hYaE+cBo/KJi4iERW89MB1tlaOJfxjuMAPacWxkNjXrg30K2Fa/QnPIT5cTn5LI PDH6ZkgMVfyCRGGgM7T4gbSnsjeQR/0cNKln9qvtgT9mdwIM7jZuPhZLeAINjwpxnbmY t8RQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=mime-version:content-transfer-encoding:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=7FDYl1QEIMrcLJFHcgNi9qL1pytf0njsBF01vlkpO0s=; fh=y3TfVeQA9QYtIzD5Inxiyys5ebBfyOgjQPZaP/+mSiI=; b=LdH1HK9BlUctLMwogW5mohgeVtTX480awbB8hwn3Bx4cyGZC+Xi+v14Ze4S/tvFzOP a0ha1Wyaxp53NgEqS0vFRXU3hrctGeAodqPEFRqLyhpgikTRLmi5nekSVkov1+Xk8+nf vmBUI9l0+yRrXwcV8FDIreSZ+dAGfhcN2KegnpQr7iQzSKv1FAWobSJczb8zOgRLE09g vuQqkPrhD/QW6W7+1ORlrvDQw/2yIKTvxY2Q8hFD9QLXyUyo6eSadwj/NzYAx8j3EDUQ 1Kbe5rkQynOZGtxLT99jK7SaN7BJLAzhyEPHKWzSJBy8YJi4R2z5k4bnaJSKdQgE//Ky dkaQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=selector2 header.b="ZqXvinF/"; arc=pass (i=1 spf=pass spfdomain=virtuozzo.com dkim=pass dkdomain=virtuozzo.com dmarc=pass fromdomain=virtuozzo.com); spf=pass (google.com: domain of ptikhomirov@virtuozzo.com designates 2a01:111:f403:c20f::7 as permitted sender) smtp.mailfrom=ptikhomirov@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com Received: from OSPPR02CU001.outbound.protection.outlook.com (mail-norwayeastazlp170130007.outbound.protection.outlook.com. [2a01:111:f403:c20f::7]) by mx.google.com with ESMTPS id 00721157ae682-8b30995cd5esi17346857b3.120.2026.11.04.04.10.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Nov 2026 04:10:59 -0800 (PST) Received-SPF: pass (google.com: domain of ptikhomirov@virtuozzo.com designates 2a01:111:f403:c20f::7 as permitted sender) client-ip=2a01:111:f403:c20f::7; Authentication-Results: mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=selector2 header.b="ZqXvinF/"; arc=pass (i=1 spf=pass spfdomain=virtuozzo.com dkim=pass dkdomain=virtuozzo.com dmarc=pass fromdomain=virtuozzo.com); spf=pass (google.com: domain of ptikhomirov@virtuozzo.com designates 2a01:111:f403:c20f::7 as permitted sender) smtp.mailfrom=ptikhomirov@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yUHA3fqXQry4g6BPht5rGVrsU3ZoOAz1BadxbnKBCEPzxjv65fgOjNYqsycyjcWYPFc9H0bQ7Wb2gCTHTYHGmAHZSMKoklhlloie24q2mZXJTTucqQhH0BLNn7If74PcUe+MlRdE9MYZGMtP65RvHr/MTYt1kWy7l8Krd+4YMj0g2alDeM9sJQulIMnAGM1vrlP9cqYbPHJK/vygFUgJpT3dPJ1d6OGR7awYRliX5JPKMG2LGu1J40tVotJJug5pQrc0Ud1wJtjnCE0xYf93+poOLsd/yk9ZlWSxFu+fnoSm77wrvV1HcrapjlaWh6n5bLKENAMJZPfM+sGZqhY4yw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7FDYl1QEIMrcLJFHcgNi9qL1pytf0njsBF01vlkpO0s=; b=MCZXVNre4wJh1gY4kRLS1CcBGzfUgJASjqKv5OhZ8cE+rmooTk/NIGeQdrHaAIG9BaYRN4Z1D+V/ujkbsjspu9XJWiMuo9BVreYcCjbdaEJgaBMzYT5UFJeQ0AbC1awk0MKavuRqiZErqBnJVdXGV1f5+SryLQx4Y0vUbPM6/NXOj/wrZ3O8ziCgK1T6aRcs9PI1waGHwYAo36fgOTN0JGGsqP2ozc9+kPxNkc5SRTBJHgEQjQ1sgAgQvSGaBKL2zpsvHOVjajL/Kn3y006Qx8mwoBb0N39n+P78Q1sVi+X5ehKE26N5J6JIh/6QfoFFmySLV4Ikpccgg/zVbKRzkg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=virtuozzo.com; dmarc=pass action=none header.from=virtuozzo.com; dkim=pass header.d=virtuozzo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7FDYl1QEIMrcLJFHcgNi9qL1pytf0njsBF01vlkpO0s=; b=ZqXvinF/DqVfM27dYe/5n7t8WapTwY/8Qv4yePovxXVkSkUnAKDp+rN8Ri78XjyAcOIgyILYNQGmGcR0SExwP8UwLxoqrg5EjTQhqNVCGgp35Ku+oTEK0OYml0yyi180A1YTxdng/NQ/fHhitCVV8zYVxMinEUaf/3oV2+DUzkHYlONyB2zdRhhCawFQ0ieeQ7XCD0hYf11NcfDPESYKhv5l/OVTRuPpmO70YOA8+2rYW/iRKDOkPFKRWMev7DDq71u/tmW1D2nd2HLw52yskyLQL1hCyHHHOBxO3kR2Gl96UwZPL1Mx5CFM6Xlhg0YEVhunVJ7UfcQ+xVoKuXkZ1A== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=virtuozzo.com; Received: from AM0PR08MB11804.eurprd08.prod.outlook.com (2603:10a6:20b:747::14) by BESPR08MB11958.eurprd08.prod.outlook.com (2603:10a6:b10:fd::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.592.8; Wed, 4 Nov 2026 12:10:55 +0000 Received: from AM0PR08MB11804.eurprd08.prod.outlook.com ([fe80::bf46:f137:f6d2:5b3a]) by AM0PR08MB11804.eurprd08.prod.outlook.com ([fe80::bf46:f137:f6d2:5b3a%7]) with mapi id 15.21.0592.007; Wed, 4 Nov 2026 12:10:55 +0000 From: Pavel Tikhomirov To: Konstantin Khorenko Date: Wed, 4 Nov 2026 13:10:39 +0100 Message-ID: <20261104121039.3697558-1-ptikhomirov@virtuozzo.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261028-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v7-8-5c48672191c2@virtuozzo.com> References: <20261028-ptikhomirov-vstor-149268-cbt-account-at-rq-start-v7-8-5c48672191c2@virtuozzo.com> X-ClientProxiedBy: BE1P281CA0455.DEUP281.PROD.OUTLOOK.COM (2603:10a6:b10:7f::27) To AM0PR08MB11804.eurprd08.prod.outlook.com (2603:10a6:20b:747::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AM0PR08MB11804:EE_|BESPR08MB11958:EE_ X-MS-Office365-Filtering-Correlation-Id: 81b426b5-cc08-4d93-717c-08df3a6cadc6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; ARA:13230040|366016|1800799024|52116014|23010399003|376014|38350700014|22082099003|10067099003|18002099003|56012099006|261022105027099003|261009223027099003|261022075727099003|261102130727099003|261026121527099003|261028144727099003|261026140227099003|261021121427099003|261102123227099003; X-Microsoft-Antispam-Message-Info: himddVjgzYwEa6y+Q1h2EVj8Hn2Mi3ry/v5hhlfFMq/NSJ0EuBuVJLV9r/aoMrkoDec/IYS0r+8qN76A52eX77l4qJnus1qb1TnIpFNMVDSm9rhYhumTNmAsnnFwaK5ni4lXmHoo9xv8A/Z2CJbUjFdrwvdDMIegpR7ts0psPR131abdvYzwbgwj6Ke3N1kBSdZzo8u+QtOVIn8LN0o7pxfwC27HOi6ASLzbQSkJ59tBhTV84Ijnm1KBIg1Wo55hDKwJAyxCg1xRV5hhQ9qSm6xl6DNAHCqMPa9zzvSLt1Rhxa723fziWIPQVdB4wDIp72tLCNfGuz6V46rXp+v3FK+KxwxDwEJ94h+JF0rUr5YTPq7C1N480o9qTIQU43KU5eLjevFJfOJgc44ZhPSY5rMiNzxLibqkjFCAPtOWVqtpyTIBGYhzmWSlxGwy1IRUsmN3HinnwQhE9q0iaq/a1W2JhkFxah1u4y//Fqm9Y6R6CFhtADNEUuiCGGLOan3DrBE9eIDfJcOhKEK9CiTEHALToBQj2oZ8gGAvdhNXVWdbDcoOMvQ3Cmf4upotdpREdw287u7ymKxcU3xP1BOoUO8qOz46YddIRndRRkZ+ZbGrfcvl8Mqq+REVZfbDDf5UTO1q15tc0wOovRDFPADt9yTufiLa87HrItDkVWpB+BU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR08MB11804.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(1800799024)(52116014)(23010399003)(376014)(38350700014)(22082099003)(10067099003)(18002099003)(56012099006)(261022105027099003)(261009223027099003)(261022075727099003)(261102130727099003)(261026121527099003)(261028144727099003)(261026140227099003)(261021121427099003)(261102123227099003); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zBiLjz/zOYLnjZ9g4QDNSltcn9xsrSbPpmmydrgdM4wHbfgjFMt0rNTRXAxR?= =?us-ascii?Q?VpMvJLz62kAiSxdsR7f1e20U2r0NAM75LQrVvcwYYW4wMxQjuaBPzx1Dl4Q1?= =?us-ascii?Q?X1Pz6NHUoE0Orw/CAVITjLRsHfajuPxcBpRqchDCcStU0nDOyrZw/VTPuHK3?= =?us-ascii?Q?Fn4UM0XboLD4r2H69Iy0N6NtK28XpLIfMB6AqtZpGIiQwrr6V1gIaeTOW7IM?= =?us-ascii?Q?ajrmF0ttWiBBoJae8Uc/XIp5TzK3g29v+0PfeXhQ2bZpS8bpJRvtkMioCiSj?= =?us-ascii?Q?NlZM3V1dZRkGlMJdrVYbCQ7m/Vp+Jd0h6LOXBTxH9v1jynvNDiy9N2ovkdKR?= =?us-ascii?Q?5XXny80r7QkLR7BMId5P1t1agbKL8Sd2J3CM2GNHP9gyi7BKERtgl8IZOkot?= =?us-ascii?Q?/j1XfHRXkQ1m6nQFYPkOHNDVec0mvx7JkXuvmW2SiYjAwUWM1zu6EIea8FDH?= =?us-ascii?Q?4dCiHnc+4A0wW10iDuaWXwfTqF8ALK1ysfjBH4WtjwS69SfAtV/4cytRn8wp?= =?us-ascii?Q?DkC/UvBpOTLkKuK2bqIfzZlOadojQiNnwsYP5zWukJ7UuX6o0vlb3yZNHLqj?= =?us-ascii?Q?1fksPYcUNs+ho36DhyQo4tk00cuWcSizfVBfgxTtzlNdoLTsd0gdk5KWFVX1?= =?us-ascii?Q?5HjrAHaa0veu+IqIWnDt/53th3HKKO4YY44Hn4KETKSyNvkd7GG4529TZQ8N?= =?us-ascii?Q?wRWCFybldVBvYGTzb/RMN3VJZWH0p2BfhQhjGbWnXE4S76Q6GyWD+j205W3W?= =?us-ascii?Q?LKD5dz5BqUB3N355O2P5gIEzTzWpEG7nPFit1aDpthQbXh7ohfC13xBh4pFn?= =?us-ascii?Q?7Vv7lBZ7/ibhE6Y/UHvYMerLtm+YNV0htoXDji/1wAn4FHc1N8MdcuGBEC8S?= =?us-ascii?Q?kKktwG4tx8X91atS7k2aiYP4Ui8SjCOeVIJvMRcNskog2kZBzHr8yKT0GMI7?= =?us-ascii?Q?DmHvxDC6omlrqO5zfnxf9dN1Sk1SZkCbe32ziA4OH8f1kD/2OpRHbzhbToKF?= =?us-ascii?Q?23GbqERIXl+eb6F7PjYl65XEEX6ectF+iwPXzHwTEt1rq0MZ0XmHml7J3i59?= =?us-ascii?Q?Z8Sc3KJ2cdcKxjm6K5XkuSYYcSY4awhkV/3YBzLY2cSGUjCnBlyN6lqjION2?= =?us-ascii?Q?0gm4qSWBOrN9AyOUcoQUbU7CpiuGY1cR7TdWct0E79sLczEhu4x61dUDsG2Z?= =?us-ascii?Q?UHwfuXUy5Xax36OWhQvPiScoDhNvLvdbyg/q4eYyxWudxETt+NnchqglHelk?= =?us-ascii?Q?BOOVQiPw6TGO0+xQf5oGH3l4qKXGuJl+CVDt0Zu/1QC1ZLwJwY8ezNuuz++W?= =?us-ascii?Q?woWyBr8M6KR25BOZXFsneUoQxZEUED9n/ByATu9r+wUWIvkMtARfVmJOeB7e?= =?us-ascii?Q?3bMkQnADliiqd71LuxQfjkDmIhSBFx3ZHEK66iPAPOVoKvrAs2PK2Eq9/yoL?= =?us-ascii?Q?GngUMIybdOYf1gRnlyIjOygrVIBhL0I0s0UzXXg60/FB4exKo2GcjUKoGx8r?= =?us-ascii?Q?6QzpswuJOoyC58uP81gzwxlEe9ePegWVYeFrYKlVDd23bp5RnQtxVihF58Ou?= =?us-ascii?Q?6masRKRWv1dMaBCgfUhk2gcM8LSQLrgyudNqVf1yTuoapiyIRD0lpLtj1l2b?= =?us-ascii?Q?CvYgDXnlFggdxPPDACIu/PMY+wy5Ua0t7PnOqu0Dq45uUhRtNQKVRikkV3SX?= =?us-ascii?Q?WXf/H9jAvGv085ul6bqiKE70MD/MOJPDtP19yPS/QuaFZuumWIUgb5syQJRy?= =?us-ascii?Q?/9VzAcNpcg=3D=3D?= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 81b426b5-cc08-4d93-717c-08df3a6cadc6 X-MS-Exchange-CrossTenant-AuthSource: AM0PR08MB11804.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Nov 2026 12:10:55.7177 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aZ9crnjVtPYUQXozBIS5haGAt/ztPTXhFucRy/gKsA/CIw612EMNclEsahlY+RxqQDynOP8jWe/KeURv++fO2znpPMZRkLUT6EhqmNMlFvk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BESPR08MB11958 X-OZ-Fwd: true Cc: devel@openvz.org Subject: [Devel] [PATCH hci-8.1 v8 8/8] block/blk-cbt: account a write request also when dm starts it X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org CBT accounts a write at bio submission. dm suspend of request based dm waits only for started requests, so a request submitted before the suspend but dispatched after resume is accounted before a backup point taken under the suspend (BLKCBTSTART, CBT_SNAP_CREATE) while its data lands after it. push_backup then saves the old data, and no incremental backup copies the new one. Account a write again when dm starts it. This runs under the dispatch rcu_read_lock(), so it can't allocate: it sets the bits directly in the bitmap pages, and the allocation of missing pages is now queued to the kblockd workqueue already at submission, also for a nowait write failed with BLK_STS_AGAIN, so that its retry finds the page. If a page is still missing when dm starts the request (the work did not run yet, or the write was submitted before BLKCBTSTART), dm returns BLK_STS_RESOURCE, so that blk-mq dispatches the request again later. https://virtuozzo.atlassian.net/browse/VSTOR-149268 Feature: cbt: changed block tracking (for backup) Signed-off-by: Pavel Tikhomirov --- v8: queue the page allocation above the nowait branch of blk_cbt_add(), so that a nowait write failed with BLK_STS_AGAIN gets its page for the retry. Patches 1-7 of v7 are unchanged. --- block/blk-cbt.c | 178 +++++++++++++++++++++++++++++++++++++++++ drivers/md/dm-rq.c | 3 + include/linux/blkdev.h | 5 ++ 3 files changed, 186 insertions(+) diff --git a/block/blk-cbt.c b/block/blk-cbt.c index 7b6441fad08f9..baa56ab1a5389 100644 --- a/block/blk-cbt.c +++ b/block/blk-cbt.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -58,6 +59,10 @@ struct cbt_info { struct page **snp_map; blkcnt_t snp_block_max; + /* Map pages to allocate, see cbt_alloc_work_fn() */ + unsigned long *missing; + struct work_struct alloc_work; + spinlock_t lock; struct list_head list; }; @@ -231,6 +236,85 @@ static int __blk_cbt_set(struct cbt_info *cbt, blkcnt_t block, return (pages_missed && *pages_missed) ? -EAGAIN : 0; } +/* Drop a cbt->count reference, release the cbt if it is dead */ +static void cbt_put(struct cbt_info *cbt) +{ + unsigned long flags; + bool release; + + spin_lock_irqsave(&cbt->lock, flags); + release = !--cbt->count && test_bit(CBT_DEAD, &cbt->flags); + spin_unlock_irqrestore(&cbt->lock, flags); + if (release) + call_srcu(&cbt_srcu, &cbt->rcu, &cbt_release_callback); +} + +/* Allocate the map pages queued in cbt->missing */ +static void cbt_alloc_work_fn(struct work_struct *work) +{ + struct cbt_info *cbt = container_of(work, struct cbt_info, alloc_work); + unsigned long idx; + + for_each_set_bit(idx, cbt->missing, NR_PAGES(cbt->block_max)) { + struct page *page; + + clear_bit(idx, cbt->missing); + if (CBT_PAGE(cbt, idx)) + continue; + + page = alloc_page(GFP_NOIO | __GFP_ZERO); + spin_lock_irq(&cbt->lock); + if (!CBT_PAGE(cbt, idx) && !test_bit(CBT_DEAD, &cbt->flags)) { + if (page) { + /* Release: lockless readers use the page */ + smp_store_release(&cbt->map[idx], page); + page = NULL; + } else { + set_bit(CBT_ERROR, &cbt->flags); + } + } + spin_unlock_irq(&cbt->lock); + if (page) + __free_page(page); + } + cbt_put(cbt); +} + +static void cbt_queue_alloc(struct cbt_info *cbt, unsigned long idx) +{ + unsigned long flags; + + /* Already queued: the work allocates it */ + if (test_and_set_bit(idx, cbt->missing)) + return; + spin_lock_irqsave(&cbt->lock, flags); + if (test_bit(CBT_DEAD, &cbt->flags)) { + spin_unlock_irqrestore(&cbt->lock, flags); + return; + } + cbt->count++; + spin_unlock_irqrestore(&cbt->lock, flags); + + /* The queued work drops the reference, or drop it if already queued */ + if (!kblockd_schedule_work(&cbt->alloc_work)) + cbt_put(cbt); +} + +/* Queue the allocation of the missing map pages for [block, block + count) */ +static void cbt_queue_alloc_pages(struct cbt_info *cbt, blkcnt_t block, + blkcnt_t count) +{ + unsigned long idx, last; + + if (unlikely(!count || block + count > cbt->block_max)) + return; + + last = (block + count - 1) / BITS_PER_PAGE; + for (idx = block / BITS_PER_PAGE; idx <= last; idx++) + if (!CBT_PAGE(cbt, idx)) + cbt_queue_alloc(cbt, idx); +} + /* * Return true if all the map pages for [block, block + count) are there, * or for an empty or out of range request, which needs no page. @@ -264,6 +348,11 @@ static bool blk_cbt_add(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len, end = DIV_ROUND_UP(start + len, 1UL << cbt->block_bits); start >>= cbt->block_bits; len = end - start; + /* + * Queue the allocation early, so blk_cbt_rq_start() and a retry of a + * nowait write find the pages + */ + cbt_queue_alloc_pages(cbt, start, len); if (unlikely(nowait)) { /* Don't sleep: set the bits directly, only with all pages there */ if (!cbt_pages_present(cbt, start, len)) @@ -317,6 +406,86 @@ bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio, bool nowait) return wouldblock; } +/* + * Set the bits of [start, start + len) in the present pages, without + * allocating. Return true if a page is missing, its allocation is queued. + */ +static bool cbt_set_present(struct cbt_info *cbt, blkcnt_t start, blkcnt_t len) +{ + blkcnt_t block, end; + unsigned long flags; + bool missing = false; + + end = DIV_ROUND_UP(start + len, 1UL << cbt->block_bits); + block = start >> cbt->block_bits; + if (unlikely(end > cbt->block_max)) { + set_bit(CBT_ERROR, &cbt->flags); + return false; + } + + while (block < end) { + unsigned long idx = block / BITS_PER_PAGE; + unsigned long off = block & (BITS_PER_PAGE - 1); + unsigned long n = min_t(unsigned long, BITS_PER_PAGE - off, + end - block); + struct page *page = CBT_PAGE(cbt, idx); + + if (page) { + local_irq_save(flags); + spin_lock_page(page); + set_bits(page_address(page), off, n, 1); + unlock_page(page); + local_irq_restore(flags); + } else { + cbt_queue_alloc(cbt, idx); + missing = true; + } + block += n; + } + + return missing; +} + +/* + * Account a write again when request based dm starts it: dm suspend waits + * only for started requests, so one accounted at submission before a + * backup point taken under suspend may still reach the disk after it. + * + * Runs under the dispatch rcu_read_lock() (blk_mq_run_dispatch_ops()), so + * it must not sleep: if a bitmap page is missing, its allocation is queued + * and true is returned, for dm to retry the request later. + */ +bool blk_cbt_rq_start(struct request *rq) +{ + struct request_queue *q = rq->q; + struct cbt_info *cbt; + struct bio *bio; + bool retry = false; + int idx; + + if (likely(list_empty(&q->cbt_list))) + return false; + if (rq_data_dir(rq) == READ || blk_rq_is_passthrough(rq)) + return false; + + idx = srcu_read_lock(&cbt_srcu); + list_for_each_entry_srcu(cbt, &q->cbt_list, list, + srcu_read_lock_held(&cbt_srcu)) { + if (unlikely(test_bit(CBT_ERROR, &cbt->flags))) + continue; + __rq_for_each_bio(bio, rq) { + if (bio->bi_iter.bi_size && + cbt_set_present(cbt, bio->bi_iter.bi_sector << 9, + bio->bi_iter.bi_size)) + retry = true; + } + } + srcu_read_unlock(&cbt_srcu, idx); + + return retry; +} +EXPORT_SYMBOL_GPL(blk_cbt_rq_start); + static struct cbt_info *do_cbt_alloc(struct request_queue *q, __u8 *name, loff_t size, loff_t blocksize) { @@ -339,8 +508,16 @@ static struct cbt_info *do_cbt_alloc(struct request_queue *q, __u8 *name, goto err_pcpu; memset(cbt->map, 0, NR_PAGES(cbt->block_max) * sizeof(void*)); + + cbt->missing = bitmap_zalloc(NR_PAGES(cbt->block_max), GFP_KERNEL); + if (!cbt->missing) + goto err_map; + INIT_WORK(&cbt->alloc_work, cbt_alloc_work_fn); + cbt->queue = q; return cbt; +err_map: + vfree(cbt->map); err_pcpu: free_percpu(cbt->cache); err_cbt: @@ -722,6 +899,7 @@ static void cbt_release_callback(struct rcu_head *head) free_map(cbt->snp_map, nr_pages); } + bitmap_free(cbt->missing); free_percpu(cbt->cache); kfree(cbt); } diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index 20f495bf5399d..5a8cd59ec257f 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -522,6 +522,9 @@ static blk_status_t dm_mq_queue_rq(struct blk_mq_hw_ctx *hctx, if (ti->type->busy && ti->type->busy(ti)) return BLK_STS_RESOURCE; + if (blk_cbt_rq_start(rq)) + return BLK_STS_RESOURCE; + dm_start_request(md, rq); /* Init tio using md established in .init_request */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 885555f480462..9698a582356a9 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1628,6 +1628,7 @@ extern void blk_cbt_update_size(struct block_device *bdev); extern void blk_cbt_release(struct request_queue *q); extern bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio, bool nowait); +extern bool blk_cbt_rq_start(struct request *rq); extern int blk_cbt_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg); #else /* CONFIG_BLK_DEV_CBT */ static inline void blk_cbt_update_size(struct block_device *bdev) @@ -1641,6 +1642,10 @@ static inline bool blk_cbt_bio_queue(struct request_queue *q, struct bio *bio, { return false; } +static inline bool blk_cbt_rq_start(struct request *rq) +{ + return false; +} static inline int blk_cbt_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg) { -- 2.55.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel