From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (unknown [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id 329438015B for ; Mon, 21 Sep 2026 11:29:54 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68LBSXGx005655; Mon, 21 Sep 2026 14:28:34 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68LBSXGx005655 Authentication-Results: mail.openvz.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=virtuozzo.com header.i=@virtuozzo.com header.b="Bh2BMaFA" Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68LBSWlh005651 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Mon, 21 Sep 2026 14:28:32 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68LBSWlh005651 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-487038a4a88so802051f8f.0 for ; Mon, 21 Sep 2026 04:28:32 -0700 (PDT) X-Gm-Message-State: AFuF++ndak5HpKwLPoSBICg01NB0ZqieWGt4xL4rRySfXNCC2OjxYHI7 hhU09pxKJeq2fD+UgjJXeJHpv2qR/bndaXwsm7RPbHJUhWKZgDoiDiGcbxufYj6T9X90DCfD6X6 EA3mYPevLRu343lMP2JClxrC6EZzfbLssT/Jn/rsnFvcoFweS6vuR7Q== X-Gm-Gg: AYBFou1QKV8VCUd4Ux66AFbxgEVmIv63+nG65IZyQ9SmSrv7TsNEwLRjJjncZvLkBb7 n+Sux4PJMKNnIAWg6g/EWzhRRdMOausrj9A6jlzWUmoAU5fdQVuE/UlasLEOa9/Rn4PnxS/pvKw Gvpr92GtTTGk57Zo654iljSLYzpXUXRqNAx3BsgjHzmjG8RovtExeNQNrrDrUR4/PZbykiOYttz nn8wmUcH9Gk/RsTcaosnyk2qdwhhWFccAX6JMPKfhsvJ04EenHM5bOIk7/Drlg5Y6mjVxxh7/ja TfzJ6iKIvpGpFfFZBUFQkzIVJyKRF827h3iSUG/lGfWmDGBdMKdK4zHLtGdgpfnasfQVvBEOnhd aDoCjOS/4mSaxEgWa X-Received: by 2002:a05:6000:4610:b0:486:e920:6726 with SMTP id ffacd0b85a97d-4871e37b86bmr13962940f8f.51.1789990112084; Mon, 21 Sep 2026 04:28:32 -0700 (PDT) X-Received: by 2002:a05:6000:4610:b0:486:e920:6726 with SMTP id ffacd0b85a97d-4871e37b86bmr13962907f8f.51.1789990111615; Mon, 21 Sep 2026 04:28:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1789990111; cv=none; d=google.com; s=arc-20260327; b=sV/FeR/dy3FpX7587T5AyrIPNSR65q0RIC0kL2Ls8TrMW3+dWwJxwTUMPvfd8b7oD2 350CI61AyA8F0SM0Rtrru1LGq81V2FeFNPo5rKtXDVAouWM/jsHfNO97ChB1CLYn9puR WJJYJdtNMF3woGYCB9n2v5LLPAui9cqpOm/hpWD9ZZT61kjabI1hv2nFkRPqD/cAkdx8 DZeXg/j0eGJgN9h/11uih1X5/WGH8mDItuY0epMWdCUY7htrzv8IgVk/nBkTUbx+CEy6 DFnbba5PYO0GTw5XtiVFoxB8hrqt2pZwOQpi2qJFGi6a13Yp68mAdItLmJKIGbu7QLbW inbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=cc:to:message-id:content-transfer-encoding:mime-version:subject :date:from:dkim-signature; bh=tcCL9Yoetmk7U1YM0tw9vUcVhAm8wDH8hzDz2qXWK4g=; fh=6JpqpGxFeFNaDh+k4t2UfZyo3rKe/54dyQ52T8vVmSc=; b=R7FPsvnubI5+leGxl7/QYHw6R6PuXPvfcrckOL+5v5v+KhMsF9mG2EUgI481OBIIKw 4H/8vDT1JuY8/w7Rh0giid+VhRL1Xxia53IRB4x1Xj6SESOfbzTa80xfX82pVzvNUrXW VhJZYc0fr8c04E2qTLI63pnuuerDOdMOfQVGdD2dXRojewll5eYdnQrTVPVRRtepkWMP r0afWxtexBsa9LxQuRzq/fY3rCmJu+invmrzhe+zFnrH/7UVJR2aPtjsd9LGgGLU1LAQ hkhIWHnEVxbjXODvIILocLa+1f9dMCS3M5jd40D9sgh4s3mA4PrNbtIjkHLK5k8jAL+F e7fw==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=Bh2BMaFA; spf=pass (google.com: domain of kui.liu@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=kui.liu@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com Received: from relay.virtuozzo.com (relay.virtuozzo.com. [130.117.225.111]) by mx.google.com with ESMTPS id ffacd0b85a97d-4885be01fe5si4294652f8f.206.2026.09.21.04.28.31 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 04:28:31 -0700 (PDT) Received-SPF: pass (google.com: domain of kui.liu@virtuozzo.com designates 130.117.225.111 as permitted sender) client-ip=130.117.225.111; Authentication-Results: mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=Bh2BMaFA; spf=pass (google.com: domain of kui.liu@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=kui.liu@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=Message-Id:Content-Type:MIME-Version:Subject:Date :From; bh=tcCL9Yoetmk7U1YM0tw9vUcVhAm8wDH8hzDz2qXWK4g=; b=Bh2BMaFAu24Qk5y5s+r Fr4G4uH/HcM/wZVuxVpkkTwZ09CgBHmxqs++WDk5RfArDiC1u3YcSXOCaIVTgzbh/YghIHXBYPg3g 9UiBpEqnFTlztH3dcbTB8dHavt9OX8GCDt0Jf8oY5VytUaaEJKgtA5JMhuxppzlq5I/IhJdFbMJCw CJ14ZRwBP/l+8GKj8EQKvTrwoDdePxUgiWxukmkAvTxEiqrMfsVsnI+1XocNHoaNZCc6oDc9iNOwx uI4p6I5EBVH60fKHvLaoFYfgmbjlUpVXcDGf4vqm1uABOB3xwzR3c/1UYzPG51gtTCcHFMPG6XCNq rn4Xq8ZAfItXUIA==; Received: from ch-vpn.virtuozzo.com ([130.117.225.6] helo=LK-MBP.local) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1x8c9M-00GkRs-2i; Mon, 21 Sep 2026 13:28:22 +0200 From: Liu Kui Date: Mon, 21 Sep 2026 19:28:07 +0800 MIME-Version: 1.0 Message-Id: <20260921-acssp872-batch-connect-v1-1-ded37d934f28@virtuozzo.com> X-B4-Tracking: v=1; b=H4sIAAAAAAAC/yWMywrCMBAAf6Xs2S1J0D78leIh2azNekhLtopQ+ u9GPQ7DzA7KRVjh2uxQ+CUqS65gTw1Q8nlmlFgZnHGdGZ1FT6rr0DsMfqOEtOTMtGFgMhceYzD nCDVeC9/l/RtPkEhwaC3c/kKf4VGb7xaO4wOD/5cbgwAAAA== X-Change-ID: 20260921-acssp872-batch-connect-bec05e9db04d To: Konstantin Khorenko X-Mailer: b4 0.16.0 X-OZ-Fwd: true Cc: azaitsev@virtuozzo.com, devel@openvz.org, Liu Kui Subject: [Devel] [PATCH hci-8.1] fs/fuse kio: start connect once per rpc_queue_work batch X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org pcs_rpc_connect() is synchronous: it performs a full connect+auth cycle while holding ep->mutex. rpc_queue_work() splices the entire ep->input_queue onto an on-stack list and used to trigger the connect from pcs_rpc_send() on the first staged message. When the connect fails, pcs_rpc_reset() purges ep->input_queue and ep->state_queue, but the messages still sitting on the on-stack list are invisible to the purge - each of them re-triggers another blocking connect attempt under the same mutex hold. With a large backlog and an unreachable peer this keeps ep->mutex held for O(messages * connect timeout): in one crashdump the rescuer of pcs_wq ground through such a batch for days, starving calendar_work/rpc_close_work on the same ep and wedging umount in flush_workqueue(pcs_wq). Stage all messages from the batch on ep->state_queue first and start the connect once afterwards. A failed connect now aborts the whole batch in a single rpc_abort(), bounding the mutex hold to one connect attempt. This also restores the meaning of state_queue, which otherwise never held more than one message. https://virtuozzo.atlassian.net/browse/VSTOR-135626 Signed-off-by: Liu Kui --- Fix a pathological case seen in ACSSP-872 (and ASUP-3562) crashdumps: a single rpc_queue_work() invocation could hold ep->mutex for days, re-running a blocking connect+auth once per queued message to an unreachable peer, wedging the pcs_wq rescuer, calendar/close works and umount in flush_workqueue(). https://virtuozzo.atlassian.net/browse/ACSSP-872 --- fs/fuse/kio/pcs/pcs_rpc.c | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/fs/fuse/kio/pcs/pcs_rpc.c b/fs/fuse/kio/pcs/pcs_rpc.c index 540225006726..c50e31c08a31 100644 --- a/fs/fuse/kio/pcs/pcs_rpc.c +++ b/fs/fuse/kio/pcs/pcs_rpc.c @@ -702,9 +702,6 @@ static void pcs_rpc_send(struct pcs_rpc * ep, struct pcs_msg * msg, bool requeue list_add_tail(&msg->list, &ep->state_queue); msg->stage = PCS_MSG_STAGE_UNSENT; - - if (ep->state == PCS_RPC_UNCONN) - pcs_rpc_connect(ep); } void pcs_rpc_kick_queue(struct pcs_rpc * ep) @@ -930,6 +927,21 @@ static void rpc_queue_work(struct work_struct *w) list_del_init(&msg->list); pcs_rpc_send(ep, msg, 0); } + + /* pcs_rpc_connect() is synchronous: it can block for a full + * connect+auth cycle while holding ep->mutex. Start it once, after + * every message from input_q has been staged on ep->state_queue, + * instead of from pcs_rpc_send() on the first message. This way a + * failed connect aborts the whole batch at once: rpc_abort() purges + * ep->state_queue, whereas messages still sitting on the on-stack + * input_q are invisible to the purge and would each trigger another + * blocking connect attempt under the same mutex hold. It also + * restores the meaning of state_queue, which otherwise never holds + * more than one message. + */ + if (ep->state == PCS_RPC_UNCONN && !list_empty(&ep->state_queue)) + pcs_rpc_connect(ep); + list_splice_tail_init(&ep->state_queue, &state_q); while (!list_empty(&state_q)) { struct pcs_msg * msg = list_first_entry(&state_q, struct pcs_msg, list); --- base-commit: 3851feefb321d7dee9e57cbe7926410aa6dd043f change-id: 20260921-acssp872-batch-connect-bec05e9db04d Best regards, -- Liu Kui _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel