From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (unknown [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id 437838015D for ; Mon, 14 Sep 2026 18:43:08 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68EIfesC030935; Mon, 14 Sep 2026 21:41:41 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68EIfesC030935 Authentication-Results: mail.openvz.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=virtuozzo.com header.i=@virtuozzo.com header.b="i+TB3v2x" Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68EIfcCV030931 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Mon, 14 Sep 2026 21:41:38 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68EIfcCV030931 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-49cd55ee2f7so31007165e9.0 for ; Mon, 14 Sep 2026 11:41:38 -0700 (PDT) X-Gm-Message-State: AFuF++k1ezhV8re3Qkjitm0+vipO3/RWw161iTyHpSgH/DtnM3lKYCJE gyhPLF1zNbH+eDkCHijy0tsjChZyn+0Q4pA2MLN54pxNpG8T2RnKblz6EXI8BvVU+ucawJ6QIEW jHe0A0mSIw1+AWI3dWZqC/mO+PEq3WbDKvkI4bWyTOG3OH6I6eCmQng== X-Gm-Gg: AYBFou1AkFYy412kg0D5MO57J1XteghSJFx7npsGcg/pTF7IRJFIuG63BUj4sf1p0u+ hzFCKrMIY6+0EhZpvTDvSBLJWj5QV4T+e0lENKDRZeMzbKw/AiZ96i4nyHCK+TxUXE3zHm/Py3D wVyWZZJItMKbDCcgiIil4tvGuKfCVLpAor0iK/Z8m+1jcEsD7Y5VHoiUQWPfDlpkNch6N1mWQBh vk2p1l8+EenjYJm0xV/fIGIjYt2NFOnjY1h2iHfzDZ8rrYM0BTN1by3qXVOkxRKus26ZeatG9x1 mSaqsTUP/dYrHq5F7SZF++1D9ZctdrOOI2JOXSDL5EfKyIYqc3BGAFD7cJYUemkCE4qQUrwvTXq YiSN8wIgmcNdQAXT5 X-Received: by 2002:a05:600c:4e90:b0:49c:fa21:1c7c with SMTP id 5b1f17b1804b1-49e7a68c831mr51812475e9.17.1789411298122; Mon, 14 Sep 2026 11:41:38 -0700 (PDT) X-Received: by 2002:a05:600c:4e90:b0:49c:fa21:1c7c with SMTP id 5b1f17b1804b1-49e7a68c831mr51812065e9.17.1789411297567; Mon, 14 Sep 2026 11:41:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1789411297; cv=none; d=google.com; s=arc-20260327; b=RVKs3bsFbrQyEh5rEmFZSF6jzz0hapDxFrFT2kBIUyMHxKvS87d2/CJ/5VXWqkjle0 WPC0D9J3xKnuY65i7NfcPmBXtGbU+sex+lwb/6mApIaj/QBlqlPDzcR9LLUebVx/1Efx vN7KBGPUgwiSfytzMtoS9SjCFr4c5MaeiL2bA6YyUOsM89hAtI/sv25bgnuvhuoCZXhD OZAJZ9Zm+r2a+PGdi7YE0aJ01YYqQMaVvt1jFhm1eG+XYwp5rPGvAKcCuBZPI76mZ296 8MYFjklUGhPU1k/NZWme/lCDCMO24c6Mj0mv0L3y+F7c2MivFSn37zc+hCFn2VSF2WuP 6fRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=message-id:date:references:in-reply-to:cc:to:from:subject :content-transfer-encoding:mime-version:dkim-signature; bh=i03K0RZ1JMj5eeVytrC88yVpUkqJO5/XeJ6wfTPkvzc=; fh=IbaWj82zqOZm03F/eoPTlWBn4zwBRsoZlMDBqCRFIUM=; b=UsuL5tyqzFOvmi4fsAm0geAKHNbmqkniiE+4J4ibAY2tUrUIz8EluZdXTI7QLvXVP8 a9mNcHUVvbzFZsoSfr35mt//dOjKm2U3uV5pWnJEIIszW2DMKHvNs8UxaLrlBVL5DXXU lb+iA2iVu11DEXhphX6K9e/tNk76TG+KzyWHJH/43BIib8sVw9veXm4E0x3bcnyM0liS eBgvhA0pgLAEiZJh/Arc5QFmQlNpnxqLd9wykqExN9fPRDQALlz6AwAtDnaYXZJ1bae1 wmWR0zk9EAyUu+p3y93ao+J0ZJ3oU7Nv8n48FvRhc/doEw+RJC3JuljiytqV4y5tWYbX rVjQ==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=i+TB3v2x; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com Received: from relay.virtuozzo.com (relay.virtuozzo.com. [130.117.225.111]) by mx.google.com with ESMTPS id 5b1f17b1804b1-49e7d688d12si7167325e9.22.2026.09.14.11.41.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 11:41:37 -0700 (PDT) Received-SPF: pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) client-ip=130.117.225.111; Authentication-Results: mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=i+TB3v2x; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=Message-Id:Date:From:Subject:Content-Type: MIME-Version; bh=i03K0RZ1JMj5eeVytrC88yVpUkqJO5/XeJ6wfTPkvzc=; b=i+TB3v2xcE8M +iSD24Z8kl7nhCu8leMr7VI9jjPtrmq33MBXcqDI4CxAadYonviDh0DLbxotOc3nNryVWfLt7VQxd ZD3TBsSxgZVOsRZ3qOjc5nUeLxXoABSub4PjBtXggfJURFlRyMcM17SoYXnp/+qM/1X3Lw292ibkc xmjLj1/tR4u0kdjTN/RLyCHoBBiUKDMbsQ9BPCB7NAAkudiNvR2qaP2lUOqu5gYMKXV/biLkLypiQ mo9Wa/9livaTnzebnlednqxZhqrFIJ4X2kmoK6BRMXKE0ciUtQQNIWWzPQJotPNl3lIzEBJ/LW4GM o3f6EAiJyD0tu61rF24ygw==; Received: from ch-demo-asa.virtuozzo.com ([130.117.225.8] helo=f0.sw.ru) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1x6BZp-00847y-11; Mon, 14 Sep 2026 20:41:28 +0200 MIME-Version: 1.0 From: Konstantin Khorenko To: Liu Kui In-Reply-To: <20260914104002.53475-1-kui.liu@virtuozzo.com> References: <20260914104002.53475-1-kui.liu@virtuozzo.com> Date: Mon, 14 Sep 2026 20:41:32 +0200 Message-Id: <178941129294.2343294.4750086745771737260.b4-review@b4> X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4783; i=khorenko@virtuozzo.com; h=from:subject:message-id; bh=c9lWC2hPPMdN5HndDhbzIdqgyBCaA+UOR27R4I4/WW8=; b=owEBbQKS/ZANAwAKAVGWCkf5YVwqAcsmYgBqqD/g53yApaDRoa3DGlT+26eOFe0xN3PSsfa33 orPANRTCwSJAjMEAAEKAB0WIQRWD23IPcXT0GO9CCxRlgpH+WFcKgUCaqg/4AAKCRBRlgpH+WFc Kn40D/9qijETDZJI4N40OGdHlVJ+rMZZK6tyEQXtwp+W7eTR+6xn9hYKL0Q36Jsn1Z+BWiH0ZO8 95zTu9DXB5TJB9SaWi/rJfXSGyKMON6EP/y0Qo4y0wCEPx9SLbZ4aMxlwEq3+RR+uCSTgTV0e21 j//GaSW5HHyUPU9Xpvlt01uSwCEAABV+cOTjH3RM1cJyCO0Z6/kOUYBxfPkeYq+OGFziJdtPM4o nOQXZaRJ4O703oL9E0hRdP6P10EeJ4XqUF9EiCIuL+X0bcZRN5PC0crfd4nvi2IhXI5DoOjQ76p 9bSsxVxLSvTHMg0u8M8Fq2YQ6+v1cJSEsNmOmiIC+oW3YZGp6LRXkoNYbX1aLam6tCM3a2+QShs jlMjYoZxQnBONkUORc7o6LX6/SLwFR1Ha2LOSdNCgtMLOLuBa0HSHa2BLKHKTlKgmE/vHasfATR juaTvRf2PTLwrli+iQYCORm38rFK3XE6cliOq8IQN9Lc70mN+0fxniNOW5oq1IPl9R4pWVjYa5Y xlrq0TM34jX+W3e1AdFeeuohX/SNK2cdOTSjcWx5Ih+TgIscOk+lESYSi74vzjiYZKs9DwTZeaD 0nyig6IVeDDpuPfik+NmK/SyLqG4ElkT6zeIYqMuBWoJpi/vNK5US3By50UchByvwTrmowwKhQc I4IctvobwYGtQ2Q== X-Developer-Key: i=khorenko@virtuozzo.com; a=openpgp; fpr=560F6DC83DC5D3D063BD082C51960A47F9615C2A X-OZ-Fwd: true Cc: azaitsev@virtuozzo.com, devel@openvz.org Subject: Re: [Devel] [PATCH VZ10] fs/fuse kio: fix RPC response drop for large messages sent over RDMA X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org fs/fuse kio: fix RPC response drop for large messages sent over RDMA The patch moves the ->done() completion of a large outgoing message from rio_handle_tx(), where it waited for both the local TX completion and the peer's RDMA_READ_ACK, into rio_handle_rx_read_ack(), so the request reaches the rpc pending queue before its response can arrive. The tx itself is still recycled only after both events. > diff --git a/fs/fuse/kio/pcs/pcs_rdma_io.c b/fs/fuse/kio/pcs/pcs_rdma_io.c > index 1c464e0e60f3e..14beab8fda8c9 100644 > --- a/fs/fuse/kio/pcs/pcs_rdma_io.c > +++ b/fs/fuse/kio/pcs/pcs_rdma_io.c > @@ -797,7 +797,6 @@ static inline void rio_kick_write_queue(struct pcs_rdmaio *rio) > > static void rio_handle_tx(struct pcs_rdmaio *rio, struct rio_tx *tx, int ok) > { > - struct pcs_msg *msg = tx->msg; > u64 xid = tx->xid; > > /* override remote success if we already aborted */ > @@ -825,13 +824,6 @@ static void rio_handle_tx(struct pcs_rdmaio *rio, struct rio_tx *tx, int ok) > default: > BUG(); > } > - > - if (msg) { > - if (!ok) > - pcs_set_local_error(&msg->error, PCS_ERR_NET_ABORT); > - > - rio_msg_sent(rio, NULL, msg, 1); > - } > } [Severity: Critical] Does this also drop the completion of messages received over RDMA READ? The removed block ran for every tx_state, not only for the two states of an outgoing large message. A tx in state TX_SUBMIT_RDMA_READ_ACK carries the *incoming* message that the local side pulls from the peer, and it is set up in rio_submit_rdma_read(): fs/fuse/kio/pcs/pcs_rdma_io.c:rio_submit_rdma_read() { ... tx->tx_state = TX_SUBMIT_RDMA_READ_ACK; tx->msg = msg; tx->xid = rb->xid; ... } The msg there comes from rio_handle_rx_immediate(), which calls rio->netio.getmsg() and then completes the message itself only when the whole body arrived inline: fs/fuse/kio/pcs/pcs_rdma_io.c:rio_handle_rx_immediate() { ... if (len == msg->size) { msg->done(msg); } else if (rio_submit_rdma_read(rio, msg, offset, rb, true) == -EAGAIN) { ... } So for a message that needs an RDMA READ, the delivery to the rpc layer used to happen when the read completed: rio_tx_done() -> rio_handle_tx() -> case TX_SUBMIT_RDMA_READ_ACK rio_put_tx() /* clears tx->msg */ rio_submit(SUBMIT_RDMA_READ_ACK) ... if (msg) rio_msg_sent(rio, NULL, msg, 1); /* msg->done(msg) */ After this patch nothing calls ->done() on that path. rio_handle_rx_read_ack() does not cover it: it walks rio->active_txs, and a TX_SUBMIT_RDMA_READ_ACK tx is never put on that list - only rio_msg_sent(..., done == 0) adds a tx there, and that is reached only from the rio_submit() send path. Does this mean every received message larger than RDMA_THRESHOLD is now never handed to rpc_work_input()/the response handler, so the msg allocated by rpc_get_hdr() leaks and the corresponding request only completes when the kill calendar fires? The same block was also the only place that reported a failed RDMA READ to that message: if (!ok) pcs_set_local_error(&msg->error, PCS_ERR_NET_ABORT); Would keeping the msg handling for the TX_SUBMIT_RDMA_READ_ACK case, and clearing tx->msg on the ack path instead, cover both message directions? > @@ -931,6 +923,7 @@ static int rio_handle_rx_read_ack(struct pcs_rdmaio *rio, > > list_for_each_entry(tx, &rio->active_txs, list) > if (tx->xid == rack->xid) { > + rio_msg_sent(rio, NULL, tx->msg, 1); > list_del(&tx->list); > rio_handle_tx(rio, tx, !rack->status); > return 0; [Severity: Low] This isn't a bug today, but is there a reason to complete the message before unlinking the tx? rio_msg_sent() calls msg->done(msg), after which the msg may be freed, while the tx is still on rio->active_txs with tx->msg pointing at it. Both users of that list, rio_cleanup() and pcs_rdma_next_timeout(), dereference tx->msg: fs/fuse/kio/pcs/pcs_rdma_io.c:pcs_rdma_next_timeout() { ... if (!list_empty(&rio->active_txs)) { tx = list_first_entry(&rio->active_txs, struct rio_tx, list); return tx->msg->start_time + rio->send_timeout; } ... } Doing the list_del() first would keep the completed msg unreachable from the list. tx->msg is also left pointing at the completed message until the TX completion reaches rio_put_tx(). [Severity: Low] The commit fixes a bug that came in with the original RDMA transport support. Should it carry Fixes: bcf01a42ff8c ("fs/fuse kio: implement support RDMA transport") The Feature: vStorage tag used by the neighbouring fs/fuse kio commits, and present in the reverted 21b2e4b35dfa ("fs/fuse kio: fix lost rpc response over rdma io"), is missing as well. -- Konstantin Khorenko _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel