From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.openvz.org (unknown [69.168.225.77]) by lore.virtuozzo.com (Postfix) with ESMTPS id 0CFDB8015C for ; Thu, 10 Sep 2026 16:18:21 +0000 (UTC) Received: from mail.openvz.org (localhost [127.0.0.1]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68AGH5Vv026602; Thu, 10 Sep 2026 19:17:06 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68AGH5Vv026602 Authentication-Results: mail.openvz.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=virtuozzo.com header.i=@virtuozzo.com header.b="KY9lFrPp" Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by mail.openvz.org (8.14.4/8.14.4) with ESMTP id 68AGH3Ns026598 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Thu, 10 Sep 2026 19:17:03 +0300 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.openvz.org 68AGH3Ns026598 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-485f6788029so1422402f8f.1 for ; Thu, 10 Sep 2026 09:17:03 -0700 (PDT) X-Forwarded-Encrypted: i=2; AKwUvBx4zYHYen7gH+Rpet0zlyXoUR1UwSfPo63Js/zPpFETRTwqPOoAp25vWeYJHKZvhtncIfuZMQ==@openvz.org X-Gm-Message-State: AFuF++k/TEOZCtSbe6ClMZYXfi7b0h0KROPswk4/P6k6M3+XpLOFxO5C VlJwthxCc5AuaWkp1J2QZnMJOu2sLiAae9Q+YzKMsPQUj1EMii+VuGqDYwnkq6v0GsSe8MftiWI 8LYZtAiYFNvDcsa+tTkNlrAZu1c8Kc1CeBXqyHN4Ffjx1sN2HP6UR2A== X-Gm-Gg: AYBFou0VW9WqQrg3yMvPriJWhyjxFB/zGmUIFRsgoKZIScGib6I4iN3CI6Qderk9f8t nGfc36tv2Fs0XYx1gfgukQ0VxFbMTkpIFgsGNCGvmgLT8RJm0nwHf721CpLy7gh0Tn25dbe1Tyk xkk2J4yPjzjaDzYGB9sWdb+sXbZaz8YsZQfyQOMDwjDOOiriTR9kRVH+z7Gf23DAgTqfdaBMD6t Uc7hwImPFn369Z1LDu0MewJ1zKNkROcw62xVwAwkDCl/bJ0osSm9l4GfcC0pV3pk1rqQgC8MGIm KNRHf9pPa/Ov8G+6/NKhmnzzQYHfJCPWYl948amz42ZdrOjJBWG4DSlpS3r0wO/nyp/x6VtSGjx Vx7letCOgEtzDfTJe X-Received: by 2002:a5d:5e82:0:b0:486:e708:1c30 with SMTP id ffacd0b85a97d-486e70826aamr5955079f8f.2.1789057022949; Thu, 10 Sep 2026 09:17:02 -0700 (PDT) X-Received: by 2002:a5d:5e82:0:b0:486:e708:1c30 with SMTP id ffacd0b85a97d-486e70826aamr5954942f8f.2.1789057022332; Thu, 10 Sep 2026 09:17:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1789057022; cv=none; d=google.com; s=arc-20260327; b=Npj0gE/3isCPNxrBrxMPtK8KqR6hrmt04P0j8VaDp/GjFDdGr/pDg80ahErosen+E4 Nc+7cTSCcFCJPAdODOZt7X4YFy7Yv7iwcGAkRtqfoKza3n9KFy7mu3PTz4wpcmlao2zx wuwGO/gnUvUpsymW0uvUNl2AcV4LXpBJTxoLtG0QeTav+hujOpGm/fI59HQh8NrgkWSm FX4zuBuixvDYboBn7LBx/HJZ+APsS77YonEXUXHuZsRTPhHB1NJepNNzLVjzdHv8zP68 fGa3BhV+KZLFuHb/rv9ET2msj984H5j2fBUV6/olGwxhOPYFLbYWTs6UrsAxlULNRPjz 7EaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=message-id:date:references:in-reply-to:cc:to:from:subject :content-transfer-encoding:mime-version:dkim-signature; bh=K2qaH7eADqKqpf9cNjlLfn4vOk44OkNRp86aBAkxsqk=; fh=Ckz6O6c6QYEPyVDKFr1GNv2Cu3Qk9zNx3WW4wvWrLr0=; b=fSZnkbOAh3VsSjb4+TVQc3daNYC2IAGxrR1B4mIVBBEtoaePhww3o/Xh1WtWtgtXZK AavM69vUOUmYJsPQtKrSVfSBeF2dQsUdvlJul2Ox5U6XenXLJHSOhgw4dJubNsGDTInK aZLgsC3Osf/mpyJJiLjmjHSjmYoALCh+w+U2FrgClNpBqk8IPdfhNSgBHvQcP8qCzfJE Evk+je1gniCKJ5l4pOYvHoO+3xLcL/NhCEuuc/8xjs0tMbyZ/Xy44wn2omBdLETxqrjE Zhx381/IW0/l9Q2MoO9nX/dl/7dSXU5gjn5+CdWJCLhAJP+4XLVclRY/ItZwjBliLEHy mgOg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=KY9lFrPp; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com Received: from relay.virtuozzo.com (relay.virtuozzo.com. [130.117.225.111]) by mx.google.com with ESMTPS id ffacd0b85a97d-485885dd798si39974290f8f.400.2026.09.10.09.17.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 09:17:01 -0700 (PDT) Received-SPF: pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) client-ip=130.117.225.111; Authentication-Results: mx.google.com; dkim=pass header.i=@virtuozzo.com header.s=relay header.b=KY9lFrPp; spf=pass (google.com: domain of khorenko@virtuozzo.com designates 130.117.225.111 as permitted sender) smtp.mailfrom=khorenko@virtuozzo.com; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=virtuozzo.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=Message-Id:Date:From:Subject:Content-Type: MIME-Version; bh=K2qaH7eADqKqpf9cNjlLfn4vOk44OkNRp86aBAkxsqk=; b=KY9lFrPphLmU +glMUhiaykJZdWh6GzolugTzKlcIW5jz9C9AHPbWlCm9j3ZUgIGbWn9LXyEiJ6dX1jxACH8YQV3ul 8B8mfn+kGGq65Md9TNA0NJdgmgwkn0m9/tUfgEcVogrDGQLu79uTYoO+H2yOQ1Wb20Mg/UDykEbFs tVK8twljJhJpTQVusBBDLqHdTREAJ24oo40ZSukRfe8qgobVmzcVHGgzUBZcE/2HteLQay0IfUv7o 4gcTeim7E7u5gEE46gdNWwVppkYCoRBMoybEmqRUs9BRXeH5fx56u95JTSyGQFdtIaB3XZrEfstgi Zn93ewGt5e3jfOpDJHwc2Q==; Received: from ch-demo-asa.virtuozzo.com ([130.117.225.8] helo=f0.sw.ru) by relay.virtuozzo.com with esmtp (Exim 4.96) (envelope-from ) id 1x4hPn-0006V0-37; Thu, 10 Sep 2026 18:16:59 +0200 MIME-Version: 1.0 From: Konstantin Khorenko To: Aleksandr Kharten In-Reply-To: <20260910-b4-vstor-141673-fix-lost-rdma-reply-v1-1-5cdd9b8180ab@virtuozzo.com> References: <20260910-b4-vstor-141673-fix-lost-rdma-reply-v1-1-5cdd9b8180ab@virtuozzo.com> Date: Thu, 10 Sep 2026 18:16:57 +0200 Message-Id: <178905701788.2120798.14628650852525213710.b4-review@b4> X-Mailer: b4 0.16.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4233; i=khorenko@virtuozzo.com; h=from:subject:message-id; bh=WI+1z4OGjbeuHR1em+MNMZAsMe+PEoOaCrlX4MWjuKY=; b=owEBbQKS/ZANAwAKAVGWCkf5YVwqAcsmYgBqotf8f/jDa/AXcUf/aMWNjrVzbScF2BswLC1tP 1lNOSS0WOqJAjMEAAEKAB0WIQRWD23IPcXT0GO9CCxRlgpH+WFcKgUCaqLX/AAKCRBRlgpH+WFc KgSfD/4lau1HxSodjQUxsHD/BhxJtmiid1ZBZ8sh2+1AOxDDOV9MK/4L1HG0vDcaHDlSucujAK7 GGabHcJMa4S7PeTU1nbdl07ufX/ad/gnW2XUgwZyqHzHLa5g4A4R9cKB4/bZRo+xB1tVSwBamCu KCzVQcEZ+WF4tDm4tXprN2lQSnC5oNOTcgrvmc2KV5++oA24lT2ZMJ/EBv1Rki+ttD5Ar2ENTC+ aXF2RBaXztSkziucSWaF0u43rVUO6ZWZ6X43Cu+guGxk6SWObV3U8W2BL+XtSAhrBSHOaHbRF0x 7gKxSnh1bIqcnxeth5MmOcve+xtRdEsuqWjxaoW3Tvx7eWtltReJzALVrbSZqvAgoAtsX9BMgTL UAOwS3JSpu2wo3Ad7Mmm5hBJPbDY67+xNZX6eWnEzuA+LX1fvJAzcwywpbQOQqYVDrWuSmOHB7L SOWN7/hta/lZu9RMGjFWMkQf+t9XGqWSBFJredVWp/sqotSJbgr2axMI4rZWPn1ndOppWElH2oo a6H1FtFPJMwy2v6Y2t1MaTs1zLhiO94GbCQM2zFWL0Lb/Cws9uN02ANm41VaS7IFmV/7qhoklJB VzJsFi3GkxMPfMemL8QNtBzTrTHvRAp36nzGgz4ss6h5IB/3as8xiltVKw6SnCMhBuywlc3PuUs avFJ7wQ/SozSWoA== X-Developer-Key: i=khorenko@virtuozzo.com; a=openpgp; fpr=560F6DC83DC5D3D063BD082C51960A47F9615C2A X-OZ-Fwd: true Cc: Andrey Zaitsev , devel@openvz.org, Kui Liu Subject: Re: [Devel] [PATCH vz10] fs/fuse kio: fix lost rpc response over rdma io X-BeenThere: devel@openvz.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: OpenVZ development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: devel-bounces@openvz.org Errors-To: devel-bounces@openvz.org > Call ->done() when the peer's RDMA_READ_ACK arrives rather than waiting > for our own TX completion as well, so that a large message is reported > as sent before its response can come back. The tx itself is still > released only once both events are in, as the HCA may still be using > its MR. > > https://virtuozzo.atlassian.net/browse/VSTOR-141673 You have inserted "---" into the commit message, please note, the automation drops all below first seen "---" at the beginning of the line. > > diff --git a/fs/fuse/kio/pcs/pcs_rdma_io.c b/fs/fuse/kio/pcs/pcs_rdma_io.c > index 1c464e0e60f3e..171e0bf3098bb 100644 > --- a/fs/fuse/kio/pcs/pcs_rdma_io.c > +++ b/fs/fuse/kio/pcs/pcs_rdma_io.c > @@ -60,8 +60,9 @@ struct rio_rx { > > enum { > TX_FREE, /* free tx request available for use */ > - TX_WAIT_FOR_TX_COMPL, /* tx request sent, wait for TX completion */ > - TX_WAIT_FOR_READ_ACK, /* wait for peer to ack RDMA read */ > + TX_WAIT_FOR_TX_COMPL, /* wait for TX completion and RDMA read ack */ > + TX_WAIT_FOR_READ_ACK, /* TX completed, wait for RDMA read ack */ > + TX_READ_ACKED, /* msg completed on ack, wait for TX completion */ > TX_MSG_DONE, /* default: call msg->done() */ > TX_SUBMIT_RDMA_READ_ACK, /* let our peer know that our RDMA_READ is done */ > }; > @@ -720,10 +721,9 @@ static int rio_submit(struct pcs_rdmaio *rio, struct pcs_msg *msg, int type, u64 > * lose an error. > * > * But, if the message is big enough to trigger RDMA READ > - * transfer, we are going to call ->done() callback after we > - * receive RDMA_READ_ACK message from our peer. Since messages > - * in a single RX queue are guaranteed to come in order, there > - * is no race in this case. > + * transfer, ->done() cannot be called here: the peer has not > + * pulled the payload yet. It is called from rio_handle_tx() > + * when RDMA_READ_ACK arrives, see the comment there. > */ > rio_msg_sent(rio, tx, msg, rb == NULL); > } > @@ -795,7 +795,21 @@ static inline void rio_kick_write_queue(struct pcs_rdmaio *rio) > rio_enable_kick(rio); > } > > -static void rio_handle_tx(struct pcs_rdmaio *rio, struct rio_tx *tx, int ok) > +/* Events a tx waits for, see rio_handle_tx() */ > +enum rio_tx_event { > + TX_EV_SEND_CQE, /* TX completion from our HCA */ > + TX_EV_READ_ACK, /* RDMA_READ_ACK from peer */ > +}; > + > +/* > + * Handle one of the two events @tx waits for. They are not ordered: the peer > + * may ack the RDMA read, and send its response, before our HCA reports the > + * send. Hence msg->done() is called on TX_EV_READ_ACK, which the response > + * cannot overtake as both come from the same RX queue, while the tx is > + * released only after both events, as the HCA may still be using its MR. > + */ > +static void rio_handle_tx(struct pcs_rdmaio *rio, struct rio_tx *tx, int ok, > + enum rio_tx_event ev) > { > struct pcs_msg *msg = tx->msg; > u64 xid = tx->xid; > @@ -814,9 +828,25 @@ static void rio_handle_tx(struct pcs_rdmaio *rio, struct rio_tx *tx, int ok) > rio->n_rdma_read_ongoing--; > break; > case TX_WAIT_FOR_TX_COMPL: > - case TX_WAIT_FOR_READ_ACK: > - if (++tx->tx_state != TX_MSG_DONE) > + if (ev == TX_EV_SEND_CQE) { > + /* Still waiting for the peer to ack RDMA read */ > + tx->tx_state = TX_WAIT_FOR_READ_ACK; > return; > + } > + /* > + * The peer is done with the payload, complete msg now > + * and keep the tx until our own TX completion arrives. > + * The tx must keep no reference to msg, which may be > + * freed by the completion below. > + */ > + tx->tx_state = TX_READ_ACKED; > + tx->msg = NULL; > + pcs_rdma_msg_detach(&tx->send.msg); > + break; > + case TX_WAIT_FOR_READ_ACK: > + rio_put_tx(rio->dev, tx); > + break; > + case TX_READ_ACKED: > rio_put_tx(rio->dev, tx); > break; > case TX_MSG_DONE: 3 identical cases here, may be to merged them? i will apply the patch as is but think about it and rework if you consider that. -- Konstantin Khorenko _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel