[Devel] [PATCH RK hci-7.4 0/2] fix node crash on RK replace: pcs RDMA handlers left in the unloaded RK

["Denis V. Lunev" <den@openvz.org>]

Replacing RK v8.51..v8.53 on 5.14.0-427.106.1.vz9.90.3 crashes nodes
with live pcs RDMA connections on the next CM event of a connection
opened under the replaced RK (ASUP-3835):

  BUG: unable to handle page fault for address: ffffffffc170ef30
  #PF: supervisor instruction fetch in kernel mode
  RIP: 0010:0xffffffffc170ef30
   cma_cm_event_handler+0x1e/0xd0 [rdma_cm]
  [last unloaded: kpatch_cumulative_8_51_r1]

TRACE() embeds __LINE__, so the VSTOR-141673 hunk ("fix RPC response
drop for large messages sent over RDMA") drags every TRACE() user below
rio_handle_tx() into the RK, pcs_rdma_create() and the RDMA event
handlers among them. The patched pcs_rdma_create() then stores the
addresses of the RK copies of the handlers in rio->id, the CQ and the
QP, and the RK's KBUILD_MODNAME as restrack name of the PD and CQ, which
the MRs inherit. "kpatch replace" unloads the old RK right after the new
one is enabled and all of these dangle.

Patch 1 keeps the line numbers stable, so these functions drop out of
the RK and new connections get the fuse_kio_pcs originals. Patch 2
repairs connections opened under an older RK from a post-patch
callback. Afterwards nothing points into any RK, so a later replace,
an unload or a downgrade is safe whatever the next RK contains. Both
are needed: without patch 1 the callback would resolve the handlers to
the RK's own copies.

The series goes on top of kpatch-patch.vzs
5.14.0-427.106.1.vz9.90.3-v8.53, after
0002-fs-fuse-kio-fix-RPC-response-drop-for-large-messages.patch; the
#line values are computed for that state. Patch 2 has to stay in all
later 90.3 RKs, since nodes may jump straight from v8.51..v8.53 to any
of them. It is idempotent where there is nothing to repair.

Tested on a 90.3 node without RDMA, built with kpatch-build 0.9.9 from
the kernel SRPM and the RK config:
- pcs_rdma_io.o contributes only rio_handle_tx() and rio_handle_rx()
  plus the fixup; the handlers reach the module only as klp relocations
  against fuse_kio_pcs;
- no code in the module stores or hands out an address of the module
  itself, apart from the TRACE() format strings below;
- atomic replace over the shipped v8.53 leaves rio_handle_tx() and
  rio_handle_rx() as the only redirected pcs_rdma_io.c functions,
  removing v8.53 and then unloading the new RK are clean.
Repair of live RDMA connections is not tested here.

https://virtuozzo.atlassian.net/browse/VSTOR-151314
Cc: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com>
Cc: Konstantin Khorenko <khorenko@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>

Denis V. Lunev (2):
  fs/fuse kio: keep pcs_rdma_io.c line numbers stable in the RK
  fs/fuse kio: repoint RDMA handlers and restrack names on RK load

 fs/fuse/kio/pcs/pcs_fuse_kdirect.c | 88 ++++++++++++++++++++++++++++++
 fs/fuse/kio/pcs/pcs_rdma_io.c      | 46 ++++++++++++++++
 2 files changed, 134 insertions(+)

-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel