[Devel] [PATCH RK hci-7.4 0/2] fix node crash on RK replace: pcs RDMA handlers left in the unloaded RK

[Devel] [PATCH RK hci-7.4 0/2] fix node crash on RK replace: pcs RDMA handlers left in the unloaded RK

[Denis V. Lunev]

Replacing RK v8.51..v8.53 on 5.14.0-427.106.1.vz9.90.3 crashes nodes
with live pcs RDMA connections on the next CM event of a connection
opened under the replaced RK (ASUP-3835):

  BUG: unable to handle page fault for address: ffffffffc170ef30
  #PF: supervisor instruction fetch in kernel mode
  RIP: 0010:0xffffffffc170ef30
   cma_cm_event_handler+0x1e/0xd0 [rdma_cm]
  [last unloaded: kpatch_cumulative_8_51_r1]

TRACE() embeds __LINE__, so the VSTOR-141673 hunk ("fix RPC response
drop for large messages sent over RDMA") drags every TRACE() user below
rio_handle_tx() into the RK, pcs_rdma_create() and the RDMA event
handlers among them. The patched pcs_rdma_create() then stores the
addresses of the RK copies of the handlers in rio->id, the CQ and the
QP, and the RK's KBUILD_MODNAME as restrack name of the PD and CQ, which
the MRs inherit. "kpatch replace" unloads the old RK right after the new
one is enabled and all of these dangle.

Patch 1 keeps the line numbers stable, so these functions drop out of
the RK and new connections get the fuse_kio_pcs originals. Patch 2
repairs connections opened under an older RK from a post-patch
callback. Afterwards nothing points into any RK, so a later replace,
an unload or a downgrade is safe whatever the next RK contains. Both
are needed: without patch 1 the callback would resolve the handlers to
the RK's own copies.

The series goes on top of kpatch-patch.vzs
5.14.0-427.106.1.vz9.90.3-v8.53, after
0002-fs-fuse-kio-fix-RPC-response-drop-for-large-messages.patch; the
#line values are computed for that state. Patch 2 has to stay in all
later 90.3 RKs, since nodes may jump straight from v8.51..v8.53 to any
of them. It is idempotent where there is nothing to repair.

Tested on a 90.3 node without RDMA, built with kpatch-build 0.9.9 from
the kernel SRPM and the RK config:
- pcs_rdma_io.o contributes only rio_handle_tx() and rio_handle_rx()
  plus the fixup; the handlers reach the module only as klp relocations
  against fuse_kio_pcs;
- no code in the module stores or hands out an address of the module
  itself, apart from the TRACE() format strings below;
- atomic replace over the shipped v8.53 leaves rio_handle_tx() and
  rio_handle_rx() as the only redirected pcs_rdma_io.c functions,
  removing v8.53 and then unloading the new RK are clean.
Repair of live RDMA connections is not tested here.

https://virtuozzo.atlassian.net/browse/VSTOR-151314
Cc: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com>
Cc: Konstantin Khorenko <khorenko@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>

Denis V. Lunev (2):
  fs/fuse kio: keep pcs_rdma_io.c line numbers stable in the RK
  fs/fuse kio: repoint RDMA handlers and restrack names on RK load

 fs/fuse/kio/pcs/pcs_fuse_kdirect.c | 88 ++++++++++++++++++++++++++++++
 fs/fuse/kio/pcs/pcs_rdma_io.c      | 46 ++++++++++++++++
 2 files changed, 134 insertions(+)

-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH RK hci-7.4 1/2] fs/fuse kio: keep pcs_rdma_io.c line numbers stable in the RK

[Denis V. Lunev]

TRACE() embeds __LINE__, so the net line shift of "fix RPC response
drop for large messages sent over RDMA" changes the trace string of
every TRACE() user below rio_handle_tx(). kpatch-build then treats
pcs_rdma_create() and the RDMA event handlers as changed and puts
copies of them into the patch module.

The patched pcs_rdma_create() stores the addresses of those copies in
rio->id.event_handler, cq->event_handler and qp->event_handler, which
live as long as the connection. Once the RK is replaced and its module
unloaded, the next CM event on such a connection jumps into freed
memory:

  BUG: unable to handle page fault for address: ffffffffc170ef30
  #PF: supervisor instruction fetch in kernel mode
  RIP: 0010:0xffffffffc170ef30
   cma_cm_event_handler+0x1e/0xd0 [rdma_cm]

Reset the line counter with #line after the two changed functions so
everything below them is identical to the base kernel and stays out of
the patch.

https://virtuozzo.atlassian.net/browse/VSTOR-151314
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 fs/fuse/kio/pcs/pcs_rdma_io.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/fs/fuse/kio/pcs/pcs_rdma_io.c b/fs/fuse/kio/pcs/pcs_rdma_io.c
index 63fb1e7..67f3aad 100644
--- a/fs/fuse/kio/pcs/pcs_rdma_io.c
+++ b/fs/fuse/kio/pcs/pcs_rdma_io.c
@@ -830,6 +830,7 @@ static void rio_handle_tx(struct pcs_rdmaio *rio, struct rio_tx *tx, int ok)
 			BUG();
 	}
 }
+#line 836
 
 /*
  * rio wire header is already stripped, buf points to payload data (pcs_rpc hdr)
@@ -937,6 +938,7 @@ static int rio_handle_rx_read_ack(struct pcs_rdmaio *rio,
 
 	return PCS_ERR_NET_ABORT;
 }
+#line 941
 
 /*
  * When we see RX coming from the wire very first time, flag "pended" is
-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH RK hci-7.4 1/2] fs/fuse kio: keep pcs_rdma_io.c line numbers stable in the RK

[no-reply]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] fuse kio: #line directives skew __LINE__ and debug line info in pcs_rdma_io.c
--

commit 09b1a08eb3317faf9482b38abadd97a11ab747d0
Author: Denis V. Lunev <den@openvz.org>

fs/fuse kio: keep pcs_rdma_io.c line numbers stable in the RK

This patch adds two #line directives to pcs_rdma_io.c. They make the
lines below rio_handle_rx_read_ack() keep the line numbers of the base
kernel. The aim is that kpatch-build does not treat pcs_rdma_create()
and the RDMA event handlers as changed, because TRACE() embeds __LINE__.

Link: https://virtuozzo.atlassian.net/browse/VSTOR-151314

> diff --git a/fs/fuse/kio/pcs/pcs_rdma_io.c b/fs/fuse/kio/pcs/pcs_rdma_io.c
> --- a/fs/fuse/kio/pcs/pcs_rdma_io.c
> +++ b/fs/fuse/kio/pcs/pcs_rdma_io.c

[ ... ]

> @@ -831,6 +831,7 @@ static void rio_handle_tx(struct pcs_rdmaio *rio, struct rio_tx *tx, int ok)
>  			BUG();
>  	}
>  }
> +#line 836
>  
>  /*
>   * rio wire header is already stripped, buf points to payload data (pcs_rpc hdr)

[Severity: Low]
This isn't a bug, but can the source say why this directive is here?

The #line 836 after rio_handle_tx() makes the logical line numbers differ
from the physical ones. Everything up to the second directive is numbered
one higher than its physical line. Nothing in the file explains this. A
later reader will see __LINE__, BUG() and WARN locations, compiler
diagnostics and addr2line output that don't match the source.

[ ... ]

> @@ -938,6 +939,7 @@ static int rio_handle_rx_read_ack(struct pcs_rdmaio *rio,
>  
>  	return PCS_ERR_NET_ABORT;
>  }
> +#line 941
>  
>  /*
>   * When we see RX coming from the wire very first time, flag "pended" is

[Severity: Low]
Is this skew limited to the kpatch build?

The directives are unconditional, so every build of the file is affected.
After #line 941, logical numbering is two lower than physical for the rest
of the file. That covers the TRACE() users in pcs_rdma_create() and the
RDMA event handlers, plus the BUG() and WARN sites there.

There is no other #line directive under fs/fuse, so there is no in-tree
precedent for this. The follow-up patch "fs/fuse kio: repoint RDMA
handlers and restrack names on RK load" doesn't remove them.

Could a comment be added next to each directive giving the reason? For
example, it could say that the numbers are pinned to the base kernel so
that kpatch-build doesn't pick up the functions below.

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/cover.1792700354.git.den@openvz.org?part=1

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH RK hci-7.4 2/2] fs/fuse kio: repoint RDMA handlers and restrack names on RK load

[Denis V. Lunev]

Up to v8.53 the RK carried its own copies of pcs_rdma_io_event_handler(),
pcs_rdma_cq_event_handler() and pcs_rdma_qp_event_handler(), and every
RDMA connection opened while such an RK was active points at them.
"kpatch replace" unloads the replaced RK right after the new one is
enabled, so these connections are left with dangling handlers and crash
the node on their next CM, QP or CQ event.

The same connections carry that RK's KBUILD_MODNAME as restrack name of
their PD and CQ, and the MRs allocated from the PD inherit it. Any
"rdma resource show pd|cq|mr", which needs no privileges, reads it.
The PD and CQ hang off the connection. MRs in use are on no pcs list,
so they are found through the device's restrack MR table.

From a post-patch callback, rewrite the handlers of every live
connection to the fuse_kio_pcs originals, and the PD, CQ and MR names to
the name of the fuse_kio_pcs module itself. Afterwards nothing points
into any RK module, this one included, so neither a later replace nor
an unload depends on callbacks of the outgoing RK, which "kpatch replace"
does not run anyway.

At that point the transition is complete, so the old pcs_rdma_create()
can no longer run, while the outgoing RK text is still mapped, so a
racing caller still lands in valid code. The values are written
unconditionally, which also repairs connections already pointing into an
RK that has been unloaded.

All endpoints are reached through pcs_client_list and the engine hash.
Each one is fixed under ep->mutex: pcs_rdmaconnect_start() holds it
from pcs_rdma_create() until ep->conn is set, every RDMA netio op runs
under it, so no MR can be allocated from the PD meanwhile, and a
connection is detached from ep->conn and queued on rio_destroy_list
under the same mutex, so flushing rio_destroy_work afterwards covers the
rest. synchronize_rcu_tasks() then waits for anyone preempted between
loading an old handler and calling it.

https://virtuozzo.atlassian.net/browse/VSTOR-151314
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 fs/fuse/kio/pcs/pcs_fuse_kdirect.c | 88 ++++++++++++++++++++++++++++++
 fs/fuse/kio/pcs/pcs_rdma_io.c      | 44 +++++++++++++++
 2 files changed, 132 insertions(+)

diff --git a/fs/fuse/kio/pcs/pcs_fuse_kdirect.c b/fs/fuse/kio/pcs/pcs_fuse_kdirect.c
index 1031c43..5c7d461 100644
--- a/fs/fuse/kio/pcs/pcs_fuse_kdirect.c
+++ b/fs/fuse/kio/pcs/pcs_fuse_kdirect.c
@@ -2121,3 +2121,91 @@ module_exit(kpcs_mod_exit);
 
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR("Virtuozzo <devel@openvz.org>");
+
+#include <linux/rcupdate.h>
+#include "kpatch-macros.h"
+
+void pcs_rdma_rk_fixup_ep(struct pcs_rpc *ep, const char *name);
+void pcs_rdma_rk_fixup_flush(void);
+
+/*
+ * [RK] Grab a reference to every RPC endpoint. Returns false if more
+ * endpoints than @cap were found, the caller has to retry then.
+ */
+static bool rk_collect_eps(struct pcs_rpc **eps, unsigned int cap,
+			   unsigned int *nr)
+{
+	struct pcs_fuse_cluster *pfc;
+	struct pcs_rpc *ep;
+	unsigned int n = 0, i;
+	bool fit = true;
+
+	spin_lock(&pcs_clients_lock);
+	list_for_each_entry(pfc, &pcs_client_list, list) {
+		struct pcs_rpc_engine *eng = &pfc->cc.eng;
+
+		spin_lock(&eng->lock);
+		for (i = 0; i <= PCS_RPC_HASH_SIZE && fit; i++) {
+			struct hlist_head *head = i < PCS_RPC_HASH_SIZE ?
+						  &eng->ht[i] : &eng->unhashed;
+
+			hlist_for_each_entry(ep, head, link) {
+				if (n == cap) {
+					fit = false;
+					break;
+				}
+				if (atomic_inc_not_zero(&ep->refcnt))
+					eps[n++] = ep;
+			}
+		}
+		spin_unlock(&eng->lock);
+		if (!fit)
+			break;
+	}
+	spin_unlock(&pcs_clients_lock);
+
+	*nr = n;
+	return fit;
+}
+
+static unsigned int rk_count_eps(void)
+{
+	struct pcs_fuse_cluster *pfc;
+	unsigned int n = 0;
+
+	spin_lock(&pcs_clients_lock);
+	list_for_each_entry(pfc, &pcs_client_list, list)
+		n += READ_ONCE(pfc->cc.eng.nrpcs);
+	spin_unlock(&pcs_clients_lock);
+
+	return n;
+}
+
+static void rk_fixup_rdma(patch_object *obj)
+{
+	struct pcs_rpc **eps;
+	unsigned int cap, n, i;
+	bool done;
+
+	do {
+		cap = rk_count_eps() + 16;
+		while (!(eps = kvmalloc_array(cap, sizeof(*eps), GFP_KERNEL)))
+			msleep(100);
+
+		done = rk_collect_eps(eps, cap, &n);
+		for (i = 0; i < n; i++) {
+			mutex_lock(&eps[i]->mutex);
+			pcs_rdma_rk_fixup_ep(eps[i], obj->mod->name);
+			mutex_unlock(&eps[i]->mutex);
+			pcs_rpc_put(eps[i]);
+		}
+		kvfree(eps);
+	} while (!done);
+
+	/* rio detached from its ep under ep->mutex sit here until destroyed */
+	pcs_rdma_rk_fixup_flush();
+
+	/* Wait for anyone preempted between loading a handler and calling it */
+	synchronize_rcu_tasks();
+}
+KPATCH_POST_PATCH_CALLBACK(rk_fixup_rdma);
diff --git a/fs/fuse/kio/pcs/pcs_rdma_io.c b/fs/fuse/kio/pcs/pcs_rdma_io.c
index 67f3aad..cf12ede 100644
--- a/fs/fuse/kio/pcs/pcs_rdma_io.c
+++ b/fs/fuse/kio/pcs/pcs_rdma_io.c
@@ -1823,3 +1823,47 @@ struct pcs_netio_tops pcs_rdma_netio_tops = {
 	.sync_send		= pcs_rdma_sync_send,
 	.sync_recv		= pcs_rdma_sync_recv,
 };
+
+#include "../../../../drivers/infiniband/core/restrack.h"
+
+/*
+ * [RK] Connections created while an older RK was active point to that RK's
+ * copies of the handlers below, and their PD, CQ and MRs carry that RK's
+ * KBUILD_MODNAME as restrack name. Repoint them into fuse_kio_pcs before the
+ * RK module is unloaded.
+ */
+void pcs_rdma_rk_fixup_ep(struct pcs_rpc *ep, const char *name)
+{
+	struct pcs_rdmaio *rio;
+	struct rdma_restrack_entry *res;
+	struct ib_pd *pd;
+	struct xarray *xa;
+	unsigned long index;
+
+	BUG_ON(!mutex_is_locked(&ep->mutex));
+
+	if (!ep->conn || ((struct pcs_netio *)ep->conn)->tops != &pcs_rdma_netio_tops)
+		return;
+
+	rio = rio_from_ioconn(ep->conn);
+	WRITE_ONCE(rio->id.event_handler, pcs_rdma_io_event_handler);
+	WRITE_ONCE(rio->cq->event_handler, pcs_rdma_cq_event_handler);
+	WRITE_ONCE(rio->cmid->qp->event_handler, pcs_rdma_qp_event_handler);
+
+	pd = rio->dev->pd;
+	WRITE_ONCE(pd->res.kern_name, name);
+	WRITE_ONCE(rio->cq->res.kern_name, name);
+
+	xa = &pd->device->res[RDMA_RESTRACK_MR].xa;
+	xa_lock(xa);
+	xa_for_each(xa, index, res) {
+		if (container_of(res, struct ib_mr, res)->pd == pd)
+			WRITE_ONCE(res->kern_name, name);
+	}
+	xa_unlock(xa);
+}
+
+void pcs_rdma_rk_fixup_flush(void)
+{
+	flush_work(&rio_destroy_work);
+}
-- 
2.53.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH RK hci-7.4 2/2] fs/fuse kio: repoint RDMA handlers and restrack names on RK load

[no-reply]

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [Medium] fs/fuse kio: RK fixup misses RDMA endpoints of clusters being torn down
- [Low] fuse kio: pcs_rdma_rk_fixup_ep() and _flush() are non-static with no shared prototype
- [Low] fuse kio: unguarded kpatch-macros.h include breaks regular module build
- [Low] fs/fuse kio: RK post-patch callback takes pcs_clients_lock before spin_lock_init()
- [Low] fuse kio: pcs_rdma_rk_fixup_ep() rescans the device MR restrack table per endpoint
--

commit 27821832ddfd0c8f4b03f262b44ec34021d6ecc0
Author: Denis V. Lunev <den@openvz.org>

fs/fuse kio: repoint RDMA handlers and restrack names on RK load

This patch adds a kpatch post-patch callback. The callback rewrites the RDMA
event handlers of live pcs connections to the fuse_kio_pcs originals. It
also rewrites the restrack names of their PDs, CQs and MRs to the
fuse_kio_pcs module name. Endpoints are found through pcs_client_list and
the engine hash tables.

Link: https://virtuozzo.atlassian.net/browse/VSTOR-151314

> diff --git a/fs/fuse/kio/pcs/pcs_fuse_kdirect.c b/fs/fuse/kio/pcs/pcs_fuse_kdirect.c
> --- a/fs/fuse/kio/pcs/pcs_fuse_kdirect.c
> +++ b/fs/fuse/kio/pcs/pcs_fuse_kdirect.c

[ ... ]

> @@ -2139,3 +2139,91 @@ module_exit(kpcs_mod_exit);
>  
>  MODULE_LICENSE("GPL v2");
>  MODULE_AUTHOR("Virtuozzo <devel@openvz.org>");
> +
> +#include <linux/rcupdate.h>
> +#include "kpatch-macros.h"

[Severity: Low]
Is this include expected to build outside of kpatch-build? A search of the
tree finds no kpatch-macros.h. fs/fuse/kio/pcs/Makefile has no include path
or define that would provide it. The include is not under an #ifdef, and
rk_fixup_rdma() uses patch_object and KPATCH_POST_PATCH_CALLBACK. A regular
build of fs/fuse/kio/pcs/pcs_fuse_kdirect.c with CONFIG_FUSE_KIO_PCS could
therefore fail with "kpatch-macros.h: No such file or directory".

Should this code be guarded, or moved somewhere that is only built for the
RK?

> +
> +void pcs_rdma_rk_fixup_ep(struct pcs_rpc *ep, const char *name);
> +void pcs_rdma_rk_fixup_flush(void);

[Severity: Low]
This isn't a bug, but pcs_rdma_rk_fixup_ep() and pcs_rdma_rk_fixup_flush()
are defined non-static in pcs_rdma_io.c with no prototype in scope. They
are declared by hand here instead. That triggers -Wmissing-prototypes under
W=1, and the compiler cannot check that the two signatures agree.

Could both prototypes go into pcs_rdma_io.h?

> +
> +/*
> + * [RK] Grab a reference to every RPC endpoint. Returns false if more
> + * endpoints than @cap were found, the caller has to retry then.
> + */
> +static bool rk_collect_eps(struct pcs_rpc **eps, unsigned int cap,
> +			   unsigned int *nr)
> +{
> +	struct pcs_fuse_cluster *pfc;
> +	struct pcs_rpc *ep;
> +	unsigned int n = 0, i;
> +	bool fit = true;
> +
> +	spin_lock(&pcs_clients_lock);
> +	list_for_each_entry(pfc, &pcs_client_list, list) {

[Severity: Medium]
Can this walk miss endpoints that still have live RDMA connections? The
commit message says "All endpoints are reached through pcs_client_list and
the engine hash". Consider the teardown path in kpcs_conn_fini():

    WRITE_ONCE(fc->kio.ctx, NULL);
    unregister_client(pfc);
    synchronize_rcu();
    flush_workqueue(pcs_wq);
    flush_workqueue(pcs_cpu_wq);
    flush_workqueue(pcs_cleanup_wq);
    pcs_cluster_fini(pfc);

unregister_client() removes the pfc from pcs_client_list. The endpoints and
their rios are only closed and destroyed afterwards, in pcs_cluster_fini().
Their rio->id.event_handler can still be an old RK copy and can still
receive CM events. For example, rio_abort() calls rdma_disconnect() during
this teardown.

If rk_fixup_rdma() runs inside that window, rk_collect_eps() never sees the
cluster. Those connections keep their old-RK handlers, and the outgoing RK
can then be unloaded while CM, QP or CQ events are still being delivered.
Is there something that covers clusters that have left pcs_client_list but
whose endpoints have not been destroyed yet?

[ ... ]

> +static unsigned int rk_count_eps(void)
> +{
> +	struct pcs_fuse_cluster *pfc;
> +	unsigned int n = 0;
> +
> +	spin_lock(&pcs_clients_lock);

[Severity: Low]
Is pcs_clients_lock initialized when this callback can run? The lock is
declared as a plain spinlock_t and only initialized by spin_lock_init() in
kpcs_mod_init().

klp_module_coming() can call klp_post_patch_callback() while the module is
still MODULE_STATE_COMING, before the init function runs. This happens if
an RK that patches fuse_kio_pcs is already enabled when fuse_kio_pcs is
loaded.

In that case rk_count_eps() and rk_collect_eps() take an uninitialized lock.
On CONFIG_DEBUG_SPINLOCK or lockdep kernels this would produce a splat. The
callback has nothing to fix up in that scenario anyway.

Would DEFINE_SPINLOCK() for pcs_clients_lock, or skipping the fixup when
obj->mod->state is not MODULE_STATE_LIVE, be appropriate?

[ ... ]

> +			mutex_lock(&eps[i]->mutex);
> +			pcs_rdma_rk_fixup_ep(eps[i], obj->mod->name);

[Severity: Low]
This isn't a bug, but pcs_rdma_rk_fixup_ep() does a full scan of the
device-wide MR restrack table for every endpoint. The scan runs under the
device-wide xa_lock. The total cost is endpoints times device MRs. That
includes MRs owned by other users of the HCA. The same xa_lock is taken by
rdma_restrack_add() and rdma_restrack_del(), so concurrent MR allocations
can stall behind it.

Would it be better to collect the PDs first and do a single pass over the MR
table?

[ ... ]

> diff --git a/fs/fuse/kio/pcs/pcs_rdma_io.c b/fs/fuse/kio/pcs/pcs_rdma_io.c
> --- a/fs/fuse/kio/pcs/pcs_rdma_io.c
> +++ b/fs/fuse/kio/pcs/pcs_rdma_io.c

[ ... ]

> +#include "../../../../drivers/infiniband/core/restrack.h"
> +
> +void pcs_rdma_rk_fixup_ep(struct pcs_rpc *ep, const char *name)
> +{

[ ... ]

> +	xa = &pd->device->res[RDMA_RESTRACK_MR].xa;
> +	xa_lock(xa);
> +	xa_for_each(xa, index, res) {
> +		if (container_of(res, struct ib_mr, res)->pd == pd)
> +			WRITE_ONCE(res->kern_name, name);
> +	}
> +	xa_unlock(xa);
> +}

See the comment above on rescanning the whole MR table per endpoint.

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/cover.1792700354.git.den@openvz.org?part=2

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel