All Virtuozzo development lists (kernel + QEMU)
 help / color / mirror / Atom feed
* [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs
@ 2026-11-13 15:08 Pavel Tikhomirov
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE Pavel Tikhomirov
                   ` (2 more replies)
  0 siblings, 3 replies; 11+ messages in thread
From: Pavel Tikhomirov @ 2026-11-13 15:08 UTC (permalink / raw)
  To: Konstantin Khorenko; +Cc: devel

A CT with an AIO context fails to restore on a node with more possible
CPUs: io_setup() makes a bigger ring there, and CRIU needs the ring of
the dumped size. Skip the CPU heuristic of the ring size while the VE
is pseudosuper, and add a selftest for it.

https://virtuozzo.atlassian.net/browse/VSTOR-154129

---
Pavel Tikhomirov (2):
      fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE
      selftests/ve: check the AIO ring size in a pseudosuper VE

 fs/aio.c                                      |   7 +-
 tools/testing/selftests/ve/.gitignore         |   1 +
 tools/testing/selftests/ve/Makefile           |   1 +
 tools/testing/selftests/ve/ve_aio_ring_test.c | 214 ++++++++++++++++++++++++++
 4 files changed, 222 insertions(+), 1 deletion(-)
---
base-commit: 3874c7f41089f93b796592c13b410ded6c987ec8
change-id: 20261113-pr-ptikhomirov-vstor-154129-aio-pseudosuper-ring-ae2133423a88

Best regards,
--  
Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE
  2026-11-13 15:08 [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Pavel Tikhomirov
@ 2026-11-13 15:08 ` Pavel Tikhomirov
  2026-11-13 15:14   ` no-reply
  2026-11-13 19:41   ` Konstantin Khorenko
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring " Pavel Tikhomirov
  2026-11-13 19:41 ` [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Konstantin Khorenko
  2 siblings, 2 replies; 11+ messages in thread
From: Pavel Tikhomirov @ 2026-11-13 15:08 UTC (permalink / raw)
  To: Konstantin Khorenko; +Cc: devel

io_setup() makes the ring at least 4 events per possible CPU of the
node. The ring is mapped into the task memory, and CRIU restores it with
io_setup() and moves the new mapping to the address of the old one, so
it needs a ring of the same size. On a node with more possible CPUs it
gets a bigger ring, and the restore fails. E.g. io_setup(1) makes a
ring of one page, 127 events, on 4 CPUs, and of two pages, 255 events,
on 16 CPUs.

Skip the CPU heuristic while the VE is pseudosuper, which it is only for
the restore. Keep the other adjustments, the doubling and the rounding
up to pages: they don't depend on the node, and the queue depth CRIU
estimates from the dumped ring already offsets them, so the restore gets
the ring of the dumped size.

After a migration to a node with more possible CPUs an AIO context can
be slightly less reliable: io_submit() can return EAGAIN before the ring
is full, while free slots sit in the per-CPU caches of other CPUs.

https://virtuozzo.atlassian.net/browse/VSTOR-154129

Feature: ve: CRIU support
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 fs/aio.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/fs/aio.c b/fs/aio.c
index 901cb622fe2c4..6e303e6612b6c 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -761,8 +761,13 @@ static struct kioctx *ioctx_alloc(unsigned nr_events)
 	 * and unavailable, double nr_events so userspace sees what they
 	 * expected: additionally, we move req_batch slots to/from percpu
 	 * counters at a time, so make sure that isn't 0:
+	 *
+	 * CRIU restores a ring with io_setup() and needs the size it had on
+	 * dump, which can be below the minimum of this node due to the CPU
+	 * heuristic below, so skip the heuristic on restore.
 	 */
-	nr_events = max(nr_events, num_possible_cpus() * 4);
+	if (ve_is_super(ve) || !ve->is_pseudosuper)
+		nr_events = max(nr_events, num_possible_cpus() * 4);
 	nr_events *= 2;
 
 	/* Prevent overflows */

-- 
2.56.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring size in a pseudosuper VE
  2026-11-13 15:08 [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Pavel Tikhomirov
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE Pavel Tikhomirov
@ 2026-11-13 15:08 ` Pavel Tikhomirov
  2026-11-13 15:15   ` no-reply
  2026-11-13 19:41   ` Konstantin Khorenko
  2026-11-13 19:41 ` [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Konstantin Khorenko
  2 siblings, 2 replies; 11+ messages in thread
From: Pavel Tikhomirov @ 2026-11-13 15:08 UTC (permalink / raw)
  To: Konstantin Khorenko; +Cc: devel

Check that the host and a VE keep the CPU heuristic of the io_setup()
ring and a pseudosuper VE skips it, and that io_setup((nr - 2) / 2)
there gives back a ring of nr events for rings of 1 to 4 pages.

The rings differ only when the CPU heuristic makes a one-page ring
bigger, e.g. with 16 possible CPUs and 4K pages, else the test is
skipped. Before the fix the pseudosuper tests fail on 16 CPUs with a
ring of 255 events instead of 127.

https://virtuozzo.atlassian.net/browse/VSTOR-154129

Feature: ve: CRIU support
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
---
 tools/testing/selftests/ve/.gitignore         |   1 +
 tools/testing/selftests/ve/Makefile           |   1 +
 tools/testing/selftests/ve/ve_aio_ring_test.c | 214 ++++++++++++++++++++++++++
 3 files changed, 216 insertions(+)

diff --git a/tools/testing/selftests/ve/.gitignore b/tools/testing/selftests/ve/.gitignore
index 3d1179bd2c5f4..2ff027048a6ae 100644
--- a/tools/testing/selftests/ve/.gitignore
+++ b/tools/testing/selftests/ve/.gitignore
@@ -4,3 +4,4 @@ ve_mount_accounting_test
 ve_cpuid_override_test
 ve_vtty_stress
 ve_numa_migrate_test
+ve_aio_ring_test
diff --git a/tools/testing/selftests/ve/Makefile b/tools/testing/selftests/ve/Makefile
index e96320a183844..124b4a918016c 100644
--- a/tools/testing/selftests/ve/Makefile
+++ b/tools/testing/selftests/ve/Makefile
@@ -7,6 +7,7 @@ TEST_GEN_PROGS += ve_perms_test
 TEST_GEN_PROGS += ve_mount_accounting_test
 TEST_GEN_PROGS += ve_vtty_stress
 TEST_GEN_PROGS += ve_numa_migrate_test
+TEST_GEN_PROGS += ve_aio_ring_test
 
 # CPUID faulting is x86-64 only. ARCH says "x86" for 32-bit too and lib.mk
 # picks the clang target by its emptiness, so decide by the compiler.
diff --git a/tools/testing/selftests/ve/ve_aio_ring_test.c b/tools/testing/selftests/ve/ve_aio_ring_test.c
new file mode 100644
index 0000000000000..c370941574224
--- /dev/null
+++ b/tools/testing/selftests/ve/ve_aio_ring_test.c
@@ -0,0 +1,214 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ve_aio_ring selftest
+ *
+ * io_setup() makes the ring at least 4 events per possible CPU. CRIU
+ * restores a ring with io_setup() and needs the size it had on dump, so
+ * a VE in pseudosuper mode (restore) gets the ring of exactly the size
+ * asked for. The tests check that:
+ *
+ *   - the host keeps the CPU heuristic
+ *   - a VE keeps the CPU heuristic
+ *   - a VE in pseudosuper mode skips the CPU heuristic
+ *   - a VE in pseudosuper mode gets a ring of nr events from
+ *     io_setup((nr - 2) / 2), for rings of 1 to 4 pages
+ *
+ * The rings differ only when the CPU heuristic makes a small ring bigger,
+ * e.g. with 16 possible CPUs and 4K pages, else the test is skipped.
+ */
+#define _GNU_SOURCE
+#include <sched.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+#include <errno.h>
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <linux/aio_abi.h>
+
+#include "../kselftest_harness.h"
+#include "ve_selftest.h"
+
+/* The ring header mapped by io_setup(), a copy of the one in fs/aio.c. */
+struct aio_ring {
+	unsigned int id;
+	unsigned int nr;	/* ring capacity in io_events */
+	unsigned int head;
+	unsigned int tail;
+	unsigned int magic;
+	unsigned int compat_features;
+	unsigned int incompat_features;
+	unsigned int header_length;
+	struct io_event io_events[];
+};
+
+#define PAGE_ALIGN(size, ps)	(((size) + ((ps) - 1)) & ~((ps) - 1))
+
+/* The biggest ring, in pages, the pseudosuper restore test covers. */
+#define RING_PAGES_MAX		4
+
+struct aio_ring_arg {
+	unsigned int nr_events;	/* in: io_setup() nr_events */
+	unsigned int ring_nr;	/* out: ring capacity, struct aio_ring::nr */
+};
+
+FIXTURE(ve_aio_ring)
+{
+	int cgv2_fd;
+	int ctid;
+	struct namespace_ctx ns_ctx;
+	unsigned int cpus;
+	unsigned long page_size;
+	struct aio_ring_arg *arg;	/* shared with the child in the VE */
+};
+
+/* The ring capacity aio_setup_ring() makes for @nr_events. */
+static unsigned int calculate_ring_nr(FIXTURE_DATA(ve_aio_ring) * self,
+				      unsigned int nr_events)
+{
+	unsigned long size;
+
+	size = sizeof(struct aio_ring) + (nr_events + 2) * sizeof(struct io_event);
+	size = PAGE_ALIGN(size, self->page_size);
+	return (size - sizeof(struct aio_ring)) / sizeof(struct io_event);
+}
+
+/* The ring capacity io_setup(@nr_events) makes. */
+static unsigned int ring_nr(FIXTURE_DATA(ve_aio_ring) * self,
+			    unsigned int nr_events)
+{
+	unsigned int cpu_events = self->cpus * 4;
+
+	if (nr_events < cpu_events)
+		nr_events = cpu_events;
+	return calculate_ring_nr(self, 2 * nr_events);
+}
+
+/* The ring capacity io_setup(@nr_events) makes without the CPU heuristic. */
+static unsigned int ring_nr_pseudosuper(FIXTURE_DATA(ve_aio_ring) * self,
+					unsigned int nr_events)
+{
+	return calculate_ring_nr(self, 2 * nr_events);
+}
+
+/* Create an AIO context of arg->nr_events, store its ring capacity. */
+static int aio_ring_nr(void *data)
+{
+	struct aio_ring_arg *arg = data;
+	aio_context_t ctx = 0;
+
+	if (syscall(__NR_io_setup, arg->nr_events, &ctx)) {
+		fprintf(stderr, "%s: io_setup(%u): %s\n", __func__,
+			arg->nr_events, strerror(errno));
+		return -1;
+	}
+	arg->ring_nr = ((struct aio_ring *)ctx)->nr;
+
+	if (syscall(__NR_io_destroy, ctx)) {
+		fprintf(stderr, "%s: io_destroy: %s\n", __func__,
+			strerror(errno));
+		return -1;
+	}
+	return 0;
+}
+
+/* The ring capacity io_setup(@nr_events) makes in the VE, or 0. */
+static unsigned int ve_ring_nr(FIXTURE_DATA(ve_aio_ring) * self,
+			       unsigned int nr_events)
+{
+	self->arg->nr_events = nr_events;
+	self->arg->ring_nr = 0;
+	if (run_in_ve(self->cgv2_fd, self->ctid, &self->ns_ctx, CLONE_NEWVE,
+		      aio_ring_nr, self->arg))
+		return 0;
+	return self->arg->ring_nr;
+}
+
+static int set_pseudosuper(FIXTURE_DATA(ve_aio_ring) * self, int value)
+{
+	char path[64];
+
+	snprintf(path, sizeof(path), "%d/ve.pseudosuper", self->ctid);
+	return write_u64_at(self->cgv2_fd, path, value);
+}
+
+FIXTURE_SETUP(ve_aio_ring)
+{
+	long cpus;
+
+	self->ctid = -1;
+	self->arg = MAP_FAILED;
+
+	self->page_size = sysconf(_SC_PAGESIZE);
+	cpus = sysconf(_SC_NPROCESSORS_CONF);
+	ASSERT_GT(cpus, 0);
+	self->cpus = cpus;
+	if (ring_nr(self, 1) == ring_nr_pseudosuper(self, 1))
+		SKIP(return, "%u possible CPUs are too few for the CPU heuristic to matter",
+		     self->cpus);
+
+	self->arg = mmap(NULL, sizeof(*self->arg), PROT_READ | PROT_WRITE,
+			 MAP_SHARED | MAP_ANONYMOUS, -1, 0);
+	ASSERT_NE(self->arg, MAP_FAILED);
+
+	self->cgv2_fd = mount_cg2_fd();
+	ASSERT_GE(self->cgv2_fd, 0);
+
+	ASSERT_EQ(write_file_at(self->cgv2_fd, "cgroup.subtree_control",
+				VE_CONTROLLERS), 0);
+
+	self->ctid = make_ve(self->cgv2_fd, CTID_MIN);
+	ASSERT_GE(self->ctid, 0);
+
+	ASSERT_EQ(prepare_namespaces(&self->ns_ctx, CLONE_NEWVE,
+				     self->cgv2_fd, self->ctid), 0);
+}
+
+FIXTURE_TEARDOWN(ve_aio_ring)
+{
+	if (self->ctid >= 0) {
+		set_pseudosuper(self, 0);
+		close_namespace_fds(&self->ns_ctx);
+		destroy_ve(self->cgv2_fd, self->ctid);
+		close(self->cgv2_fd);
+	}
+	if (self->arg != MAP_FAILED)
+		munmap(self->arg, sizeof(*self->arg));
+}
+
+TEST_F(ve_aio_ring, host_keeps_cpu_heuristic)
+{
+	self->arg->nr_events = 1;
+	EXPECT_EQ(aio_ring_nr(self->arg), 0);
+	EXPECT_EQ(self->arg->ring_nr, ring_nr(self, 1));
+}
+
+TEST_F(ve_aio_ring, ve_keeps_cpu_heuristic)
+{
+	EXPECT_EQ(ve_ring_nr(self, 1), ring_nr(self, 1));
+}
+
+TEST_F(ve_aio_ring, pseudosuper_ve_skips_cpu_heuristic)
+{
+	EXPECT_EQ(set_pseudosuper(self, 1), 0);
+	EXPECT_EQ(ve_ring_nr(self, 1), ring_nr_pseudosuper(self, 1));
+}
+
+/*
+ * A ring of nr events is restored with io_setup((nr - 2) / 2), check
+ * it gives back a ring of nr events.
+ */
+TEST_F(ve_aio_ring, pseudosuper_ve_restores_ring_size)
+{
+	unsigned int pages, nr;
+
+	EXPECT_EQ(set_pseudosuper(self, 1), 0);
+	for (pages = 1; pages <= RING_PAGES_MAX; pages++) {
+		nr = (pages * self->page_size - sizeof(struct aio_ring)) /
+			sizeof(struct io_event);
+		EXPECT_EQ(ve_ring_nr(self, (nr - 2) / 2), nr)
+			TH_LOG("ring of %u pages", pages);
+	}
+}
+
+TEST_HARNESS_MAIN

-- 
2.56.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE Pavel Tikhomirov
@ 2026-11-13 15:14   ` no-reply
  2026-11-13 19:41   ` Konstantin Khorenko
  1 sibling, 0 replies; 11+ messages in thread
From: no-reply @ 2026-11-13 15:14 UTC (permalink / raw)
  To: Pavel Tikhomirov, Konstantin Khorenko; +Cc: devel

> io_setup() makes the ring at least 4 events per possible CPU of the
> node. The ring is mapped into the task memory, and CRIU restores it with
> io_setup() and moves the new mapping to the address of the old one, so
> it needs a ring of the same size. On a node with more possible CPUs it
> gets a bigger ring, and the restore fails. E.g. io_setup(1) makes a
> ring of one page, 127 events, on 4 CPUs, and of two pages, 255 events,
> on 16 CPUs.
> 
> Skip the CPU heuristic while the VE is pseudosuper, which it is only for
> the restore. Keep the other adjustments, the doubling and the rounding
> up to pages: they don't depend on the node, and the queue depth CRIU
> estimates from the dumped ring already offsets them, so the restore gets
> the ring of the dumped size.
> 
> After a migration to a node with more possible CPUs an AIO context can
> be slightly less reliable: io_submit() can return EAGAIN before the ring
> is full, while free slots sit in the per-CPU caches of other CPUs.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-154129
> 
> Feature: ve: CRIU support
> Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113-pr-ptikhomirov-vstor-154129-aio-pseudosuper-ring-v1-0-7d35d143fdc3@virtuozzo.com?part=1


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring size in a pseudosuper VE
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring " Pavel Tikhomirov
@ 2026-11-13 15:15   ` no-reply
  2026-11-13 19:41   ` Konstantin Khorenko
  1 sibling, 0 replies; 11+ messages in thread
From: no-reply @ 2026-11-13 15:15 UTC (permalink / raw)
  To: Pavel Tikhomirov, Konstantin Khorenko; +Cc: devel

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [Low] selftests/ve: cgroup2 mount fd leaked on early FIXTURE_SETUP
  failure
- [Low] selftests/ve: VE cgroup left behind when FIXTURE_SETUP fails
  after make_ve()
- [Low] selftests/ve: test models possible CPUs with
  sysconf(_SC_NPROCESSORS_CONF)
--

commit 5c946723cd7135b742e593bb7b560ea1869bd370
Author: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>

selftests/ve: check the AIO ring size in a pseudosuper VE

This adds a selftest, ve_aio_ring_test, that checks the io_setup() ring size on the host, in a VE, and in a pseudosuper VE. It checks that the host and a VE keep the CPU heuristic, that a pseudosuper VE skips it, and that io_setup((nr - 2) / 2) in a pseudosuper VE gives a ring of nr events for rings of 1 to 4 pages.

> diff --git a/tools/testing/selftests/ve/ve_aio_ring_test.c b/tools/testing/selftests/ve/ve_aio_ring_test.c
> new file mode 100644
> index 000000000000..c37094157422
> --- /dev/null
> +++ b/tools/testing/selftests/ve/ve_aio_ring_test.c

[ ... ]

> +FIXTURE_SETUP(ve_aio_ring)
> +{
> +	long cpus;
> +
> +	self->ctid = -1;
> +	self->arg = MAP_FAILED;
> +
> +	self->page_size = sysconf(_SC_PAGESIZE);
> +	cpus = sysconf(_SC_NPROCESSORS_CONF);
> +	ASSERT_GT(cpus, 0);
> +	self->cpus = cpus;

[Severity: Low]
Does sysconf(_SC_NPROCESSORS_CONF) really give the number of possible CPUs that ring_nr() needs? The comments and the skip message call it "possible CPUs", and ioctx_alloc() uses num_possible_cpus().

As far as I can tell, glibc derives this value from the cpuN entries under /sys/devices/system/cpu. Those are registered only for present CPUs (see cpu_dev_register_generic()). On a VM or host with possible-but-not-present CPUs, such as hot-pluggable vCPUs or possible_cpus= on the command line, the count can be lower than num_possible_cpus(). The expected ring sizes then no longer match the kernel's, so the test fails spuriously, or the skip check wrongly skips the pseudosuper checks.

Would parsing /sys/devices/system/cpu/possible be more reliable?

> +	if (ring_nr(self, 1) == ring_nr_pseudosuper(self, 1))
> +		SKIP(return, "%u possible CPUs are too few for the CPU heuristic to matter",
> +		     self->cpus);
> +
> +	self->arg = mmap(NULL, sizeof(*self->arg), PROT_READ | PROT_WRITE,
> +			 MAP_SHARED | MAP_ANONYMOUS, -1, 0);
> +	ASSERT_NE(self->arg, MAP_FAILED);
> +
> +	self->cgv2_fd = mount_cg2_fd();
> +	ASSERT_GE(self->cgv2_fd, 0);
> +
> +	ASSERT_EQ(write_file_at(self->cgv2_fd, "cgroup.subtree_control",
> +				VE_CONTROLLERS), 0);
> +
> +	self->ctid = make_ve(self->cgv2_fd, CTID_MIN);
> +	ASSERT_GE(self->ctid, 0);

[Severity: Low]
Does this leak the cgroup2 fd when one of these ASSERTs fails? If write_file_at() or make_ve() fails after mount_cg2_fd() succeeded, ctid is still -1. FIXTURE_TEARDOWN() then skips close(self->cgv2_fd), because the close sits under "if (self->ctid >= 0)". The harness also does not run teardown at all when setup does not complete, so the fd stays open either way. The impact is small, since the test process exits soon afterwards.

> +
> +	ASSERT_EQ(prepare_namespaces(&self->ns_ctx, CLONE_NEWVE,
> +				     self->cgv2_fd, self->ctid), 0);
> +}

[Severity: Low]
Is the VE left behind if prepare_namespaces() fails? make_ve() has already created the VE cgroup by then. The harness runs FIXTURE_TEARDOWN() only if setup completed, so destroy_ve() never runs. The CTID then stays in the cgroup2 hierarchy, and later runs find it already taken.

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261113-pr-ptikhomirov-vstor-154129-aio-pseudosuper-ring-v1-0-7d35d143fdc3@virtuozzo.com?part=2

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE Pavel Tikhomirov
  2026-11-13 15:14   ` no-reply
@ 2026-11-13 19:41   ` Konstantin Khorenko
  1 sibling, 0 replies; 11+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:41 UTC (permalink / raw)
  To: Pavel Tikhomirov; +Cc: devel

I won't apply this version, please send v2 with the changes below.

On Fri, 13 Nov 2026 16:08:44 +0100, Pavel Tikhomirov wrote:
> After a migration to a node with more possible CPUs an AIO context can
> be slightly less reliable: io_submit() can return EAGAIN before the ring
> is full, while free slots sit in the per-CPU caches of other CPUs.

Item 1 of the cover letter reply: keep a restored small ring out of
the per-CPU slot caches, or state the real bound here.

Why: the doubling relies on the heuristic: with at least 8 events per
possible CPU, the slots cached on other CPUs stay below half the ring.
A restored ring has no such floor. req_batch drops to 1, and
put_reqs_available() leaves one slot on every CPU a completion runs on,
which only a submit from that CPU takes back.

A one-page ring dumped on 4 CPUs comes from io_setup() of up to 62
events and has 126 usable slots. Restored on a big node,
io_submit() returns EAGAIN with fewer requests in flight than the
application asked for once completions have run on more than 64 other
CPUs. Once 126 CPUs hold a slot, it returns EAGAIN with nothing in
flight, and an application that then waits for a completion hangs.
Before the patch this migration failed cleanly and the CT stayed on
the source node.

> Skip the CPU heuristic while the VE is pseudosuper, which it is only for
> the restore. Keep the other adjustments, the doubling and the rounding

Item 2 of the cover letter reply: the VE is also pseudosuper during
every CT start.

Why: libvzctl do_env_create() sets ve.pseudosuper on every start, and
the CT's first task clears it just before it execs init. vzctl runs no
AIO in that window, so the code is fine, but the message and the
comment ("skip the heuristic on restore") say otherwise.

> gets a bigger ring, and the restore fails. E.g. io_setup(1) makes a

Item 3 of the cover letter reply: say what the user sees.

Why: CRIU's restore_aio_ring() compares the new ring with the dumped
one, reports "wrong aio" and the migration is aborted.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring size in a pseudosuper VE
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring " Pavel Tikhomirov
  2026-11-13 15:15   ` no-reply
@ 2026-11-13 19:41   ` Konstantin Khorenko
  1 sibling, 0 replies; 11+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:41 UTC (permalink / raw)
  To: Pavel Tikhomirov; +Cc: devel

I won't apply this version, please send v2 with the changes below.

On Fri, 13 Nov 2026 16:08:45 +0100, Pavel Tikhomirov wrote:
> +	cpus = sysconf(_SC_NPROCESSORS_CONF);

Item 4 of the cover letter reply: read
/sys/devices/system/cpu/possible here.

Why: the kernel sizes the ring by cpu_possible_mask, and what
_SC_NPROCESSORS_CONF returns depends on the libc. On a VM with CPU
hotplug slots the two can differ, and the test then skips when it
should run, or expects the wrong ring size.

> +	self->ctid = make_ve(self->cgv2_fd, CTID_MIN);
> +	ASSERT_GE(self->ctid, 0);
> +
> +	ASSERT_EQ(prepare_namespaces(&self->ns_ctx, CLONE_NEWVE,

Item 5 of the cover letter reply: clean up the VE cgroup when setup
fails after make_ve().

Why: the harness skips FIXTURE_TEARDOWN when setup fails, so a failed
prepare_namespaces() leaves the VE cgroup in the host hierarchy.

> +	EXPECT_EQ(set_pseudosuper(self, 1), 0);
> +	EXPECT_EQ(ve_ring_nr(self, 1), ring_nr_pseudosuper(self, 1));

Item 5 of the cover letter reply: ASSERT_EQ here.

Why: if the write fails, the test goes on and reports a ring size
mismatch, which hides the cause.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs
  2026-11-13 15:08 [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Pavel Tikhomirov
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE Pavel Tikhomirov
  2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring " Pavel Tikhomirov
@ 2026-11-13 19:41 ` Konstantin Khorenko
  2026-11-16 10:33   ` Pavel Tikhomirov
  2026-11-16 10:35   ` Pavel Tikhomirov
  2 siblings, 2 replies; 11+ messages in thread
From: Konstantin Khorenko @ 2026-11-13 19:41 UTC (permalink / raw)
  To: Pavel Tikhomirov; +Cc: devel

I won't apply this version, please send v2 with the changes below.

On Fri, 13 Nov 2026 16:08:43 +0100, Pavel Tikhomirov wrote:
> Pavel Tikhomirov (2):
>       fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE
>       selftests/ve: check the AIO ring size in a pseudosuper VE

The ring sizing is right: with the CPU heuristic skipped, CRIU's
(nr - 2) / 2 gets back the dumped ring for any page count and CPU
counts, and the host, running CTs and restores to a node with fewer
CPUs are unchanged. What to change for v2:

1. Patch 1, code: a restored ring below this node's minimum must not
   lose slots in the per-CPU caches. E.g. mark such a context in
   ioctx_alloc() and let put_reqs_available() and
   __get_reqs_available() use ctx->reqs_available directly for it.
   If you keep the cache, the commit message must state the real
   bound instead of "slightly less reliable".

2. Patch 1, commit message and comment: the VE is pseudosuper not only
   for the restore. vzctl sets ve.pseudosuper on every CT start and
   clears it just before it execs init. Say "while the CT is being
   started or restored", or similar.

3. Patch 1, commit message: say what fails without the patch: CRIU's
   restore_aio_ring() reports "wrong aio" and the migration is
   aborted.

4. Patch 2: read the CPU count from /sys/devices/system/cpu/possible,
   the mask the kernel uses, not sysconf(_SC_NPROCESSORS_CONF).

5. Patch 2: do not leave the VE cgroup behind when FIXTURE_SETUP fails
   after make_ve(), and use ASSERT_EQ for set_pseudosuper().

Separate patches, not needed for v2 (I created a task):

1. vz misses mainline b44341e39d856 ("aio: avoid request count
   overflow in ioctx_alloc()"), which touches the lines next to
   patch 1.
   https://virtuozzo.atlassian.net/browse/VSTOR-155110

Details are in the replies to the patches.

-- 
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs
  2026-11-13 19:41 ` [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Konstantin Khorenko
@ 2026-11-16 10:33   ` Pavel Tikhomirov
  2026-11-16 10:35     ` Pavel Tikhomirov
  2026-11-16 10:35   ` Pavel Tikhomirov
  1 sibling, 1 reply; 11+ messages in thread
From: Pavel Tikhomirov @ 2026-11-16 10:33 UTC (permalink / raw)
  To: Konstantin Khorenko; +Cc: devel

On 11/13/26 20:41, Konstantin Khorenko wrote:
> 1. Patch 1, code: a restored ring below this node's minimum must not
>    lose slots in the per-CPU caches. E.g. mark such a context in
>    ioctx_alloc() and let put_reqs_available() and
>    __get_reqs_available() use ctx->reqs_available directly for it.
>    If you keep the cache, the commit message must state the real
>    bound instead of "slightly less reliable".

Done as you suggest: ioctx_alloc() sets ctx->no_percpu_reqs when it
skips the heuristic and the heuristic would have made the ring bigger,
and put_reqs_available() and __get_reqs_available() then use only
ctx->reqs_available. All slots of such a ring stay usable, as on the
source node. struct kioctx is private to fs/aio.c, and the bool takes
the hole after req_batch, so its size doesn't change.

> 2. Patch 1, commit message and comment: the VE is pseudosuper not only
>    for the restore. vzctl sets ve.pseudosuper on every CT start and
>    clears it just before it execs init. Say "while the CT is being
>    started or restored", or similar.

Changed to "which it is almost only for the restore". No AIO runs in
the CT start window, so the skip matters only for the restore, and I'd
rather not spell out the start details there.

> 3. Patch 1, commit message: say what fails without the patch: CRIU's
>    restore_aio_ring() reports "wrong aio" and the migration is
>    aborted.

Not changed: the message already says that CRIU needs a ring of the same
size and that with a bigger ring the restore fails.

> 4. Patch 2: read the CPU count from /sys/devices/system/cpu/possible,
>    the mask the kernel uses, not sysconf(_SC_NPROCESSORS_CONF).

Not changed: on vz10 glibc (2.39) sysconf(_SC_NPROCESSORS_CONF) already
reads /sys/devices/system/cpu/possible (checked with strace), so it is
the same mask.

> 5. Patch 2: do not leave the VE cgroup behind when FIXTURE_SETUP fails
>    after make_ve(), and use ASSERT_EQ for set_pseudosuper().

Done, both.

-- 
Best regards, Pavel Tikhomirov
Senior Software Developer, Virtuozzo.

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs
  2026-11-16 10:33   ` Pavel Tikhomirov
@ 2026-11-16 10:35     ` Pavel Tikhomirov
  0 siblings, 0 replies; 11+ messages in thread
From: Pavel Tikhomirov @ 2026-11-16 10:35 UTC (permalink / raw)
  To: Konstantin Khorenko; +Cc: devel

please ignore, this is wrong version, copy/paste mistake on my part

On 11/16/26 11:33, Pavel Tikhomirov wrote:
> On 11/13/26 20:41, Konstantin Khorenko wrote:
>> 1. Patch 1, code: a restored ring below this node's minimum must not
>>    lose slots in the per-CPU caches. E.g. mark such a context in
>>    ioctx_alloc() and let put_reqs_available() and
>>    __get_reqs_available() use ctx->reqs_available directly for it.
>>    If you keep the cache, the commit message must state the real
>>    bound instead of "slightly less reliable".
> 
> Done as you suggest: ioctx_alloc() sets ctx->no_percpu_reqs when it
> skips the heuristic and the heuristic would have made the ring bigger,
> and put_reqs_available() and __get_reqs_available() then use only
> ctx->reqs_available. All slots of such a ring stay usable, as on the
> source node. struct kioctx is private to fs/aio.c, and the bool takes
> the hole after req_batch, so its size doesn't change.
> 
>> 2. Patch 1, commit message and comment: the VE is pseudosuper not only
>>    for the restore. vzctl sets ve.pseudosuper on every CT start and
>>    clears it just before it execs init. Say "while the CT is being
>>    started or restored", or similar.
> 
> Changed to "which it is almost only for the restore". No AIO runs in
> the CT start window, so the skip matters only for the restore, and I'd
> rather not spell out the start details there.
> 
>> 3. Patch 1, commit message: say what fails without the patch: CRIU's
>>    restore_aio_ring() reports "wrong aio" and the migration is
>>    aborted.
> 
> Not changed: the message already says that CRIU needs a ring of the same
> size and that with a bigger ring the restore fails.
> 
>> 4. Patch 2: read the CPU count from /sys/devices/system/cpu/possible,
>>    the mask the kernel uses, not sysconf(_SC_NPROCESSORS_CONF).
> 
> Not changed: on vz10 glibc (2.39) sysconf(_SC_NPROCESSORS_CONF) already
> reads /sys/devices/system/cpu/possible (checked with strace), so it is
> the same mask.
> 
>> 5. Patch 2: do not leave the VE cgroup behind when FIXTURE_SETUP fails
>>    after make_ve(), and use ASSERT_EQ for set_pseudosuper().
> 
> Done, both.
> 

-- 
Best regards, Pavel Tikhomirov
Senior Software Developer, Virtuozzo.

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs
  2026-11-13 19:41 ` [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Konstantin Khorenko
  2026-11-16 10:33   ` Pavel Tikhomirov
@ 2026-11-16 10:35   ` Pavel Tikhomirov
  1 sibling, 0 replies; 11+ messages in thread
From: Pavel Tikhomirov @ 2026-11-16 10:35 UTC (permalink / raw)
  To: Konstantin Khorenko; +Cc: devel

On 11/13/26 20:41, Konstantin Khorenko wrote:
> 1. Patch 1, code: a restored ring below this node's minimum must not
>    lose slots in the per-CPU caches. E.g. mark such a context in
>    ioctx_alloc() and let put_reqs_available() and
>    __get_reqs_available() use ctx->reqs_available directly for it.
>    If you keep the cache, the commit message must state the real
>    bound instead of "slightly less reliable".

Done as you suggest: ioctx_alloc() marks such a context with
ctx->no_percpu_reqs, and put_reqs_available() and
__get_reqs_available() then use ctx->reqs_available directly, so no
free slots get stuck in the per-CPU caches.

> 2. Patch 1, commit message and comment: the VE is pseudosuper not only
>    for the restore. vzctl sets ve.pseudosuper on every CT start and
>    clears it just before it execs init. Say "while the CT is being
>    started or restored", or similar.

Changed to "which it is almost only for the restore". No AIO runs in
the CT start window, so the skip matters only for the restore, and I'd
rather not spell out the start details, so the commit stays focused.

> 3. Patch 1, commit message: say what fails without the patch: CRIU's
>    restore_aio_ring() reports "wrong aio" and the migration is
>    aborted.

Not changed: the message already says that CRIU needs a ring of the same
size and that with a bigger ring the restore fails.

> 4. Patch 2: read the CPU count from /sys/devices/system/cpu/possible,
>    the mask the kernel uses, not sysconf(_SC_NPROCESSORS_CONF).

Not changed: on vz10 glibc (2.39) sysconf(_SC_NPROCESSORS_CONF) already
reads /sys/devices/system/cpu/possible (checked with strace), so it is
the same mask.

> 5. Patch 2: do not leave the VE cgroup behind when FIXTURE_SETUP fails
>    after make_ve(), and use ASSERT_EQ for set_pseudosuper().

Done, both.

-- 
Best regards, Pavel Tikhomirov
Senior Software Developer, Virtuozzo.

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2026-11-16 10:36 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-11-13 15:08 [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Pavel Tikhomirov
2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 1/2] fs/aio: restore the AIO ring of the dumped size in a pseudosuper VE Pavel Tikhomirov
2026-11-13 15:14   ` no-reply
2026-11-13 19:41   ` Konstantin Khorenko
2026-11-13 15:08 ` [Devel] [PATCH hci-8.1 2/2] selftests/ve: check the AIO ring " Pavel Tikhomirov
2026-11-13 15:15   ` no-reply
2026-11-13 19:41   ` Konstantin Khorenko
2026-11-13 19:41 ` [Devel] [PATCH hci-8.1 0/2] fs/aio: fix CT migration with AIO to a node with more CPUs Konstantin Khorenko
2026-11-16 10:33   ` Pavel Tikhomirov
2026-11-16 10:35     ` Pavel Tikhomirov
2026-11-16 10:35   ` Pavel Tikhomirov

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.