* [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT
@ 2026-10-29 10:16 Vladimir Riabchun
2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall Vladimir Riabchun
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Vladimir Riabchun @ 2026-10-29 10:16 UTC (permalink / raw)
To: Konstantin Khorenko, ptikhomirov; +Cc: devel
We observed weird nproc behavior - it reported 64 CPUs
no matter what the CT limits were or how many CPUs the host
actually had.
nproc code uses sched_getaffinity syscall under the hood[1].
It appears that this syscall returned complete nonsence when
called from VE - we ignored both num_online_vcpus and
cgroup cpuset limitations.
Now it is fixed and a new selftest is added.
[1]: https://github.com/coreutils/gnulib/blob/master/lib/nproc.c#L144
https://virtuozzo.atlassian.net/browse/VSTOR-141682
Signed-off-by: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com>
---
Changes in v3:
- Patch 1: num_online_vcpus() -> task_nr_cpus(p)
- Patch 2: updated a comment and moved CPU check in setup fixture to the top.
- Link to v2: http://lore.virtuozzo.com/20261028-ct-invalid-nrpoc-v2-0-968fe985ce27@virtuozzo.com
Changes in v2:
- Updated vCPU logic in patch 1. Now we don't return actual
cpumask, insted first min(vcpu, count(mask)) bits are set.
This is closer to the real CT resources.
- Patch 2: adjusted logic and added one more test.
- Link to v1: http://lore.virtuozzo.com/20261027-ct-invalid-nrpoc-v1-0-48727c3afe16@virtuozzo.com
---
Vladimir Riabchun (2):
sched: Fix sched_getaffinity syscall
selftests: ve: Test sched_getaffinity in VE
kernel/sched/syscalls.c | 18 +-
tools/testing/selftests/ve/.gitignore | 1 +
tools/testing/selftests/ve/Makefile | 1 +
.../selftests/ve/ve_sched_getaffinity_test.c | 211 +++++++++++++++++++++
4 files changed, 226 insertions(+), 5 deletions(-)
---
base-commit: b499efa17aa9fb8f20512d1831d1cb61b22e60d3
change-id: 20261027-ct-invalid-nrpoc-f183b3ba691d
Best regards,
--
Vladimir Riabchun <vladimir.riabchun@virtuozzo.com>
_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel
^ permalink raw reply [flat|nested] 10+ messages in thread* [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall 2026-10-29 10:16 [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT Vladimir Riabchun @ 2026-10-29 10:16 ` Vladimir Riabchun 2026-10-29 10:21 ` no-reply 2026-11-12 18:37 ` Konstantin Khorenko 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE Vladimir Riabchun 2026-11-12 18:37 ` [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT Konstantin Khorenko 2 siblings, 2 replies; 10+ messages in thread From: Vladimir Riabchun @ 2026-10-29 10:16 UTC (permalink / raw) To: Konstantin Khorenko, ptikhomirov; +Cc: devel We observed incorrect cpu numbers inside containers: [root@localhost ~]# nproc 64 even when there were much less physical/virtual CPUs. The root cause is incorrect bitmap_fill usage - it sets whole integers only, without handling bit tails, so the actual number of set bits is 64 = bits in ULONG. If we use cpumask only or vcpus only, nproc and other getaffinity users will get incorrect info about CT resources. We can't run on more than count(cpuset) CPUs and we are limited by vCPU number. So the minimum of these two is the real limit. Return first bits set - anyway they can't be used for anything except counting from inside CT. Host behavior is not changed. It is also intentional that we perform VE check against current exec env - CT users will get virtualized view and host users will get real affinity without vCPU info. https://virtuozzo.atlassian.net/browse/VSTOR-141682 Signed-off-by: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> --- kernel/sched/syscalls.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c index 6d921043e9e7..ce5200c84573 100644 --- a/kernel/sched/syscalls.c +++ b/kernel/sched/syscalls.c @@ -1316,15 +1316,23 @@ long sched_getaffinity(pid_t pid, struct cpumask *mask) if (retval) return retval; + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + cpumask_and(mask, &p->cpus_mask, cpu_active_mask); + } + if (!ve_is_super(get_exec_env())) { + /* + * setaffinity is not available inside CT, so these bits + * can only be used for counting. + * We can't have more than vcpus and we can't run on CPUs + * outside the mask. Reporting smallest will provide true + * information about CT resources. + */ + unsigned int vcpus_avail = min(task_nr_cpus(p), cpumask_weight(mask)); cpumask_clear(mask); - bitmap_fill(cpumask_bits(mask), num_online_vcpus()); - return 0; + bitmap_set(cpumask_bits(mask), 0, vcpus_avail); } - guard(raw_spinlock_irqsave)(&p->pi_lock); - cpumask_and(mask, &p->cpus_mask, cpu_active_mask); - return 0; } -- 2.52.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall Vladimir Riabchun @ 2026-10-29 10:21 ` no-reply 2026-10-29 10:48 ` Vladimir Riabchun 2026-11-12 18:37 ` Konstantin Khorenko 1 sibling, 1 reply; 10+ messages in thread From: no-reply @ 2026-10-29 10:21 UTC (permalink / raw) To: ptikhomirov, Vladimir Riabchun, Konstantin Khorenko; +Cc: devel Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Low] sched: sched_getaffinity() takes vCPU limit from target task, not caller - [Low] sched: sched_getaffinity() in CT can return an empty mask with no floor of 1 -- commit 89d21190c29ff2e196f8d22db77072e4bd2c2544 Author: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> sched: Fix sched_getaffinity syscall This patch changes sched_getaffinity() so that, inside a container, the reported mask is a prefix of set bits. The number of bits is the smaller of the task's vCPU limit and the weight of cpus_mask & cpu_active_mask. Host behavior is unchanged. Link: https://virtuozzo.atlassian.net/browse/VSTOR-141682 > diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c > --- a/kernel/sched/syscalls.c > +++ b/kernel/sched/syscalls.c [ ... ] > @@ -1316,15 +1316,23 @@ long sched_getaffinity(pid_t pid, struct cpumask *mask) > if (retval) > return retval; > > + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { > + cpumask_and(mask, &p->cpus_mask, cpu_active_mask); > + } > + > if (!ve_is_super(get_exec_env())) { > + /* > + * setaffinity is not available inside CT, so these bits > + * can only be used for counting. > + * We can't have more than vcpus and we can't run on CPUs > + * outside the mask. Reporting smallest will provide true > + * information about CT resources. > + */ > + unsigned int vcpus_avail = min(task_nr_cpus(p), cpumask_weight(mask)); [Severity: Low] Previously the vCPU limit came from num_online_vcpus(), which is task_nr_cpus(current), so it was based on the caller. Now it comes from task_nr_cpus(p), the target task, while the VE check still uses the caller's exec env. Can a task in a CT that calls sched_getaffinity(pid) for another task in a different task group now get a different CPU count than it gets for itself? For example, a task in a child cpu cgroup has nr_cpus of 0, which is clamped to num_online_cpus(). The CT root group may have a lower limit. Callers that query themselves, such as nproc, see no change. The other users of num_online_vcpus() (/proc/cpuinfo, sysfs, net-procfs) still use the caller's limit. Is this change intentional? The commit message only mentions that the VE check is done against the current exec env, and does not mention that the vCPU limit is now taken from the target task. > cpumask_clear(mask); > + bitmap_set(cpumask_bits(mask), 0, vcpus_avail); [Severity: Low] Can this return an empty mask inside a CT? vcpus_avail depends on cpumask_weight(mask), which is the weight of p->cpus_mask & cpu_active_mask. If that intersection is empty, vcpus_avail is 0 and bitmap_set(..., 0, 0) leaves the mask all zero. One way this can happen is during a CPU hotplug window. The task's cpus_mask may only hold CPUs that are already cleared from cpu_active_mask. select_fallback_rq() does not repair the mask until the task next wakes. The old CT path used num_online_vcpus(), which is task_nr_cpus(current). That function is clamped to at least num_online_cpus(), so it never returned an empty mask. In this case nproc and similar tools would report 0. Would a floor of 1 be appropriate here? > } > > - guard(raw_spinlock_irqsave)(&p->pi_lock); > - cpumask_and(mask, &p->cpus_mask, cpu_active_mask); > - > return 0; > } -- Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261029-ct-invalid-nrpoc-v3-0-89d4b0b7d520@virtuozzo.com?part=1 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall 2026-10-29 10:21 ` no-reply @ 2026-10-29 10:48 ` Vladimir Riabchun 0 siblings, 0 replies; 10+ messages in thread From: Vladimir Riabchun @ 2026-10-29 10:48 UTC (permalink / raw) To: ptikhomirov, Konstantin Khorenko; +Cc: devel On 10/29/26 11:21, no-reply@sashiko.virtuozzo.com wrote: > Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: > - [Low] sched: sched_getaffinity() takes vCPU limit from target task, not caller > - [Low] sched: sched_getaffinity() in CT can return an empty mask with no floor of 1 > -- > > commit 89d21190c29ff2e196f8d22db77072e4bd2c2544 > Author: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> > > sched: Fix sched_getaffinity syscall > > This patch changes sched_getaffinity() so that, inside a container, the > reported mask is a prefix of set bits. The number of bits is the smaller of > the task's vCPU limit and the weight of cpus_mask & cpu_active_mask. Host > behavior is unchanged. > > Link: https://virtuozzo.atlassian.net/browse/VSTOR-141682 > >> diff --git a/kernel/sched/syscalls.c b/kernel/sched/syscalls.c >> --- a/kernel/sched/syscalls.c >> +++ b/kernel/sched/syscalls.c > > [ ... ] > >> @@ -1316,15 +1316,23 @@ long sched_getaffinity(pid_t pid, struct cpumask *mask) >> if (retval) >> return retval; >> >> + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { >> + cpumask_and(mask, &p->cpus_mask, cpu_active_mask); >> + } >> + >> if (!ve_is_super(get_exec_env())) { >> + /* >> + * setaffinity is not available inside CT, so these bits >> + * can only be used for counting. >> + * We can't have more than vcpus and we can't run on CPUs >> + * outside the mask. Reporting smallest will provide true >> + * information about CT resources. >> + */ >> + unsigned int vcpus_avail = min(task_nr_cpus(p), cpumask_weight(mask)); > > [Severity: Low] > Previously the vCPU limit came from num_online_vcpus(), which is > task_nr_cpus(current), so it was based on the caller. Now it comes from > task_nr_cpus(p), the target task, while the VE check still uses the > caller's exec env. > > Can a task in a CT that calls sched_getaffinity(pid) for another task > in a different task group now get a different CPU count than it gets for > itself? > > For example, a task in a child cpu cgroup has nr_cpus of 0, which is > clamped to num_online_cpus(). The CT root group may have a lower limit. > > Callers that query themselves, such as nproc, see no change. The other > users of num_online_vcpus() (/proc/cpuinfo, sysfs, net-procfs) still use > the caller's limit. > > Is this change intentional? The commit message only mentions that the VE > check is done against the current exec env, and does not mention that the > vCPU limit is now taken from the target task. Yes, it is intentional. I think the current approach is correct. > >> cpumask_clear(mask); >> + bitmap_set(cpumask_bits(mask), 0, vcpus_avail); > > [Severity: Low] > Can this return an empty mask inside a CT? Already replied in v2. It is fine. > > vcpus_avail depends on cpumask_weight(mask), which is the weight of > p->cpus_mask & cpu_active_mask. If that intersection is empty, vcpus_avail > is 0 and bitmap_set(..., 0, 0) leaves the mask all zero. > > One way this can happen is during a CPU hotplug window. The task's > cpus_mask may only hold CPUs that are already cleared from > cpu_active_mask. select_fallback_rq() does not repair the mask until the > task next wakes. > > The old CT path used num_online_vcpus(), which is task_nr_cpus(current). > That function is clamped to at least num_online_cpus(), so it never > returned an empty mask. In this case nproc and similar tools would report > 0. > > Would a floor of 1 be appropriate here? > >> } >> >> - guard(raw_spinlock_irqsave)(&p->pi_lock); >> - cpumask_and(mask, &p->cpus_mask, cpu_active_mask); >> - >> return 0; >> } > -- Best regards, Riabchun Vladimir Linux Kernel Developer, Virtuozzo _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall Vladimir Riabchun 2026-10-29 10:21 ` no-reply @ 2026-11-12 18:37 ` Konstantin Khorenko 1 sibling, 0 replies; 10+ messages in thread From: Konstantin Khorenko @ 2026-11-12 18:37 UTC (permalink / raw) To: Vladimir Riabchun; +Cc: devel I won't apply this version, please send v4 with the changes below. On Thu, 29 Oct 2026 11:16:06 +0100, Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> wrote: > The root cause is incorrect bitmap_fill usage - it sets whole > integers only, without handling bit tails, so the actual number of > set bits is 64 = bits in ULONG. Items 1 and 2 of the cover letter reply: add the Fixes: and Feature: trailers, write "whole unsigned longs", and say what a CT sees now. Why: 87211cbff4b4 added the CT branch with bitmap_fill(). Since v3 the vCPU limit comes from the target task and the count follows the cpuset, so nproc may differ from /proc/cpuinfo. A reader of the message should not have to find that in the cover changelog. > + /* > + * setaffinity is not available inside CT, so these bits > + * can only be used for counting. Item 3: sched_setaffinity() returns 0 in a CT and does nothing, it is not "not available". -- Konstantin Khorenko, Virtuozzo Linux Kernel Team _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE 2026-10-29 10:16 [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT Vladimir Riabchun 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall Vladimir Riabchun @ 2026-10-29 10:16 ` Vladimir Riabchun 2026-10-29 10:22 ` no-reply 2026-11-12 18:37 ` Konstantin Khorenko 2026-11-12 18:37 ` [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT Konstantin Khorenko 2 siblings, 2 replies; 10+ messages in thread From: Vladimir Riabchun @ 2026-10-29 10:16 UTC (permalink / raw) To: Konstantin Khorenko, ptikhomirov; +Cc: devel Add a new selftest that verifies that sched_getaffinity respects both cgroup cpuset and nr_cpus limits. https://virtuozzo.atlassian.net/browse/VSTOR-141682 Signed-off-by: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> --- tools/testing/selftests/ve/.gitignore | 1 + tools/testing/selftests/ve/Makefile | 1 + .../selftests/ve/ve_sched_getaffinity_test.c | 211 +++++++++++++++++++++ 3 files changed, 213 insertions(+) diff --git a/tools/testing/selftests/ve/.gitignore b/tools/testing/selftests/ve/.gitignore index 8ab1bfe6f5e4..178de0ee1822 100644 --- a/tools/testing/selftests/ve/.gitignore +++ b/tools/testing/selftests/ve/.gitignore @@ -3,3 +3,4 @@ ve_perms_test ve_mount_accounting_test ve_cpuid_override_test ve_vtty_stress +ve_sched_getaffinity_test diff --git a/tools/testing/selftests/ve/Makefile b/tools/testing/selftests/ve/Makefile index 51f9316beb99..d53e724205f7 100644 --- a/tools/testing/selftests/ve/Makefile +++ b/tools/testing/selftests/ve/Makefile @@ -6,6 +6,7 @@ TEST_GEN_PROGS += ve_ns_owner_test TEST_GEN_PROGS += ve_perms_test TEST_GEN_PROGS += ve_mount_accounting_test TEST_GEN_PROGS += ve_vtty_stress +TEST_GEN_PROGS += ve_sched_getaffinity_test # CPUID faulting is x86-64 only. ARCH says "x86" for 32-bit too and lib.mk # picks the clang target by its emptiness, so decide by the compiler. diff --git a/tools/testing/selftests/ve/ve_sched_getaffinity_test.c b/tools/testing/selftests/ve/ve_sched_getaffinity_test.c new file mode 100644 index 000000000000..98acc246ede0 --- /dev/null +++ b/tools/testing/selftests/ve/ve_sched_getaffinity_test.c @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * ve_sched_getaffinity selftests + * + * Tests to check cpuset virtualization. + * Not suitable for running inside CT. + */ +#define _GNU_SOURCE +#include <asm/unistd.h> +#include <linux/sched.h> +#include <sys/syscall.h> +#include <sys/stat.h> +#include <errno.h> +#include <sched.h> +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> +#include <string.h> + +#include "../kselftest_harness.h" +#include "ve_selftest.h" + +FIXTURE(ve_getaff_test) +{ + int cgv2_fd; + int ctid; + struct namespace_ctx ns_ctx; + cpu_set_t orig_cpuset; + int orig_cpucount; +}; + +static int __getaffinity_ve(void *cpuset) +{ + cpu_set_t *dst = cpuset; + + return sched_getaffinity(getpid(), sizeof(*dst), dst); +} + +static int getaffinity_ve(FIXTURE_DATA(ve_getaff_test) * self, cpu_set_t *dst) +{ + int ret = -1; + cpu_set_t *tmp = mmap(NULL, sizeof(cpu_set_t), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + + if (tmp == MAP_FAILED) { + fprintf(stderr, "Failed to mmap shared region: %s\n", strerror(errno)); + return -1; + } + CPU_ZERO(tmp); + CPU_ZERO(dst); + + ret = run_in_ve(self->cgv2_fd, self->ctid, &self->ns_ctx, + CLONE_NEWVE, __getaffinity_ve, tmp); + + if (!ret) + memcpy(dst, tmp, sizeof(cpu_set_t)); + + munmap(tmp, sizeof(cpu_set_t)); + return ret; +} + +static int check_cpumask_is_prefix(cpu_set_t *mask, int expected) +{ + int i, count = CPU_COUNT(mask); + + if (expected >= 0 && expected != count) { + fprintf(stderr, "cpumask count mismatch: %d != %d\n", expected, count); + return 0; + } + for (i = 0; i < count; i++) { + if (!CPU_ISSET(i, mask)) { + fprintf(stderr, "cpumask is not a prefix: %d is OFF\n", i); + return 0; + } + } + return 1; +} + +static int get_nth_cpu(cpu_set_t *mask, int n) +{ + int i; + + for (i = 0; i < sizeof(*mask); i++) { + if (!CPU_ISSET(i, mask) || --n) + continue; + + return i; + } + return -1; +} + +FIXTURE_SETUP(ve_getaff_test) +{ + ASSERT_EQ(sched_getaffinity(getpid(), sizeof(cpu_set_t), &self->orig_cpuset), 0); + self->orig_cpucount = CPU_COUNT(&self->orig_cpuset); + if (self->orig_cpucount < 2) + SKIP(return, "Not enough CPUs available"); + + self->cgv2_fd = mount_cg2_fd(); + ASSERT_GE(self->cgv2_fd, 0); + + ASSERT_EQ(write_file_at(self->cgv2_fd, "cgroup.subtree_control", + VE_CONTROLLERS), 0); + + self->ctid = make_ve(self->cgv2_fd, CTID_MIN); + ASSERT_GE(self->ctid, 0); + + ASSERT_EQ(prepare_namespaces(&self->ns_ctx, CLONE_NEWVE, + self->cgv2_fd, self->ctid), 0); +} + +FIXTURE_TEARDOWN(ve_getaff_test) +{ + close_namespace_fds(&self->ns_ctx); + destroy_ve(self->cgv2_fd, self->ctid); + close(self->cgv2_fd); +} + +/* Test correct processing of cpuset.cpus updates */ +TEST_F(ve_getaff_test, update_cg_cpuset) +{ + char path[PATH_MAX]; + int cpu_leave; + cpu_set_t in_ve; + + snprintf(path, sizeof(path), "%d/cpuset.cpus", self->ctid); + if (faccessat(self->cgv2_fd, path, F_OK, 0)) + SKIP(return, "CT cgroup doesn't have cpuset.cpus entry"); + + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, self->orig_cpucount)); + + /* Find two CPUs and leave the second via cgroup interface */ + cpu_leave = get_nth_cpu(&self->orig_cpuset, 2); + ASSERT_GE(cpu_leave, 0); + ASSERT_EQ(write_u64_at(self->cgv2_fd, path, cpu_leave), 0); + + /* Verify sched_getaffinity sees 1 CPU, that is the first one */ + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, 1)); +} + +/* Test correct processing of cpu.nr_cpus updates */ +TEST_F(ve_getaff_test, nr_proc_bound) +{ + char path[PATH_MAX]; + cpu_set_t in_ve; + + snprintf(path, sizeof(path), "%d/cpu.nr_cpus", self->ctid); + if (faccessat(self->cgv2_fd, path, F_OK, 0)) + SKIP(return, "CT cgroup doesn't have cpu.nr_cpus entry"); + + /* We expect here CPU_COUNT >= 2 */ + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, self->orig_cpucount)); + + /* Leave one vCPU */ + ASSERT_EQ(write_u64_at(self->cgv2_fd, path, 1), 0); + + /* Verify sched_getaffinity sees only one vCPU */ + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, 1)); +} + +/* Test correct interaction of cpu.nr_cpus and cpuset.cpus */ +TEST_F(ve_getaff_test, limits_interaction) +{ + char path_nr[PATH_MAX], path_set[PATH_MAX], tmp[128]; + int c1, c3; + cpu_set_t in_ve; + + snprintf(path_nr, sizeof(path_nr), "%d/cpu.nr_cpus", self->ctid); + if (faccessat(self->cgv2_fd, path_nr, F_OK, 0)) + SKIP(return, "CT cgroup doesn't have cpu.nr_cpus entry"); + + snprintf(path_set, sizeof(path_set), "%d/cpuset.cpus", self->ctid); + if (faccessat(self->cgv2_fd, path_set, F_OK, 0)) + SKIP(return, "CT cgroup doesn't have cpuset.cpus entry"); + + if (self->orig_cpucount < 3) + SKIP(return, "This test requires at least 3 CPUs"); + + /* We expect here CPU_COUNT >= 3 */ + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, self->orig_cpucount)); + + /* Leave two CPUs in cpuset */ + c1 = get_nth_cpu(&self->orig_cpuset, 1); + c3 = get_nth_cpu(&self->orig_cpuset, 3); + ASSERT_TRUE(c1 >= 0 && c3 >= 0); + + snprintf(tmp, sizeof(tmp), "%d,%d", c1, c3); + ASSERT_EQ(write_file_at(self->cgv2_fd, path_set, tmp), 0); + + /* Verify sched_getaffinity sees two vCPU */ + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, 2)); + + /* Set vCPU limit to 2 and check that we still have two vCPUs */ + ASSERT_EQ(write_u64_at(self->cgv2_fd, path_nr, 2), 0); + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, 2)); + + /* Set vCPU limit to 1 and check that we have one vCPU only */ + ASSERT_EQ(write_u64_at(self->cgv2_fd, path_nr, 1), 0); + ASSERT_EQ(getaffinity_ve(self, &in_ve), 0); + ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, 1)); + +} + +TEST_HARNESS_MAIN -- 2.52.0 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE Vladimir Riabchun @ 2026-10-29 10:22 ` no-reply 2026-10-29 10:50 ` Vladimir Riabchun 2026-11-12 18:37 ` Konstantin Khorenko 1 sibling, 1 reply; 10+ messages in thread From: no-reply @ 2026-10-29 10:22 UTC (permalink / raw) To: Vladimir Riabchun, Konstantin Khorenko, ptikhomirov; +Cc: devel Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Medium] selftests/ve: get_nth_cpu() bounds loop by sizeof(cpu_set_t) bytes, not CPU_SETSIZE - [Medium] selftests/ve: expected CT CPU count taken from caller's affinity, not CT cpuset - [Low] selftests/ve: FIXTURE_SETUP leaks CT cgroup if prepare_namespaces() ASSERT fails -- commit 55fc819ea7f7954caf6a8e968697a2cc48fd1982 Author: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> selftests: ve: Test sched_getaffinity in VE This adds a selftest that checks sched_getaffinity() inside a VE respects both the cgroup cpuset and the cpu.nr_cpus limit. It adds three test cases and registers the binary in the Makefile and .gitignore. Link: https://virtuozzo.atlassian.net/browse/VSTOR-141682 > diff --git a/tools/testing/selftests/ve/ve_sched_getaffinity_test.c b/tools/testing/selftests/ve/ve_sched_getaffinity_test.c > new file mode 100644 > index 000000000000..98acc246ede0 > --- /dev/null > +++ b/tools/testing/selftests/ve/ve_sched_getaffinity_test.c [ ... ] > +static int get_nth_cpu(cpu_set_t *mask, int n) > +{ > + int i; > + > + for (i = 0; i < sizeof(*mask); i++) { > + if (!CPU_ISSET(i, mask) || --n) > + continue; > + > + return i; > + } > + return -1; > +} [Severity: Medium] Is the loop bound in get_nth_cpu() correct? sizeof(*mask) is the size of cpu_set_t in bytes (128), while CPU_ISSET() takes a bit index and the set holds CPU_SETSIZE (1024) bits: for (i = 0; i < sizeof(*mask); i++) { if (!CPU_ISSET(i, mask) || --n) The scan stops at CPU 127. If the 2nd or 3rd allowed CPU in orig_cpuset is numbered 128 or higher (for example under taskset, a restricted cpuset, or a large sparse host), get_nth_cpu() returns -1. update_cg_cpuset then fails ASSERT_GE(cpu_leave, 0), and limits_interaction fails ASSERT_TRUE(c1 >= 0 && c3 >= 0), without sched_getaffinity() in the VE being exercised. The loop also compares a signed int with a size_t. Should the bound be CPU_SETSIZE? > +FIXTURE_SETUP(ve_getaff_test) > +{ > + ASSERT_EQ(sched_getaffinity(getpid(), sizeof(cpu_set_t), &self->orig_cpuset), 0); > + self->orig_cpucount = CPU_COUNT(&self->orig_cpuset); [Severity: Medium] Is orig_cpucount the right expected value for the CT? It is taken from the runner's own affinity mask, not from the CT cgroup's effective cpuset. In getaffinity_ve(), run_in_ve() forks a child that does: enter_cgroup(cgv2_fd, ctid); enter_namespaces(ctx, flags); ret = fn(arg); Nothing in the child restores the parent's affinity after joining the cgroup. So the mask that __getaffinity_ve() reports derives from the CT cgroup's cpuset, which is inherited from the parent cgroup. I did not read cpuset_attach() in this tree. From memory of mainline, it resets cpus_mask to the cgroup's effective_cpus on attach, but that is unverified. If the runner was restricted, for example with taskset -c 0,1 on a host with more CPUs, the CT may report more CPUs than orig_cpucount. Then this fails in update_cg_cpuset, nr_proc_bound and limits_interaction: ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, self->orig_cpucount)); Would it be better to derive the expected count from the CT cgroup's effective cpuset, or to skip when the two differ? Separately, sched_getaffinity() with a fixed sizeof(cpu_set_t) returns EINVAL when nr_cpu_ids is greater than 1024. FIXTURE_SETUP asserts on that call rather than skipping. [ ... ] > + self->ctid = make_ve(self->cgv2_fd, CTID_MIN); > + ASSERT_GE(self->ctid, 0); > + > + ASSERT_EQ(prepare_namespaces(&self->ns_ctx, CLONE_NEWVE, > + self->cgv2_fd, self->ctid), 0); > +} [Severity: Low] Can this leak the CT cgroup created by make_ve()? If prepare_namespaces() fails, the ASSERT_EQ() aborts FIXTURE_SETUP before setup_completed is set. In kselftest_harness.h, __TEST_F_IMPL only calls the teardown when _metadata->setup_completed is true: if (_metadata->setup_completed && !fixture_name##_teardown_parent && \ __sync_bool_compare_and_swap(teardown, false, true)) \ fixture_name##_teardown(_metadata, self, variant->data); \ So FIXTURE_TEARDOWN(ve_getaff_test) never runs. close_namespace_fds(), destroy_ve() and close(cgv2_fd) are all skipped. I inferred, but did not verify in this tree, that the cgroup persists after the process exits. If so, each leaked cgroup keeps the first free id in the CTID_MIN..CTID_MAX range that make_ve() scans: for (id = from; id < CTID_MAX; id++) { snprintf(name, sizeof(name), "%d", id); if (faccessat(cgv2_fd, name, F_OK, 0) != 0 && errno == ENOENT) break; } This only happens on a failure path. The same pattern exists in other ve selftests such as ve_ns_owner_test and ve_perms_test, so this patch adds another instance rather than a new class of problem. Could the cleanup be done on that failure path, for example by destroying the CT before asserting? -- Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261029-ct-invalid-nrpoc-v3-0-89d4b0b7d520@virtuozzo.com?part=2 _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE 2026-10-29 10:22 ` no-reply @ 2026-10-29 10:50 ` Vladimir Riabchun 0 siblings, 0 replies; 10+ messages in thread From: Vladimir Riabchun @ 2026-10-29 10:50 UTC (permalink / raw) To: Konstantin Khorenko, ptikhomirov; +Cc: devel On 10/29/26 11:22, no-reply@sashiko.virtuozzo.com wrote: > Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: > - [Medium] selftests/ve: get_nth_cpu() bounds loop by sizeof(cpu_set_t) bytes, not CPU_SETSIZE Replied in v2. > - [Medium] selftests/ve: expected CT CPU count taken from caller's affinity, not CT cpuset This is not a CT. This test is a root-only, no need for complications. > - [Low] selftests/ve: FIXTURE_SETUP leaks CT cgroup if prepare_namespaces() ASSERT fails This is fine IMO. With skip it was correct to move it to the top. Here if something fails we should leave all resources, to simplify debugging. > -- > > commit 55fc819ea7f7954caf6a8e968697a2cc48fd1982 > Author: Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> > > selftests: ve: Test sched_getaffinity in VE > > This adds a selftest that checks sched_getaffinity() inside a VE respects > both the cgroup cpuset and the cpu.nr_cpus limit. It adds three test cases > and registers the binary in the Makefile and .gitignore. > > Link: https://virtuozzo.atlassian.net/browse/VSTOR-141682 > >> diff --git a/tools/testing/selftests/ve/ve_sched_getaffinity_test.c b/tools/testing/selftests/ve/ve_sched_getaffinity_test.c >> new file mode 100644 >> index 000000000000..98acc246ede0 >> --- /dev/null >> +++ b/tools/testing/selftests/ve/ve_sched_getaffinity_test.c > > [ ... ] > >> +static int get_nth_cpu(cpu_set_t *mask, int n) >> +{ >> + int i; >> + >> + for (i = 0; i < sizeof(*mask); i++) { >> + if (!CPU_ISSET(i, mask) || --n) >> + continue; >> + >> + return i; >> + } >> + return -1; >> +} > > [Severity: Medium] > Is the loop bound in get_nth_cpu() correct? sizeof(*mask) is the size of > cpu_set_t in bytes (128), while CPU_ISSET() takes a bit index and the set > holds CPU_SETSIZE (1024) bits: > > for (i = 0; i < sizeof(*mask); i++) { > if (!CPU_ISSET(i, mask) || --n) > > The scan stops at CPU 127. If the 2nd or 3rd allowed CPU in orig_cpuset > is numbered 128 or higher (for example under taskset, a restricted cpuset, > or a large sparse host), get_nth_cpu() returns -1. > > update_cg_cpuset then fails ASSERT_GE(cpu_leave, 0), and limits_interaction > fails ASSERT_TRUE(c1 >= 0 && c3 >= 0), without sched_getaffinity() in the > VE being exercised. > > The loop also compares a signed int with a size_t. Should the bound be > CPU_SETSIZE? > >> +FIXTURE_SETUP(ve_getaff_test) >> +{ >> + ASSERT_EQ(sched_getaffinity(getpid(), sizeof(cpu_set_t), &self->orig_cpuset), 0); >> + self->orig_cpucount = CPU_COUNT(&self->orig_cpuset); > > [Severity: Medium] > Is orig_cpucount the right expected value for the CT? It is taken from the > runner's own affinity mask, not from the CT cgroup's effective cpuset. > > In getaffinity_ve(), run_in_ve() forks a child that does: > > enter_cgroup(cgv2_fd, ctid); > enter_namespaces(ctx, flags); > ret = fn(arg); > > Nothing in the child restores the parent's affinity after joining the > cgroup. So the mask that __getaffinity_ve() reports derives from the CT > cgroup's cpuset, which is inherited from the parent cgroup. > > I did not read cpuset_attach() in this tree. From memory of mainline, it > resets cpus_mask to the cgroup's effective_cpus on attach, but that is > unverified. > > If the runner was restricted, for example with taskset -c 0,1 on a host > with more CPUs, the CT may report more CPUs than orig_cpucount. Then this > fails in update_cg_cpuset, nr_proc_bound and limits_interaction: > > ASSERT_TRUE(check_cpumask_is_prefix(&in_ve, self->orig_cpucount)); > > Would it be better to derive the expected count from the CT cgroup's > effective cpuset, or to skip when the two differ? > > Separately, sched_getaffinity() with a fixed sizeof(cpu_set_t) returns > EINVAL when nr_cpu_ids is greater than 1024. FIXTURE_SETUP asserts on that > call rather than skipping. > > [ ... ] > >> + self->ctid = make_ve(self->cgv2_fd, CTID_MIN); >> + ASSERT_GE(self->ctid, 0); >> + >> + ASSERT_EQ(prepare_namespaces(&self->ns_ctx, CLONE_NEWVE, >> + self->cgv2_fd, self->ctid), 0); >> +} > > [Severity: Low] > Can this leak the CT cgroup created by make_ve()? If prepare_namespaces() > fails, the ASSERT_EQ() aborts FIXTURE_SETUP before setup_completed is set. > > In kselftest_harness.h, __TEST_F_IMPL only calls the teardown when > _metadata->setup_completed is true: > > if (_metadata->setup_completed && !fixture_name##_teardown_parent && \ > __sync_bool_compare_and_swap(teardown, false, true)) \ > fixture_name##_teardown(_metadata, self, variant->data); \ > > So FIXTURE_TEARDOWN(ve_getaff_test) never runs. close_namespace_fds(), > destroy_ve() and close(cgv2_fd) are all skipped. > > I inferred, but did not verify in this tree, that the cgroup persists after > the process exits. If so, each leaked cgroup keeps the first free id in > the CTID_MIN..CTID_MAX range that make_ve() scans: > > for (id = from; id < CTID_MAX; id++) { > snprintf(name, sizeof(name), "%d", id); > if (faccessat(cgv2_fd, name, F_OK, 0) != 0 && errno == ENOENT) > break; > } > > This only happens on a failure path. The same pattern exists in other ve > selftests such as ve_ns_owner_test and ve_perms_test, so this patch adds > another instance rather than a new class of problem. > > Could the cleanup be done on that failure path, for example by destroying > the CT before asserting? > -- Best regards, Riabchun Vladimir Linux Kernel Developer, Virtuozzo _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE Vladimir Riabchun 2026-10-29 10:22 ` no-reply @ 2026-11-12 18:37 ` Konstantin Khorenko 1 sibling, 0 replies; 10+ messages in thread From: Konstantin Khorenko @ 2026-11-12 18:37 UTC (permalink / raw) To: Vladimir Riabchun; +Cc: devel I won't apply this version, please send v4 with the changes below. On Thu, 29 Oct 2026 11:16:07 +0100, Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> wrote: > + for (i = 0; i < sizeof(*mask); i++) { Item 4 of the cover letter reply: loop to CPU_SETSIZE. Why: sizeof(*mask) is 128 bytes, CPU_ISSET() takes a CPU number up to 1024. It is a one-token fix, and a test on a host with CPUs above 127 should not fail for a reason unrelated to what it checks. > +#include <string.h> Item 5: include <sys/mman.h> for mmap(), as promised in the v2 reply. It builds today only because kselftest_harness.h pulls it in. -- Konstantin Khorenko, Virtuozzo Linux Kernel Team _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT 2026-10-29 10:16 [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT Vladimir Riabchun 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall Vladimir Riabchun 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE Vladimir Riabchun @ 2026-11-12 18:37 ` Konstantin Khorenko 2 siblings, 0 replies; 10+ messages in thread From: Konstantin Khorenko @ 2026-11-12 18:37 UTC (permalink / raw) To: Vladimir Riabchun; +Cc: devel I won't apply this version, please send v4 with the changes below. On Thu, 29 Oct 2026 11:16:05 +0100, Vladimir Riabchun <vladimir.riabchun@virtuozzo.com> wrote: > nproc code uses sched_getaffinity syscall under the hood[1]. > It appears that this syscall returned complete nonsence when > called from VE - we ignored both num_online_vcpus and > cgroup cpuset limitations. The code of patch 1 is right. What to change for v4: 1. Patch 1: add a Fixes tag for 87211cbff4b4 ("sched: Return only virtual cpus in sched_getaffinity()") and the Feature line of that commit, "sched: virtualize CT cpu affinity management". 2. Patch 1: fix the message. bitmap_fill() fills whole unsigned longs, not integers. Say what a CT sees now: the first min(vCPUs, CPUs of its cpuset) bits, with the vCPU limit of the target task. nproc can then be lower than /proc/cpuinfo and /sys/devices/system/cpu/online, say that this is intended. 3. Patch 1: the code comment says setaffinity is "not available" inside a CT. sched_setaffinity() returns 0 there and does nothing, write that. 4. Patch 2: loop to CPU_SETSIZE in get_nth_cpu(). 5. Patch 2: include <sys/mman.h>, as you promised in the v2 reply, and add a Feature: line. 6. Cover letter: - The old code did use num_online_vcpus(), the bug was the rounding. Ignoring the cpuset was the design of 87211cbff4b4. - "64 no matter what" holds up to 64 vCPUs, above that it was 128. - Typos: "nonsence", "insted". - Link [1] points at a line of gnulib master, which will drift. Name the function instead. Separate patches, not needed for this series; I created a task for it: 1. task_nr_cpus() reads nr_cpus of the task's own cpu cgroup, while vzctl --cpus sets it on the CT root cgroup only. In a nested cpu cgroup inside a CT (docker in a CT, a systemd unit with CPUQuota=) nproc, /proc/cpuinfo and the sysfs cpu lists show the host CPU count. tg_nr_cpus() in cpuacct.c already goes through ve_root_tg(), task_nr_cpus() should do the same. https://virtuozzo.atlassian.net/browse/VSTOR-154938 -- Konstantin Khorenko, Virtuozzo Linux Kernel Team _______________________________________________ Devel mailing list Devel@openvz.org https://lists.openvz.org/mailman/listinfo/devel ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2026-11-12 18:39 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2026-10-29 10:16 [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT Vladimir Riabchun 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 1/2] sched: Fix sched_getaffinity syscall Vladimir Riabchun 2026-10-29 10:21 ` no-reply 2026-10-29 10:48 ` Vladimir Riabchun 2026-11-12 18:37 ` Konstantin Khorenko 2026-10-29 10:16 ` [Devel] [PATCH hci-8.1 v3 2/2] selftests: ve: Test sched_getaffinity in VE Vladimir Riabchun 2026-10-29 10:22 ` no-reply 2026-10-29 10:50 ` Vladimir Riabchun 2026-11-12 18:37 ` Konstantin Khorenko 2026-11-12 18:37 ` [Devel] [PATCH hci-8.1 v3 0/2] sched: Fix incorrect nproc report inside CT Konstantin Khorenko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox