[Devel] [PATCH vz10 v2 0/1] selftests/core: fix unshare_test with large fs.nr_open

[Devel] [PATCH vz10 v2 0/1] selftests/core: fix unshare_test with large fs.nr_open

[Konstantin Khorenko]

unshare_test assumes fs.nr_open is close to the default 1048576, but
systemd bumps it to its largest possible value on boot (e.g.
1073741816) since v240, which makes the test's dup2() past nr_open
try to allocate a ~1 billion entry fd table and fail with ENOMEM.

Changes from v1:
 - v1 capped nr_open in place and reused the same buffer (buf/n) both
   for the capped test value and for the original fs.nr_open readout,
   so the "restore fs.nr_open" write done by the child before calling
   unshare() ended up writing the capped value back instead of the
   real original one. Since fs.nr_open is a global, non-namespaced
   sysctl, this permanently lowered it on any system where it was
   originally set above 1 MiB, as a side effect of running the test.
 - v2 keeps buf/n untouched as the real original value and caps only
   the local nr_open variable used for the test's own arithmetic. The
   child still restores the capped baseline before calling unshare()
   (that's still needed to trigger EMFILE), but we now additionally
   restore the real original fs.nr_open value in the parent once the
   test has completed, so the test no longer leaves the sysctl
   modified after a run.

Eva Kurchatova (1):
  selftests/core: fix unshare_test with large fs.nr_open

 tools/testing/selftests/core/unshare_test.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

-- 
2.43.0

[Devel] [PATCH vz10 v2 1/1] selftests/core: fix unshare_test with large fs.nr_open

[Konstantin Khorenko]

From: Eva Kurchatova <eva.kurchatova@virtuozzo.com>

The test assumes fs.nr_open is close to the default 1048576, but some
systems set it much higher (e.g. 1073741816). This is systemd's doing:
since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to
their largest possible values on boot, as file descriptors are already
accounted for by memcg [1].

In that case, dup2() to nr_open + 64 requires the kernel to allocate a
file descriptor table with ~1 billion entries, which fails with ENOMEM.

Cap the nr_open value used for the test's own arithmetic to a known
reasonable base value (1048576) and restore the true original value
once the test has completed.

[1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
    ("main: bump fs.nr_open + fs.max-file to their largest possible values")

https://virtuozzo.atlassian.net/browse/VSTOR-132444

Feature: fix kselftests
Signed-off-by: Eva Kurchatova <eva.kurchatova@virtuozzo.com>
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 tools/testing/selftests/core/unshare_test.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/core/unshare_test.c b/tools/testing/selftests/core/unshare_test.c
index 7fec9dfb1b0e3..6716e0f74d7da 100644
--- a/tools/testing/selftests/core/unshare_test.c
+++ b/tools/testing/selftests/core/unshare_test.c
@@ -40,6 +40,14 @@ TEST(unshare_EMFILE)
 
 	ASSERT_EQ(sscanf(buf, "%d", &nr_open), 1);
 
+	/*
+	 * Cap nr_open for the duration of the test to avoid ENOMEM from a
+	 * huge fd table allocation; buf/n keep the real original value so
+	 * fs.nr_open can be restored to it once the test is done.
+	 */
+	if (nr_open > 1024 * 1024)
+		nr_open = 1024 * 1024;
+
 	ASSERT_EQ(0, getrlimit(RLIMIT_NOFILE, &rlimit));
 
 	/* bump fs.nr_open */
@@ -73,10 +81,13 @@ TEST(unshare_EMFILE)
 
 	if (pid == 0) {
 		int err;
+		char buf3[32];
+		ssize_t n3;
 
-		/* restore fs.nr_open */
+		/* restore fs.nr_open to the (possibly capped) test baseline */
+		n3 = sprintf(buf3, "%d\n", nr_open);
 		lseek(fd, 0, SEEK_SET);
-		write(fd, buf, n);
+		write(fd, buf3, n3);
 		/* ... and now unshare(CLONE_FILES) must fail with EMFILE */
 		err = unshare(CLONE_FILES);
 		EXPECT_EQ(err, -1)
@@ -89,6 +100,10 @@ TEST(unshare_EMFILE)
 	EXPECT_EQ(waitpid(pid, &status, 0), pid);
 	EXPECT_EQ(true, WIFEXITED(status));
 	EXPECT_EQ(0, WEXITSTATUS(status));
+
+	/* restore the real fs.nr_open value */
+	lseek(fd, 0, SEEK_SET);
+	write(fd, buf, n);
 }
 
 TEST_HARNESS_MAIN
-- 
2.43.0

[Devel] [PATCH RHEL10 COMMIT] selftests/core: fix unshare_test with large fs.nr_open

[Konstantin Khorenko]

The commit is pushed to "branch-rh10-6.12.0-211.39.1.16.x.vz10-ovz" and will appear at git at bitbucket.org:openvz/vzkernel.git
after rh10-6.12.0-211.39.1.16.4.vz10
------>
commit cade8d2ddfe60d4ea67bc95b44ea9ed0585989ab
Author: Eva Kurchatova <eva.kurchatova@virtuozzo.com>
Date:   Fri Aug 14 14:35:43 2026 +0200

    selftests/core: fix unshare_test with large fs.nr_open
    
    The test assumes fs.nr_open is close to the default 1048576, but some
    systems set it much higher (e.g. 1073741816). This is systemd's doing:
    since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to
    their largest possible values on boot, as file descriptors are already
    accounted for by memcg [1].
    
    In that case, dup2() to nr_open + 64 requires the kernel to allocate a
    file descriptor table with ~1 billion entries, which fails with ENOMEM.
    
    Cap the nr_open value used for the test's own arithmetic to a known
    reasonable base value (1048576) and restore the true original value
    once the test has completed.
    
    [1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
        ("main: bump fs.nr_open + fs.max-file to their largest possible values")
    
    https://virtuozzo.atlassian.net/browse/VSTOR-132444
    
    Feature: fix kselftests
    Signed-off-by: Eva Kurchatova <eva.kurchatova@virtuozzo.com>
    Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 tools/testing/selftests/core/unshare_test.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/core/unshare_test.c b/tools/testing/selftests/core/unshare_test.c
index 7fec9dfb1b0e3..6716e0f74d7da 100644
--- a/tools/testing/selftests/core/unshare_test.c
+++ b/tools/testing/selftests/core/unshare_test.c
@@ -40,6 +40,14 @@ TEST(unshare_EMFILE)
 
 	ASSERT_EQ(sscanf(buf, "%d", &nr_open), 1);
 
+	/*
+	 * Cap nr_open for the duration of the test to avoid ENOMEM from a
+	 * huge fd table allocation; buf/n keep the real original value so
+	 * fs.nr_open can be restored to it once the test is done.
+	 */
+	if (nr_open > 1024 * 1024)
+		nr_open = 1024 * 1024;
+
 	ASSERT_EQ(0, getrlimit(RLIMIT_NOFILE, &rlimit));
 
 	/* bump fs.nr_open */
@@ -73,10 +81,13 @@ TEST(unshare_EMFILE)
 
 	if (pid == 0) {
 		int err;
+		char buf3[32];
+		ssize_t n3;
 
-		/* restore fs.nr_open */
+		/* restore fs.nr_open to the (possibly capped) test baseline */
+		n3 = sprintf(buf3, "%d\n", nr_open);
 		lseek(fd, 0, SEEK_SET);
-		write(fd, buf, n);
+		write(fd, buf3, n3);
 		/* ... and now unshare(CLONE_FILES) must fail with EMFILE */
 		err = unshare(CLONE_FILES);
 		EXPECT_EQ(err, -1)
@@ -89,6 +100,10 @@ TEST(unshare_EMFILE)
 	EXPECT_EQ(waitpid(pid, &status, 0), pid);
 	EXPECT_EQ(true, WIFEXITED(status));
 	EXPECT_EQ(0, WEXITSTATUS(status));
+
+	/* restore the real fs.nr_open value */
+	lseek(fd, 0, SEEK_SET);
+	write(fd, buf, n);
 }
 
 TEST_HARNESS_MAIN