[Devel] [PATCH RHEL10 COMMIT] selftests/core: fix unshare_test with large fs.nr_open

[Konstantin Khorenko <khorenko@virtuozzo.com>]

The commit is pushed to "branch-rh10-6.12.0-211.39.1.16.x.vz10-ovz" and will appear at git at bitbucket.org:openvz/vzkernel.git
after rh10-6.12.0-211.39.1.16.4.vz10
------>
commit cade8d2ddfe60d4ea67bc95b44ea9ed0585989ab
Author: Eva Kurchatova <eva.kurchatova@virtuozzo.com>
Date:   Fri Aug 14 14:35:43 2026 +0200

    selftests/core: fix unshare_test with large fs.nr_open
    
    The test assumes fs.nr_open is close to the default 1048576, but some
    systems set it much higher (e.g. 1073741816). This is systemd's doing:
    since systemd v240 (2018), PID 1 bumps fs.nr_open and fs.file-max to
    their largest possible values on boot, as file descriptors are already
    accounted for by memcg [1].
    
    In that case, dup2() to nr_open + 64 requires the kernel to allocate a
    file descriptor table with ~1 billion entries, which fails with ENOMEM.
    
    Cap the nr_open value used for the test's own arithmetic to a known
    reasonable base value (1048576) and restore the true original value
    once the test has completed.
    
    [1] https://github.com/systemd/systemd/commit/a8b627aaed409a15260c25988970c795bf963812
        ("main: bump fs.nr_open + fs.max-file to their largest possible values")
    
    https://virtuozzo.atlassian.net/browse/VSTOR-132444
    
    Feature: fix kselftests
    Signed-off-by: Eva Kurchatova <eva.kurchatova@virtuozzo.com>
    Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 tools/testing/selftests/core/unshare_test.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/core/unshare_test.c b/tools/testing/selftests/core/unshare_test.c
index 7fec9dfb1b0e3..6716e0f74d7da 100644
--- a/tools/testing/selftests/core/unshare_test.c
+++ b/tools/testing/selftests/core/unshare_test.c
@@ -40,6 +40,14 @@ TEST(unshare_EMFILE)
 
 	ASSERT_EQ(sscanf(buf, "%d", &nr_open), 1);
 
+	/*
+	 * Cap nr_open for the duration of the test to avoid ENOMEM from a
+	 * huge fd table allocation; buf/n keep the real original value so
+	 * fs.nr_open can be restored to it once the test is done.
+	 */
+	if (nr_open > 1024 * 1024)
+		nr_open = 1024 * 1024;
+
 	ASSERT_EQ(0, getrlimit(RLIMIT_NOFILE, &rlimit));
 
 	/* bump fs.nr_open */
@@ -73,10 +81,13 @@ TEST(unshare_EMFILE)
 
 	if (pid == 0) {
 		int err;
+		char buf3[32];
+		ssize_t n3;
 
-		/* restore fs.nr_open */
+		/* restore fs.nr_open to the (possibly capped) test baseline */
+		n3 = sprintf(buf3, "%d\n", nr_open);
 		lseek(fd, 0, SEEK_SET);
-		write(fd, buf, n);
+		write(fd, buf3, n3);
 		/* ... and now unshare(CLONE_FILES) must fail with EMFILE */
 		err = unshare(CLONE_FILES);
 		EXPECT_EQ(err, -1)
@@ -89,6 +100,10 @@ TEST(unshare_EMFILE)
 	EXPECT_EQ(waitpid(pid, &status, 0), pid);
 	EXPECT_EQ(true, WIFEXITED(status));
 	EXPECT_EQ(0, WEXITSTATUS(status));
+
+	/* restore the real fs.nr_open value */
+	lseek(fd, 0, SEEK_SET);
+	write(fd, buf, n);
 }
 
 TEST_HARNESS_MAIN