[Devel] [PATCH hci-8.1 v3 0/2] mm/memcontrol: numa_migrate destinations of the source folio order

[Devel] [PATCH hci-8.1 v3 0/2] mm/memcontrol: numa_migrate destinations of the source folio order

[Konstantin Khorenko]

memory.numa_migrate, our memcg file that vcmmd uses to move a cgroup's
memory between NUMA nodes, allocates a 2 MB destination for every large
folio it migrates. Page cache folios of a smaller order get a
destination of the wrong size, and the page cache falls apart around
it. Patch 1 is the fix, patch 2 a selftest that catches it without
crashing the node.

What the customer hit
=====================

A two-node VMware host runs VMs with vcmmd's NUMA policy enabled, so
every few minutes vcmmd writes memory.numa_migrate for a VM's cgroup.
qemu writes its backup scratch file on NFS, and NFS sizes each page
cache folio to the write, so that file is full of order-1 to order-5
folios. Closing the scratch file dies in nfs4_evict_inode():

  kernel BUG at fs/inode.c:737!
  RIP: 0010:clear_inode+0x75/0x80

mapping->nrpages of the inode is negative, and the vmcore holds
order-9 folios of that file with an index not aligned to 512, the
refcount of a small folio, and whole xarray ranges of live folios that
are no longer in the tree. Reads of the migrated ranges return garbage,
so the backups taken on that host while the policy was active should
be verified. The same crash was filed as VSTOR-151676.

Where the 2 MB comes from
=========================

memcg_numa_migrate_new_folio() is the migrate_pages() allocation
callback. Its large folio branch dates from vz8 (4.18) and follows what
new_page() in mm/mempolicy.c did on the 4.x kernels:

  if (thp_migration_supported() && PageTransHuge(page))
          alloc ... HPAGE_PMD_ORDER

A compound page on the LRU could only be a PMD-sized THP then, so the
test implied the order. Upstream replaced that with folio_order() of
the source when its allocators moved to folios, ffe06786b540
("mm/migrate: Use a folio in alloc_migration_target()"). Our copy kept
the shape, only PageTransHuge() became folio_test_large(), which is
folio_test_head() and is true for any order from 1 up. NFS and XFS
large folios, shmem and anonymous mTHP now all reach it.

Why an oversized destination breaks the page cache
==================================================

The two sides of the page cache size the folio differently.

On insertion __folio_migrate_mapping() takes everything from the
source: nr = folio_nr_pages(old), the xarray state at the old index
with no xas_set_order(), newfolio->index = old->index,
folio_ref_add(newfolio, nr), and xas_store() replaces exactly the slots
the source occupied. folio_copy() copies the source pages only. So a
2 MB folio enters the cache as, say, a 4-page one: 4 xarray slots, 4
extra references, nrpages unchanged, 4 pages of valid data.

Everything afterwards asks the folio itself, and it answers order 9:

- page_cache_delete(): xas_set_order(index, 9) and nrpages -= 512.
  512 slots wiped instead of 4, with the index rounded down to 512 so
  that other live folios lose their slots, and 512 subtracted instead
  of 4. That is the negative nrpages and the orphaned folios.
- __remove_mapping(): expects 1 + 512 references and finds 1 + 4.
  Reclaim can never free the folio.
- filemap_free_folio(): drops 512 references from a handful at
  eviction. The refcount goes negative and the folio leaks.
- reads: folio_file_page() picks the page by its offset inside the
  folio, but only the first pages were copied.

The data damage shows on the first read of the folio: filemap_read()
takes the page at ki_pos & (folio_size - 1) of the destination, which
is uninitialised unless the index is 2 MB aligned, and a longer read
that starts there continues through the destination past the
neighbours. The structural damage waits for the first operation that
applies folio_nr_pages() or folio_order() to it: invalidation,
truncate, reclaim, inode eviction. On NFS cache invalidation is
routine (an attribute change), so the bug shows up there instead of
staying a silent 2 MB leak per folio.

In short: inserted by the source size, removed by the destination
size. That only works when the two match, which for a real THP they
always did. A debug kernel says the same on the first migration, in
mem_cgroup_migrate():

  VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new))

The patches
===========

Patch 1 allocates the destination with folio_order() of the source,
GFP_TRANSHUGE_LIGHT for large folios as before. The
thp_migration_supported() test goes: migrate_pages() splits PMD-mappable
folios itself when THP migration is unsupported, and every other large
folio still needs a matching destination.

Patch 2 adds tools/testing/selftests/ve/ve_numa_migrate_test.c. It
binds a cgroup to one NUMA node, writes a file on a loop-mounted XFS
image in 64K chunks so that the page cache holds large folios, widens
cpuset.mems to both nodes (the destination allocation honours the
writer's cpuset), migrates the cgroup to the other node through
memory.numa_migrate and compares what reads back. The chunks at 2 MB
aligned offsets are left out: a folio there reads right even from an
oversized destination. memory.current cannot serve as the detector:
mem_cgroup_migrate() only transfers the charge, whatever the size
mismatch. On an unfixed kernel the comparison fails and the test leaves
the image mounted on purpose, since evicting the inode would hit the
BUG_ON.

The folio sizes are not taken on faith: the test reads the compound
flags of the file's pages through /proc/self/pagemap and
/proc/kpageflags, and after the migration also the node of every page
through move_pages() in query mode. A file without a large folio below
the PMD size, or a migration that moved none of them to the target
node in one piece, fails the test (PMD-sized folios got a correct
destination from the old code too, and migrate_pages() may split or
skip folios under memory pressure). That is deliberate: a skip would
hide that the test no longer reaches the bug, a failure says the setup
needs another way to get large folios. memory.numa_stat is only
logged, polled for a few seconds since memcg flushes its per-cpu
counters lazily. It needs root, two NUMA nodes with memory
(numa=fake=2 is enough), cgroup2 with memory and cpuset, and mkfs.xfs;
it skips otherwise.

Testing
=======

On a VM booted with numa=fake=2:

- the reproducer (64K and 8K writes on XFS from a node-bound cgroup,
  memory.numa_migrate to the other node, unlink) kills the unfixed
  debug kernel on the VM_BUG_ON above and the release kernel in
  clear_inode(); with patch 1 migration succeeds and unlink and
  eviction complete;
- the selftest passes on the fixed kernel and fails on the comparison
  on 22.11 without taking the node down.

Other branches
==============

The same patch applies as is to the 20.x branch (the customer's
kernel); the function is a plain callback, so it is a ReadyKernel
candidate.
vz9 carries the same PageTransHuge() code, but the file systems we
ship on 5.14 (XFS, NFS, ext4, shmem) put no non-PMD large folios on
the LRU, so it is not reachable there in practice.

https://virtuozzo.atlassian.net/browse/VSTOR-151607

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
Changes in v3:
- Patch 2: the large folio guard requires a folio below the PMD size
  and the page size comes from sysconf() (Sashiko). After the
  migration the test checks through move_pages() that at least one
  such folio sits on the target node in one piece, instead of
  requiring the whole file in memory.numa_stat, which migrate_pages()
  does not guarantee (Sashiko); the stat is logged only.
- Patch 2: the chunks at 2 MB aligned offsets are not written, so an
  oversized destination always shows in the read-back, and the image
  stays mounted until the last check passes; the folio sizes per node
  are logged before and after the migration; the mkfs.xfs check goes,
  a failing mkfs skips the test anyway (independent review).
- Patch 2: Reviewed-by tags dropped, the test changed too much since
  the review.
- Link to v2: http://lore.virtuozzo.com/20261028-b4-prep-khorenko-vstor-151607-numa-migrate-order-v2-0-b973a82a5b0c@virtuozzo.com

Changes in v2:
- Patch 1: the commit message says why the thp_migration_supported()
  check goes (Pavel).
- Patch 2: the folio sizes are checked through /proc/self/pagemap and
  /proc/kpageflags before the migration; a file without a large folio
  fails the test on purpose (Pavel).
- Patch 2: the read-back comparison runs first and the memory.numa_stat
  check after it, polled for a few seconds, since memory_numa_stat_show()
  flushes the per-cpu counters only past a per-CPU threshold (Vasileios).
- Reviewed-by from Pavel and Vasileios on both patches.
- Link to v1: http://lore.virtuozzo.com/20261028-b4-prep-khorenko-vstor-151607-numa-migrate-order-v1-0-268336526677@virtuozzo.com

---
Konstantin Khorenko (2):
      mm/memcontrol: allocate numa_migrate targets of the source folio order
      selftests/ve: check memory.numa_migrate keeps the folio order

 mm/memcontrol.c                                   |  11 +-
 tools/testing/selftests/ve/Makefile               |   1 +
 tools/testing/selftests/ve/ve_numa_migrate_test.c | 494 ++++++++++++++++++++++
 3 files changed, 501 insertions(+), 5 deletions(-)
---
base-commit: b499efa17aa9fb8f20512d1831d1cb61b22e60d3
change-id: 20261028-b4-prep-khorenko-vstor-151607-numa-migrate-order-4e3a664a89d4

Best regards,
--  
Konstantin Khorenko,
Virtuozzo Linux Kernel Team

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v3 1/2] mm/memcontrol: allocate numa_migrate targets of the source folio order

[Konstantin Khorenko]

memcg_numa_migrate_new_folio() treats every large folio as a PMD-sized
THP and allocates an HPAGE_PMD_ORDER destination for it. The branch is
the 4.18 pattern of new_page() in mm/mempolicy.c: a compound page on
the LRU could only be a THP then, so PageTransHuge() implied
HPAGE_PMD_ORDER.

Today the page cache holds large folios of any order (NFS sizes a folio
to the write, XFS and shmem have them too) and anonymous memory has
mTHP, while folio_test_large() is still just folio_test_head() and is
true for any order from 1 up. migrate_pages() never checks that the
destination order matches the source.

For an order-1 or order-4 file folio the destination is an order-9
folio that inherits the source index and page cache reference count,
so every later page cache operation that sizes the folio by its order
goes wrong: nrpages underflows, neighbouring xarray slots get wiped,
the folio can never be freed.

On a two-node host a VM migration to another NUMA node along with a
backup to NFS ends up with a negative nrpages, and closing the backup
file dies in nfs4_evict_inode():

  kernel BUG at fs/inode.c:737!
  RIP: 0010:clear_inode+0x75/0x80
  Call Trace:
   nfs4_evict_inode+0x21/0x80 [nfsv4]
   evict+0x115/0x2a0
   nfs4_delegreturn_release+0x55/0xa0 [nfsv4]
   rpc_free_task+0x39/0x60 [sunrpc]
   __rpc_execute+0x3d1/0x3e0 [sunrpc]

A debug build dies earlier, on the first migration of a large folio of
any file system, in mem_cgroup_migrate():

  VM_BUG_ON_FOLIO(folio_nr_pages(old) != folio_nr_pages(new))

Make memcg_numa_migrate_new_folio() handle large folios properly.
The thp_migration_supported() check goes: migrate_pages() splits PMD
THPs itself when unsupported, other large folios migrate whole anyway.

Fixes: 9f8a754ad6c2 ("mm: memcontrol: add memory.numa_migrate file")
https://virtuozzo.atlassian.net/browse/VSTOR-151607

Feature: mm: interface to migrate memory between NUMA nodes upon userspace request
Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Reviewed-by: Vasileios Almpanis <vasileios.almpanis@virtuozzo.com>
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 mm/memcontrol.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 0734c0fa21d05..d8a87913ace37 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3480,6 +3480,7 @@ static struct folio *memcg_numa_migrate_new_folio(struct folio *old_folio,
 {
 	struct memcg_numa_migrate_struct *ms = (void *)private;
 	gfp_t gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_NORETRY | __GFP_NOWARN;
+	unsigned int order = 0;
 
 	ms->current_node = next_node(ms->current_node, *ms->target_nodes);
 	if (ms->current_node >= MAX_NUMNODES) {
@@ -3487,13 +3488,13 @@ static struct folio *memcg_numa_migrate_new_folio(struct folio *old_folio,
 		VM_BUG_ON(ms->current_node >= MAX_NUMNODES);
 	}
 
-	if (thp_migration_supported() && folio_test_large(old_folio)) {
-		return __folio_alloc(GFP_TRANSHUGE_LIGHT | __GFP_THISNODE,
-				HPAGE_PMD_ORDER, ms->current_node,
-				ms->target_nodes);
+	if (folio_test_large(old_folio)) {
+		gfp_mask = GFP_TRANSHUGE_LIGHT | __GFP_THISNODE;
+		order = folio_order(old_folio);
 	}
 
-	return __folio_alloc(gfp_mask, 0, ms->current_node, ms->target_nodes);
+	return __folio_alloc(gfp_mask, order, ms->current_node,
+			     ms->target_nodes);
 }
 
 /*

-- 
2.43.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

[Devel] [PATCH hci-8.1 v3 2/2] selftests/ve: check memory.numa_migrate keeps the folio order

[Konstantin Khorenko]

memcg_numa_migrate_new_folio() used to give every large folio a
PMD-sized destination. A smaller page cache folio then read back as
garbage, and its inode died in clear_inode() on eviction.

The test writes a file on an XFS image in 64K chunks, which gives
order-4 folios, from a cgroup bound to one NUMA node. It migrates the
cgroup to a second node with memory.numa_migrate and compares the file
with the written pattern.

Two checks make sure the test reaches the bug:

- before the migration the page cache must hold a large folio below
  the PMD size, read through /proc/self/pagemap and /proc/kpageflags;
- after it at least one such folio must sit on the target node in one
  piece, its node read through move_pages().

A failed check is a failure, not a skip, so that a setup where the
test stops reaching the bug shows up. The chunks at 2 MB aligned
offsets are not written: a folio there reads right even from an
oversized destination.

On an unfixed kernel the comparison fails and the test leaves the
image mounted, because an eviction would crash the node.

Needs root, two NUMA nodes with memory (numa=fake=2 is enough),
cgroup2 with the memory and cpuset controllers, and mkfs.xfs; skips
otherwise.

https://virtuozzo.atlassian.net/browse/VSTOR-151607

Feature: ve selftests
Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 tools/testing/selftests/ve/Makefile               |   1 +
 tools/testing/selftests/ve/ve_numa_migrate_test.c | 494 ++++++++++++++++++++++
 2 files changed, 495 insertions(+)

diff --git a/tools/testing/selftests/ve/Makefile b/tools/testing/selftests/ve/Makefile
index 51f9316beb99b..e96320a183844 100644
--- a/tools/testing/selftests/ve/Makefile
+++ b/tools/testing/selftests/ve/Makefile
@@ -6,6 +6,7 @@ TEST_GEN_PROGS += ve_ns_owner_test
 TEST_GEN_PROGS += ve_perms_test
 TEST_GEN_PROGS += ve_mount_accounting_test
 TEST_GEN_PROGS += ve_vtty_stress
+TEST_GEN_PROGS += ve_numa_migrate_test
 
 # CPUID faulting is x86-64 only. ARCH says "x86" for 32-bit too and lib.mk
 # picks the clang target by its emptiness, so decide by the compiler.
diff --git a/tools/testing/selftests/ve/ve_numa_migrate_test.c b/tools/testing/selftests/ve/ve_numa_migrate_test.c
new file mode 100644
index 0000000000000..2c6499195b9ba
--- /dev/null
+++ b/tools/testing/selftests/ve/ve_numa_migrate_test.c
@@ -0,0 +1,494 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * memory.numa_migrate must give a large page cache folio a destination of
+ * the same order. An oversized destination inherits the index and the
+ * reference count of the source, only the source pages are copied into it,
+ * a read of the folio is served from the destination by file offset and
+ * lands in its uninitialised part, and the inode dies in clear_inode() on
+ * eviction.
+ *
+ * Write a file on an XFS image (XFS allocates large folios for 64K writes)
+ * from a cgroup bound to one NUMA node, migrate the cgroup to another node
+ * and check that the data reads back intact.
+ *
+ * Needs root, two NUMA nodes with memory (numa=fake=2 will do), cgroup2
+ * with the memory and cpuset controllers, and mkfs.xfs. The folio sizes
+ * are read through /proc/self/pagemap and /proc/kpageflags before the
+ * migration, and the node of every page through move_pages() after it: a
+ * file without large folios below the PMD size, or a migration that
+ * moved none of them in one piece, fails the test, as it would pass
+ * without touching the bug.
+ */
+#define _GNU_SOURCE
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sched.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/mount.h>
+#include <sys/mman.h>
+#include <sys/syscall.h>
+#include <linux/sched.h>
+#include <linux/kernel-page-flags.h>
+
+#include "../kselftest_harness.h"
+#include "ve_selftest.h"
+
+#define CG_NAME		"ve_numa_migrate_test"
+#define DIR_TEMPLATE	"/var/tmp/ve_numa_migrate.XXXXXX"
+/* mkfs.xfs refuses anything smaller; the image is sparse. */
+#define IMG_SIZE	(320UL << 20)
+#define FILE_SIZE	(4UL << 20)
+#define CHUNK_SIZE	(64UL << 10)
+#define PM_PFN_MASK	((1ULL << 55) - 1)
+#ifndef MADV_POPULATE_READ
+#define MADV_POPULATE_READ	22
+#endif
+#define PM_PRESENT	(1ULL << 63)
+/* The periodic memcg stats flush runs every two seconds. */
+#define STAT_RETRIES	50
+#define STAT_RETRY_US	100000
+
+FIXTURE(numa_migrate)
+{
+	int cgv2_fd;
+	int cg_fd;
+	int node_from;
+	int node_to;
+	char dir[64];
+	char img[80];
+	char mnt[80];
+	int mounted;
+	int leave_mounted;
+	int data_fd;
+};
+
+/* First two node ids of a sysfs node list such as "0-1" or "0,2-3". */
+static int two_nodes_with_memory(int *from, int *to)
+{
+	char buf[256], *tok, *save;
+	int found = 0;
+
+	if (read_file_at(AT_FDCWD, "/sys/devices/system/node/has_memory",
+			 buf, sizeof(buf)) <= 0)
+		return -1;
+
+	for (tok = strtok_r(buf, ",\n", &save); tok && found < 2;
+	     tok = strtok_r(NULL, ",\n", &save)) {
+		int a, b, n;
+
+		if (sscanf(tok, "%d-%d", &a, &b) != 2) {
+			if (sscanf(tok, "%d", &a) != 1)
+				return -1;
+			b = a;
+		}
+		for (n = a; n <= b && found < 2; n++) {
+			if (found == 0)
+				*from = n;
+			else
+				*to = n;
+			found++;
+		}
+	}
+	return found == 2 ? 0 : -1;
+}
+
+static int run_cmd(const char *fmt, ...)
+{
+	char cmd[256];
+	va_list ap;
+	int status;
+
+	va_start(ap, fmt);
+	vsnprintf(cmd, sizeof(cmd), fmt, ap);
+	va_end(ap);
+
+	status = system(cmd);
+	if (status < 0 || !WIFEXITED(status))
+		return -1;
+	return WEXITSTATUS(status);
+}
+
+/* Bytes of file pages the cgroup has on @node, from memory.numa_stat. */
+static int file_bytes_on_node(int cg_fd, int node, unsigned long long *out)
+{
+	static char buf[32768];
+	char key[16], *p;
+
+	if (read_file_at(cg_fd, "memory.numa_stat", buf, sizeof(buf)) <= 0)
+		return -1;
+	p = strstr(buf, "\nfile ");
+	if (!p)
+		return -1;
+	snprintf(key, sizeof(key), " N%d=", node);
+	p = strstr(p, key);
+	if (!p)
+		return -1;
+	*out = strtoull(p + strlen(key), NULL, 10);
+	return 0;
+}
+
+static unsigned long page_size;
+
+/* A folio at a PMD-aligned index reads right from an oversized one. */
+static bool chunk_is_hole(unsigned long off, unsigned long pmd_size)
+{
+	return !((off / CHUNK_SIZE * CHUNK_SIZE) % pmd_size);
+}
+
+#define SCAN_ORDERS	16
+
+struct folio_scan {
+	long large;	/* pages in large folios */
+	long smallest;	/* pages in the smallest large folio */
+	long on_node;	/* large folios below @pmd_pages that sit on @node */
+	long hist[2][SCAN_ORDERS];	/* folios by order, on @node / elsewhere */
+};
+
+static void folio_scan_end_run(struct folio_scan *fs, long run, int run_node,
+			       int node, long pmd_pages)
+{
+	int order = 0;
+
+	if (!run)
+		return;
+	while ((1L << order) < run && order < SCAN_ORDERS - 1)
+		order++;
+	fs->hist[run_node != node][order]++;
+	if (run == 1)
+		return;
+	if (!fs->smallest || run < fs->smallest)
+		fs->smallest = run;
+	if (run < pmd_pages && run_node == node)
+		fs->on_node++;
+}
+
+/* "64 x 64K, 3 x 4K" or "none" */
+static const char *folio_hist_str(const long *hist, char *buf, size_t len)
+{
+	size_t pos = 0;
+	int order;
+
+	for (order = SCAN_ORDERS - 1; order >= 0; order--) {
+		if (!hist[order])
+			continue;
+		pos += snprintf(buf + pos, len - pos, "%s%ld x %luK",
+				pos ? ", " : "", hist[order],
+				(page_size << order) / 1024);
+		if (pos >= len)
+			break;
+	}
+	return pos ? buf : "none";
+}
+
+/*
+ * Walk the page cache of @fd through /proc/self/pagemap, /proc/kpageflags
+ * and move_pages(2) in query mode, leaving the holes alone: faulting them
+ * in would add folios to the file. Returns -1 when that is not possible.
+ */
+static int scan_file_folios(int fd, unsigned long size, int node,
+			    unsigned long pmd_size, struct folio_scan *fs)
+{
+	unsigned long nr = size / page_size, i, off;
+	long pmd_pages = pmd_size / page_size;
+	long run = 0;
+	int run_node = -1, ret = -1;
+	int pagemap, kpageflags, *status;
+	unsigned char *map;
+	void **pages;
+	uint64_t *pm;
+
+	memset(fs, 0, sizeof(*fs));
+	pagemap = open("/proc/self/pagemap", O_RDONLY);
+	kpageflags = open("/proc/kpageflags", O_RDONLY);
+	pm = malloc(nr * sizeof(*pm));
+	pages = malloc(nr * sizeof(*pages));
+	status = malloc(nr * sizeof(*status));
+	map = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
+	if (pagemap < 0 || kpageflags < 0 || !pm || !pages || !status ||
+	    map == MAP_FAILED)
+		goto out;
+	for (off = 0; off < size; off += CHUNK_SIZE) {
+		if (!chunk_is_hole(off, pmd_size) &&
+		    madvise(map + off, CHUNK_SIZE, MADV_POPULATE_READ))
+			goto out;
+	}
+	if (pread(pagemap, pm, nr * sizeof(*pm),
+		  (unsigned long)map / page_size * sizeof(*pm)) !=
+	    (ssize_t)(nr * sizeof(*pm)))
+		goto out;
+	for (i = 0; i < nr; i++)
+		pages[i] = map + i * page_size;
+	if (syscall(__NR_move_pages, 0, nr, pages, NULL, status, 0))
+		goto out;
+	for (i = 0; i < nr; i++) {
+		uint64_t flags, pfn = pm[i] & PM_PFN_MASK;
+
+		if (chunk_is_hole(i * page_size, pmd_size)) {
+			folio_scan_end_run(fs, run, run_node, node, pmd_pages);
+			run = 0;
+			continue;
+		}
+		if (!(pm[i] & PM_PRESENT) || !pfn || status[i] < 0 ||
+		    pread(kpageflags, &flags, sizeof(flags),
+			  pfn * sizeof(flags)) != sizeof(flags))
+			goto out;
+		if (flags & (1ULL << KPF_COMPOUND_HEAD)) {
+			folio_scan_end_run(fs, run, run_node, node, pmd_pages);
+			fs->large++;
+			run = 1;
+			run_node = status[i];
+		} else if (flags & (1ULL << KPF_COMPOUND_TAIL)) {
+			fs->large++;
+			run++;
+		} else {
+			folio_scan_end_run(fs, run, run_node, node, pmd_pages);
+			folio_scan_end_run(fs, 1, status[i], node, pmd_pages);
+			run = 0;
+		}
+	}
+	folio_scan_end_run(fs, run, run_node, node, pmd_pages);
+	ret = 0;
+out:
+	if (map != MAP_FAILED)
+		munmap(map, size);
+	free(status);
+	free(pages);
+	free(pm);
+	if (pagemap >= 0)
+		close(pagemap);
+	if (kpageflags >= 0)
+		close(kpageflags);
+	return ret;
+}
+
+static void cleanup(FIXTURE_DATA(numa_migrate) *self)
+{
+	char path[PATH_MAX];
+
+	if (self->data_fd >= 0) {
+		close(self->data_fd);
+		self->data_fd = -1;
+	}
+	if (self->dir[0]) {
+		snprintf(path, sizeof(path), "%s/data", self->mnt);
+		unlink(path);
+		if (self->mounted)
+			umount(self->mnt);
+		unlink(self->img);
+		rmdir(self->mnt);
+		rmdir(self->dir);
+		self->dir[0] = '\0';
+	}
+	if (self->cg_fd >= 0) {
+		close(self->cg_fd);
+		self->cg_fd = -1;
+	}
+	if (self->cgv2_fd >= 0) {
+		/* The test task may still sit in the cgroup after a failure. */
+		write_u64_at(self->cgv2_fd, "cgroup.procs", getpid());
+		unlinkat(self->cgv2_fd, CG_NAME, AT_REMOVEDIR);
+		close(self->cgv2_fd);
+		self->cgv2_fd = -1;
+	}
+}
+
+/*
+ * Returns 0, or 1 with everything undone when the host cannot run the test.
+ * The harness runs no teardown after a failed setup, so the undo is here.
+ */
+static int prepare(FIXTURE_DATA(numa_migrate) *self, const char **why)
+{
+	int fd;
+
+	self->cgv2_fd = mount_cg2_fd();
+	if (self->cgv2_fd < 0) {
+		*why = "cannot mount cgroup2";
+		goto fail;
+	}
+	if (write_file_at(self->cgv2_fd, "cgroup.subtree_control",
+			  "+memory +cpuset")) {
+		*why = "cannot enable the memory and cpuset controllers";
+		goto fail;
+	}
+	if (mkdirat(self->cgv2_fd, CG_NAME, 0755) && errno != EEXIST) {
+		*why = "cannot create the test cgroup";
+		goto fail;
+	}
+	self->cg_fd = openat(self->cgv2_fd, CG_NAME, O_PATH | O_DIRECTORY);
+	if (self->cg_fd < 0) {
+		*why = "cannot open the test cgroup";
+		goto fail;
+	}
+	if (faccessat(self->cg_fd, "memory.numa_migrate", W_OK, 0)) {
+		*why = "no memory.numa_migrate";
+		goto fail;
+	}
+	if (write_u64_at(self->cg_fd, "cpuset.mems", self->node_from)) {
+		*why = "cannot bind the cgroup to a node";
+		goto fail;
+	}
+
+	snprintf(self->dir, sizeof(self->dir), DIR_TEMPLATE);
+	if (!mkdtemp(self->dir)) {
+		self->dir[0] = '\0';
+		*why = "cannot create a temporary directory";
+		goto fail;
+	}
+	snprintf(self->img, sizeof(self->img), "%s/img", self->dir);
+	snprintf(self->mnt, sizeof(self->mnt), "%s/mnt", self->dir);
+	if (mkdir(self->mnt, 0755)) {
+		*why = "cannot create the mount point";
+		goto fail;
+	}
+	fd = open(self->img, O_CREAT | O_RDWR, 0600);
+	if (fd < 0) {
+		*why = "cannot create the image file";
+		goto fail;
+	}
+	if (ftruncate(fd, IMG_SIZE)) {
+		close(fd);
+		*why = "cannot size the image file";
+		goto fail;
+	}
+	close(fd);
+	if (run_cmd("mkfs.xfs -q -f %s", self->img)) {
+		*why = "mkfs.xfs failed";
+		goto fail;
+	}
+	if (run_cmd("mount -o loop %s %s", self->img, self->mnt)) {
+		*why = "cannot loop-mount the XFS image";
+		goto fail;
+	}
+	self->mounted = 1;
+	return 0;
+fail:
+	cleanup(self);
+	return 1;
+}
+
+FIXTURE_SETUP(numa_migrate)
+{
+	const char *why;
+
+	self->cgv2_fd = -1;
+	self->cg_fd = -1;
+	self->data_fd = -1;
+	self->dir[0] = '\0';
+	page_size = sysconf(_SC_PAGESIZE);
+
+	if (geteuid())
+		SKIP(return, "needs root");
+	if (two_nodes_with_memory(&self->node_from, &self->node_to))
+		SKIP(return, "needs two NUMA nodes with memory");
+	if (prepare(self, &why))
+		SKIP(return, "%s", why);
+}
+
+FIXTURE_TEARDOWN(numa_migrate)
+{
+	if (self->leave_mounted) {
+		TH_LOG("leaving %s mounted, eviction would hit BUG_ON(nrpages)",
+		       self->mnt);
+		self->dir[0] = '\0';
+	}
+	cleanup(self);
+}
+
+TEST_F(numa_migrate, large_folio_order)
+{
+	unsigned long long on_to_before, on_to_after;
+	unsigned char *buf, *check;
+	char path[PATH_MAX], list[64];
+	unsigned long long pmd_size;
+	struct folio_scan fs;
+	char h0[128], h1[128];
+	long pmd_pages;
+	unsigned long off;
+	int fd, i;
+
+	ASSERT_EQ(write_u64_at(self->cg_fd, "cgroup.procs", getpid()), 0);
+
+	buf = malloc(CHUNK_SIZE);
+	check = malloc(CHUNK_SIZE);
+	ASSERT_NE(buf, NULL);
+	ASSERT_NE(check, NULL);
+
+	/* The old code allocated PMD-sized destinations: those were fine. */
+	if (read_u64_at(AT_FDCWD,
+			"/sys/kernel/mm/transparent_hugepage/hpage_pmd_size",
+			&pmd_size))
+		pmd_size = 2UL << 20;
+	pmd_pages = pmd_size / page_size;
+
+	snprintf(path, sizeof(path), "%s/data", self->mnt);
+	fd = open(path, O_CREAT | O_RDWR | O_TRUNC, 0600);
+	ASSERT_GE(fd, 0);
+	self->data_fd = fd;
+	for (off = 0; off < FILE_SIZE; off += CHUNK_SIZE) {
+		if (chunk_is_hole(off, pmd_size))
+			continue;
+		memset(buf, (off / CHUNK_SIZE) & 0xff, CHUNK_SIZE);
+		ASSERT_EQ(pwrite(fd, buf, CHUNK_SIZE, off), (ssize_t)CHUNK_SIZE);
+	}
+	ASSERT_EQ(fsync(fd), 0);
+	ASSERT_EQ(scan_file_folios(fd, FILE_SIZE, self->node_to, pmd_size, &fs), 0)
+		TH_LOG("cannot read the page cache folio sizes");
+	TH_LOG("before: on node %d: %s, elsewhere: %s", self->node_to,
+	       folio_hist_str(fs.hist[0], h0, sizeof(h0)),
+	       folio_hist_str(fs.hist[1], h1, sizeof(h1)));
+	ASSERT_GT(fs.large, 0)
+		TH_LOG("no large folio in the page cache, the bug is out of reach");
+	ASSERT_LT(fs.smallest, pmd_pages)
+		TH_LOG("only PMD-sized folios in the page cache, the bug is out of reach");
+
+	ASSERT_EQ(file_bytes_on_node(self->cg_fd, self->node_to,
+				     &on_to_before), 0);
+
+	/* From here on a failure means eviction would crash the kernel. */
+	self->leave_mounted = 1;
+	snprintf(list, sizeof(list), "%d,%d", self->node_from, self->node_to);
+	ASSERT_EQ(write_file_at(self->cg_fd, "cpuset.mems", list), 0);
+	ASSERT_EQ(write_u64_at(self->cg_fd, "memory.numa_migrate",
+			       self->node_to), 0);
+
+	for (off = 0; off < FILE_SIZE; off += CHUNK_SIZE) {
+		if (chunk_is_hole(off, pmd_size))
+			continue;
+		memset(buf, (off / CHUNK_SIZE) & 0xff, CHUNK_SIZE);
+		ASSERT_EQ(pread(fd, check, CHUNK_SIZE, off), (ssize_t)CHUNK_SIZE);
+		ASSERT_EQ(memcmp(buf, check, CHUNK_SIZE), 0);
+	}
+
+	/* memory.numa_stat flushes per-cpu counters lazily, give it time. */
+	for (i = 0; i < STAT_RETRIES; i++) {
+		ASSERT_EQ(file_bytes_on_node(self->cg_fd, self->node_to,
+					     &on_to_after), 0);
+		if (on_to_after >= on_to_before + FILE_SIZE)
+			break;
+		usleep(STAT_RETRY_US);
+	}
+	TH_LOG("file bytes on node %d: %llu -> %llu", self->node_to,
+	       on_to_before, on_to_after);
+
+	ASSERT_EQ(scan_file_folios(fd, FILE_SIZE, self->node_to, pmd_size, &fs), 0)
+		TH_LOG("cannot read the page cache folio sizes");
+	TH_LOG("after: on node %d: %s, elsewhere: %s", self->node_to,
+	       folio_hist_str(fs.hist[0], h0, sizeof(h0)),
+	       folio_hist_str(fs.hist[1], h1, sizeof(h1)));
+	ASSERT_GT(fs.on_node, 0)
+		TH_LOG("no folio below the PMD size reached node %d in one piece",
+		       self->node_to);
+	self->leave_mounted = 0;
+
+	free(buf);
+	free(check);
+}
+
+TEST_HARNESS_MAIN

-- 
2.43.0

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v3 1/2] mm/memcontrol: allocate numa_migrate targets of the source folio order

[no-reply]

> memcg_numa_migrate_new_folio() treats every large folio as a PMD-sized
> THP and allocates an HPAGE_PMD_ORDER destination for it. The branch is
> the 4.18 pattern of new_page() in mm/mempolicy.c: a compound page on
> the LRU could only be a THP then, so PageTransHuge() implied
> HPAGE_PMD_ORDER.
> 
> Today the page cache holds large folios of any order (NFS sizes a folio
> to the write, XFS and shmem have them too) and anonymous memory has
> mTHP, while folio_test_large() is still just folio_test_head() and is
> true for any order from 1 up. migrate_pages() never checks that the
> destination order matches the source.
> 
> For an order-1 or order-4 file folio the destination is an order-9
> folio that inherits the source index and page cache reference count,
> so every later page cache operation that sizes the folio by its order
> [ ... ]
> 
> Feature: mm: interface to migrate memory between NUMA nodes upon userspace request
> Reviewed-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
> Reviewed-by: Vasileios Almpanis <vasileios.almpanis@virtuozzo.com>
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261028-b4-prep-khorenko-vstor-151607-numa-migrate-order-v3-0-893494e8db76@virtuozzo.com?part=1


_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v3 2/2] selftests/ve: check memory.numa_migrate keeps the folio order

[no-reply]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Low] selftests/ve: numa_stat poll threshold FILE_SIZE exceeds the data written
- [Low] selftests/ve: test does not compare folio order before and after migration
--

commit d2b3960189f82e4c1867ccecec0c5fbdf30b860e
Author: Konstantin Khorenko <khorenko@virtuozzo.com>

selftests/ve: check memory.numa_migrate keeps the folio order

This commit adds ve_numa_migrate_test. The test writes a file in 64K chunks on an XFS image from a cgroup bound to one NUMA node. It then runs memory.numa_migrate to a second node and compares the file contents. It also checks folio sizes and nodes through pagemap, kpageflags and move_pages().

Link: https://virtuozzo.atlassian.net/browse/VSTOR-151607

> diff --git a/tools/testing/selftests/ve/ve_numa_migrate_test.c b/tools/testing/selftests/ve/ve_numa_migrate_test.c
> new file mode 100644
> --- /dev/null
> +++ b/tools/testing/selftests/ve/ve_numa_migrate_test.c

[ ... ]

> +static void folio_scan_end_run(struct folio_scan *fs, long run, int run_node,
> +			       int node, long pmd_pages)
> +{
> +	int order = 0;
> +
> +	if (!run)
> +		return;
> +	while ((1L << order) < run && order < SCAN_ORDERS - 1)
> +		order++;
> +	fs->hist[run_node != node][order]++;
> +	if (run == 1)
> +		return;
> +	if (!fs->smallest || run < fs->smallest)
> +		fs->smallest = run;
> +	if (run < pmd_pages && run_node == node)
> +		fs->on_node++;

[Severity: Low]
The commit message says the test checks that memory.numa_migrate keeps
the folio order. Is that what this checks?

In folio_scan_end_run(), any folio with 1 < run < pmd_pages that sits
on the target node increments on_node. Nothing compares the order seen
before the migration with the order seen after it.

A regression that gave an order-4 source a destination of a different
non-PMD order, or split it into smaller folios, would still satisfy
the final check:

	ASSERT_GT(fs.on_node, 0)
		TH_LOG("no folio below the PMD size reached node %d in one piece",

The before and after histograms are only printed with TH_LOG.

Could the test compare the two histograms, or could the commit message
be narrowed to say it catches the PMD-sized destination case?

> +}

[ ... ]

> +TEST_F(numa_migrate, large_folio_order)
> +{

[ ... ]

> +	for (off = 0; off < FILE_SIZE; off += CHUNK_SIZE) {
> +		if (chunk_is_hole(off, pmd_size))
> +			continue;
> +		memset(buf, (off / CHUNK_SIZE) & 0xff, CHUNK_SIZE);
> +		ASSERT_EQ(pwrite(fd, buf, CHUNK_SIZE, off), (ssize_t)CHUNK_SIZE);
> +	}

[ ... ]

> +	/* memory.numa_stat flushes per-cpu counters lazily, give it time. */
> +	for (i = 0; i < STAT_RETRIES; i++) {
> +		ASSERT_EQ(file_bytes_on_node(self->cg_fd, self->node_to,
> +					     &on_to_after), 0);
> +		if (on_to_after >= on_to_before + FILE_SIZE)
> +			break;
> +		usleep(STAT_RETRY_US);
> +	}

[Severity: Low]
Can this break ever be taken?

The write loop skips the chunks where chunk_is_hole() is true. That is
the chunks at offsets 0 and 2 MiB. At most FILE_SIZE minus 2 * 64 KiB
(3.875 MiB) of the file's pages can therefore be on node_to.

The threshold is on_to_before + FILE_SIZE (4 MiB). Unless unrelated
file pages of the cgroup, such as XFS metadata or loop backing-file
pages, move to node_to and make up the difference, the loop sleeps for
the full STAT_RETRIES x STAT_RETRY_US, about 5 seconds.

The result is also only printed:

	TH_LOG("file bytes on node %d: %llu -> %llu", self->node_to,
	       on_to_before, on_to_after);

Should the threshold be the number of bytes actually written, or should
the loop be dropped or its outcome asserted?

[ ... ]

-- 
Sashiko AI review · https://sashiko.vzint.dev/#/patchset/20261028-b4-prep-khorenko-vstor-151607-numa-migrate-order-v3-0-893494e8db76@virtuozzo.com?part=2

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel

Re: [Devel] [PATCH hci-8.1 v3 2/2] selftests/ve: check memory.numa_migrate keeps the folio order

[Vladimir Riabchun]

On 10/28/26 18:11, Konstantin Khorenko wrote:
> memcg_numa_migrate_new_folio() used to give every large folio a
> PMD-sized destination. A smaller page cache folio then read back as
> garbage, and its inode died in clear_inode() on eviction.
> 
> The test writes a file on an XFS image in 64K chunks, which gives
> order-4 folios, from a cgroup bound to one NUMA node. It migrates the
> cgroup to a second node with memory.numa_migrate and compares the file
> with the written pattern.
> 
> Two checks make sure the test reaches the bug:
> 
> - before the migration the page cache must hold a large folio below
>    the PMD size, read through /proc/self/pagemap and /proc/kpageflags;
> - after it at least one such folio must sit on the target node in one
>    piece, its node read through move_pages().
> 
> A failed check is a failure, not a skip, so that a setup where the
> test stops reaching the bug shows up. The chunks at 2 MB aligned
> offsets are not written: a folio there reads right even from an
> oversized destination.
> 
> On an unfixed kernel the comparison fails and the test leaves the
> image mounted, because an eviction would crash the node.
> 
> Needs root, two NUMA nodes with memory (numa=fake=2 is enough),
> cgroup2 with the memory and cpuset controllers, and mkfs.xfs; skips
> otherwise.
> 
> https://virtuozzo.atlassian.net/browse/VSTOR-151607
> 
> Feature: ve selftests
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
> ---
>   tools/testing/selftests/ve/Makefile               |   1 +
>   tools/testing/selftests/ve/ve_numa_migrate_test.c | 494 ++++++++++++++++++++++

Please, update gitignore as well with test executable.

>   2 files changed, 495 insertions(+)
> 
> diff --git a/tools/testing/selftests/ve/Makefile b/tools/testing/selftests/ve/Makefile
> index 51f9316beb99b..e96320a183844 100644
> --- a/tools/testing/selftests/ve/Makefile
> +++ b/tools/testing/selftests/ve/Makefile
> @@ -6,6 +6,7 @@ TEST_GEN_PROGS += ve_ns_owner_test
>   TEST_GEN_PROGS += ve_perms_test
>   TEST_GEN_PROGS += ve_mount_accounting_test
>   TEST_GEN_PROGS += ve_vtty_stress
> +TEST_GEN_PROGS += ve_numa_migrate_test
>   
>   # CPUID faulting is x86-64 only. ARCH says "x86" for 32-bit too and lib.mk
>   # picks the clang target by its emptiness, so decide by the compiler.
> diff --git a/tools/testing/selftests/ve/ve_numa_migrate_test.c b/tools/testing/selftests/ve/ve_numa_migrate_test.c
> new file mode 100644
> index 0000000000000..2c6499195b9ba
> --- /dev/null
> +++ b/tools/testing/selftests/ve/ve_numa_migrate_test.c
> @@ -0,0 +1,494 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * memory.numa_migrate must give a large page cache folio a destination of
> + * the same order. An oversized destination inherits the index and the
> + * reference count of the source, only the source pages are copied into it,
> + * a read of the folio is served from the destination by file offset and
> + * lands in its uninitialised part, and the inode dies in clear_inode() on
> + * eviction.
> + *
> + * Write a file on an XFS image (XFS allocates large folios for 64K writes)
> + * from a cgroup bound to one NUMA node, migrate the cgroup to another node
> + * and check that the data reads back intact.
> + *
> + * Needs root, two NUMA nodes with memory (numa=fake=2 will do), cgroup2
> + * with the memory and cpuset controllers, and mkfs.xfs. The folio sizes
> + * are read through /proc/self/pagemap and /proc/kpageflags before the
> + * migration, and the node of every page through move_pages() after it: a
> + * file without large folios below the PMD size, or a migration that
> + * moved none of them in one piece, fails the test, as it would pass
> + * without touching the bug.
> + */
> +#define _GNU_SOURCE
> +#include <stdbool.h>
> +#include <stdint.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <unistd.h>
> +#include <sched.h>
> +#include <fcntl.h>
> +#include <errno.h>
> +#include <sys/stat.h>
> +#include <sys/wait.h>
> +#include <sys/mount.h>
> +#include <sys/mman.h>
> +#include <sys/syscall.h>
> +#include <linux/sched.h>
> +#include <linux/kernel-page-flags.h>
> +
> +#include "../kselftest_harness.h"
> +#include "ve_selftest.h"
> +
> +#define CG_NAME		"ve_numa_migrate_test"
> +#define DIR_TEMPLATE	"/var/tmp/ve_numa_migrate.XXXXXX"
> +/* mkfs.xfs refuses anything smaller; the image is sparse. */
> +#define IMG_SIZE	(320UL << 20)
> +#define FILE_SIZE	(4UL << 20)
> +#define CHUNK_SIZE	(64UL << 10)
> +#define PM_PFN_MASK	((1ULL << 55) - 1)
> +#ifndef MADV_POPULATE_READ
> +#define MADV_POPULATE_READ	22
> +#endif
> +#define PM_PRESENT	(1ULL << 63)
> +/* The periodic memcg stats flush runs every two seconds. */
> +#define STAT_RETRIES	50
> +#define STAT_RETRY_US	100000
> +
> +FIXTURE(numa_migrate)
> +{
> +	int cgv2_fd;
> +	int cg_fd;
> +	int node_from;
> +	int node_to;
> +	char dir[64];
> +	char img[80];
> +	char mnt[80];
> +	int mounted;
> +	int leave_mounted;
> +	int data_fd;
> +};
> +
> +/* First two node ids of a sysfs node list such as "0-1" or "0,2-3". */
> +static int two_nodes_with_memory(int *from, int *to)
> +{
> +	char buf[256], *tok, *save;
> +	int found = 0;
> +
> +	if (read_file_at(AT_FDCWD, "/sys/devices/system/node/has_memory",
> +			 buf, sizeof(buf)) <= 0)
> +		return -1;
> +
> +	for (tok = strtok_r(buf, ",\n", &save); tok && found < 2;
> +	     tok = strtok_r(NULL, ",\n", &save)) {
> +		int a, b, n;
> +
> +		if (sscanf(tok, "%d-%d", &a, &b) != 2) {
> +			if (sscanf(tok, "%d", &a) != 1)
> +				return -1;
> +			b = a;
> +		}
> +		for (n = a; n <= b && found < 2; n++) {
> +			if (found == 0)
> +				*from = n;
> +			else
> +				*to = n;
> +			found++;
> +		}
> +	}
> +	return found == 2 ? 0 : -1;
> +}
> +
> +static int run_cmd(const char *fmt, ...)
> +{
> +	char cmd[256];
> +	va_list ap;
> +	int status;
> +
> +	va_start(ap, fmt);
> +	vsnprintf(cmd, sizeof(cmd), fmt, ap);
> +	va_end(ap);
> +
> +	status = system(cmd);
> +	if (status < 0 || !WIFEXITED(status))
> +		return -1;
> +	return WEXITSTATUS(status);
> +}
> +
> +/* Bytes of file pages the cgroup has on @node, from memory.numa_stat. */
> +static int file_bytes_on_node(int cg_fd, int node, unsigned long long *out)
> +{
> +	static char buf[32768];
> +	char key[16], *p;
> +
> +	if (read_file_at(cg_fd, "memory.numa_stat", buf, sizeof(buf)) <= 0)
> +		return -1;
> +	p = strstr(buf, "\nfile ");
> +	if (!p)
> +		return -1;
> +	snprintf(key, sizeof(key), " N%d=", node);
> +	p = strstr(p, key);
> +	if (!p)
> +		return -1;
> +	*out = strtoull(p + strlen(key), NULL, 10);
> +	return 0;
> +}
> +
> +static unsigned long page_size;
> +
> +/* A folio at a PMD-aligned index reads right from an oversized one. */
> +static bool chunk_is_hole(unsigned long off, unsigned long pmd_size)
> +{
> +	return !((off / CHUNK_SIZE * CHUNK_SIZE) % pmd_size);
> +}
> +
> +#define SCAN_ORDERS	16
> +
> +struct folio_scan {
> +	long large;	/* pages in large folios */
> +	long smallest;	/* pages in the smallest large folio */
> +	long on_node;	/* large folios below @pmd_pages that sit on @node */
> +	long hist[2][SCAN_ORDERS];	/* folios by order, on @node / elsewhere */
> +};
> +
> +static void folio_scan_end_run(struct folio_scan *fs, long run, int run_node,
> +			       int node, long pmd_pages)
> +{
> +	int order = 0;
> +
> +	if (!run)
> +		return;
> +	while ((1L << order) < run && order < SCAN_ORDERS - 1)
> +		order++;
> +	fs->hist[run_node != node][order]++;
> +	if (run == 1)
> +		return;
> +	if (!fs->smallest || run < fs->smallest)
> +		fs->smallest = run;
> +	if (run < pmd_pages && run_node == node)
> +		fs->on_node++;
> +}
> +
> +/* "64 x 64K, 3 x 4K" or "none" */
> +static const char *folio_hist_str(const long *hist, char *buf, size_t len)
> +{
> +	size_t pos = 0;
> +	int order;
> +
> +	for (order = SCAN_ORDERS - 1; order >= 0; order--) {
> +		if (!hist[order])
> +			continue;
> +		pos += snprintf(buf + pos, len - pos, "%s%ld x %luK",
> +				pos ? ", " : "", hist[order],
> +				(page_size << order) / 1024);
> +		if (pos >= len)
> +			break;
> +	}
> +	return pos ? buf : "none";
> +}
> +
> +/*
> + * Walk the page cache of @fd through /proc/self/pagemap, /proc/kpageflags
> + * and move_pages(2) in query mode, leaving the holes alone: faulting them
> + * in would add folios to the file. Returns -1 when that is not possible.
> + */
> +static int scan_file_folios(int fd, unsigned long size, int node,
> +			    unsigned long pmd_size, struct folio_scan *fs)
> +{
> +	unsigned long nr = size / page_size, i, off;
> +	long pmd_pages = pmd_size / page_size;
> +	long run = 0;
> +	int run_node = -1, ret = -1;
> +	int pagemap, kpageflags, *status;
> +	unsigned char *map;
> +	void **pages;
> +	uint64_t *pm;
> +
> +	memset(fs, 0, sizeof(*fs));
> +	pagemap = open("/proc/self/pagemap", O_RDONLY);
> +	kpageflags = open("/proc/kpageflags", O_RDONLY);
> +	pm = malloc(nr * sizeof(*pm));
> +	pages = malloc(nr * sizeof(*pages));
> +	status = malloc(nr * sizeof(*status));
> +	map = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
> +	if (pagemap < 0 || kpageflags < 0 || !pm || !pages || !status ||
> +	    map == MAP_FAILED)
> +		goto out;
> +	for (off = 0; off < size; off += CHUNK_SIZE) {
> +		if (!chunk_is_hole(off, pmd_size) &&
> +		    madvise(map + off, CHUNK_SIZE, MADV_POPULATE_READ))
> +			goto out;
> +	}
> +	if (pread(pagemap, pm, nr * sizeof(*pm),
> +		  (unsigned long)map / page_size * sizeof(*pm)) !=
> +	    (ssize_t)(nr * sizeof(*pm)))
> +		goto out;
> +	for (i = 0; i < nr; i++)
> +		pages[i] = map + i * page_size;
> +	if (syscall(__NR_move_pages, 0, nr, pages, NULL, status, 0))
> +		goto out;
> +	for (i = 0; i < nr; i++) {
> +		uint64_t flags, pfn = pm[i] & PM_PFN_MASK;
> +
> +		if (chunk_is_hole(i * page_size, pmd_size)) {
> +			folio_scan_end_run(fs, run, run_node, node, pmd_pages);
> +			run = 0;
> +			continue;
> +		}
> +		if (!(pm[i] & PM_PRESENT) || !pfn || status[i] < 0 ||
> +		    pread(kpageflags, &flags, sizeof(flags),
> +			  pfn * sizeof(flags)) != sizeof(flags))
> +			goto out;
> +		if (flags & (1ULL << KPF_COMPOUND_HEAD)) {
> +			folio_scan_end_run(fs, run, run_node, node, pmd_pages);
> +			fs->large++;
> +			run = 1;
> +			run_node = status[i];
> +		} else if (flags & (1ULL << KPF_COMPOUND_TAIL)) {
> +			fs->large++;
> +			run++;
> +		} else {
> +			folio_scan_end_run(fs, run, run_node, node, pmd_pages);
> +			folio_scan_end_run(fs, 1, status[i], node, pmd_pages);
> +			run = 0;
> +		}
> +	}
> +	folio_scan_end_run(fs, run, run_node, node, pmd_pages);
> +	ret = 0;
> +out:
> +	if (map != MAP_FAILED)
> +		munmap(map, size);
> +	free(status);
> +	free(pages);
> +	free(pm);
> +	if (pagemap >= 0)
> +		close(pagemap);
> +	if (kpageflags >= 0)
> +		close(kpageflags);
> +	return ret;
> +}
> +
> +static void cleanup(FIXTURE_DATA(numa_migrate) *self)
> +{
> +	char path[PATH_MAX];
> +
> +	if (self->data_fd >= 0) {
> +		close(self->data_fd);
> +		self->data_fd = -1;
> +	}
> +	if (self->dir[0]) {
> +		snprintf(path, sizeof(path), "%s/data", self->mnt);
> +		unlink(path);
> +		if (self->mounted)
> +			umount(self->mnt);
> +		unlink(self->img);
> +		rmdir(self->mnt);
> +		rmdir(self->dir);
> +		self->dir[0] = '\0';
> +	}
> +	if (self->cg_fd >= 0) {
> +		close(self->cg_fd);
> +		self->cg_fd = -1;
> +	}
> +	if (self->cgv2_fd >= 0) {
> +		/* The test task may still sit in the cgroup after a failure. */
> +		write_u64_at(self->cgv2_fd, "cgroup.procs", getpid());
> +		unlinkat(self->cgv2_fd, CG_NAME, AT_REMOVEDIR);
> +		close(self->cgv2_fd);
> +		self->cgv2_fd = -1;
> +	}
> +}
> +
> +/*
> + * Returns 0, or 1 with everything undone when the host cannot run the test.
> + * The harness runs no teardown after a failed setup, so the undo is here.
> + */
> +static int prepare(FIXTURE_DATA(numa_migrate) *self, const char **why)
> +{
> +	int fd;
> +
> +	self->cgv2_fd = mount_cg2_fd();
> +	if (self->cgv2_fd < 0) {
> +		*why = "cannot mount cgroup2";
> +		goto fail;
> +	}
> +	if (write_file_at(self->cgv2_fd, "cgroup.subtree_control",
> +			  "+memory +cpuset")) {
> +		*why = "cannot enable the memory and cpuset controllers";
> +		goto fail;
> +	}
> +	if (mkdirat(self->cgv2_fd, CG_NAME, 0755) && errno != EEXIST) {
> +		*why = "cannot create the test cgroup";
> +		goto fail;
> +	}
> +	self->cg_fd = openat(self->cgv2_fd, CG_NAME, O_PATH | O_DIRECTORY);
> +	if (self->cg_fd < 0) {
> +		*why = "cannot open the test cgroup";
> +		goto fail;
> +	}
> +	if (faccessat(self->cg_fd, "memory.numa_migrate", W_OK, 0)) {
> +		*why = "no memory.numa_migrate";
> +		goto fail;
> +	}
> +	if (write_u64_at(self->cg_fd, "cpuset.mems", self->node_from)) {
> +		*why = "cannot bind the cgroup to a node";
> +		goto fail;
> +	}
> +
> +	snprintf(self->dir, sizeof(self->dir), DIR_TEMPLATE);
> +	if (!mkdtemp(self->dir)) {
> +		self->dir[0] = '\0';
> +		*why = "cannot create a temporary directory";
> +		goto fail;
> +	}
> +	snprintf(self->img, sizeof(self->img), "%s/img", self->dir);
> +	snprintf(self->mnt, sizeof(self->mnt), "%s/mnt", self->dir);
> +	if (mkdir(self->mnt, 0755)) {
> +		*why = "cannot create the mount point";
> +		goto fail;
> +	}
> +	fd = open(self->img, O_CREAT | O_RDWR, 0600);
> +	if (fd < 0) {
> +		*why = "cannot create the image file";
> +		goto fail;
> +	}
> +	if (ftruncate(fd, IMG_SIZE)) {
> +		close(fd);
> +		*why = "cannot size the image file";
> +		goto fail;
> +	}
> +	close(fd);
> +	if (run_cmd("mkfs.xfs -q -f %s", self->img)) {
> +		*why = "mkfs.xfs failed";
> +		goto fail;
> +	}
> +	if (run_cmd("mount -o loop %s %s", self->img, self->mnt)) {
> +		*why = "cannot loop-mount the XFS image";
> +		goto fail;
> +	}
> +	self->mounted = 1;
> +	return 0;
> +fail:
> +	cleanup(self);
> +	return 1;
> +}
> +
> +FIXTURE_SETUP(numa_migrate)
> +{
> +	const char *why;
> +
> +	self->cgv2_fd = -1;
> +	self->cg_fd = -1;
> +	self->data_fd = -1;
> +	self->dir[0] = '\0';
> +	page_size = sysconf(_SC_PAGESIZE);
> +
> +	if (geteuid())
> +		SKIP(return, "needs root");
> +	if (two_nodes_with_memory(&self->node_from, &self->node_to))
> +		SKIP(return, "needs two NUMA nodes with memory");
> +	if (prepare(self, &why))
> +		SKIP(return, "%s", why);
> +}
> +
> +FIXTURE_TEARDOWN(numa_migrate)
> +{
> +	if (self->leave_mounted) {
> +		TH_LOG("leaving %s mounted, eviction would hit BUG_ON(nrpages)",
> +		       self->mnt);
> +		self->dir[0] = '\0';
> +	}
> +	cleanup(self);
> +}
> +
> +TEST_F(numa_migrate, large_folio_order)
> +{
> +	unsigned long long on_to_before, on_to_after;
> +	unsigned char *buf, *check;
> +	char path[PATH_MAX], list[64];
> +	unsigned long long pmd_size;
> +	struct folio_scan fs;
> +	char h0[128], h1[128];
> +	long pmd_pages;
> +	unsigned long off;
> +	int fd, i;
> +
> +	ASSERT_EQ(write_u64_at(self->cg_fd, "cgroup.procs", getpid()), 0);
> +
> +	buf = malloc(CHUNK_SIZE);
> +	check = malloc(CHUNK_SIZE);
> +	ASSERT_NE(buf, NULL);
> +	ASSERT_NE(check, NULL);
> +
> +	/* The old code allocated PMD-sized destinations: those were fine. */
> +	if (read_u64_at(AT_FDCWD,
> +			"/sys/kernel/mm/transparent_hugepage/hpage_pmd_size",
> +			&pmd_size))
> +		pmd_size = 2UL << 20;
> +	pmd_pages = pmd_size / page_size;
> +
> +	snprintf(path, sizeof(path), "%s/data", self->mnt);
> +	fd = open(path, O_CREAT | O_RDWR | O_TRUNC, 0600);
> +	ASSERT_GE(fd, 0);
> +	self->data_fd = fd;
> +	for (off = 0; off < FILE_SIZE; off += CHUNK_SIZE) {
> +		if (chunk_is_hole(off, pmd_size))
> +			continue;
> +		memset(buf, (off / CHUNK_SIZE) & 0xff, CHUNK_SIZE);
> +		ASSERT_EQ(pwrite(fd, buf, CHUNK_SIZE, off), (ssize_t)CHUNK_SIZE);
> +	}
> +	ASSERT_EQ(fsync(fd), 0);
> +	ASSERT_EQ(scan_file_folios(fd, FILE_SIZE, self->node_to, pmd_size, &fs), 0)
> +		TH_LOG("cannot read the page cache folio sizes");
> +	TH_LOG("before: on node %d: %s, elsewhere: %s", self->node_to,
> +	       folio_hist_str(fs.hist[0], h0, sizeof(h0)),
> +	       folio_hist_str(fs.hist[1], h1, sizeof(h1)));
> +	ASSERT_GT(fs.large, 0)
> +		TH_LOG("no large folio in the page cache, the bug is out of reach");
> +	ASSERT_LT(fs.smallest, pmd_pages)
> +		TH_LOG("only PMD-sized folios in the page cache, the bug is out of reach");
> +
> +	ASSERT_EQ(file_bytes_on_node(self->cg_fd, self->node_to,
> +				     &on_to_before), 0);
> +
> +	/* From here on a failure means eviction would crash the kernel. */
> +	self->leave_mounted = 1;
> +	snprintf(list, sizeof(list), "%d,%d", self->node_from, self->node_to);
> +	ASSERT_EQ(write_file_at(self->cg_fd, "cpuset.mems", list), 0);
> +	ASSERT_EQ(write_u64_at(self->cg_fd, "memory.numa_migrate",
> +			       self->node_to), 0);
> +
> +	for (off = 0; off < FILE_SIZE; off += CHUNK_SIZE) {
> +		if (chunk_is_hole(off, pmd_size))
> +			continue;
> +		memset(buf, (off / CHUNK_SIZE) & 0xff, CHUNK_SIZE);
> +		ASSERT_EQ(pread(fd, check, CHUNK_SIZE, off), (ssize_t)CHUNK_SIZE);
> +		ASSERT_EQ(memcmp(buf, check, CHUNK_SIZE), 0);
> +	}
> +
> +	/* memory.numa_stat flushes per-cpu counters lazily, give it time. */
> +	for (i = 0; i < STAT_RETRIES; i++) {
> +		ASSERT_EQ(file_bytes_on_node(self->cg_fd, self->node_to,
> +					     &on_to_after), 0);
> +		if (on_to_after >= on_to_before + FILE_SIZE)
> +			break;
> +		usleep(STAT_RETRY_US);
> +	}
> +	TH_LOG("file bytes on node %d: %llu -> %llu", self->node_to,
> +	       on_to_before, on_to_after);
> +
> +	ASSERT_EQ(scan_file_folios(fd, FILE_SIZE, self->node_to, pmd_size, &fs), 0)
> +		TH_LOG("cannot read the page cache folio sizes");
> +	TH_LOG("after: on node %d: %s, elsewhere: %s", self->node_to,
> +	       folio_hist_str(fs.hist[0], h0, sizeof(h0)),
> +	       folio_hist_str(fs.hist[1], h1, sizeof(h1)));
> +	ASSERT_GT(fs.on_node, 0)
> +		TH_LOG("no folio below the PMD size reached node %d in one piece",
> +		       self->node_to);
> +	self->leave_mounted = 0;
> +
> +	free(buf);
> +	free(check);
> +}
> +
> +TEST_HARNESS_MAIN
> 

-- 
Best regards, Riabchun Vladimir
Linux Kernel Developer, Virtuozzo

_______________________________________________
Devel mailing list
Devel@openvz.org
https://lists.openvz.org/mailman/listinfo/devel