NUMA Balancing: add page promotion counter

Patch series "NUMA balancing: optimize memory placement for memory tiering system", v13

With the advent of various new memory types, some machines will have
multiple types of memory, e.g.  DRAM and PMEM (persistent memory).  The
memory subsystem of these machines can be called memory tiering system,
because the performance of the different types of memory are different.

After commit b62f840a5a6f ("device-dax: "Hotplug" persistent memory for
use like normal RAM"), the PMEM could be used as the cost-effective
volatile memory in separate NUMA nodes.  In a typical memory tiering
system, there are CPUs, DRAM and PMEM in each physical NUMA node.  The
CPUs and the DRAM will be put in one logical node, while the PMEM will
be put in another (faked) logical node.

To optimize the system overall performance, the hot pages should be
placed in DRAM node.  To do that, we need to identify the hot pages in
the PMEM node and migrate them to DRAM node via NUMA migration.

In the original NUMA balancing, there are already a set of existing
mechanisms to identify the pages recently accessed by the CPUs in a node
and migrate the pages to the node.  So we can reuse these mechanisms to
build the mechanisms to optimize the page placement in the memory
tiering system.  This is implemented in this patchset.

At the other hand, the cold pages should be placed in PMEM node.  So, we
also need to identify the cold pages in the DRAM node and migrate them
to PMEM node.

In commit a1ca6f335d1c ("mm/migrate: demote pages during reclaim"), a
mechanism to demote the cold DRAM pages to PMEM node under memory
pressure is implemented.  Based on that, the cold DRAM pages can be
demoted to PMEM node proactively to free some memory space on DRAM node
to accommodate the promoted hot PMEM pages.  This is implemented in this
patchset too.

We have tested the solution with the pmbench memory accessing benchmark
with the 80:20 read/write ratio and the Gauss access address
distribution on a 2 socket Intel server with Optane DC Persistent Memory
Model.  The test results shows that the pmbench score can improve up to
95.9%.

This patch (of 3):

In a system with multiple memory types, e.g.  DRAM and PMEM, the CPU
and DRAM in one socket will be put in one NUMA node as before, while
the PMEM will be put in another NUMA node as described in the
description of the commit b62f840a5a6f ("device-dax: "Hotplug"
persistent memory for use like normal RAM").  So, the NUMA balancing
mechanism will identify all PMEM accesses as remote access and try to
promote the PMEM pages to DRAM.

To distinguish the number of the inter-type promoted pages from that of
the inter-socket migrated pages.  A new vmstat count is added.  The
counter is per-node (count in the target node).  So this can be used to
identify promotion imbalance among the NUMA nodes.

Link: https://lkml.kernel.org/r/20220301085329.3210428-1-ying.huang@intel.com
Link: https://lkml.kernel.org/r/20220221084529.1052339-1-ying.huang@intel.com
Link: https://lkml.kernel.org/r/20220221084529.1052339-2-ying.huang@intel.com
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Reviewed-by: Yang Shi <shy828301@gmail.com>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: zhongjiang-ali <zhongjiang-ali@linux.alibaba.com>
Cc: Feng Tang <feng.tang@intel.com>
Cc: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index c9e6a50109b99e58de09633b189212e0bd46932d..310b6e7ce58aa30ee43deb98eca108c738135fc0 100644 (file)
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -221,6 +221,9 @@ enum node_stat_item {
        NR_PAGETABLE,           /* used for pagetables */
 #ifdef CONFIG_SWAP
        NR_SWAPCACHE,
+#endif
+#ifdef CONFIG_NUMA_BALANCING
+       PGPROMOTE_SUCCESS,      /* promote successfully */
 #endif
        NR_VM_NODE_STAT_ITEMS
 };

diff --git a/include/linux/node.h b/include/linux/node.h
index bb21fd631b1621110953fd4dffebc43766b5ca00..81bbf1c0afd37553f95586ea0c607fac16130e40 100644 (file)
--- a/include/linux/node.h
+++ b/include/linux/node.h
@@ -181,4 +181,9 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,
 
 #define to_node(device) container_of(device, struct node, dev)
 
+static inline bool node_is_toptier(int node)
+{
+       return node_state(node, N_CPU);
+}
+
 #endif /* _LINUX_NODE_H_ */

diff --git a/mm/migrate.c b/mm/migrate.c
index c0d16f050fec2ba2a5f744ece3c3421ab71482ea..dc4adf9792018d6fa4702e5ff2d6776ec1d75ebb 100644 (file)
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2069,6 +2069,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,
        pg_data_t *pgdat = NODE_DATA(node);
        int isolated;
        int nr_remaining;
+       unsigned int nr_succeeded;
        LIST_HEAD(migratepages);
        new_page_t *new;
        bool compound;
@@ -2107,7 +2108,8 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,
 
        list_add(&page->lru, &migratepages);
        nr_remaining = migrate_pages(&migratepages, *new, NULL, node,
-                                    MIGRATE_ASYNC, MR_NUMA_MISPLACED, NULL);
+                                    MIGRATE_ASYNC, MR_NUMA_MISPLACED,
+                                    &nr_succeeded);
        if (nr_remaining) {
                if (!list_empty(&migratepages)) {
                        list_del(&page->lru);
@@ -2116,8 +2118,13 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,
                        putback_lru_page(page);
                }
                isolated = 0;
-       } else
-               count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_pages);
+       }
+       if (nr_succeeded) {
+               count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+               if (!node_is_toptier(page_to_nid(page)) && node_is_toptier(node))
+                       mod_node_page_state(pgdat, PGPROMOTE_SUCCESS,
+                                           nr_succeeded);
+       }
        BUG_ON(!list_empty(&migratepages));
        return isolated;
 

diff --git a/mm/vmstat.c b/mm/vmstat.c
index 4057372745d04755629ae1f52923f537bac42e03..846b670dd346a6bbee0d16ff9423ddaa4829e551 100644 (file)
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1242,6 +1242,9 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_SWAP
        "nr_swapcached",
 #endif
+#ifdef CONFIG_NUMA_BALANCING
+       "pgpromote_success",
+#endif
 
        /* enum writeback_stat_item counters */
        "nr_dirty_threshold",