We found that when we turned off ConstrainKmemSpace=no in the slurm configuration there were no crashes over the weekend. Therefore it appears that the kmem cg is strongly implicated as the source of the problem. I asked LLNL's kernel engineer to apply a44cb9449182fd7b25bf5f1cc38b7f19e0b96f6d which could deal with at least part of the problem. ref: https://bugzilla.redhat.com/show_bug.cgi?id=1442618

From the reporter:

I did some testing on a 7.4 node today, and I was able to trivially recreate the behavior where setting a kmem limit in a cgroup causes a number of slab caches to get created that don't go away when the cgroup is removed.  The steps were,
 # mkdir /sys/fs/cgroup/memory/kmem_test
 # echo 137438953472 > /sys/fs/cgroup/memory/kmem_test/memory.kmem.max_usage_in_bytes (128GB, total RAM in the node)
 # Spawn a shell and echo its pid into /sys/fs/cgroup/memory/kmem_test/tasks to put the shell into the cgroup.
 # Do a little work in the shell. Pinging the management node seemed to be enough.
 # 'ls /sys/kernel/slab | grep kmem_test' will show a bunch of new slab caches
 # Exit the shell.  When I tried this, it took the shell 30-60 seconds to exit.  Not sure what was up there.
 # rmdir /sys/fs/cgroup/memory/kmem_test
 # 'ls /sys/kernel/slab | grep kmem_test' shows the slabs are still there.

No panics/messages to syslog though, and no broken symlinks in /sys/kernel/slab.  I'm not rightly certain whether all of the child slab caches are supposed to go away when the cgroup is removed or not, but letting them build up into the hundreds of thousands probably isn't good behavior.

----------

Is that the expected behavior or do we have our reproducer here? What is supposed to reap these now abandoned slab caches? It seems like the build up of these slab caches is what ultimately crashes the node. So the reproducer that ultimately leads to the crash is probably something like what is described above but iterated many times.

Good morning,

I hope you don't mind if I butt in.  We have a cluster of 80 Intel OPA/PSM2 nodes running 7.4, slurm 17.04 with memory enforcement and with hfi1 drivers updated to Intel's latest 10.6.x.x that's experiencing similar issues. E.g. - after a while lots of:

[684578.150960] cache_from_obj: Wrong slab cache. kmem_cache_node but object is from kmalloc-64(6281:step_0)
[684578.162326] cache_from_obj: Wrong slab cache. kmem_cache_node but object is from kmalloc-64(6281:step_0)
[684578.173546] cache_from_obj: Wrong slab cache. kmem_cache but object is from kmalloc-256(6281:step_0)
[684578.186536] cache_from_obj: Wrong slab cache. kmem_cache_node but object is from kmalloc-64(6281:step_0)
[684578.197869] cache_from_obj: Wrong slab cache. kmem_cache_node but object is from kmalloc-64(6281:step_0)
[684578.210447] cache_from_obj: Wrong slab cache. kmem_cache but object is from kmalloc-256(6281:step_0)

in logs and we see huge /sys/kernel/slab dirs (e.g. 120k entries). No node crashes but we do get weird program crashes that we cannot easily explain any other way. 

Anyway, I just wanted to report that we tried applying a44cb9449182fd7b25bf5f1cc38b7f19e0b96f6d to 3.10.0-693.5.2.el7 and while it does not fix the problem fully it does help.  We ran repeatedly a simple MPI program (report rank) and on node without patch
ls /sys/kernel/slab|grep step|wc -l
grew at twice the rate vs the node with patch (after an hour 2043 vs 965). We'd be happy to try more things if that might help.

Thanks,

Josko

Hi,

Can you both try this patch and report back it's effect on the issue:

commit f57947d27711451a7739a25bba6cddc8a385e438
Author: Li Zefan <lizefan>
Date:   Tue Jun 18 18:41:53 2013 +0800

    cgroup: fix memory leak in cgroup_rm_cftypes()
    
    The memory allocated in cgroup_add_cftypes() should be freed. The
    effect of this bug is we leak a bit memory everytime we unload
    cfq-iosched module if blkio cgroup is enabled.

I was advised the above might help.

Travis

Travis, before we ask LLNL to test random patches. Can we do some testing to make sure that we don't continue to see evidence of the problem using the reproducer provided?

Good afternoon,

just finished testing also with the 2nd patch and there isn't a huge difference.  After 60 MPI runs I get 3175 items in /sys/kernel/slab on unpatched, 1285 with first patch, 1290 with both.  I also kept list of /sys/kernel/slab dir after every test - if you want to compare what's happening - and can put it up somewhere.

Thanks for the suggestion - if it can help keep'em coming.

encountered this issue with nvidia-docker(cgroup open) and 3.10.0-514.el7.x86_64 on centos

[6728212.703168]  [<ffffffff811c35c4>] __kmem_cache_shutdown+0x14/0x80
[6728212.703196]  [<ffffffff8118ceaf>] kmem_cache_destroy+0x3f/0xe0
[6728212.703222]  [<ffffffff811d4949>] kmem_cache_destroy_memcg_children+0x89/0xb0
[6728212.703252]  [<ffffffff8118ce84>] kmem_cache_destroy+0x14/0xe0
[6728212.703284]  [<ffffffffa167a507>] deinit_chunk_split_cache+0x77/0xa0 [nvidia_uvm]
[6728212.703321]  [<ffffffffa167c11e>] uvm_pmm_gpu_deinit+0x3e/0x70 [nvidia_uvm]
[6728212.703355]  [<ffffffffa1650b10>] remove_gpu+0x220/0x300 [nvidia_uvm]
[6728212.703387]  [<ffffffffa1650e31>] uvm_gpu_release_locked+0x21/0x30 [nvidia_uvm]
[6728212.703423]  [<ffffffffa1654588>] uvm_va_space_destroy+0x348/0x3b0 [nvidia_uvm]
[6728212.703458]  [<ffffffffa164a401>] uvm_release+0x11/0x20 [nvidia_uvm]
[6728212.703486]  [<ffffffff811e0329>] __fput+0xe9/0x270
[6728212.703509]  [<ffffffff811e05ee>] ____fput+0xe/0x10
[6728212.703532]  [<ffffffff810a22f4>] task_work_run+0xc4/0xe0
[6728212.703557]  [<ffffffff810815eb>] do_exit+0x2cb/0xa60
[6728212.703580]  [<ffffffff810e1f22>] ? __unqueue_futex+0x32/0x70
[6728212.703605]  [<ffffffff810e2f7d>] ? futex_wait+0x11d/0x280
[6728212.704619]  [<ffffffff81081dff>] do_group_exit+0x3f/0xa0
[6728212.705598]  [<ffffffff81092c10>] get_signal_to_deliver+0x1d0/0x6d0
[6728212.706559]  [<ffffffff81014417>] do_signal+0x57/0x6c0
[6728212.707495]  [<ffffffff8110b796>] ? __audit_syscall_exit+0x1e6/0x280
[6728212.708407]  [<ffffffff81014adf>] do_notify_resume+0x5f/0xb0

Ben,

Can you please try the attached patches (non obsolete)?
The following Linus' commits are included:

 363a044 memcg, slab: kmem_cache_create_memcg(): fix memleak on fail path
 1aa1325 memcg, slab: clean up memcg cache initialization/destruction
 2edefe1 memcg, slab: fix races in per-memcg cache creation/destruction
 b852990 memcg, slab: do not destroy children caches if parent has aliases

Hullo,

I am not Ben but I gave it a try nevertheless.  Patches apply cleanly to 693.17.1 kernel but don't seem to help at all.  Number of entries in /sys/kernel/slab increases at almost exactly the same rate as for clean 693.17.1 kernel.

Josko

There are a few reports of the same thing happening when running Docker here: https://github.com/moby/moby/issues/35446

Kernel versions reported there are all different, but they look recent.

#!/bin/bash 
mkdir -p pages
for x in `seq 1280000`; do
        [ $((x % 1000)) -eq 0 ] && echo $x
        mkdir /sys/fs/cgroup/memory/foo
        # echo 1M > /sys/fs/cgroup/memory/foo/memory.limit_in_bytes
        echo 100M > /sys/fs/cgroup/memory/foo/memory.kmem.limit_in_bytes
        echo $$ >/sys/fs/cgroup/memory/foo/cgroup.procs
        memhog 4K &>/dev/null
        echo trex>pages/$x
        echo $$ >/sys/fs/cgroup/memory/cgroup.procs 
        rmdir /sys/fs/cgroup/memory/foo 
done




[root@localhost ~]# cat /proc/mounts 
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=4077588k,nr_inodes=1019397,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/mapper/centos-root / xfs rw,relatime,attr2,inode64,noquota 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=300,minproto=5,maxproto=5,direct 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
/dev/vda1 /boot xfs rw,relatime,attr2,inode64,noquota 0 0


i encountered same issue use sciript above 

i got this scripit from
https://github.com/torvalds/linux/commit/73f576c04b9410ed19660f74f97521bee6e1c546

and did some changes to reproduce 

"Docker leaking cgroups causing no space left on device? #29638"
https://github.com/moby/moby/issues/29638

unfortunately,encountered  this bug .

on Longterm stable 4.4.121 ,i did not see the same scence.

FYI.

Encountering this same issue with Centos 7, kernel 3.10.0-693.21.1.el7.x86_64, and docker-ce-17.12.1.ce-1.el7.centos.x86_64. The behavior we see is after a period of heavy container usage on hosts, the /sys/kernel/slab dir will have over 1 mil files/dirs, and new containers won't startup anymore. 

Also the following is logged heavily:
May 14 19:50:06 hostname kernel: cache_from_obj: Wrong slab cache. kmalloc-256 but object is from kmem_cache(5071:9934c4c3821bcb474f5a113d5a1822c276fcf196d6c2e31b1b8dd969df9d50d2)
May 14 19:50:06 hostname kernel: cache_from_obj: Wrong slab cache. kmalloc-256 but object is from kmem_cache(5071:9934c4c3821bcb474f5a113d5a1822c276fcf196d6c2e31b1b8dd969df9d50d2)
May 14 19:50:06 hostname kernel: cache_from_obj: Wrong slab cache. kmalloc-256 but object is from kmem_cache(5071:9934c4c3821bcb474f5a113d5a1822c276fcf196d6c2e31b1b8dd969df9d50d2)
May 14 19:50:06 hostname kernel: cache_from_obj: Wrong slab cache. kmalloc-256 but object is from kmem_cache(5071:9934c4c3821bcb474f5a113d5a1822c276fcf196d6c2e31b1b8dd969df9d50d2)

Looks like Kubernetes users are also experiencing this issue, where the recent changes of enabling kmem limit in Kubernetes cause cgroups to leak. For details please refer to https://github.com/kubernetes/kubernetes/issues/61937.

Is this just pending someone testing the patches?

1442618 referenced up near the top is a private bug -- can the relevant parts be copied here (or has that already been done), or is there another bug that has them?

This is biting us and at least one of our customers via the Kubernetes issue linked above. The following script repros it reliably on a Kubernetes node running kernel 3.10.0-862.9.1.el7.x86_64 :

#!/bin/bash
run_some_pods () {
  for j in {1..20}
  do
    kubectl run -i -t busybox-${j} --image=busybox --restart=Never -- echo "hi" &> /dev/null &
  done
  sleep 1
  wait
  for j in {1..20}
  do
    kubectl delete pods busybox-${j} --grace-period=0 --force &> /dev/null || true
  done
}
for i in {1..3279}
do
  echo "Running pod batch $i"
  run_some_pods
done

We'd be happy to test any patches.

We were able to work around the issue in the context of kubernetes 1.8.12 by doing the following on startup. Before running the v1.8 kubelet, we run a v1.6 kubelet with suitable arguments, and verify that it completes enough of its startup process to create /sys/fs/cgroup/memory/kubepods . Then, we start up the kubernetes 1.8.12 cluster as normal.


We tried creating those cgroup folders manually as a simpler version of the above workaround using the following script:

#!/bin/bash

for thing in memory blkio "cpu,cpuacct" cpuset devices freezer hugetlb memory net_cls perf_event systemd
do
   mkdir-p /sys/fs/cgroup/${thing}/kubepods
   mkdir -p /sys/fs/cgroup/${thing}/kubepods/burstable
   mkdir -p /sys/fs/cgroup/${thing}/kubepods/besteffort
done 

However, this simpler attempt did not prevent the repro script above from showing the issue.


What would we need to add to this simpler workaround to prevent the issue as starting a v1.6 kubelet does?

Posting stats from one of the machines that shows this problem:

Problem:
> [root@vd0916 ~]# mkdir /sys/fs/cgroup/memory/b6c438c7ba469d5c3bad89c4a81d732a68a375b096d3791c4847cd517f8eee7c
> mkdir: cannot create directory ‘/sys/fs/cgroup/memory/b6c438c7ba469d5c3bad89c4a81d732a68a375b096d3791c4847cd517f8eee7c’: No space left on device

Some metrics:
> [root@vd0916 ~]# ll /sys/kernel/slab | wc -l 
> 2587635

> [root@vd0916 ~]# slabtop -s -c

 Active / Total Objects (% used)    : 87188577 / 222858888 (39.1%)
 Active / Total Slabs (% used)      : 4716565 / 4716565 (100.0%)
 Active / Total Caches (% used)     : 77 / 118 (65.3%)
 Active / Total Size (% used)       : 17769913.34K / 63916938.41K (27.8%)
 Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
29129856 15635925  53%    0.19K 693568       42   5548544K dentry
23371439 5516568  23%    0.57K 417348       56  13355136K radix_tree_node
22468896 22386411  99%    0.11K 624136       36   2496544K sysfs_dir_cache
19747455 13294931  67%    0.10K 506345       39   2025380K buffer_head
17578139 5788599  32%    0.15K 331663       53   2653304K xfs_ili
15482112  88288   0%    0.25K 241908	   64   3870528K kmalloc-256
11236544  44493   0%    0.50K 175571	   64   5618272K kmalloc-512
11229792 1416428  12%    0.38K 267376       42   4278016K blkdev_requests
11110932 5855107  52%    0.09K 264546       42   1058184K kmalloc-96
9687120 1467586  15%    1.06K 322904	   30  10332928K xfs_inode
9019200 3112167  34%    0.12K 140925	   64   1127400K kmalloc-128
8103040 894521  11%    0.06K 126610	  64    506440K kmalloc-64
7442190 2117485  28%    0.19K 177195	   42   1417560K kmalloc-192
7131390  32130   0%    0.23K 101877	  70   1630032K cfq_queue
4258608 753366  17%    0.66K  88721	  48   2839072K shmem_inode_cache
3663990 2814407  76%    0.58K  66618	   55   2131776K inode_cache
3575296 382122  10%    0.01K   6983	 512     27932K kmalloc-8
3378048 2483023  73%    0.03K  26391	  128    105564K kmalloc-32
1172800 1168793  99%    0.06K  18325	   64     73300K kmem_cache_node
660992 152980  23%    0.02K   2582	256     10328K kmalloc-16
588736 587894  99%    0.25K   9199	 64    147184K kmem_cache
570556 336689  59%    0.64K  11644	 49    372608K proc_inode_cache
527904  28209   5%    1.00K  16497	 32    527904K kmalloc-1024
308160   6622   2%    1.56K  15408	 20    493056K mm_struct
225215   4247   1%    1.03K   7265	 31    232480K nfs_inode_cache
201952 133963  66%    4.00K  25244        8    807808K kmalloc-4096
132600   3240   2%    0.39K   3315	 40     53040K xfs_efd_item
123368 123368 100%    0.07K   2203	 56	 8812K Acpi-ParseExt
110048  37821  34%    2.00K   6878	 16    220096K kmalloc-2048
 71055   3090   4%    2.06K   4737	 15    151584K idr_layer_cache
 51684  50881  98%    0.05K    708	 73	 2832K fanotify_event_info
 51000  50933  99%    0.12K    750	 68	 6000K dnotify_mark
 37504  36803  98%    0.06K    586	 64	 2344K anon_vma
 33303  33303 100%    0.08K    653	 51	 2612K selinux_inode_security
 28083  27187  96%    0.21K    759	 37	 6072K vm_area_struct
 27534  18839  68%    0.81K    706	 39     22592K task_xstate
 26214  21624  82%    0.04K    257	102	 1028K Acpi-Namespace
 26163  25143  96%    0.31K    513	 51	 8208K nf_conntrack_ffffffff81a26d80
 25959  25808  99%    0.62K    509	 51     16288K sock_inode_cache
 25530  25312  99%    0.09K    555	 46	 2220K configfs_dir_cache
 24565  24225  98%    0.05K    289	 85	 1156K shared_policy_node
 23205  22976  99%    0.10K    595	 39	 2380K blkdev_ioc
 20416  20197  98%    0.18K    464	 44	 3712K xfs_log_ticket


> cat /proc/slabinfo

   <<<< See file attached >>>>

Created attachment 1479631 [details]
Output of /proc/slabinfo on the machine showing the issue

Kernel version of the host that I was able to reproduce the issue.

> [root@vd0916 ~]# uname -r
> 3.10.0-327.36.3.el7.x86_64

Reproducibility steps:

> I was running a script similar to what @Anand has posted https://bugzilla.redhat.com/show_bug.cgi?id=1507149#c44.
> By starting a lot of kubernetes pods and killing them in a tight loop.

The source of the strange workaround in comment 45 was this article: http://www.linuxfly.org/kubernetes-19-conflict-with-centos7/? which one of our colleagues translated for us. His summary was:

The reason that upgrading k8s from 1.6 to 1.9 alone does not reproduce the cgroup memory leak issue is that without restarting the node kubelet will re-use the

/sys/fs/cgroup/memory/kubepods

directory that was created by k8s 1.6 kubelet, which has cgroup kernel memory limit disabled by default. Restarting the nodes will result in

/sys/fs/cgroup/memory/kubepods

directory being re-created by k8s 1.9 kubelet and hence will have cgroup kernel memory limit enabled by default.

Tomorrow I'll start a test to double-check the workaround on a second node.

I tried the script in Comment 32 and was able to have the same issue (where a docker run fails due to ENOSPC). The script is copied below for ease of reading:

#!/bin/bash 
mkdir -p pages
for x in `seq 1280000`; do
        [ $((x % 1000)) -eq 0 ] && echo $x
        mkdir /sys/fs/cgroup/memory/foo
        # echo 1M > /sys/fs/cgroup/memory/foo/memory.limit_in_bytes
        echo 100M > /sys/fs/cgroup/memory/foo/memory.kmem.limit_in_bytes
        echo $$ >/sys/fs/cgroup/memory/foo/cgroup.procs
        memhog 4K &>/dev/null
        echo trex>pages/$x
        echo $$ >/sys/fs/cgroup/memory/cgroup.procs 
        rmdir /sys/fs/cgroup/memory/foo 
done

I suspected that this could be due to short-lived processes (our use case potentially has many short-lived containers in an error-handling path). I modified the script to the following with the sleeper being a compiled C program that prints pid (getpid()) and then sleeps for a second:

#!/bin/bash
mkdir -p pages
for x in `seq 1280000`; do
        [ $((x % 1000)) -eq 0 ] && echo $x
        mkdir /sys/fs/cgroup/memory/test2
        # echo 1M > /sys/fs/cgroup/memory/test2/memory.limit_in_bytes
        echo 100M > /sys/fs/cgroup/memory/test2/memory.kmem.limit_in_bytes
        ./sleeper >/sys/fs/cgroup/memory/test2/cgroup.procs
        memhog 4K &>/dev/null
        echo trex>pages/$x
        echo $$ >/sys/fs/cgroup/memory/cgroup.procs
        rmdir /sys/fs/cgroup/memory/test2
done

---------------------------------------
On a machine without the ENOSPC:

ls -lrt /sys/kernel/slab/ | wc -l
251

slabtop -s l -o
 Active / Total Objects (% used)    : 533404 / 535929 (99.5%)
 Active / Total Slabs (% used)      : 18399 / 18399 (100.0%)
 Active / Total Caches (% used)     : 65 / 95 (68.4%)
 Active / Total Size (% used)       : 182723.21K / 183691.12K (99.5%)
 Minimum / Average / Maximum Object : 0.01K / 0.34K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
124131 123889  99%    0.19K   5911       21     23644K dentry
 89908  89908 100%    0.15K   3458       26     13832K xfs_ili
 95280  95280 100%    1.06K   3176       30    101632K xfs_inode
 89427  89427 100%    0.10K   2293       39      9172K buffer_head
 16335  16147  98%    0.58K    605       27      9680K inode_cache
 18072  18072 100%    0.11K    502       36      2008K sysfs_dir_cache


---------------------------------------
On a machine with the ENOSPC:

ls -lrt /sys/kernel/slab/ | wc -l
2491573

sudo slabtop -s l -o
 Active / Total Objects (% used)    : 99818530 / 209669755 (47.6%)
 Active / Total Slabs (% used)      : 4532638 / 4532638 (100.0%)
 Active / Total Caches (% used)     : 125 / 161 (77.6%)
 Active / Total Size (% used)       : 21303475.11K / 59862027.02K (35.6%)
 Minimum / Average / Maximum Object : 0.01K / 0.29K / 8.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
30771153 17915952  58%    0.19K 732647       42   5861176K dentry
22472028 22454051  99%    0.11K 624223       36   2496892K sysfs_dir_cache
22912383 19079451  83%    0.10K 587497       39   2349988K buffer_head
18623542 3876899  20%    0.57K 333069       56  10658208K radix_tree_node
8240850 837650  10%    1.06K 274695       30   8790240K xfs_inode
10468080 6422369  61%    0.09K 249240       42    996960K kmalloc-96
10072062 677935   6%    0.38K 239811       42   3836976K blkdev_requests
13732880 415473   3%    0.25K 214577       64   3433232K kmalloc-256

---------------------------------------

I can see that with a slightly long-lived process (as shown below), the case has not been hit even though we have run through 90k iterations. The active slabs is at 18k for 1s long processes while for the short-lived process the active slabs was at 400k at a similar point.

---------------------------------------
time ./sleeper
4427

real	0m1.001s
user	0m0.000s
sys	0m0.001s
---------------------------------------
time echo $$
29978

real	0m0.000s
user	0m0.000s
sys	0m0.000s
---------------------------------------

Based on a superficial understanding of the comments in the bug and the test results, it seems that the 'leak' is exercised in cases where there are many short-lived processes / containers. Is this assumption right? We can implement this as a workaround if this reduces the frequency of the leak.

In continuation to Comment 50, I just tried to run the script with the short-lived process on the side. The 'Active / Total Slabs (% used)' jumped from 18k to 27k in about 1k iterations of the script and does not show any signs of going down. So I strongly suspect that a quickly exiting process could somehow lead to this state.

Could someone please corroborate this based on better understanding of the issue and the patch? We could implement a workaround based on this.

I cannot view External Bug ID: Red Hat Knowledge Base (Solution) 3562061

I cannot see that KB article either? Can you please update this?

(In reply to Vamsee Yarlagadda from comment #48)
> Kernel version of the host that I was able to reproduce the issue.
> > [root@vd0916 ~]# uname -r
> > 3.10.0-327.36.3.el7.x86_64

As per Bug 1502818, can you attempt to reproduce the reported issue under
kernel-3.10.0-862.el7?

We seem to be experiencing this issue with short lived containers with kernel 3.10.0-862.9.1.el7.x86_64 and Docker version 18.03.1-ce, build 9ee9f40

So, as far as I know kernel memory controller in RHEL7 kernel is very old, useless*, and buggy and it either needs to be disabled entirely, or fixed by backported changes from 4.x kernels (where it was basically rewritten from scratch). A workaround is NOT to enable/use kmem; I am working on that in https://github.com/opencontainers/runc/pull/1921 (or https://github.com/opencontainers/runc/pull/1920).

* it is useless as there's no reclamation of kernel memory, i.e. dcache and icache entries are not being reclaimed once we run out of kernel memory in this cgroup.

RHEL 7.6 kernel should have version number of 3.10.0-957 -- can anyone check if the bug is fixed in there?

The issue is a bit different, but same symptoms. Upstream the slab caches aren't
immediately freed but only when memory pressure occours. Because of changes
present upstream, it's possible to free the memcg id as soon as it goes offline
so the slab caches and corresponding memcg keep existing until everything else
is freed.

Two things are happening in RHEL7: kmem caches aren't being emptied even during
memory pressure and the memcg id isn't being freed so the -ENOMEM and failure
to create new memcgs come from the fact that there're 65535 memory cgroups still
using an id waiting for the slab caches to be freed.

This has not been fixed in 7.6 and I'm working on the fix. It's possible that
just solving the first problem will address both, but I'm not sure yet.

Test kernel: http://people.redhat.com/~arozansk/a18a3f62f17ae367d23647cad92c05f0b5ca39dc/

Known issues: dentry isn't being accounted; moving a process while it's allocating kmem with
existing or new slab caches will leak both cache and the old memory cgroup.

In practical terms this test kernel should reduce significantly the kmem cache and memory cgroup leaks in short lived containers where the processes running in the container finish before the
cgroup is removed.

I'll keep working on the remaining issues.

Aristeu, is there any decent way to see which backports have you applied (other than to download src.rpm and try to figure it out)?

In general, all related upstream fixes should be from Vladimir Davydov (vdavydov and vdavydov.dev).

Created attachment 1578367 [details]
Test patches.

Attached.

Folks, can I get some feedback on the test kernel please?

Created attachment 1591753 [details]
Slab active/Total Graph

Created attachment 1591754 [details]
Slab active/Total Graph data

Patch(es) committed on kernel-3.10.0-1075.el7

Quick question about the kernel the update is committed on... I see that 7.7 was released with kernel 3.10.0-1062.el7 earlier this month and in https://bugzilla.redhat.com/show_bug.cgi?id=1507149#c101 Jan Stancek mentioned the patch was included in kernel 3.10.0-1075.el7. RHEL 8.0 starts with the new 4.18.0 kernel. Typically the kernel updates are 1062.x.x releases, i.e. the latest kernel for 7.6 is 3.10.0-957.27.4.el7.

My question is, which release of RHEL is 3.10.0-1075.el7 slated to be included in? 7.8?

(In reply to Aaron from comment #105)
> Quick question about the kernel the update is committed on... I see that 7.7
> was released with kernel 3.10.0-1062.el7 earlier this month and in
> https://bugzilla.redhat.com/show_bug.cgi?id=1507149#c101 Jan Stancek
> mentioned the patch was included in kernel 3.10.0-1075.el7. RHEL 8.0 starts
> with the new 4.18.0 kernel. Typically the kernel updates are 1062.x.x
> releases, i.e. the latest kernel for 7.6 is 3.10.0-957.27.4.el7.
> 
> My question is, which release of RHEL is 3.10.0-1075.el7 slated to be
> included in? 7.8?

7.8. These fixes are likely to be present in a future zstream release.

*** Bug 1739145 has been marked as a duplicate of this bug. ***

https://bugzilla.redhat.com/show_bug.cgi?id=1507149#c106

Thanks @Aristeu, I see https://bugzilla.redhat.com/show_bug.cgi?id=1748237 was created for a z-stream update to 7.5 with status of "ON_QA". Since 7.8 has not yet been released, is there a z-stream 3.10.0-1062.x release in the works as well?

Hi Aaron,

(In reply to Aaron from comment #118)
> https://bugzilla.redhat.com/show_bug.cgi?id=1507149#c106
> 
> Thanks @Aristeu, I see https://bugzilla.redhat.com/show_bug.cgi?id=1748237
> was created for a z-stream update to 7.5 with status of "ON_QA". Since 7.8
> has not yet been released, is there a z-stream 3.10.0-1062.x release in the
> works as well?

yes, there's a BZ to include it on 7.7 kernel as well (1752421)

Verified with kernel-3.10.0-1098.el7:
============================================
slableakv1 - ppc64le:
......
Wait for SUnreclaim stabilized
SUnreclaim stabilized now
One exec per cpu on 8 - Iterations: 4000 - Before: SUnreclaim: 164672 kB - After: SUnreclaim: 190400 kB
Total Leak: 25728 kB - Leak per execute: .8040 kB - Total time: 5863s
Kernel: 3.10.0-1098.el7.ppc64le - podman: podman version 1.4.4

slableakv2 - ppc64le:
......
Wait for SUnreclaim stabilized
SUnreclaim stabilized now
One exec per cpu on 8 - Iterations: 4000 - Before: SUnreclaim: 168384 kB - After: SUnreclaim: 199488 kB
Total Leak: 31104 kB - Leak per execute: .9720 kB - Total time: 5172s
Kernel: 3.10.0-1098.el7.ppc64le - podman: podman version 1.4.4

slableakv1 - x86_64:
......
Wait for SUnreclaim stabilized
SUnreclaim stabilized now
One exec per cpu on 12 - Iterations: 6000 - Before: SUnreclaim: 96400 kB - After: SUnreclaim: 158772 kB
Total Leak: 62372 kB - Leak per execute: .8662 kB - Total time: 10442s
Kernel: 3.10.0-1098.el7.x86_64 - podman: podman version 1.4.4

slableakv2 - x86_64:
......
Wait for SUnreclaim stabilized
SUnreclaim stabilized now
One exec per cpu on 12 - Iterations: 6000 - Before: SUnreclaim: 82584 kB - After: SUnreclaim: 115444 kB
Total Leak: 32860 kB - Leak per execute: .4563 kB - Total time: 5274s
Kernel: 3.10.0-1098.el7.x86_64 - podman: podman version 1.4.4


Both slableakv1(podman exec) and slableakv2(shortlive containers) tests showing good result. Move to VERIFIED.

Beaker Jobs:
https://beaker.engineering.redhat.com/jobs/3807009

*** Bug 1392283 has been marked as a duplicate of this bug. ***

------- Comment From mjwolf.com 2017-05-17 17:25 EDT-------
This should not be with Gustavo on the IBM side.  Re-assigning the bug

------- Comment From gmgaydos.com 2017-05-26 14:06 EDT-------
Red Hat:  This bug has been reassigned on May 25 in order to improve response time.

------- Comment From chavez.com 2017-05-30 18:21 EDT-------
Hello,

How much RAM should we have installed in the system? Also, does this problem happen with PowerVM, KVM and/or bare metal? Any information you can give us on the config will help us set up a similar environment for a recreate. Thanks.

------- Comment From chavez.com 2017-06-01 18:37 EDT-------
(In reply to comment #10)
> Hello,
> How much RAM should we have installed in the system? Also, does this problem
> happen with PowerVM, KVM and/or bare metal? Any information you can give us
> on the config will help us set up a similar environment for a recreate.
> Thanks.

Just to clarify, this question was addressed to the Red Hat reporter, Wang Shu. The needinfo flag in the Red Hat bug was already set accordingly.

------- Comment From diegodo.com 2017-06-13 16:10 EDT-------

Mauricio did a check on the LPAR used to reproduce the problem. This system has 16 CPUs & 2677 MB RAM.

I was able to reproduce this problem using my local system (KVM guest):

Red Hat Enterprise Linux Server 7.4 Beta (Maipo)

[root@rhel74snapshot2 ~]# lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    8
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Model:                 2.0 (pvr 004d 0200)
Model name:            POWER8 (raw), altivec supported
Hypervisor vendor:     KVM
Virtualization type:   para
L1d cache:             64K
L1i cache:             32K
NUMA node0 CPU(s):     0-31

[root@rhel74snapshot2 ~]# free -m
total        used        free      shared  buff/cache   available
Mem:            498         193          25           4         278          20
Swap:          8192         136        8056

I wasn't able to reproduce with less cpus.

The steps were the same as documented in the first comment:

[root@rhel74snapshot2 ~]# systemctl start cgconfig
[root@rhel74snapshot2 ~]# for i in $(seq 1 1000); do mkdir /sys/fs/cgroup/cpuacct/cgroup_$i; done

------- Comment From diegodo.com 2017-06-23 11:41 EDT-------
UPDATE

I couldn't reproduce this problem on kernel >= 4.1.
I was able to increase the "for" execution in 10x and nothing happened.

Now I'm trying to find the problem and creating a backport to kernel 3.10.

------- Comment From diegodo.com 2017-06-28 14:09 EDT-------
UPDATE:

I did some research using a basic "free -m" command, and there is a huge difference between kernel 3.10 and kernel >= 4.1 - the creation of 1000 dirs in 3.10 consumes about 100MB, while in 4.1 consumes 20MB.

Because there are a lot of changes between these kernels in this area of cgroup, I'm trying to figure out what exactly did the difference in this size and trying to create a backport to this improvement.

Please feel free to suggest/ask something.

------- Comment From diegodo.com 2017-07-11 12:59 EDT-------
UPDATE

I did some tests on x86 arch and I figured out that the creation of 1000 cgroups have almost the same memory consumption (100MB) as we can see below:

-------------------------------------------------------------------------------
Before:

total        used        free      shared  buff/cache   available
Mem:            570         126         290           4         154         293
Swap:             0           0           0

After:

total        used        free      shared  buff/cache   available
Mem:            570         125         185           4         259         193
Swap:             0           0           0

[root@localhost ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             32
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 94
Model name:            Intel Core Processor (Skylake)
Stepping:              3
CPU MHz:               2496.000
BogoMIPS:              4992.00
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0-31
[root@localhost ~]#

-------------------------------------------------------------------------------

Before:
total        used        free      shared  buff/cache   available
Mem:          14560         313       13774          12         472       13947
Swap:          8192           0        8192

After:
total        used        free      shared  buff/cache   available
Mem:          14560         329       13659          12         571       13836
Swap:          8192           0        8192

Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    8
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Model:                 2.0 (pvr 004d 0200)
Model name:            POWER8 (raw), altivec supported
Hypervisor vendor:     KVM
Virtualization type:   para
L1d cache:             64K
L1i cache:             32K
NUMA node0 CPU(s):     0-31

---------------------------------------------

So, it seems that memory consumption are almost the same for both architectures.

------- Comment From diegodo.com 2017-11-22 14:30 EDT-------
(In reply to comment #21)
> It seems that the bug can be reproduced only on systems where the first NUMA
> node has no CPUs assigned:
> $ uname -r
> 3.10.0-693.el7.ppc64le
>
> $ cat test.sh
> echo 3 > /proc/sys/vm/drop_caches
> grep ^MemAvail /proc/meminfo
> for i in $(seq 10); do
> for j in $(seq 100); do
> mkdir /sys/fs/cgroup/cpuacct/cg_${i}_${j}
> done
> grep ^MemAvail /proc/meminfo
> done
>
> === First system ===
>
> $ lscpu
> Architecture:          ppc64le
> Byte Order:            Little Endian
> CPU(s):                16
> On-line CPU(s) list:   0-15
> Thread(s) per core:    8
> Core(s) per socket:    1
> Socket(s):             2
> NUMA node(s):          1
> Model:                 2.1 (pvr 004b 0201)
> Model name:            POWER8 (architected), altivec supported
> Hypervisor vendor:     (null)
> Virtualization type:   full
> L1d cache:             64K
> L1i cache:             32K
> L2 cache:              512K
> L3 cache:              8192K
> NUMA node0 CPU(s):     0-15
>
> $ ./test.sh
> MemAvailable:   16756096 kB
> MemAvailable:   16715200 kB
> MemAvailable:   16668544 kB
> MemAvailable:   16627712 kB
> MemAvailable:   16580992 kB
> MemAvailable:   16528448 kB
> MemAvailable:   16475840 kB
> MemAvailable:   16423360 kB
> MemAvailable:   16379648 kB
> MemAvailable:   16327040 kB
> MemAvailable:   16280384 kB
>
> === Second system ===
>
> $ lscpu
> Architecture:          ppc64le
> Byte Order:            Little Endian
> CPU(s):                16
> On-line CPU(s) list:   0-15
> Thread(s) per core:    8
> Core(s) per socket:    1
> Socket(s):             2
> NUMA node(s):          2
> Model:                 2.1 (pvr 004b 0201)
> Model name:            POWER8 (architected), altivec supported
> Hypervisor vendor:     (null)
> Virtualization type:   full
> L1d cache:             64K
> L1i cache:             32K
> NUMA node0 CPU(s):     0-15
> NUMA node3 CPU(s):
>
> $ ./test.sh
> MemAvailable:   16766720 kB
> MemAvailable:   16725888 kB
> MemAvailable:   16679168 kB
> MemAvailable:   16618304 kB
> MemAvailable:   16571712 kB
> MemAvailable:   16519296 kB
> MemAvailable:   16466816 kB
> MemAvailable:   16420160 kB
> MemAvailable:   16367680 kB
> MemAvailable:   16318144 kB
> MemAvailable:   16265664 kB
>
> === Third system ===
>
> $ lscpu
> Architecture:          ppc64le
> Byte Order:            Little Endian
> CPU(s):                16
> On-line CPU(s) list:   0-15
> Thread(s) per core:    8
> Core(s) per socket:    1
> Socket(s):             2
> NUMA node(s):          2
> Model:                 2.1 (pvr 004b 0201)
> Model name:            POWER8 (architected), altivec supported
> Hypervisor vendor:     (null)
> Virtualization type:   full
> L1d cache:             64K
> L1i cache:             32K
> L2 cache:              512K
> L3 cache:              8192K
> NUMA node0 CPU(s):
> NUMA node2 CPU(s):     0-15
>
> $ uname -r
> 3.10.0-693.el7.ppc64le
>
> $ ./test.sh
> MemAvailable:   22156736 kB
> MemAvailable:   16632896 kB
> MemAvailable:   11104064 kB
> MemAvailable:    5566976 kB
> MemAvailable:      34112 kB
> !!! OOM !!!

I was not able to reproduce the result with Numa Node0 with no Cpu using qemu:

[root@localhost ~]# lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             16
NUMA node(s):          2
Model:                 2.0 (pvr 004d 0200)
Model name:            POWER8 (raw), altivec supported
Hypervisor vendor:     KVM
Virtualization type:   para
L1d cache:             64K
L1i cache:             32K
NUMA node0 CPU(s):
NUMA node1 CPU(s):     0-15

[root@localhost ~]# ./test.sh
MemAvailable:      86848 kB
MemAvailable:      31168 kB
MemAvailable:      56704 kB
MemAvailable:      51328 kB
MemAvailable:      44608 kB
MemAvailable:      40576 kB
MemAvailable:      37632 kB
MemAvailable:      27328 kB
MemAvailable:      29440 kB
MemAvailable:      30720 kB
MemAvailable:      32064 kB

------- Comment From diegodo.com 2018-02-28 06:56 EDT-------
Hi Li Wang,

I need help to reproduce this problem. I wasn't able to reproduce this problem how you can see in Comment #24.

Can anyone help me to reproduce this?

Thank you

Guys,
any news when it will be backported to 7.7?

Hi Jean-Philippe,

(In reply to Jean-Philippe Menil from comment #124)
> Guys,
> any news when it will be backported to 7.7?

It was backported to RHEL-7.7 in BZ 1752421.
Upgrade to kernel-3.10.0-1062.4.1.el7 from RHSA-2019:3055
https://access.redhat.com/errata/RHSA-2019:3055

Yeah i guess it was fixed:

https://github.com/docker/for-linux/issues/841

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-1062.7.1.el7.x86_64 root=/dev/mapper/rootvg-lvroot ro crashkernel=auto rd.lvm.lv=rootvg/lvroot rd.lvm.lv=rootvg/lvswap rhgb quiet LANG=en_US.UTF-8 namespace.unpriv_enable=1 user_namespace.enable=1

uname -a
Linux hsv5234s 3.10.0-1062.7.1.el7.x86_64 #1 SMP Wed Nov 13 08:44:42 EST 2019 x86_64 x86_64 x86_64 GNU/Linux

Dec 18 08:20:41 hsv5234s kernel: [1080660.051671] runc:[1:CHILD]: page allocation failure: order:7, mode:0xc0d0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051679] CPU: 17 PID: 16064 Comm: runc:[1:CHILD] Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.7.1.el7.x86_64 #1
Dec 18 08:20:41 hsv5234s kernel: [1080660.051682] Hardware name: LENOVO System x3650 M5 -[8871AC1]-/01KN179, BIOS -[TCE140D-2.90]- 02/13/2019
Dec 18 08:20:41 hsv5234s kernel: [1080660.051684] Call Trace:
Dec 18 08:20:41 hsv5234s kernel: [1080660.051696]  [<ffffffffbcb7ac23>] dump_stack+0x19/0x1b
Dec 18 08:20:41 hsv5234s kernel: [1080660.051704]  [<ffffffffbc5c3e50>] warn_alloc_failed+0x110/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.051708]  [<ffffffffbc5c8a5f>] __alloc_pages_nodemask+0x9df/0xbe0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051714]  [<ffffffffbc616c08>] alloc_pages_current+0x98/0x110
Dec 18 08:20:41 hsv5234s kernel: [1080660.051719]  [<ffffffffbc5e3c08>] kmalloc_order+0x18/0x40
Dec 18 08:20:41 hsv5234s kernel: [1080660.051723]  [<ffffffffbc622136>] kmalloc_order_trace+0x26/0xa0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051727]  [<ffffffffbc6266f1>] __kmalloc+0x211/0x230
Dec 18 08:20:41 hsv5234s kernel: [1080660.051732]  [<ffffffffbc63ee41>] memcg_alloc_cache_params+0x81/0xb0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051735]  [<ffffffffbc5e38b4>] do_kmem_cache_create+0x74/0xf0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051738]  [<ffffffffbc5e3a32>] kmem_cache_create+0x102/0x1b0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051753]  [<ffffffffc06b4dc1>] nf_conntrack_init_net+0xf1/0x260 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.051759]  [<ffffffffc06b56c4>] nf_conntrack_pernet_init+0x14/0x150 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.051764]  [<ffffffffbca44054>] ops_init+0x44/0x150
Dec 18 08:20:41 hsv5234s kernel: [1080660.051767]  [<ffffffffbca44203>] setup_net+0xa3/0x160
Dec 18 08:20:41 hsv5234s kernel: [1080660.051770]  [<ffffffffbca449a5>] copy_net_ns+0xb5/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.051774]  [<ffffffffbc4cb469>] create_new_namespaces+0xf9/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.051777]  [<ffffffffbc4cb6aa>] unshare_nsproxy_namespaces+0x5a/0xc0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051781]  [<ffffffffbc49ae8b>] SyS_unshare+0x1cb/0x340
Dec 18 08:20:41 hsv5234s kernel: [1080660.051785]  [<ffffffffbcb8dede>] system_call_fastpath+0x25/0x2a
Dec 18 08:20:41 hsv5234s kernel: [1080660.051787] Mem-Info:
Dec 18 08:20:41 hsv5234s kernel: [1080660.051800] active_anon:603062 inactive_anon:658360 isolated_anon:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051800]  active_file:2900635 inactive_file:2830989 isolated_file:4
Dec 18 08:20:41 hsv5234s kernel: [1080660.051800]  unevictable:0 dirty:1534 writeback:0 unstable:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051800]  slab_reclaimable:341029 slab_unreclaimable:390332
Dec 18 08:20:41 hsv5234s kernel: [1080660.051800]  mapped:56516 shmem:134670 pagetables:12006 bounce:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051800]  free:46786 free_pcp:2175 free_cma:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051807] Node 0 DMA free:15880kB min:44kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 18 08:20:41 hsv5234s kernel: [1080660.051815] lowmem_reserve[]: 0 971 15057 15057
Dec 18 08:20:41 hsv5234s kernel: [1080660.051820] Node 0 DMA32 free:60060kB min:2808kB low:3508kB high:4212kB active_anon:94876kB inactive_anon:149864kB active_file:233272kB inactive_file:218016kB unevictable:0kB isolated(anon):0kB isolated(file):16kB present:1226660kB managed:995204kB mlocked:0kB dirty:112kB writeback:0kB mapped:14792kB shmem:16016kB slab_reclaimable:112532kB slab_unreclaimable:89812kB kernel_stack:2080kB pagetables:2928kB unstable:0kB bounce:0kB free_pcp:1384kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:23 all_unreclaimable? no
Dec 18 08:20:41 hsv5234s kernel: [1080660.051828] lowmem_reserve[]: 0 0 14085 14085
Dec 18 08:20:41 hsv5234s kernel: [1080660.051831] Node 0 Normal free:51656kB min:40708kB low:50884kB high:61060kB active_anon:1375020kB inactive_anon:1401180kB active_file:4987756kB inactive_file:4918788kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14423212kB mlocked:0kB dirty:3076kB writeback:0kB mapped:94496kB shmem:141648kB slab_reclaimable:612400kB slab_unreclaimable:685176kB kernel_stack:10704kB pagetables:32204kB unstable:0kB bounce:0kB free_pcp:4540kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 18 08:20:41 hsv5234s kernel: [1080660.051839] lowmem_reserve[]: 0 0 0 0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051843] Node 1 Normal free:59548kB min:46548kB low:58184kB high:69820kB active_anon:942352kB inactive_anon:1082396kB active_file:6381128kB inactive_file:6187152kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16495992kB mlocked:0kB dirty:2948kB writeback:0kB mapped:116776kB shmem:381016kB slab_reclaimable:639184kB slab_unreclaimable:786324kB kernel_stack:9344kB pagetables:12892kB unstable:0kB bounce:0kB free_pcp:2744kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
Dec 18 08:20:41 hsv5234s kernel: [1080660.051851] lowmem_reserve[]: 0 0 0 0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051854] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15880kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051866] Node 0 DMA32: 667*4kB (UEM) 594*8kB (UEM) 263*16kB (UEM) 222*32kB (UEM) 258*64kB (UEM) 132*128kB (UEM) 31*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60076kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051878] Node 0 Normal: 8586*4kB (UEM) 2023*8kB (UM) 6*16kB (M) 6*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 50816kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051888] Node 1 Normal: 3685*4kB (UEM) 5529*8kB (UEM) 24*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59356kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051899] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051901] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051903] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051905] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051906] 5869920 total pagecache pages
Dec 18 08:20:41 hsv5234s kernel: [1080660.051910] 3487 pages in swap cache
Dec 18 08:20:41 hsv5234s kernel: [1080660.051913] Swap cache stats: add 38962826, delete 38956860, find 14019018/18441754
Dec 18 08:20:41 hsv5234s kernel: [1080660.051914] Free swap  = 8253408kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051915] Total swap = 8388604kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.051917] 8174980 pages RAM
Dec 18 08:20:41 hsv5234s kernel: [1080660.051918] 0 pages HighMem/MovableOnly
Dec 18 08:20:41 hsv5234s kernel: [1080660.051920] 192404 pages reserved
Dec 18 08:20:41 hsv5234s kernel: [1080660.051923] kmem_cache_create(nf_conntrack_ffff98b468a20000) failed with error -12
Dec 18 08:20:41 hsv5234s kernel: [1080660.051926] CPU: 17 PID: 16064 Comm: runc:[1:CHILD] Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.7.1.el7.x86_64 #1
Dec 18 08:20:41 hsv5234s kernel: [1080660.051928] Hardware name: LENOVO System x3650 M5 -[8871AC1]-/01KN179, BIOS -[TCE140D-2.90]- 02/13/2019
Dec 18 08:20:41 hsv5234s kernel: [1080660.051930] Call Trace:
Dec 18 08:20:41 hsv5234s kernel: [1080660.051933]  [<ffffffffbcb7ac23>] dump_stack+0x19/0x1b
Dec 18 08:20:41 hsv5234s kernel: [1080660.051936]  [<ffffffffbc5e3ab7>] kmem_cache_create+0x187/0x1b0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051942]  [<ffffffffc06b4dc1>] nf_conntrack_init_net+0xf1/0x260 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.051947]  [<ffffffffc06b56c4>] nf_conntrack_pernet_init+0x14/0x150 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.051950]  [<ffffffffbca44054>] ops_init+0x44/0x150
Dec 18 08:20:41 hsv5234s kernel: [1080660.051953]  [<ffffffffbca44203>] setup_net+0xa3/0x160
Dec 18 08:20:41 hsv5234s kernel: [1080660.051956]  [<ffffffffbca449a5>] copy_net_ns+0xb5/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.051959]  [<ffffffffbc4cb469>] create_new_namespaces+0xf9/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.051961]  [<ffffffffbc4cb6aa>] unshare_nsproxy_namespaces+0x5a/0xc0
Dec 18 08:20:41 hsv5234s kernel: [1080660.051965]  [<ffffffffbc49ae8b>] SyS_unshare+0x1cb/0x340
Dec 18 08:20:41 hsv5234s kernel: [1080660.051967]  [<ffffffffbcb8dede>] system_call_fastpath+0x25/0x2a
Dec 18 08:20:41 hsv5234s kernel: [1080660.051970] Unable to create nf_conn slab cache
Dec 18 08:20:41 hsv5234s kernel: [1080660.119124] runc:[1:CHILD]: page allocation failure: order:7, mode:0xc0d0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119135] CPU: 4 PID: 16094 Comm: runc:[1:CHILD] Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.7.1.el7.x86_64 #1
Dec 18 08:20:41 hsv5234s kernel: [1080660.119139] Hardware name: LENOVO System x3650 M5 -[8871AC1]-/01KN179, BIOS -[TCE140D-2.90]- 02/13/2019
Dec 18 08:20:41 hsv5234s kernel: [1080660.119142] Call Trace:
Dec 18 08:20:41 hsv5234s kernel: [1080660.119156]  [<ffffffffbcb7ac23>] dump_stack+0x19/0x1b
Dec 18 08:20:41 hsv5234s kernel: [1080660.119167]  [<ffffffffbc5c3e50>] warn_alloc_failed+0x110/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.119174]  [<ffffffffbc5c8a5f>] __alloc_pages_nodemask+0x9df/0xbe0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119182]  [<ffffffffbc616c08>] alloc_pages_current+0x98/0x110
Dec 18 08:20:41 hsv5234s kernel: [1080660.119190]  [<ffffffffbc5e3c08>] kmalloc_order+0x18/0x40
Dec 18 08:20:41 hsv5234s kernel: [1080660.119196]  [<ffffffffbc622136>] kmalloc_order_trace+0x26/0xa0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119202]  [<ffffffffbc6266f1>] __kmalloc+0x211/0x230
Dec 18 08:20:41 hsv5234s kernel: [1080660.119209]  [<ffffffffbc63ee41>] memcg_alloc_cache_params+0x81/0xb0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119215]  [<ffffffffbc5e38b4>] do_kmem_cache_create+0x74/0xf0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119221]  [<ffffffffbc5e3a32>] kmem_cache_create+0x102/0x1b0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119238]  [<ffffffffc06b4dc1>] nf_conntrack_init_net+0xf1/0x260 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.119249]  [<ffffffffc06b56c4>] nf_conntrack_pernet_init+0x14/0x150 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.119256]  [<ffffffffbca44054>] ops_init+0x44/0x150
Dec 18 08:20:41 hsv5234s kernel: [1080660.119260]  [<ffffffffbca44203>] setup_net+0xa3/0x160
Dec 18 08:20:41 hsv5234s kernel: [1080660.119265]  [<ffffffffbca449a5>] copy_net_ns+0xb5/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.119271]  [<ffffffffbc4cb469>] create_new_namespaces+0xf9/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.119276]  [<ffffffffbc4cb6aa>] unshare_nsproxy_namespaces+0x5a/0xc0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119286]  [<ffffffffbc49ae8b>] SyS_unshare+0x1cb/0x340
Dec 18 08:20:41 hsv5234s kernel: [1080660.119293]  [<ffffffffbcb8dede>] system_call_fastpath+0x25/0x2a
Dec 18 08:20:41 hsv5234s kernel: [1080660.119296] Mem-Info:
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309] active_anon:606064 inactive_anon:659529 isolated_anon:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  active_file:2897344 inactive_file:2822214 isolated_file:4
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  unevictable:0 dirty:2854 writeback:0 unstable:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  slab_reclaimable:340962 slab_unreclaimable:390372
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  mapped:56516 shmem:134670 pagetables:12006 bounce:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  free:55605 free_pcp:512 free_cma:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119318] Node 0 DMA free:15880kB min:44kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15980kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 18 08:20:41 hsv5234s kernel: [1080660.119331] lowmem_reserve[]: 0 971 15057 15057
Dec 18 08:20:41 hsv5234s kernel: [1080660.119338] Node 0 DMA32 free:62628kB min:2808kB low:3508kB high:4212kB active_anon:94796kB inactive_anon:149968kB active_file:233240kB inactive_file:216416kB unevictable:0kB isolated(anon):0kB isolated(file):16kB present:1226660kB managed:995204kB mlocked:0kB dirty:112kB writeback:0kB mapped:14792kB shmem:16016kB slab_reclaimable:112628kB slab_unreclaimable:89972kB kernel_stack:2080kB pagetables:2928kB unstable:0kB bounce:0kB free_pcp:316kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:66 all_unreclaimable? no
Dec 18 08:20:41 hsv5234s kernel: [1080660.119351] lowmem_reserve[]: 0 0 14085 14085
Dec 18 08:20:41 hsv5234s kernel: [1080660.119357] Node 0 Normal free:61096kB min:40708kB low:50884kB high:61060kB active_anon:1381536kB inactive_anon:1404520kB active_file:4987048kB inactive_file:4901144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14680064kB managed:14423212kB mlocked:0kB dirty:3780kB writeback:0kB mapped:94496kB shmem:141648kB slab_reclaimable:612400kB slab_unreclaimable:685176kB kernel_stack:10704kB pagetables:32204kB unstable:0kB bounce:0kB free_pcp:1144kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:76 all_unreclaimable? no
Dec 18 08:20:41 hsv5234s kernel: [1080660.119369] lowmem_reserve[]: 0 0 0 0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119375] Node 1 Normal free:82816kB min:46548kB low:58184kB high:69820kB active_anon:947596kB inactive_anon:1084024kB active_file:6369088kB inactive_file:6170932kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16495992kB mlocked:0kB dirty:7524kB writeback:0kB mapped:116776kB shmem:381016kB slab_reclaimable:638820kB slab_unreclaimable:786324kB kernel_stack:9344kB pagetables:12892kB unstable:0kB bounce:0kB free_pcp:584kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 18 08:20:41 hsv5234s kernel: [1080660.119386] lowmem_reserve[]: 0 0 0 0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119391] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15880kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119412] Node 0 DMA32: 1166*4kB (UEM) 677*8kB (UEM) 266*16kB (UEM) 225*32kB (UEM) 258*64kB (UEM) 131*128kB (UEM) 31*256kB (UEM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 62752kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119432] Node 0 Normal: 9639*4kB (UEM) 2269*8kB (UEM) 251*16kB (UEM) 83*32kB (UEM) 3*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 63572kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119450] Node 1 Normal: 5940*4kB (UEM) 7125*8kB (UEM) 221*16kB (UEM) 30*32kB (UEM) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85320kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119470] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119474] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119477] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119481] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119483] 5858268 total pagecache pages
Dec 18 08:20:41 hsv5234s kernel: [1080660.119488] 3481 pages in swap cache
Dec 18 08:20:41 hsv5234s kernel: [1080660.119492] Swap cache stats: add 38962827, delete 38956867, find 14019018/18441754
Dec 18 08:20:41 hsv5234s kernel: [1080660.119494] Free swap  = 8253408kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119496] Total swap = 8388604kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119499] 8174980 pages RAM
Dec 18 08:20:41 hsv5234s kernel: [1080660.119516] 0 pages HighMem/MovableOnly
Dec 18 08:20:41 hsv5234s kernel: [1080660.119518] 192404 pages reserved
Dec 18 08:20:41 hsv5234s kernel: [1080660.119523] kmem_cache_create(nf_conntrack_ffff98b287b81480) failed with error -12
Dec 18 08:20:41 hsv5234s kernel: [1080660.119535] CPU: 4 PID: 16094 Comm: runc:[1:CHILD] Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.7.1.el7.x86_64 #1
Dec 18 08:20:41 hsv5234s kernel: [1080660.119552] Hardware name: LENOVO System x3650 M5 -[8871AC1]-/01KN179, BIOS -[TCE140D-2.90]- 02/13/2019
Dec 18 08:20:41 hsv5234s kernel: [1080660.119567] Call Trace:
Dec 18 08:20:41 hsv5234s kernel: [1080660.119590]  [<ffffffffbcb7ac23>] dump_stack+0x19/0x1b
Dec 18 08:20:41 hsv5234s kernel: [1080660.119611]  [<ffffffffbc5e3ab7>] kmem_cache_create+0x187/0x1b0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119630]  [<ffffffffc06b4dc1>] nf_conntrack_init_net+0xf1/0x260 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.119663]  [<ffffffffc06b56c4>] nf_conntrack_pernet_init+0x14/0x150 [nf_conntrack]
Dec 18 08:20:41 hsv5234s kernel: [1080660.119680]  [<ffffffffbca44054>] ops_init+0x44/0x150
Dec 18 08:20:41 hsv5234s kernel: [1080660.119706]  [<ffffffffbca44203>] setup_net+0xa3/0x160
Dec 18 08:20:41 hsv5234s kernel: [1080660.119721]  [<ffffffffbca449a5>] copy_net_ns+0xb5/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.119735]  [<ffffffffbc4cb469>] create_new_namespaces+0xf9/0x180
Dec 18 08:20:41 hsv5234s kernel: [1080660.119752]  [<ffffffffbc4cb6aa>] unshare_nsproxy_namespaces+0x5a/0xc0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119772]  [<ffffffffbc49ae8b>] SyS_unshare+0x1cb/0x340
Dec 18 08:20:41 hsv5234s kernel: [1080660.119790]  [<ffffffffbcb8dede>] system_call_fastpath+0x25/0x2a
Dec 18 08:20:41 hsv5234s kernel: [1080660.119807] Unable to create nf_conn slab cache

Do you wan me to open a new bug for the same issue ?

Hello Jean-Philippe Menil,

Are you using ip_vs in your env? There is a know issue with ip_vs + user ns+ net ns, BZ#1540585.

Thanks,
Chao

nope:

lsmod | grep ip_vs

return nothing :)

By the way, BZ#1540585 is restricted ...

(In reply to Jean-Philippe Menil from comment #128)
> nope:
> 
> lsmod | grep ip_vs
> 
> return nothing :)
> 
> By the way, BZ#1540585 is restricted ...

Thanks you, Jean-Philippe Menil,

Could you help try boot with 'cgroup.memory=nokmem' and see if this issue still exist. Also, can you try to run container without nf_conntrack loaded. I'll check with our network qe, and see if we could reproduce this in house. Many thanks!

Chao

Chao,

i will try the cgroup.memory=nokmem parameter, effectively, could be a good candidate.
Just to let you know that the issue is intermittent and took us around a week to trig it.

There is absolutely no way that we run anything without nf_conntrack ...

Per log snippets pasted on comment #126, it doesn't look like to be the same issue that was coped with in this ticket, 
but a severe memory fragmentation case instead.


There's no panic, but page allocation failure warnings, for high-order blocks:

...
Dec 18 08:20:41 hsv5234s kernel: [1080660.051671] runc:[1:CHILD]: page allocation failure: order:7, mode:0xc0d0
...


and the reason is because there are no order-7 (128kB) blocks available on the zones the request could grab memory from (per mode flags), as well as there are no blocks bigger than order-7 available for buddy splits, as we can observe on the provided log splats:

...
Dec 18 08:20:41 hsv5234s kernel: [1080660.119432] Node 0 Normal: 9639*4kB (UEM) 2269*8kB (UEM) 251*16kB (UEM) 83*32kB (UEM) 3*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 63572kB
Dec 18 08:20:41 hsv5234s kernel: [1080660.119450] Node 1 Normal: 5940*4kB (UEM) 7125*8kB (UEM) 221*16kB (UEM) 30*32kB (UEM) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85320kB
...


and meminfo data doesn't suggest a bloated slab usage, but a bloated page-cache usage instead:
...
Dec 18 08:20:41 hsv5234s kernel: [1080660.119296] Mem-Info:
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309] active_anon:606064 inactive_anon:659529 isolated_anon:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  active_file:2897344 inactive_file:2822214 isolated_file:4
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  unevictable:0 dirty:2854 writeback:0 unstable:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  slab_reclaimable:340962 slab_unreclaimable:390372
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  mapped:56516 shmem:134670 pagetables:12006 bounce:0
Dec 18 08:20:41 hsv5234s kernel: [1080660.119309]  free:55605 free_pcp:512 free_cma:0
...

-- Rafael

So, what you suggest is just increase min_free_kbytes?

*** Bug 1489799 has been marked as a duplicate of this bug. ***

Chao,

the cgroup.memory=nokmem seem to do the trick.

Thanks again and best regards,

Jean-Philippe

Hi Chao,

I got a customer [case : 02605103] facing the same problem.

~~~~~
Mar 10 10:44:45 prd-k8s-worker101 kubelet: E0310 10:44:45.452884    9223 pod_workers.go:190] Error syncing pod 101ce83b-60eb-4212-9ab2-08bf2d933054 ("kafka-connector-healer-1583854800-2gb8s_
tenet(101ce83b-60eb-4212-9ab2-08bf2d933054)"), skipping: failed to ensure that the pod: 101ce83b-60eb-4212-9ab2-08bf2d933054 cgroups exist and are correctly applied: failed to create contain
er for [kubepods burstable pod101ce83b-60eb-4212-9ab2-08bf2d933054] : mkdir /sys/fs/cgroup/memory/kubepods/burstable/pod101ce83b-60eb-4212-9ab2-08bf2d933054: cannot allocate memory
~~~~~

Messages are flooding just as in above snip.

He is running 3.10.0-1062.12.1.el7.x86_64 already.

I grabbed strace when problem was present.

[root@prd-k8s-worker101 psjoberg]#  strace -f -o /root/strace_cgroup2.out cgcreate -g memory:kubepods/burstable/IODINE
cgcreate: can't create cgroup kubepods/burstable/IODINE: Cgroup, operation not allowed



[root@prd-k8s-worker101 psjoberg]# cat /root/strace_cgroup2.out 
132066 execve("/bin/cgcreate", ["cgcreate", "-g", "memory:kubepods/burstable/IODINE"], [/* 22 vars */]) = 0
132066 brk(NULL)                        = 0x555645f29000
132066 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f51d43b8000
132066 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
132066 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
132066 fstat(3, {st_mode=S_IFREG|0644, st_size=26713, ...}) = 0
132066 mmap(NULL, 26713, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f51d43b1000
132066 close(3)                         = 0
132066 open("/lib64/libcgroup.so.1", O_RDONLY|O_CLOEXEC) = 3
132066 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240D\0\0\0\0\0\0"..., 832) = 832
132066 fstat(3, {st_mode=S_IFREG|0755, st_size=104224, ...}) = 0
132066 mmap(NULL, 4665984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f51d3d24000
132066 mprotect(0x7f51d3d3b000, 2097152, PROT_NONE) = 0
132066 mmap(0x7f51d3f3b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f51d3f3b000
132066 mmap(0x7f51d3f3d000, 2466432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f51d3f3d000
132066 close(3)                         = 0
132066 open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
132066 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200m\0\0\0\0\0\0"..., 832) = 832
132066 fstat(3, {st_mode=S_IFREG|0755, st_size=142144, ...}) = 0
132066 mmap(NULL, 2208904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f51d3b08000
132066 mprotect(0x7f51d3b1f000, 2093056, PROT_NONE) = 0
132066 mmap(0x7f51d3d1e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f51d3d1e000
132066 mmap(0x7f51d3d20000, 13448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f51d3d20000
132066 close(3)                         = 0
132066 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
132066 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P&\2\0\0\0\0\0"..., 832) = 832
132066 fstat(3, {st_mode=S_IFREG|0755, st_size=2156072, ...}) = 0
132066 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f51d43b0000
132066 mmap(NULL, 3985888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f51d373a000
132066 mprotect(0x7f51d38fd000, 2097152, PROT_NONE) = 0
132066 mmap(0x7f51d3afd000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f51d3afd000
132066 mmap(0x7f51d3b03000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f51d3b03000
132066 close(3)                         = 0
132066 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f51d43af000
132066 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f51d43ad000
132066 arch_prctl(ARCH_SET_FS, 0x7f51d43adb80) = 0
132066 mprotect(0x7f51d3afd000, 16384, PROT_READ) = 0
132066 mprotect(0x7f51d3d1e000, 4096, PROT_READ) = 0
132066 mprotect(0x7f51d3f3b000, 4096, PROT_READ) = 0
132066 mprotect(0x555645ac1000, 4096, PROT_READ) = 0
132066 mprotect(0x7f51d43b9000, 4096, PROT_READ) = 0
132066 munmap(0x7f51d43b1000, 26713)    = 0
132066 set_tid_address(0x7f51d43ade50)  = 132066
132066 set_robust_list(0x7f51d43ade60, 24) = 0
132066 rt_sigaction(SIGRTMIN, {0x7f51d3b0e860, [], SA_RESTORER|SA_SIGINFO, 0x7f51d3b17630}, NULL, 8) = 0
132066 rt_sigaction(SIGRT_1, {0x7f51d3b0e8f0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f51d3b17630}, NULL, 8) = 0
132066 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
132066 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
132066 brk(NULL)                        = 0x555645f29000
132066 brk(0x555645f4a000)              = 0x555645f4a000
132066 brk(NULL)                        = 0x555645f4a000
132066 open("/proc/cgroups", O_RDONLY|O_CLOEXEC) = 3
132066 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
132066 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f51d43b7000
132066 read(3, "#subsys_name\thierarchy\tnum_cgrou"..., 1024) = 230
132066 read(3, "", 1024)                = 0
132066 open("/proc/mounts", O_RDONLY|O_CLOEXEC) = 4
132066 fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
132066 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f51d43b6000
132066 read(4, "rootfs / rootfs rw 0 0\nsysfs /sy"..., 1024) = 1024
132066 read(4, "l,nosuid,nodev,noexec,relatime,p"..., 1024) = 1024
132066 read(4, "ebugfs rw,relatime 0 0\nhugetlbfs"..., 1024) = 1024
132066 read(4, "proc rw,nosuid,nodev,noexec,rela"..., 1024) = 1024
132066 read(4, "25fee5d1d08edf87aefb18dd7a5c144f"..., 1024) = 1024
132066 read(4, "abel,relatime 0 0\ntmpfs /var/lib"..., 1024) = 1024
132066 read(4, "7f3cd7f4557a51bff97846504d968e27"..., 1024) = 1024
132066 read(4, ".io~secret/ssot-processor-token-"..., 1024) = 1024
132066 read(4, "ir=/var/lib/docker/overlay/4022a"..., 1024) = 1024
132066 read(4, "bef704f791045ca20906025d914/moun"..., 1024) = 1024
132066 read(4, "mpfs rw,seclabel,relatime 0 0\ntm"..., 1024) = 1024
132066 read(4, "e78bce0b/merged overlay rw,secla"..., 1024) = 1024
132066 read(4, "roc rw,nosuid,nodev,noexec,relat"..., 1024) = 1024
132066 read(4, "cker/containers/3594e77e3c3b353e"..., 1024) = 1024
132066 read(4, ".io~secret/ssot-processor-token-"..., 1024) = 1024
132066 read(4, "latime,lowerdir=/var/lib/docker/"..., 1024) = 1024
132066 read(4, "147daf/volumes/kubernetes.io~sec"..., 1024) = 1024
132066 read(4, "1d84afbaef3dd73e500fd212cfefb4e5"..., 1024) = 1024
132066 read(4, "bel,nosuid,nodev,noexec,relatime"..., 1024) = 1024
132066 read(4, "kubelet/pods/5229f250-0bb5-4615-"..., 1024) = 1024
132066 read(4, "rkdir=/var/lib/docker/overlay/ec"..., 1024) = 1024
132066 read(4, "owerdir=/var/lib/docker/overlay/"..., 1024) = 1024
132066 read(4, "mounts/shm tmpfs rw,seclabel,nos"..., 1024) = 1024
132066 read(4, "per,workdir=/var/lib/docker/over"..., 1024) = 1024
132066 read(4, "nodev,noexec,relatime 0 0\ntmpfs "..., 1024) = 1024
132066 read(4, "var/lib/docker/overlay/ac0ad3d37"..., 1024) = 1024
132066 read(4, "-4bc6-9c5b-1a0c2ef092dd/volumes/"..., 1024) = 1024
132066 read(4, "55e99237c7d32a8e85b4f2f47c5b2d4d"..., 1024) = 1024
132066 read(4, " /var/lib/docker/containers/3e46"..., 1024) = 1024
132066 read(4, "ar/lib/docker/overlay/708310bef3"..., 1024) = 1024
132066 read(4, "/var/lib/docker/overlay/3a7bc822"..., 1024) = 1024
132066 read(4, "10abe71d4cfa4506fffa59471a5a54e4"..., 1024) = 1024
132066 read(4, "100183f1c4cda2efc169622d625662/w"..., 1024) = 1024
132066 read(4, "b76b/merged overlay rw,seclabel,"..., 1024) = 1024
132066 read(4, "e13b53/root,upperdir=/var/lib/do"..., 1024) = 1024
132066 read(4, "/docker/overlay/3004b765e83e7c5b"..., 1024) = 1024
132066 read(4, "73da8084d3646be98da1ebf7bf32e039"..., 1024) = 1024
132066 read(4, "7071f5387761c2d09186dcb877d50ac8"..., 1024) = 1024
132066 read(4, "dbdc16d7dd5add9575579534180/uppe"..., 1024) = 1024
132066 read(4, "ork 0 0\noverlay /var/lib/docker/"..., 1024) = 1024
132066 read(4, "relatime,lowerdir=/var/lib/docke"..., 1024) = 1024
132066 read(4, "cker/overlay/3f7aee3450f473d4c94"..., 1024) = 1024
132066 read(4, "3d14643f5fb12fe46d1243ab246298dc"..., 1024) = 1024
132066 read(4, "13e047b643fc0f48a46dbb/merged ov"..., 1024) = 1024
132066 read(4, "eb6bf8d8b956019dbf8587ab/root,up"..., 1024) = 1024
132066 read(4, "r,workdir=/var/lib/docker/overla"..., 1024) = 1024
132066 read(4, "overlay/b5c2a757796e0a00e5a6b2b1"..., 1024) = 1024
132066 read(4, "r/overlay/3a7bc82268e1e432147f4a"..., 1024) = 1024
132066 read(4, "149df10503bf62d621f098e7a765e36a"..., 1024) = 1024
132066 read(4, "2f029e2616d6d67d/work 0 0\noverla"..., 1024) = 1024
132066 read(4, "erlay rw,seclabel,relatime,lower"..., 1024) = 1024
132066 read(4, "perdir=/var/lib/docker/overlay/2"..., 1024) = 1024
132066 read(4, "y/5501fd0719bf03746d78240bedd47a"..., 1024) = 1024
132066 read(4, "81a143658ac3da9afed334682ccf18a6"..., 1024) = 1024
132066 read(4, "144c26b4d72d1a0023b6def921366f54"..., 1024) = 1024
132066 read(4, "ea91e64f67495/upper,workdir=/var"..., 1024) = 1024
132066 read(4, "y /var/lib/docker/overlay/3ab88f"..., 1024) = 1024
132066 read(4, "l,relatime 0 0\ntmpfs /var/lib/ku"..., 1024) = 1024
132066 read(4, "overlay/d02e11e096cca9176e98b00d"..., 1024) = 1024
132066 read(4, "b/docker/overlay/86ea03d933cbb06"..., 1024) = 1024
132066 read(4, "f91d54c proc rw,nosuid,nodev,noe"..., 1024) = 1024
132066 read(4, "rdir=/var/lib/docker/overlay/1c5"..., 1024) = 1024
132066 read(4, "da131c59a5af78d91c476cf64c99ced1"..., 1024) = 1024
132066 read(4, "e18708ae244f60e55945e97842f1504d"..., 1024) = 1024
132066 read(4, "44a2b1e09f182b20a7502af60a08571a"..., 1024) = 1024
132066 read(4, "/overlay/732d75ed02b7f3cd7f4557a"..., 1024) = 1024
132066 read(4, " rw,seclabel,relatime 0 0\noverla"..., 1024) = 1024
132066 read(4, "18c91aa4a50241d06a2bf72820ee/upp"..., 1024) = 1024
132066 read(4, "69868c0615d4a4cd3482c36a0e40/upp"..., 1024) = 1024
132066 read(4, "ns/817eb67a936c proc rw,nosuid,n"..., 1024) = 1024
132066 read(4, "686b97c79d3b919bef603/merged ove"..., 1024) = 1024
132066 read(4, "fc5ce34c296f7f590a3598b/root,upp"..., 1024) = 1024
132066 read(4, ",workdir=/var/lib/docker/overlay"..., 1024) = 1024
132066 read(4, "6d7cd22/upper,workdir=/var/lib/d"..., 1024) = 1024
132066 read(4, "8d8b956019dbf8587ab/root,upperdi"..., 1024) = 1024
132066 read(4, "kdir=/var/lib/docker/overlay/48e"..., 1024) = 1024
132066 read(4, "/netns/5ea3bbc2414d proc rw,nosu"..., 1024) = 1024
132066 read(4, "5d79e30ba4afd6c68dc0c8752/merged"..., 1024) = 1024
132066 read(4, "023b6def921366f540a9a99897d/root"..., 1024) = 1024
132066 read(4, "/docker/netns/d243aeae8026 proc "..., 1024) = 1024
132066 read(4, "c40a0c8c1bf834bf515fe56086877a3d"..., 1024) = 1024
132066 read(4, "04cbfe0388fa7c1ba942aec189949896"..., 1024) = 1024
132066 read(4, "ffd3959467c515ec5fb3d857a8e09dda"..., 1024) = 1024
132066 read(4, "b968701cd3a58b18d790c5601/merged"..., 1024) = 1024
132066 read(4, "60a0a54b1608f500d0bd66e4ab28bd6d"..., 1024) = 1024
132066 read(4, "7d9d2d37/merged overlay rw,secla"..., 1024) = 1024
132066 read(4, "b8faf9667fc2a0bf755eb57bf4315bd/"..., 1024) = 1024
132066 read(4, "0272c09b34b6a59894bec081972aff77"..., 1024) = 1024
132066 read(4, "latime,lowerdir=/var/lib/docker/"..., 1024) = 1024
132066 read(4, "tmpfs rw,seclabel,relatime 0 0\no"..., 1024) = 1024
132066 read(4, "6ae83c0f426419532642913841d367d7"..., 1024) = 1024
132066 read(4, "61b4/work 0 0\noverlay /var/lib/d"..., 1024) = 1024
132066 read(4, "4d5e5678e4ebb09334056cf4d4bbc37e"..., 1024) = 1024
132066 read(4, "d30890fe8d9d7733/upper,workdir=/"..., 1024) = 1024
132066 read(4, "cker/overlay/07ef1c31ef2e2a4bdfa"..., 1024) = 1024
132066 read(4, "069c86989d3ed346e2bcfa3e59305ecd"..., 1024) = 1024
132066 read(4, " 0\nshm /var/lib/docker/container"..., 1024) = 1024
132066 read(4, "29db4a56add2f6f4a31cbe6302a8a96a"..., 1024) = 1024
132066 read(4, "y/4502a0aaf969ad02344062e2c30a94"..., 1024) = 1024
132066 read(4, "869ff7cfce56845562753ed8b206cef3"..., 1024) = 1024
132066 read(4, "71aca264fe7dbedf11c74df4a9c19f73"..., 1024) = 1024
132066 read(4, "cd41cdc5e1467/upper,workdir=/var"..., 1024) = 1024
132066 read(4, "y /var/lib/docker/overlay/171cc7"..., 1024) = 1024
132066 read(4, "dir=/var/lib/docker/overlay/d75f"..., 1024) = 1024
132066 read(4, "4b99d3bd80356131daca1e52fc6ae34d"..., 1024) = 1024
132066 read(4, "34ce2c5b3f0c50884d0c235b513211d5"..., 1024) = 1024
132066 read(4, "620515ad/merged overlay rw,secla"..., 1024) = 1024
132066 read(4, "b8b984f9a6735c10f02ca98161f6dfaf"..., 1024) = 1024
132066 read(4, "verlay/8eac9bf965261c6606bace18a"..., 1024) = 1024
132066 read(4, "0c6c127fbd685955f6ea262d21a89dfb"..., 1024) = 1024
132066 read(4, "1991d3d8176ad2cd/merged overlay "..., 1024) = 1024
132066 read(4, "f2d7-6bea-4300-b34b-658f0ff15f82"..., 1024) = 507
132066 read(4, "", 1024)                = 0
132066 close(3)                         = 0
132066 munmap(0x7f51d43b7000, 4096)     = 0
132066 close(4)                         = 0
132066 munmap(0x7f51d43b6000, 4096)     = 0
132066 open("/proc/mounts", O_RDONLY|O_CLOEXEC) = 3
132066 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
132066 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f51d43b7000
132066 read(3, "rootfs / rootfs rw 0 0\nsysfs /sy"..., 1024) = 1024
132066 close(3)                         = 0
132066 munmap(0x7f51d43b7000, 4096)     = 0
132066 mkdir("/sys", 0775)              = -1 EEXIST (File exists)
132066 mkdir("/sys/fs", 0775)           = -1 EEXIST (File exists)
132066 mkdir("/sys/fs/cgroup", 0775)    = -1 EEXIST (File exists)
132066 mkdir("/sys/fs/cgroup/memory", 0775) = -1 EEXIST (File exists)
132066 mkdir("/sys/fs/cgroup/memory/kubepods", 0775) = -1 EEXIST (File exists)
132066 mkdir("/sys/fs/cgroup/memory/kubepods/burstable", 0775) = -1 EEXIST (File exists)
132066 mkdir("/sys/fs/cgroup/memory/kubepods/burstable/IODINE", 0775) = -1 ENOMEM (Cannot allocate memory)                   <<=----------------
132066 stat("/sys/fs/cgroup/memory/kubepods/burstable/IODINE", 0x7ffea363eac0) = -1 ENOENT (No such file or directory)
132066 write(2, "cgcreate: can't create cgroup ku"..., 87) = 87
132066 exit_group(50007)                = ?
132066 +++ exited with 87 +++


I do see that 'ip_vs' is loaded, but I think problem is different here than bug 1540585. Could you confirm and give suggestions.

I am asking customer to test 1062.18.1.el7 because of bug 1768386 [Sanitize MM backported code for RHEL7 [rhel-7.7.z]]
I will share the result if he is able to test.

He did not got a reproducer, but it happens almost twice a week on production kubernetes server.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1016

Hello,

I am getting this bug on 3.10.0-1062.18.1.el7.x86_64.

Nov  5 09:19:21 server-with-a-problem dockerd: time="2020-11-05T09:19:21.562680365-05:00" level=error msg="Handler for POST /containers/4e00783f74ad9033b146fb5ef7a2c2499ffb74e25cf7800fe6e479a1385180c5/start returned error: OCI runtime create failed: container_linux.go:349: starting container process caused \"process_linux.go:297: applying cgroup configuration for process caused \\\"mkdir /sys/fs/cgroup/memory/kubepods/burstable/pod5e1a5f41-c6ce-47db-b640-e9db42f79b37/4e00783f74ad9033b146fb5ef7a2c2499ffb74e25cf7800fe6e479a1385180c5: cannot allocate memory\\\"\": unknown"

I am ***not using cgroup.memory=nokmem*** as I thought the bug was fixed in 1062.18.1.

Has anyone tested 1062.18.1?

I have obviously not experienced the bug with cgroup.memory=nokmem.

Thanks,
Doug