1783889 – RHCOS rebooted suddenly after a lot of running processes killed by OOM-killer

Bug 1783889 - RHCOS rebooted suddenly after a lot of running processes killed by OOM-killer

Summary: RHCOS rebooted suddenly after a lot of running processes killed by OOM-killer

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	RHCOS
Sub Component:
Version:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	medium
Target Milestone:	---
Target Release:	4.6.0
Assignee:	Colin Walters
QA Contact:	Michael Nguyen
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1186913
TreeView+	depends on / blocked

Reported:	2019-12-16 08:09 UTC by Daein Park
Modified:	2023-09-07 21:16 UTC (History)
CC List:	10 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-06-17 17:21:45 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Daein Park 2019-12-16 08:09:00 UTC

Description of problem:

OCP master node(RHCOS) rebooted suddenly after a lot of OOM-killer as follows.

e.g.>
~~~
Dec 14 00:09:11 ip-10-0-1-1 systemd[1]: systemd-logind.service: Killing process 60351 (systemd-logind) with signal SIGKILL.
Dec 14 00:09:11 ip-10-0-1-1 systemd[1]: systemd-logind.service: Killing process 60355 (systemd-logind) with signal SIGKILL.
Dec 14 00:09:11 ip-10-0-1-1 systemd[1]: systemd-logind.service: Killing process 60358 (systemd-logind) with signal SIGKILL.
Dec 14 00:09:11 ip-10-0-1-1 systemd[1]: systemd-logind.service: Killing process 60362 (systemd-logind) with signal SIGKILL.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Processes still around after final SIGKILL. Entering failed mode.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: Failed to start Login Service.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Consumed 22ms CPU time
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 46.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: Stopped Login Service.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Consumed 22ms CPU time
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Found left-over process 995 (systemd-logind) in control group while starting unit. Ignoring.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Found left-over process 59880 (systemd-logind) in control group while starting unit. Ignoring.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: systemd-logind.service: Found left-over process 60116 (systemd-logind) in control group while starting unit. Ignoring.
Dec 14 00:10:41 ip-10-0-1-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
:
Dec 14 19:20:22 ip-10-0-1-1 kernel: kthreadd invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=1, oom_score_adj=0
Dec 14 19:20:22 ip-10-0-1-1 kernel: kthreadd cpuset=/ mems_allowed=0
Dec 14 19:20:22 ip-10-0-1-1 kernel: CPU: 2 PID: 2 Comm: kthreadd Not tainted 4.18.0-80.11.2.el8_0.x86_64 #1
Dec 14 19:20:22 ip-10-0-1-1 kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Dec 14 19:20:22 ip-10-0-1-1 kernel: Call Trace:
Dec 14 19:20:22 ip-10-0-1-1 kernel:  dump_stack+0x5c/0x80
Dec 14 19:20:22 ip-10-0-1-1 kernel:  dump_header+0x6e/0x27a
Dec 14 19:20:22 ip-10-0-1-1 kernel:  oom_kill_process.cold.26+0xb/0x389
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? oom_badness+0xd9/0x130
Dec 14 19:20:22 ip-10-0-1-1 kernel:  out_of_memory+0x1b7/0x4a0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  __alloc_pages_slowpath+0xbf4/0xcd0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  __alloc_pages_nodemask+0x252/0x270
Dec 14 19:20:22 ip-10-0-1-1 kernel:  new_slab+0x608/0x660
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ___slab_alloc+0x36a/0x4e0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? copy_process.part.31+0xfc/0x15a0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? xen_clocksource_read+0x11/0x20
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? __switch_to_asm+0x40/0x70
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? copy_process.part.31+0xfc/0x15a0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  __slab_alloc+0x1c/0x30
Dec 14 19:20:22 ip-10-0-1-1 kernel:  kmem_cache_alloc_node+0xa8/0x1f0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  copy_process.part.31+0xfc/0x15a0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  __slab_alloc+0x1c/0x30
Dec 14 19:20:22 ip-10-0-1-1 kernel:  kmem_cache_alloc_node+0xa8/0x1f0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  copy_process.part.31+0xfc/0x15a0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? __switch_to_asm+0x40/0x70
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? __switch_to_asm+0x34/0x70
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? kthread_bind+0x30/0x30
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? __switch_to_asm+0x34/0x70
Dec 14 19:20:22 ip-10-0-1-1 kernel:  _do_fork+0xe2/0x390
Dec 14 19:20:22 ip-10-0-1-1 kernel:  kernel_thread+0x25/0x30
Dec 14 19:20:22 ip-10-0-1-1 kernel:  kthreadd+0x2b0/0x300
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ? kthread_create_on_cpu+0xa0/0xa0
Dec 14 19:20:22 ip-10-0-1-1 kernel:  ret_from_fork+0x35/0x40
Dec 14 19:20:22 ip-10-0-1-1 kernel: Mem-Info:
Dec 14 19:20:22 ip-10-0-1-1 kernel: active_anon:196868 inactive_anon:630 isolated_anon:0
                                         active_file:2318 inactive_file:33873 isolated_file:0
                                         unevictable:0 dirty:86 writeback:0 unstable:0
                                         slab_reclaimable:415036 slab_unreclaimable:3332606
                                         mapped:13800 shmem:3077 pagetables:12486 bounce:0
                                         free:36437 free_pcp:50 free_cma:0
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 active_anon:787472kB inactive_anon:2520kB active_file:9272kB inactive_file:135492kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:55396kB dirty:540kB writeback:0kB shmem:12308kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 100352kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 DMA free:15904kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: lowmem_reserve[]: 0 3699 15955 15955 15955
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 DMA32 free:68268kB min:15652kB low:19564kB high:23476kB active_anon:258884kB inactive_anon:316kB active_file:8kB inactive_file:30756kB unevictable:0kB writepending:40kB present:3915776kB managed:3850240kB mlocked:0kB kernel_stack:2192kB pagetables:14152kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: lowmem_reserve[]: 0 0 12255 12255 12255
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 Normal free:61060kB min:51860kB low:64824kB high:77788kB active_anon:528588kB inactive_anon:2204kB active_file:9992kB inactive_file:103860kB unevictable:0kB writepending:400kB present:12845056kB managed:12554288kB mlocked:0kB kernel_stack:10192kB pagetables:35792kB bounce:0kB free_pcp:192kB local_pcp:0kB free_cma:0kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: lowmem_reserve[]: 0 0 0 0 0
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 DMA32: 3099*4kB (UME) 3*8kB (U) 938*16kB (M) 855*32kB (UM) 200*64kB (UM) 8*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68612kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 Normal: 11310*4kB (UMEH) 2081*8kB (UMEH) 1*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 61904kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: 39207 total pagecache pages
Dec 14 19:20:22 ip-10-0-1-1 kernel: 0 pages in swap cache
Dec 14 19:20:22 ip-10-0-1-1 kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 14 19:20:22 ip-10-0-1-1 kernel: Free swap  = 0kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Total swap = 0kB
Dec 14 19:20:22 ip-10-0-1-1 kernel: 4194205 pages RAM
Dec 14 19:20:22 ip-10-0-1-1 kernel: 0 pages HighMem/MovableOnly
Dec 14 19:20:22 ip-10-0-1-1 kernel: 89097 pages reserved
Dec 14 19:20:22 ip-10-0-1-1 kernel: 0 pages hwpoisoned
Dec 14 19:20:22 ip-10-0-1-1 kernel: Unreclaimable slab info:
Dec 14 19:20:22 ip-10-0-1-1 kernel: Name                      Used          Total
Dec 14 19:20:22 ip-10-0-1-1 kernel: sw_flow               650936KB     650936KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: nf_conntrack             147KB        625KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_dqtrx                 63KB         63KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_dquot                 31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_rud_item              15KB         15KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_efd_item             265KB        265KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_buf_item              43KB         67KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_trans                372KB        372KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_da_state              63KB         63KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfs_log_ticket           201KB        201KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: scsi_sense_cache           4KB          4KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: ip6-frags                 31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: ip6_dst_cache            617KB        617KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: RAWv6                    378KB        378KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: UDPv6                    126KB        126KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: tw_sock_TCPv6             47KB         47KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: request_sock_TCPv6         31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: TCPv6                  22417KB      22620KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: sgpool-128                32KB         32KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: mqueue_inode_cache        127KB        127KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kioctx                    15KB         15KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: pid_namespace             31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: secpath_cache            492KB        492KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: xfrm_dst_cache           334KB        346KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: PING                     570KB        570KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: RAW                      535KB        535KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: UDP                      125KB        125KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: tw_sock_TCP              111KB        111KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: request_sock_TCP         354KB        354KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: TCP                      895KB        895KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: hugetlbfs_inode_cache         31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: eventpoll_pwq             23KB         23KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: request_queue             30KB         30KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: blkdev_ioc                74KB         74KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: biovec-max               344KB        400KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: biovec-128               136KB        180KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: biovec-64                132KB        152KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: dmaengine-unmap-256         30KB         30KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: dmaengine-unmap-128         31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: dmaengine-unmap-16       1376KB       2287KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: skbuff_fclone_cache        612KB        612KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: skbuff_head_cache        522KB        616KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: file_lock_cache           31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: net_namespace            257KB        257KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: shmem_inode_cache        950KB        950KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: taskstats                149KB        149KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: proc_dir_entry           680KB        680KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: pde_opener                15KB         15KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: seq_file                  88KB         88KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: sigqueue                  66KB         66KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kernfs_node_cache       6663KB       7685KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: mnt_cache               2570KB       2578KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: filp                   29304KB      29304KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: names_cache              288KB        288KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: ebitmap_node            1319KB       1456KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: avtab_node              9000KB       9000KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: avc_xperms_data        15964KB      15964KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: selinux_file_security        720KB        720KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: uts_namespace             63KB         63KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: vm_area_struct           422KB        422KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: mm_struct                255KB        255KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: files_cache              205KB        205KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: signal_cache            1475KB       1475KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: sighand_cache            773KB        773KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: task_struct             8104KB       8104KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: cred_jar                 771KB        771KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: anon_vma                 146KB        146KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: pid                      289KB        352KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Acpi-Operand             614KB        614KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Acpi-Parse                31KB         31KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Acpi-State               115KB        115KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: Acpi-Namespace        211319KB     211319KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: numa_policy                7KB          7KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: trace_event_file         173KB        173KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: ftrace_event_field        207KB        207KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: pool_workqueue        762149KB     762160KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: task_group               777KB        777KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-8192            1072KB       1120KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-4096           18256KB      18256KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-2048         2319692KB    2319692KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-1024            7774KB      10656KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-512           532168KB     532168KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-256            21776KB      21776KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-192            15868KB      15868KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-128           588236KB     588236KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-96             14785KB      14785KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-64            521100KB     521100KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-32            152571KB     152572KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-16              4972KB       4972KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmalloc-8              23344KB      23344KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmem_cache_node          936KB        936KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: kmem_cache              6548KB       6548KB
Dec 14 19:20:22 ip-10-0-1-1 kernel: [ pid ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Dec 14 19:20:22 ip-10-0-1-1 kernel: [  758]     0   758    26119      694   233472        0         -1000 systemd-udevd
Dec 14 19:20:22 ip-10-0-1-1 kernel: [  950]     0   950    15794      168   151552        0         -1000 auditd
Dec 14 19:20:22 ip-10-0-1-1 kernel: [  999]     0   999    93862        0   364544        0             0 NetworkManager
Dec 14 19:20:22 ip-10-0-1-1 kernel: [ 1025]     0  1025    20490      218   200704        0         -1000 sshd
Dec 14 19:20:22 ip-10-0-1-1 kernel: [ 1117]     0  1100   451831        0   499712        0          -999 crio
Dec 14 19:20:22 ip-10-0-1-1 kernel: [ 1174]     0  1168   581788        0   888832        0          -999 hyperkube
Dec 14 19:20:22 ip-10-0-1-1 kernel: [19710]    99 19689    32155        0   122880        0          -998 kube-rbac-proxy
Dec 14 19:20:22 ip-10-0-1-1 kernel: [112893]     0 112776   277365        0   548864        0          -998 hyperkube
Dec 14 19:20:22 ip-10-0-1-1 kernel: [43757]     0 43757    21519       82    73728        0          -999 conmon
Dec 14 19:20:22 ip-10-0-1-1 kernel: [43767]     0 43767    21519      120    69632        0          -999 conmon
Dec 14 19:20:22 ip-10-0-1-1 kernel: [44010]     0 43781   565308        0  3096576        0          -998 hyperkube
Dec 14 19:20:22 ip-10-0-1-1 kernel: [43795]     0 43795   178997     6541   532480        0          -998 hyperkube
Dec 14 19:20:22 ip-10-0-1-1 kernel: [43895]     0 43895    21519       82    65536        0          -999 conmon
Dec 14 19:20:22 ip-10-0-1-1 kernel: [43910]     0 43910   151513     5292   307200        0          -998 cluster-kube-co
Dec 14 19:20:22 ip-10-0-1-1 kernel: [43911]     0 43911    21519       81    69632        0          -999 conmon
Dec 14 19:20:22 ip-10-0-1-1 kernel: [43937]     0 43937   149439     5629   344064        0          -998 cluster-kube-ap
Dec 14 19:20:22 ip-10-0-1-1 kernel: [44008]     0 44008    21519       81    73728        0          -999 conmon
Dec 14 19:20:22 ip-10-0-1-1 kernel: [44029]     0 44029   128893     2324   266240        0          -998 cluster-kube-ap
Dec 14 19:20:22 ip-10-0-1-1 kernel: [45206]     0 45206    21519       78    73728        0          -999 conmon
Dec 14 19:20:22 ip-10-0-1-1 kernel: [45207]     0 45207   159489     1968   172032        0          -999 runc
Dec 14 19:20:22 ip-10-0-1-1 kernel: [45216]     0 45216     5065      812    77824        0          -999 exe
Dec 14 19:20:22 ip-10-0-1-1 kernel: [45224]     0 45224    21519       78    77824        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45225]     0 45225   159425     1423   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45234]     0 45234     5065      813    94208        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45245]     0 45245    21519       79    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45246]     0 45246   141344     1417   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45255]     0 45255     5065      811    90112        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45296]     0 45296    21519       77    69632        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45297]     0 45297   159841     1421   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45306]     0 45306     5065      813    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45510]     0 45510     9717      217   122880        0          -998 iptables-restor
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45543]     0 45543     7126      198    98304        0          -998 iptables
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45781]     0 45781     6607        0    98304        0          -999 iptables
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45856]     0 45856    21519       77    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45857]     0 45857   141344     1413   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [45865]     0 45865     5065      812    77824        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46140]     0 46140    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46141]     0 46141   141280     2446   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46149]     0 46149     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46160]     0 46160    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46161]     0 46161   141280     2451   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46169]     0 46169     5065      811    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46210]     0 46210    21519       77    77824        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46211]     0 46211   141344     2368   163840        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [46219]     0 46219     5065      812    81920        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47311]     0 47311    21519       77    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47312]     0 47312   140992     2414   163840        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47320]     0 47320     5065      812    81920        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47329]     0 47329    21519       77    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47330]     0 47330   159777     1430   184320        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47339]     0 47339     5065      813    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47384]     0 47384    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47385]     0 47385   141280     1352   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [47393]     0 47393     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48068]     0 48068    21519       78    69632        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48069]     0 48069   159905     1497   176128        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48079]     0 48079     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48106]     0 48106    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48107]     0 48107   159777     1343   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48116]     0 48116     5065      811    81920        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48144]     0 48144    21519       77    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48145]     0 48145   159777     1343   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48154]     0 48154     5065      811    81920        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48853]     0 48853    21519       78    77824        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48854]     0 48854   140992     1342   163840        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48862]     0 48862     5065      811    81920        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48872]     0 48872    21519       77    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48873]     0 48873   140992     1341   163840        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48881]     0 48881     5065      811    90112        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48912]     0 48912    21519       78    77824        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48913]     0 48913   159489     1426   163840        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [48922]     0 48922     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49621]     0 49621    21519       77    65536        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49622]     0 49622   141408     1473   163840        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49630]     0 49630     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49645]     0 49645    21519       81    69632        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49646]     0 49646   141280     1338   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49654]     0 49654     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49674]     0 49674    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49675]     0 49675   141280     1336   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [49683]     0 49683     5065      811    90112        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50394]     0 50394    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50395]     0 50395   141344     1413   159744        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50403]     0 50403     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50412]     0 50412    21519       77    77824        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50413]     0 50413   141344     1341   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50421]     0 50421     5065      811    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50455]     0 50455    21519       77    81920        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50456]     0 50456   141280     1343   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [50464]     0 50464     5065      811    77824        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51161]     0 51161    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51162]     0 51162   140992     1341   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51170]     0 51170     5065      811    81920        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51179]     0 51179    21519       78    77824        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51180]     0 51180   159841     1480   172032        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51189]     0 51189     5065      811    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51224]     0 51224    21519       77    69632        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51225]     0 51225   140992     1358   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51233]     0 51233     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51916]     0 51916    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51917]     0 51917   141344     1405   163840        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51925]     0 51925     5065      813    81920        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51935]     0 51935    21519       78    73728        0          -999 conmon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51936]     0 51936   141408     1415   167936        0          -999 runc
Dec 14 19:20:23 ip-10-0-1-1 kernel: [51947]     0 51947     5065      812    86016        0          -999 exe
Dec 14 19:20:23 ip-10-0-1-1 kernel: [59595]     0 59595    21745        0   204800        0             0 systemd-user-ru
Dec 14 19:20:23 ip-10-0-1-1 kernel: [59834]     0 59834    45284        0   221184        0           500 (coredump)
Dec 14 19:20:23 ip-10-0-1-1 kernel: [59881]    81 59881    17951     1733   176128        0          -900 dbus-daemon
Dec 14 19:20:23 ip-10-0-1-1 kernel: [59885]     0 59885    94106        0   356352        0             0 NetworkManager
Dec 14 19:20:23 ip-10-0-1-1 kernel: [59893]     0 59893    61668        0   225280        0             0 (ostnamed)
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60116]     0 60116    21200        0   184320        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60119]     0 60119    21200        0   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60124]     0 60124    21200        0   212992        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60127]     0 60127    21200        0   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60130]     0 60130    21200        0   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60134]     0 60134    21200        0   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60138]     0 60138    21200        0   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60142]     0 60142    21200        0   212992        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60146]     0 60146    21200        0   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60173]     0 60173    21200        0   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60211]     0 60211    21200        0   212992        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60228]     0 60228    21200        0   188416        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60260]     0 60260     4222       35    77824        0             0 agetty
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60267]     0 60267    57932    14617  1228800        0             0 systemd-journal
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60272]     0 60272     4106       41    77824        0             0 agetty
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60273]     0 60273    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60276]     0 60276    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60281]     0 60281    21200      535   192512        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60284]     0 60284    21200      541   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60287]     0 60287    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60293]     0 60293    21200      538   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60297]     0 60297    21200      537   192512        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60300]     0 60300    21200      542   217088        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60304]     0 60304    21200      537   212992        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60307]     0 60307    21200      542   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60312]     0 60312    21200      540   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60315]     0 60315    21200      539   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60318]     0 60318    21200      542   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60322]     0 60322    21200      542   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60326]     0 60326    21200      539   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60329]     0 60329    21200      541   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60333]     0 60333    21200      537   212992        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60336]     0 60336    21200      539   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60340]     0 60340    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60344]     0 60344    21200      538   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60347]     0 60347    21200      536   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60351]     0 60351    21200      537   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60355]     0 60355    21200      537   184320        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60358]     0 60358    21200      541   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60362]     0 60362    21200      539   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60365]     0 60365    21200      540   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60369]     0 60369    21200      537   188416        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60373]     0 60373    21200      537   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60376]     0 60376    21200      537   188416        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60379]     0 60379    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60384]     0 60384    21200      542   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60398]     0 60398    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60402]     0 60402    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60405]     0 60405    21200      537   188416        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60408]     0 60408    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60413]     0 60413    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60416]     0 60416    21200      537   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60419]     0 60419    21200      538   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60423]     0 60423    21200      541   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60427]     0 60427    21200      541   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60430]     0 60430    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60434]     0 60434    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60437]     0 60437    21200      542   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60441]     0 60441    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60445]     0 60445    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60448]     0 60448    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60452]     0 60452    21200      540   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60456]     0 60456    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60459]     0 60459    21200      539   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60463]     0 60463    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60466]     0 60466    21200      539   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60470]     0 60470    21200      536   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60474]     0 60474    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60477]     0 60477    21200      541   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60480]     0 60480    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60485]     0 60485    21200      536   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60488]     0 60488    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60491]     0 60491    21200      538   188416        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60495]     0 60495    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60499]     0 60499    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60503]     0 60503    21200      541   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60506]     0 60506    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60509]     0 60509    21200      542   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60514]     0 60514    21200      538   192512        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60517]     0 60517    21200      538   184320        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60520]     0 60520    21200      542   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60524]     0 60524    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60528]     0 60528    21200      536   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60532]     0 60532    21200      535   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60535]     0 60535    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60538]     0 60538    21200      542   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60543]     0 60543    21200      539   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60546]     0 60546    21200      540   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60549]     0 60549    21200      540   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60552]     0 60552    21200      541   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60557]     0 60557    21200      541   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60560]     0 60560    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60564]     0 60564    21200      541   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60567]     0 60567    21200      542   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60571]     0 60571    21200      541   188416        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60575]     0 60575    21200      542   208896        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60578]     0 60578    21200      540   192512        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60581]     0 60581    21200      539   212992        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60586]     0 60586    21200      537   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60589]     0 60589    21200      538   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60593]     0 60593    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60596]     0 60596    21200      541   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60600]     0 60600    21200      541   192512        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60604]     0 60604    21200      541   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60607]     0 60607    21200      539   196608        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60610]     0 60610    21200      542   192512        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60615]     0 60615    21200      540   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60618]     0 60618    21200      541   188416        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60621]     0 60621    21200      535   204800        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60625]     0 60625    21200      539   200704        0             0 systemd-logind
Dec 14 19:20:23 ip-10-0-1-1 kernel: [60629]     0 60629    21200      541   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60632]     0 60632    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60636]     0 60636    21200      539   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60639]     0 60639    21200      539   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60644]     0 60644    21200      535   184320        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60647]     0 60647    21200      540   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60650]     0 60650    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60654]     0 60654    21200      538   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60658]     0 60658    21200      538   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60661]     0 60661    21200      538   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60665]     0 60665    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60668]     0 60668    21200      538   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60672]     0 60672    21200      542   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60676]     0 60676    21200      540   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60679]     0 60679    21200      536   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60682]     0 60682    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60687]     0 60687    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60690]     0 60690    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60694]     0 60694    21200      541   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60697]     0 60697    21200      538   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60701]     0 60701    21200      538   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60705]     0 60705    21200      538   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60708]     0 60708    21200      541   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60711]     0 60711    21200      537   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60716]     0 60716    21200      542   212992        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60719]     0 60719    21200      541   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60723]     0 60723    21200      538   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60726]     0 60726    21200      537   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60730]     0 60730    21200      541   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60734]     0 60734    21200      538   184320        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60737]     0 60737    21200      537   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60740]     0 60740    21200      538   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60745]     0 60745    21200      539   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60748]     0 60748    21200      534   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60751]     0 60751    21200      539   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60754]     0 60754    21200      541   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60759]     0 60759    21200      535   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60762]     0 60762    21200      538   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60766]     0 60766    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60769]     0 60769    21200      541   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60773]     0 60773    21200      543   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60777]     0 60777    21200      535   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60780]     0 60780    21200      538   212992        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60783]     0 60783    21200      536   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60788]     0 60788    21200      541   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60791]     0 60791    21200      542   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60794]     0 60794    21200      538   212992        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60798]     0 60798    21200      535   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60802]     0 60802    21200      534   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60805]     0 60805    21200      539   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60809]     0 60809    21200      537   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60812]     0 60812    21200      542   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60817]     0 60817    21200      541   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60820]     0 60820    21200      542   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60823]     0 60823    21200      541   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60827]     0 60827    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60831]     0 60831    21200      540   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60834]     0 60834    21200      538   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60838]     0 60838    21200      539   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60841]     0 60841    21200      537   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60846]     0 60846    21200      538   212992        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60849]     0 60849    21200      540   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60852]     0 60852    21200      536   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60856]     0 60856    21200      539   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60860]     0 60860    21200      542   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60863]     0 60863    21200      538   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60867]     0 60867    21200      539   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60871]     0 60871    21200      537   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60875]     0 60875    21200      541   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60879]     0 60879    21200      541   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60882]     0 60882    21200      542   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60886]     0 60886    21200      538   192512        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60890]     0 60890    21200      542   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60894]     0 60894    21200      541   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60897]     0 60897    21200      539   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60900]     0 60900    21200      537   188416        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60905]     0 60905    21200      536   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60912]     0 60912    21200      536   208896        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60917]     0 60917    21200      535   204800        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60925]     0 60925    21200      538   200704        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: [60929]     0 60929    21200      534   196608        0             0 systemd-logind
Dec 14 19:20:24 ip-10-0-1-1 kernel: Out of memory: Kill process 60267 (systemd-journal) score 3 or sacrifice child
Dec 14 19:21:57 ip-10-0-1-1 systemd-journald[60933]: Journal started
Dec 14 19:21:57 ip-10-0-1-1 systemd-journald[60933]: System journal (/var/log/journal/ab62a6248306453883a5d8995fad7646) is 4.0G, max 4.0G, 0B free.
Dec 14 19:21:57 ip-10-0-1-1 kernel: Killed process 60267 (systemd-journal) total-vm:231728kB, anon-rss:2244kB, file-rss:58328kB, shmem-rss:4kB
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 30.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: Stopped Journal Service.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-journald.service: Consumed 7.033s CPU time
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: Starting Journal Service...
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-logind.service: Processes still around after final SIGKILL. Entering failed mode.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-logind.service: Failed with result 'timeout'.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: Failed to start Login Service.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-logind.service: Consumed 28.708s CPU time
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-logind.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 199.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: Stopped Login Service.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-logind.service: Consumed 28.708s CPU time
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: systemd-logind.service: Found left-over process 995 (systemd-logind) in control group while starting unit. Ignoring.
Dec 14 19:21:57 ip-10-0-1-1 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Dec 14 19:20:24 ip-10-0-1-1 systemd[1]: systemd-journald.service: Main process exited, code=killed, status=9/KILL
Dec 14 19:20:24 ip-10-0-1-1 systemd[1]: systemd-journald.service: Failed with result 'signal'.
Dec 14 19:20:24 ip-10-0-1-1 systemd[1]: systemd-journald.service: Consumed 7.033s CPU time
Dec 14 19:21:58 ip-10-0-1-1 systemd[1]: Starting Login Service...
Dec 14 19:21:58 ip-10-0-1-1 systemd[1]: Started Journal Service.
Dec 14 19:21:58 ip-10-0-1-1 systemd-logind[60936]: New seat seat0.
Dec 14 19:21:58 ip-10-0-1-1 systemd-logind[60936]: Watching system buttons on /dev/input/event0 (Power Button)
Dec 14 19:21:58 ip-10-0-1-1 systemd-logind[60936]: Watching system buttons on /dev/input/event1 (Sleep Button)
Dec 14 19:21:58 ip-10-0-1-1 systemd-logind[60936]: Watching system buttons on /dev/input/event2 (AT Translated Set 2 keyboard)
Dec 14 19:23:21 ip-10-0-1-1 systemd[1]: systemd-logind.service: Start operation timed out. Terminating.
Dec 14 19:24:52 ip-10-0-1-1 systemd[1]: systemd-logind.service: State 'stop-sigterm' timed out. Killing.
Dec 14 19:24:52 ip-10-0-1-1 systemd[1]: systemd-logind.service: Killing process 60936 (systemd-logind) with signal SIGKILL.
:
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61245]     0 61245    21200        0   196608        0             0 systemd-logind
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61298]     0 61298    21200        0   196608        0             0 systemd-logind
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61320]     0 61320    21200        0   204800        0             0 systemd-logind
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61449]     0 61449    21200        0   204800        0             0 systemd-logind
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61471]     0 61471    21200        0   184320        0             0 systemd-logind
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61708]     0 61708    21200      703   188416        0             0 systemd-logind
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61709]     0 61709    22838     1525   241664        0             0 systemd-journal
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61710]     0 61710    61668     1723   225280        0             0 (agetty)
Dec 15 01:32:20 ip-10-0-1-1 kernel: [61712]    81 61712    16284      264   159744        0          -900 dbus-daemon
Dec 15 01:32:20 ip-10-0-1-1 kernel: Out of memory: Kill process 61709 (systemd-journal) score 0 or sacrifice child
-- Reboot --
Dec 15 01:34:06 localhost kernel: Linux version 4.18.0-80.11.2.el8_0.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)) #1 SMP Sun Sep 15 11:24:21 UTC 2019
Dec 15 01:34:06 localhost kernel: Command line: BOOT_IMAGE=/ostree/rhcos-ee73255863c1e88baaa06eac3fff2d753f20da9a49139224bb88dbfe2a7ba712/vmlinuz-4.18.0-80.11.2.el8_0.x86_64 console=tty0 console=ttyS0,115200n8 rootflags=defaults,prjquota rw root=UUID=477c3d77-20c6-4ff3-8bb3-dc2543eedfbd ostree=/ostree/boot.1/rhcos/ee73255863c1e88baaa06eac3fff2d753f20da9a49139224bb88dbfe2a7ba712/0 coreos.oem.id=qemu coreos.oem.id=ec2 ignition.platform.id=ec2
Dec 15 01:34:06 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Dec 15 01:34:06 localhost kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Dec 15 01:34:06 localhost kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Dec 15 01:34:06 localhost kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Dec 15 01:34:06 localhost kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Dec 15 01:34:06 localhost kernel: BIOS-provided physical RAM map:
:
~~~

Version-Release number of selected component (if applicable):

OCP 4.2.0

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:

master node rebooted suddenly.

Expected results:

master node is running without reboot.

Additional info:

oom-killer happens 290 times.

Comment 8 Colin Walters 2020-06-10 19:02:46 UTC

How reproducible is this?  Did it just happen once?

That's a lot of systemd-logind processes.  This could be a systemd bug, but is there e.g. something logging in a lot?

The ask here is about a root cause for the reboot...I don't see anything obvious, but I would certainly try to address the root cause of the OOM pressure by e.g. setting up resource quotas if you haven't https://docs.openshift.com/container-platform/4.3/applications/quotas/quotas-setting-across-multiple-projects.html

Comment 10 Colin Walters 2020-06-17 17:21:45 UTC

The OpenShift program currently requires us to take action on every open bug.  We haven't had any other reports of this and can't justify trying to root cause this bug at this time versus e.g. working on bugs or features that apply to multiple customers or across platforms.

Please reopen if the bug reoccurs, or any substantial new information is provided.

Note You need to log in before you can comment on or make changes to this bug.