Bug 1713999

Summary: Stopping and Starting OCP 4.1 cluster on AWS kills the cluster.
Product: OpenShift Container Platform Reporter: Andrew Pitt <apitt>
Component: NodeAssignee: Ryan Phillips <rphillips>
Status: CLOSED DUPLICATE QA Contact: Jianwei Hou <jhou>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, bleanhar, eparis, jokerman, mifiedle, mmccomas, piqin, schoudha, shlao, tnozicka, xtian, xxia, yinzhou
Target Milestone: ---Keywords: Reopened
Target Release: 4.1.z   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-03 15:30:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andrew Pitt 2019-05-26 13:27:28 UTC 
Description of problem:
OpenShift 4.1.rc cluster is dead after shutting down VM nodes and restarting them.

This hss now happened three times to me on AWS.  Twice in ca-central-1 (Canada) and once in us-east-2 (Ohio).

Version-Release number of selected component (if applicable):
OpenShift 4.1 based on the following installers:
openshift-install-mac-4.1.0-rc.3
openshift-install-mac-4.1.0-rc.4
openshift-install-mac-4.1.0-rc.5

How reproducible:
For me, this has happened for each of the 3 clusters I have tried to create.  The first two clusters died after the 2nd time I tried to start them (they survived one restart just fine).  The 3rd cluster died after the 1st restart.

Steps to Reproduce:
1. With a fresh AWS region, install a new OCP cluster using the Mac installer.
2. Once the cluster has successfully been installed and tested (CLI and Console working properly), leave the cluster running for a few hours.
3. Stop the cluster by selecting all EC2 instances from the AWS console and selecting "Actions -> Instance state -> Stop"
4. Later (the next day in my case) restart the cluster by selecting all EC2 instances from the AWS dashboard and selecting "Actions -> Instance state -> Start"
5. Wait for VMs to fully boot.
6. Try to access the web console.  If it doesn't respond, check the VM logs.  You will probably see a line stating "Failed to start Kubernetes Kubelet".

Actual results:
After 1 or 2 restarts of the cluster, the cluster is dead and can't be recovered.

Expected results:
Cluster restarts shouldn't kill the cluster.

Additional info:
Logs from one of the Master nodes:

[22;1H   The selected entry will be started automatically in 1s.                     [4;79H[22;1H   The selected entry will be started automatically in 0s.                     [4;79H[?25h[H[J[1;1H[H[J[1;1H[    0.000000] Linux version 4.18.0-80.1.2.el8_0.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)) #1 SMP Sun Apr 28 09:21:22 UTC 2019
[    0.000000] Command line: BOOT_IMAGE=/ostree/rhcos-46868cfc9ed0776a5e535bc23872bc145cabe7c4f88e92729d6ff4147791a641/vmlinuz-4.18.0-80.1.2.el8_0.x86_64 console=tty0 console=ttyS0,115200n8 rootflags=defaults,prjquota rw root=UUID=b2f3c8ee-9c4b-4d07-87b3-24f4af95a40b ostree=/ostree/boot.1/rhcos/46868cfc9ed0776a5e535bc23872bc145cabe7c4f88e92729d6ff4147791a641/0 coreos.oem.id=ec2 ignition.platform.id=ec2
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000efffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000040fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[    0.000000] Hypervisor detected: Xen HVM
[    0.000000] Xen version 4.2.
[    0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[    0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[    0.000000] You might have to change the root device
[    0.000000] from /dev/hd[a-d] to /dev/xvd[a-d]
[    0.000000] in your root= kernel command line option
[    0.000000] last_pfn = 0x410000 max_arch_pfn = 0x400000000
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000000] last_pfn = 0xf0000 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fbc50-0x000fbc5f] mapped at [(____ptrval____)]
[    0.000000] Using GB pages for direct mapping
[    0.000000] RAMDISK: [mem 0x31080000-0x34837fff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen   )
[    0.000000] ACPI: XSDT 0x00000000FC00E2A0 000054 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: FACP 0x00000000FC00DF60 0000F4 (v04 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: DSDT 0x00000000FC0021C0 00BD19 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: FACS 0x00000000FC002180 000040
[    0.000000] ACPI: FACS 0x00000000FC002180 000040
[    0.000000] ACPI: APIC 0x00000000FC00E060 0000D8 (v02 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: HPET 0x00000000FC00E1B0 000038 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: WAET 0x00000000FC00E1F0 000028 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: SSDT 0x00000000FC00E220 000031 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: SSDT 0x00000000FC00E260 000033 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000040fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x40ffd5000-0x40fffefff]
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000040fffffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009dfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x00000000efffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000040fffffff]
[    0.000000] Zeroed struct page in unavailable ranges: 99 pages
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000040fffffff]
[    0.000000] ACPI: PM-Timer IO Port: 0xb008
[    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.000000] smpboot: Allowing 15 CPUs, 11 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfbffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xffffffff]
[    0.000000] [mem 0xf0000000-0xfbffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen HVM
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.000000] random: get_random_bytes called from start_kernel+0x93/0x550 with crng_init=0
[    0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:15 nr_cpu_ids:15 nr_node_ids:1
[    0.000000] percpu: Embedded 47 pages/cpu @(____ptrval____) s155648 r8192 d28672 u262144
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 4128648
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/ostree/rhcos-46868cfc9ed0776a5e535bc23872bc145cabe7c4f88e92729d6ff4147791a641/vmlinuz-4.18.0-80.1.2.el8_0.x86_64 console=tty0 console=ttyS0,115200n8 rootflags=defaults,prjquota rw root=UUID=b2f3c8ee-9c4b-4d07-87b3-24f4af95a40b ostree=/ostree/boot.1/rhcos/46868cfc9ed0776a5e535bc23872bc145cabe7c4f88e92729d6ff4147791a641/0 coreos.oem.id=ec2 ignition.platform.id=ec2
[    0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://access.redhat.com/ecosystem.
[    0.000000] Memory: 3910480K/16776820K available (12292K kernel code, 2101K rwdata, 3816K rodata, 2356K init, 3320K bss, 418136K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=15, Nodes=1
[    0.000000] Kernel/User page tables isolation: enabled
[    0.000000] ftrace: allocating 34909 entries in 137 pages
[    0.001000] rcu: Hierarchical RCU implementation.
[    0.001000] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=15.
[    0.001000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[    0.001000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=15
[    0.001000] NR_IRQS: 524544, nr_irqs: 952, preallocated irqs: 16
[    0.001000] xen:events: Using 2-level ABI
[    0.001000] xen:events: Xen HVM callback vector for event delivery is enabled
[    0.001000] Console: colour VGA+ 80x25
[    0.001000] console [tty0] enabled
[    0.001000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[    0.001000] console [ttyS0] enabled
[    0.001000] ACPI: Core revision 20180531
[    0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns
[    0.001010] APIC: Switch to symmetric I/O mode setup
[    0.005576] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.010003] Switched APIC routing to physical flat.
[    0.015000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.026000] tsc: Fast TSC calibration using PIT
[    0.043001] tsc: Detected 2300.307 MHz processor
[    0.047001] tsc: Detected 2300.072 MHz TSC
[    0.047010] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2127794b7e4, max_idle_ns: 440795254733 ns
[    0.058004] Calibrating delay loop (skipped), value calculated using timer frequency.. 4600.14 BogoMIPS (lpj=2300072)
[    0.066002] pid_max: default: 32768 minimum: 301
[    0.069044] Security Framework initialized
[    0.073002] Yama: becoming mindful.
[    0.076005] SELinux:  Initializing.
[    0.082259] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.089591] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.095061] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.100046] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.106302] mce: CPU supports 2 MCE banks
[    0.110024] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.114001] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.119001] FEATURE SPEC_CTRL Not Present
[    0.122001] FEATURE IBPB_SUPPORT Not Present
[    0.126003] Spectre V2 : Mitigation: Full generic retpoline
[    0.130001] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    0.137001] Speculative Store Bypass: Vulnerable
[    0.141031] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    0.150183] Freeing SMP alternatives memory: 32K
[    0.155185] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.162015] installing Xen timer for CPU 0
[    0.165079] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz (family: 0x6, model: 0x4f, stepping: 0x1)
[    0.166034] cpu 0 spinlock event irq 53
[    0.167118] Performance Events: unsupported p6 CPU model 79 no PMU driver, software events only.
[    0.168039] rcu: Hierarchical SRCU implementation.
[    0.169466] NMI watchdog: Perf event create on CPU 0 failed with -2
[    0.170005] NMI watchdog: Perf NMI watchdog permanently disabled
[    0.171198] smp: Bringing up secondary CPUs ...
[    0.172111] installing Xen timer for CPU 1
[    0.173056] x86: Booting SMP configuration:
[    0.174005] .... node  #0, CPUs:        #1
[    0.175058] cpu 1 spinlock event irq 59
[    0.181104] installing Xen timer for CPU 2
[    0.182065]   #2
[    0.183056] cpu 2 spinlock event irq 65
[    0.188473] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[    0.189109] installing Xen timer for CPU 3
[    0.190061]   #3
[    0.191051] cpu 3 spinlock event irq 71
[    0.196007] smp: Brought up 1 node, 4 CPUs
[    0.197004] smpboot: Max logical packages: 4
[    0.198005] smpboot: Total of 4 processors activated (18400.57 BogoMIPS)
[    0.204004] random: fast init done
[    0.308016] node 0 initialised, 3112051 pages in 109ms
[    0.312630] devtmpfs: initialized
[    0.313062] x86/mm: Memory block size: 128MB
[    0.315164] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    0.323024] futex hash table entries: 4096 (order: 6, 262144 bytes)
[    0.329168] pinctrl core: initialized pinctrl subsystem
[    0.335751] NET: Registered protocol family 16
[    0.340173] audit: initializing netlink subsys (disabled)
[    0.345020] audit: type=2000 audit(1558875282.820:1): state=initialized audit_enabled=0 res=1
[    0.353008] cpuidle: using governor menu
[    0.358062] ACPI: bus type PCI registered
[    0.362005] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.368525] PCI: Using configuration type 1 for base access
[    0.375204] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    0.376011] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    0.377069] cryptd: max_cpu_qlen set to 1000
[    0.378097] ACPI: Added _OSI(Module Device)
[    0.379006] ACPI: Added _OSI(Processor Device)
[    0.380004] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.381004] ACPI: Added _OSI(Processor Aggregator Device)
[    0.382005] ACPI: Added _OSI(Linux-Dell-Video)
[    0.386780] ACPI: 3 ACPI AML tables successfully acquired and loaded
[    0.393769] ACPI: Interpreter enabled
[    0.398018] ACPI: (supports S0 S3 S4 S5)
[    0.402005] ACPI: Using IOAPIC for interrupt routing
[    0.407026] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.418361] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.453672] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.460014] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[    0.466025] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.474000] acpiphp: Slot [0] registered
[    0.477886] acpiphp: Slot [3] registered
[    0.480335] acpiphp: Slot [4] registered
[    0.484296] acpiphp: Slot [5] registered
[    0.487349] acpiphp: Slot [6] registered
[    0.490309] acpiphp: Slot [7] registered
[    0.493322] acpiphp: Slot [8] registered
[    0.496424] acpiphp: Slot [9] registered
[    0.500526] acpiphp: Slot [10] registered
[    0.504308] acpiphp: Slot [11] registered
[    0.508310] acpiphp: Slot [12] registered
[    0.511317] acpiphp: Slot [13] registered
[    0.514317] acpiphp: Slot [14] registered
[    0.518315] acpiphp: Slot [15] registered
[    0.521300] acpiphp: Slot [16] registered
[    0.524314] acpiphp: Slot [17] registered
[    0.527296] acpiphp: Slot [18] registered
[    0.530310] acpiphp: Slot [19] registered
[    0.534300] acpiphp: Slot [20] registered
[    0.537303] acpiphp: Slot [21] registered
[    0.540318] acpiphp: Slot [22] registered
[    0.543325] acpiphp: Slot [23] registered
[    0.546323] acpiphp: Slot [24] registered
[    0.550314] acpiphp: Slot [25] registered
[    0.553324] acpiphp: Slot [26] registered
[    0.556332] acpiphp: Slot [27] registered
[    0.559318] acpiphp: Slot [28] registered
[    0.563317] acpiphp: Slot [29] registered
[    0.566317] acpiphp: Slot [30] registered
[    0.569323] acpiphp: Slot [31] registered
[    0.572332] PCI host bridge to bus 0000:00
[    0.576008] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.580006] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.584005] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.589009] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
[    0.594007] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.604559] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.609005] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.613005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.618005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.622839] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
[    0.622839] * this clock source is slow. Consider trying other clock sources
[    0.633377] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    0.649388] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)
[    0.653204] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    0.659201] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    0.664193] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)
[    0.688214] xen:balloon: Initialising balloon driver
[    0.694067] pci 0000:00:02.0: vgaarb: setting as boot VGA device
[    0.695000] pci 0000:00:02.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    0.708006] pci 0000:00:02.0: vgaarb: bridge control possible
[    0.714004] vgaarb: loaded
[    0.717123] SCSI subsystem initialized
[    0.721029] ACPI: bus type USB registered
[    0.726021] usbcore: registered new interface driver usbfs
[    0.731011] usbcore: registered new interface driver hub
[    0.735046] usbcore: registered new device driver usb
[    0.741029] pps_core: LinuxPPS API ver. 1 registered
[    0.746004] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti>
[    0.755007] PTP clock support registered
[    0.760041] EDAC MC: Ver: 3.0.0
[    0.765039] PCI: Using ACPI for IRQ routing
[    0.770004] NetLabel: Initializing
[    0.774005] NetLabel:  domain hash size = 128
[    0.779004] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    0.785019] NetLabel:  unlabeled traffic allowed by default
[    0.792486] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    0.800019] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.805005] hpet0: 3 comparators, 64-bit 62.500000 MHz counter
[    0.815035] clocksource: Switched to clocksource xen
[    0.834348] VFS: Disk quotas dquot_6.6.0
[    0.839446] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.847344] pnp: PnP ACPI init
[    0.851603] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[    0.859540] system 00:01: [io  0x08a0-0x08a3] has been reserved
[    0.866396] system 00:01: [io  0x0cc0-0x0ccf] has been reserved
[    0.873107] system 00:01: [io  0x04d0-0x04d1] has been reserved
[    0.880124] system 00:07: [io  0x10c0-0x1141] has been reserved
[    0.886986] system 00:07: [io  0xb044-0xb047] has been reserved
[    0.921209] pnp: PnP ACPI: found 8 devices
[    0.933829] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.944322] NET: Registered protocol family 2
[    0.950103] tcp_listen_portaddr_hash hash table entries: 8192 (order: 5, 131072 bytes)
[    0.959518] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.968320] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.975301] TCP: Hash tables configured (established 131072 bind 65536)
[    0.982345] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    0.988894] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    0.995842] NET: Registered protocol family 1
[    1.000788] NET: Registered protocol family 44
[    1.006070] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.013059] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    1.019934] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    1.026070] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.036017] Unpacking initramfs...
[    1.683267] Freeing initrd memory: 57056K
[    1.686445] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.690813] software IO TLB: mapped [mem 0xec000000-0xf0000000] (64MB)
[    1.697281] Initialise system trusted keyrings
[    1.700553] Key type blacklist registered
[    1.703618] workingset: timestamp_bits=36 max_order=22 bucket_order=0
[    1.709067] zbud: loaded
[    1.711561] pstore: using deflate compression
[    1.715780] Platform Keyring initialized
[    1.758910] NET: Registered protocol family 38
[    1.762242] Key type asymmetric registered
[    1.765319] Asymmetric key parser 'x509' registered
[    1.768791] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 246)
[    1.774336] io scheduler mq-deadline registered
[    1.778680] io scheduler kyber registered
[    1.782678] io scheduler bfq registered
[    1.786658] atomic64_test: passed for x86-64 platform with CX8 and with SSE
[    1.792782] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    1.798840] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    1.805697] ACPI: Power Button [PWRF]
[    1.809485] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
[    1.816542] ACPI: Sleep Button [SLPF]
[    1.821304] xen:grant_table: Grant tables using version 1 layout
[    1.826824] Grant table initialized
[    1.830665] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[    1.836168] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    1.872368] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.880187] Non-volatile memory driver v1.3
[    1.884273] Linux agpgart interface v0.103
[    1.888733] rdac: device handler registered
[    1.892846] hp_sw: device handler registered
[    1.896989] emc: device handler registered
[    1.901071] alua: device handler registered
[    1.905193] libphy: Fixed MDIO Bus: probed
[    1.909259] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.914995] ehci-pci: EHCI PCI platform driver
[    1.919444] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.925177] ohci-pci: OHCI PCI platform driver
[    1.929595] uhci_hcd: USB Universal Host Controller Interface driver
[    1.935412] usbcore: registered new interface driver usbserial_generic
[    1.941169] usbserial: USB Serial support registered for generic
[    1.946697] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    1.956828] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.962640] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.967516] mousedev: PS/2 mouse device common for all mice
[    1.973590] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[    1.980809] rtc_cmos 00:02: registered as rtc0
[    1.985007] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs
[    1.992667] hidraw: raw HID events driver (C) Jiri Kosina
[    1.999339] usbcore: registered new interface driver usbhid
[    2.006153] usbhid: USB HID core driver
[    2.010717] drop_monitor: Initializing network drop monitor service
[    2.017037] Initializing XFRM netlink socket
[    2.021425] NET: Registered protocol family 10
[    2.026014] Segment Routing with IPv6
[    2.029741] NET: Registered protocol family 17
[    2.034023] mpls_gso: MPLS GSO support
[    2.038309] RAS: Correctable Errors collector initialized.
[    2.044231] AVX2 version of gcm_enc/dec engaged.
[    2.049410] AES CTR mode by8 optimization enabled
[    2.070317] sched_clock: Marking stable (2070310884, 0)->(11470017056, -9399706172)
[    2.080384] registered taskstats version 1
[    2.085844] Loading compiled-in X.509 certificates
[    2.125397] Loaded X.509 cert 'Red Hat Enterprise Linux kernel signing key: 89dfa7de1bb9e5d722687082d7121826300fee75'
[    2.136716] Loaded X.509 cert 'Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87362bc7229d9f465321773dfd1f77a80'
[    2.149093] Loaded X.509 cert 'Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b72e3852e2014c3a676fc8'
[    2.160195] zswap: loaded using pool lzo/zbud
[    2.170325] Key type big_key registered
[    2.177155] Key type trusted registered
[    2.184069] Key type encrypted registered
[    2.188834] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[    2.194660] ima: Allocated hash algorithm: sha1
[    2.199868] evm: Initialising EVM extended attributes:
[    2.205618] evm: security.selinux
[    2.209648] evm: security.ima
[    2.213383] evm: security.capability
[    2.217782] evm: HMAC attrs: 0x1
[    2.222176] xenbus_probe_frontend: Device with no driver: device/vbd/51712
[    2.229258] xenbus_probe_frontend: Device with no driver: device/pci/0
[    2.236126] rtc_cmos 00:02: setting system clock to 2019-05-26 12:54:44 UTC (1558875284)
[    2.538036] Freeing unused kernel memory: 2356K
[    2.549083] Write protecting the kernel read-only data: 18432k
[    2.556646] Freeing unused kernel memory: 2020K
[    2.562410] Freeing unused kernel memory: 280K
[    2.575757] systemd[1]: systemd 239 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[    2.595325] systemd[1]: Detected virtualization xen.
[    2.599956] systemd[1]: Detected architecture x86-64.
[    2.604678] systemd[1]: Running in initial RAM disk.

Welcome to [0;34mRed Hat Enterprise Linux CoreOS 410.8.20190517.0 (Ootpa) dracut-049-10.git20190115.el8 (Initramfs)[0m!

[    2.619116] systemd[1]: No hostname configured.
[    2.623450] systemd[1]: Set hostname to <localhost>.
[    2.628155] random: systemd: uninitialized urandom read (16 bytes read)
[    2.633926] systemd[1]: Initializing machine ID from random generator.
[    2.692917] random: systemd: uninitialized urandom read (16 bytes read)
[    2.698089] systemd[1]: Reached target Swap.
[[0;32m  OK  [0m] Reached target Swap.
[    2.704025] random: systemd: uninitialized urandom read (16 bytes read)
[    2.709219] systemd[1]: Listening on Open-iSCSI iscsid Socket.
[[0;32m  OK  [0m] Listening on Open-iSCSI iscsid Socket.
[    2.716437] systemd[1]: Listening on Journal Socket.
[    2.719082] tsc: Refined TSC clocksource calibration: 2300.001 MHz
[    2.725527] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x212735f0517, max_idle_ns: 440795237604 ns
[[0;32m  OK  [0m] Listening on Journal Socket.
[[0;32m  OK  [0m] Listening on Open-iSCSI iscsiuio Socket.
         Starting Setup Virtual Console...
[[0;32m  OK  [0m] Reached target Slices.
[[0;32m  OK  [0m] Listening on udev Control Socket.
         Starting Create list of required st…ce nodes for the current kernel...
         Starting Apply Kernel Variables...
[[0;32m  OK  [0m] Reached target Timers.
         Starting iSCSI UserSpace I/O driver...
[[0;32m  OK  [0m] Listening on udev Kernel Socket.
[[0;32m  OK  [0m] Listening on Journal Socket (/dev/log).
         Starting Journal Service...
[[0;32m  OK  [0m] Reached target Sockets.
[[0;32m  OK  [0m] Started Dispatch Password Requests to Console Directory Watch.
[[0;32m  OK  [0m] Reached target Paths.
[[0;32m  OK  [0m] Reached target Local Encrypted Volumes.
[[0;32m  OK  [0m] Started Setup Virtual Console.
[[0;32m  OK  [0m] Started Create list of required sta…vice nodes for the current kernel.
[[0;32m  OK  [0m] Started Apply Kernel Variables.
[[0;32m  OK  [0m] Started iSCSI UserSpace I/O driver.
         Starting Create Static Device Nodes in /dev...
         Starting dracut cmdline hook...
[[0;32m  OK  [0m] Started Create Static Device Nodes in /dev.
[    2.847588] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input4
[[0;32m  OK  [0m] Started Journal Service.
[    2.859599] Loading iSCSI transport class v2.0-870.
[    2.863948] dracut-cmdline[238]: dracut-410.8.20190517.0 dracut-049-10.git20190115.el8
[    2.874493] dracut-cmdline[238]: Using kernel command line parameters: BOOT_IMAGE=/ostree/rhcos-46868cfc9ed0776a5e535bc23872bc145cabe7c4f88e92729d6ff4147791a641/vmlinuz-4.18.0-80.1.2.el8_0.x86_64 console=tty0 console=ttyS0,115200n8 rootflags=defaults,prjquota rw root=UUID=b2f3c8ee-9c4b-4d07-87b3-24f4af95a40b ostree=/ostree/boot.1/rhcos/46868cfc9ed0776a5e535bc23872bc145cabe7c4f88e92729d6ff4147791a641/0 coreos.oem.id=ec2 ignition.platform.id=ec2
[    2.983571] iscsi: registered transport (tcp)
[    3.008679] iscsi: registered transport (qla4xxx)
[    3.014921] QLogic iSCSI HBA Driver
[    3.025665] libcxgbi:libcxgbi_init_module: Chelsio iSCSI driver library libcxgbi v0.9.1-ko (Apr. 2015)
[    3.063654] Chelsio T4-T6 iSCSI Driver cxgb4i v0.9.5-ko (Apr. 2015)
[    3.071355] iscsi: registered transport (cxgb4i)
[    3.083628] cnic: QLogic cnicDriver v2.5.22 (July 20, 2015)
[    3.094977] QLogic NetXtreme II iSCSI Driver bnx2i v2.7.10.1 (Jul 16, 2014)
[    3.103402] iscsi: registered transport (bnx2i)
[    3.117645] iscsi: registered transport (be2iscsi)
[    3.123915] In beiscsi_module_init, tt=(____ptrval____)
[[    3.165068] systemd[1]: Started dracut cmdline hook.
[0;32m  OK  [0m] Started dracut cmdline hook.
         [    3.175468] systemd[1]: Starting dracut pre-udev hook...
Starting dracut pre-udev hook...
[    3.199909] device-mapper: uevent: version 1.0.3
[    3.206048] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel
[    3.239939] systemd[1]: Started dracut pre-udev hook.
[[0;32m  OK  [0m] Started dracut pre-udev hook.
         [    3.250269] systemd[1]: Starting udev Kernel Device Manager...
Starting udev Kernel Device Manager...
[    3.272408] systemd-udevd[443]: Specified group 'tty' unknown
[[    3.280477] systemd-udevd[443]: Specified group 'kmem' unknown
[0;32m  OK  [0m] [    3.287831] systemd-udevd[443]: Specified group 'input' unknown
Started udev Kernel Device Manag[    3.295435] systemd-udevd[443]: Specified group 'video' unknown
er.
         Starting dracut pre-trigger hook...
[    3.349087] random: crng init done
[    3.353862] random: 7 urandom warning(s) missed due to ratelimiting
[[0;32m  OK  [0m] Started dracut pre-trigger hook.
         Starting udev Coldplug all Devices...
         Mounting Kernel Configuration File System...
[[0;32m  OK  [0m] Mounted Kernel Configuration File System.
[[0;32m  OK  [0m] Started udev Coldplug all Devices.
         Starting udev Wait for Complete Device Initialization...
[    3.502260] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[    3.514949] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[    3.545308] ixgbevf 0000:00:03.0: 02:f2:e1:5d:8d:fe
[    3.552341] ixgbevf 0000:00:03.0: MAC: 1
[    3.558311] ixgbevf 0000:00:03.0: Intel(R) 82599 Virtual Function
[    3.613433] ixgbevf 0000:00:03.0 ens3: renamed from eth0
[    3.614023] scsi host0: ata_piix
[    3.624892] blkfront: xvda: flush diskcache: enabled; persistent grants: disabled; indirect descriptors: enabled;
[    3.626902] scsi host1: ata_piix
[    3.639032] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
[    3.644799] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
[    3.655250]  xvda: xvda1 xvda2 xvda3
[[0;32m  OK  [0m] Found device /dev/disk/by-uuid/b2f3c8ee-9c4b-4d07-87b3-24f4af95a40b.
[[0;32m  OK  [0m] Reached target Initrd Root Device.
[[0;32m  OK  [0m] Started udev Wait for Complete Device Initialization.
         Starting Device-Mapper Multipath Device Controller...
[[0;32m  OK  [0m] Started Device-Mapper Multipath Device Controller.
[[0;32m  OK  [0m] Reached target Local File Systems (Pre).
[[0;32m  OK  [0m] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Open-iSCSI...
[[0;32m  OK  [0m] Started Create Volatile Files and Directories.
[[0;32m  OK  [0m] Started Open-iSCSI.
         Starting dracut initqueue hook...
[[0;32m  OK  [0m] Reached target System Initialization.
[[0;32m  OK  [0m] Reached target Basic System.
[[0;32m  OK  [0m] Started dracut initqueue hook.
         Starting dracut pre-mount hook...
[[0;32m  OK  [0m] Reached target Remote File Systems (Pre).
[[0;32m  OK  [0m] Reached target Remote File Systems.
[[0;32m  OK  [0m] Started dracut pre-mount hook.
         Starting File System Check on /dev/…8ee-9c4b-4d07-87b3-24f4af95a40b...
[[0;32m  OK  [0m] Started File System Check on /dev/d…3c8ee-9c4b-4d07-87b3-24f4af95a40b.
         Mounting /sysroot...
[    4.001425] SGI XFS with ACLs, security attributes, no debug enabled
[    4.011119] XFS (xvda3): Mounting V5 Filesystem
[    4.034724] XFS (xvda3): Ending clean mount
[[0;32m  OK  [0m] Mounted /sysroot.
         Starting OSTree Prepare OS/...
[[0;32m  OK  [0m] Started OSTree Prepare OS/.
[[0;32m  OK  [0m] Reached target Initrd Root File System.
         Starting Reload Configuration from the Real Root...
[[0;32m  OK  [0m] Started Reload Configuration from the Real Root.
         Starting dracut mount hook...
[[0;32m  OK  [0m] Reached target Initrd File Systems.
[[0;32m  OK  [0m] Reached target Initrd Default Target.
[[0;32m  OK  [0m] Started dracut mount hook.
         Starting dracut pre-pivot and cleanup hook...
[    4.362975] dracut-pre-pivot[653]: May 26 12:54:46 | /etc/multipath.conf does not exist, blacklisting all devices.
[    4.370192] dracut-pre-pivot[653]: May 26 12:54:46 | You can run "/sbin/mpathconf --enable" to create
[    4.376420] dracut-pre-pivot[653]: May 26 12:54:46 | /etc/multipath.conf. See man mpathconf(8) for more details
[[0;32m  OK  [0m] Started dracut pre-pivot and cleanup hook.
         Starting Cleaning Up and Shutting Down Daemons...
[[0;32m  OK  [0m] Stopped dracut pre-pivot and cleanup hook.
[[0;32m  OK  [0m] Stopped target Initrd Default Target.
[[0;32m  OK  [0m] Stopped target Basic System.
[[0;32m  OK  [0m] Stopped target System Initialization.
[[0;32m  OK  [0m] Stopped target Swap.
[[0;32m  OK  [0m] Stopped Apply Kernel Variables.
[[0;32m  OK  [0m] Stopped Create Volatile Files and Directories.
[[0;32m  OK  [0m] Stopped target Local File Systems.
[[0;32m  OK  [0m] Stopped target Sockets.
[[0;32m  OK  [0m] Stopped target Timers.
[[0;32m  OK  [0m] Stopped target Initrd Root Device.
[[0;32m  OK  [0m] Stopped target Slices.
[[0;32m  OK  [0m] Stopped dracut mount hook.
[[0;32m  OK  [0m] Stopped dracut pre-mount hook.
[[0;32m  OK  [0m] Stopped target Local Encrypted Volumes.
[[0;32m  OK  [0m] Stopped target Local File Systems (Pre).
[[0;32m  OK  [0m] Stopped target Remote File Systems.
[[0;32m  OK  [0m] Stopped target Remote File Systems (Pre).
[[0;32m  OK  [0m] Stopped dracut initqueue hook.
         Stopping Open-iSCSI...
[[0;32m  OK  [0m] Stopped target Paths.
[[0;32m  OK  [0m] Stopped Dispatch Password Requests to Console Directory Watch.
[[0;32m  OK  [0m] Stopped Open-iSCSI.
         Stopping Device-Mapper Multipath Device Controller...
         Stopping iSCSI UserSpace I/O driver...
[[0;32m  OK  [0m] Closed Open-iSCSI iscsid Socket.
[[0;32m  OK  [0m] Stopped iSCSI UserSpace I/O driver.
[[0;32m  OK  [0m] Started Cleaning Up and Shutting Down Daemons.
[[0;32m  OK  [0m] Stopped Device-Mapper Multipath Device Controller.
[[0;32m  OK  [0m] Stopped udev Wait for Complete Device Initialization.
[[0;32m  OK  [0m] Stopped udev Coldplug all Devices.
[[0;32m  OK  [0m] Stopped dracut pre-trigger hook.
         Stopping udev Kernel Device Manager...
[[0;32m  OK  [0m] Closed Open-iSCSI iscsiuio Socket.
[[0;32m  OK  [0m] Stopped udev Kernel Device Manager.
[[0;32m  OK  [0m] Stopped Create Static Device Nodes in /dev.
[[0;32m  OK  [0m] Stopped Create list of required sta…vice nodes for the current kernel.
[[0;32m  OK  [0m] Stopped dracut pre-udev hook.
[[0;32m  OK  [0m] Stopped dracut cmdline hook.
[[0;32m  OK  [0m] Closed udev Control Socket.
[[0;32m  OK  [0m] Closed udev Kernel Socket.
         Starting Cleanup udevd DB...
[[0;32m  OK  [0m] Started Cleanup udevd DB.
[[0;32m  OK  [0m] Reached target Switch Root.
         Starting Switch Root...
[    4.602216] systemd-journald[236]: Received SIGTERM from PID 1 (systemd).
[    4.608459] systemd-journal: 133 output lines suppressed due to ratelimiting
[    4.636122] systemd: 23 output lines suppressed due to ratelimiting
[    4.722138] audit: type=1404 audit(1558875286.985:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295 enabled=1 old-enabled=1 lsm=selinux res=1
[    5.226717] SELinux:  Class bpf not defined in policy.
[    5.232113] SELinux:  Class xdp_socket not defined in policy.
[    5.238224] SELinux: the above unknown classes and permissions will be allowed
[    5.246034] SELinux:  policy capability network_peer_controls=1
[    5.252220] SELinux:  policy capability open_perms=1
[    5.257696] SELinux:  policy capability extended_socket_class=1
[    5.264210] SELinux:  policy capability always_check_network=0
[    5.270485] SELinux:  policy capability cgroup_seclabel=1
[    5.276519] SELinux:  policy capability nnp_nosuid_transition=1
[    5.302928] audit: type=1403 audit(1558875287.565:3): auid=4294967295 ses=4294967295 lsm=selinux res=1
[    5.305517] systemd[1]: Successfully loaded SELinux policy in 583.833ms.
[    5.348218] systemd[1]: Failed to insert module 'ip_tables': Operation not permitted
[    5.384904] systemd[1]: Relabelled /dev, /run and /sys/fs/cgroup in 24.673ms.
[    5.394264] systemd[1]: systemd 239 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[    5.412316] systemd[1]: Detected virtualization xen.
[    5.416971] systemd[1]: Detected architecture x86-64.

Welcome to [0;31mRed Hat Enterprise Linux CoreOS 410.8.20190517.0 (Ootpa)[0m!

[    5.426970] systemd[1]: Set hostname to <localhost.localdomain>.
[    5.706838] systemd[1]: systemd-journald.service: Consumed 0 CPU time
[    5.714583] systemd[1]: Stopped Switch Root.
[[0;32m  OK  [0m] Stopped Switch Root.
[    5.725467] systemd[1]: initrd-switch-root.service: Consumed 0 CPU time
[    5.733873] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart.
[    5.747244] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[    5.758172] systemd[1]: Stopped Journal Service.
[[0;32m  OK  [0m] Stopped Journal Service.
         Starting Journal Service...
[[0;32m  OK  [0m] Created slice User and Session Slice.
[[0;32m  OK  [0m] Started Forward Password Requests to Wall Directory Watch.
         Mounting Kernel Debug File System...
         Mounting POSIX Message Queue File System...
[[0;32m  OK  [0m] Started Dispatch Password Requests to Console Directory Watch.
[[0;32m  OK  [0m] Listening on Device-mapper event daemon FIFOs.
[[0;32m  OK  [0m] Listening on udev Kernel Socket.
[[0;32m  OK  [0m] Created slice system-systemd\x2dfsck.slice.
[[0;32m  OK  [0m] Listening on initctl Compatibility Named Pipe.
[[0;32m  OK  [0m] Listening on LVM2 poll daemon socket.
         Starting Create list of required st…ce nodes for the current kernel...
[[0;32m  OK  [0m] Reached target Slices.
         Starting Apply Kernel Variables...
         Mounting Huge Pages File System...
[[0;32m  OK  [0m] Reached target Swap.
[[0;32m  OK  [0m] Reached target Remote Encrypted Volumes.
[[0;32m  OK  [0m] Reached target Remote File Systems.
[[0;32m  OK  [0m] Listening on Process Core Dump Socket.
[[0;32m  OK  [0m] Reached target Local Encrypted Volumes.
[[0;32m  OK  [0m] Reached target Host and Network Name Lookups.
         Starting Monitoring of LVM2 mirrors…ng dmeventd or progress polling...
[[0;32m  OK  [0m] Listening on multipathd control socket.
[[0;32m  OK  [0m] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
[[0;32m  OK  [0m] Listening on RPCbind Server Activation Socket.
[[0;32m  OK  [0m] Created slice system-serial\x2dgetty.slice.
[[0;32m  OK  [0m] Created slice system-sshd\x2dkeygen.slice.
[[0;32m  OK  [0m] Reached target RPC Port Mapper.
[[0;32m  OK  [0m] Stopped File System Check on Root Device.
         Starting Remount Root and Kernel File Systems...
[[0;32m  OK  [0m] Created slice system-getty.slice.
[[0;32m  OK  [0m] Set up automount Arbitrary Executab…rmats File System Automount Point.
[[0;32m  OK  [0m] Stopped target Switch Root.
[[0;32m  OK  [0m] Stopped target Initrd Root File System.
[[0;32m  OK  [0m] Stopped OSTree Prepare OS/.
[[0;32m  OK  [0m] Stopped target Initrd File Systems.
[[0;32m  OK  [0m] Started Journal Service.
[[0;32m  OK  [0m] Mounted Kernel Debug File System.
[[0;32m  OK  [0m] Mounted POSIX Message Queue File System.
[[0;32m  OK  [0m] Started Create list of required sta…vice nodes for the current kernel.
[[0;32m  OK  [0m] Started Apply Kernel Variables.
[[0;32m  OK  [0m] Mounted Huge Pages File System.
[[0;32m  OK  [0m] Started Remount Root and Kernel File Systems.
         Starting OSTree Remount OS/ Bind Mounts...
         Starting Create Static Device Nodes in /dev...
[[0;32m  OK  [0m] Started OSTree Remount OS/ Bind Mounts.
[[0;32m  OK  [0m] Started Create Static Device Nodes in /dev.
         Starting udev Kernel Device Manager...
[[0;32m  OK  [0m] Started udev Coldplug all Devices.
         Starting udev Wait for Complete Device Initialization...
[[0;32m  OK  [0m] Started udev Kernel Device Manager.
[    6.224340] input: PC Speaker as /devices/platform/pcspkr/input/input5
[    6.236990] piix4_smbus 0000:00:01.3: SMBus base address uninitialized - upgrade BIOS or use force_addr=0xaddr
[    6.259621] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer
[    6.268342] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[    6.274181] RAPL PMU: hw unit of domain package 2^-14 Joules
[    6.280119] RAPL PMU: hw unit of domain dram 2^-16 Joules
[    6.311837] EDAC sbridge:  Ver: 1.1.2 
[    6.327234] intel_rapl: Found RAPL domain package
[    6.331985] intel_rapl: Found RAPL domain dram
[    6.336503] intel_rapl: DRAM domain energy unit 15300pj
[    6.360862] [TTM] Zone  kernel: Available graphics memory: 8210214 kiB
[    6.366200] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[    6.371247] [TTM] Initializing pool allocator
[    6.375025] [TTM] Initializing DMA pool allocator
[    6.379488] [drm] fb mappable at 0xF0000000
[    6.379489] [drm] vram aper at 0xF0000000
[    6.379489] [drm] size 33554432
[    6.379490] [drm] fb depth is 24
[    6.379490] [drm]    pitch is 3072
[    6.496404] fbcon: cirrusdrmfb (fb0) is primary device
[    6.535821] Console: switching to colour frame buffer device 128x48
[    6.549114] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device
[    6.556097] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0
[[0;32m  OK  [0m] Started udev Wait for Complete Device Initialization.
[[0;32m  OK  [0m] Started Monitoring of LVM2 mirrors,…sing dmeventd or progress polling.
[[0;32m  OK  [0m] Reached target Local File Systems (Pre).
         Starting File System Check on /dev/…6da-ed87-476d-bebf-aa501adad24c...
         Mounting /var...
[[0;32m  OK  [0m] Mounted /var.
         Starting Load/Save Random Seed...
         Starting Flush Journal to Persistent Storage...
[[0;32m  OK  [0m] Started Load/Save Random Seed.
[    9.202091] systemd-journald[712]: Received request to flush runtime journal from PID 1
[[0;32m  OK  [0m] Started File System Check on /dev/d…196da-ed87-476d-bebf-aa501adad24c.
         Mounting /boot...
[[0;32m  OK  [0m] Started Flush Journal to Persistent Storage.
[    9.306641] EXT4-fs (xvda2): mounted filesystem with ordered data mode. Opts: (null)
[[0;32m  OK  [0m] Mounted /boot.
[[0;32m  OK  [0m] Reached target Local File Systems.
         Starting Restore /run/initramfs on shutdown...
         Starting Create Volatile Files and Directories...
[[0;32m  OK  [0m] Started Restore /run/initramfs on shutdown.
[[0;32m  OK  [0m] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
[[0;32m  OK  [0m] Started Update UTMP about System Boot/Shutdown.
[[0;32m  OK  [0m] Reached target System Initialization.
[[0;32m  OK  [0m] Listening on D-Bus System Message Bus Socket.
[[0;32m  OK  [0m] Reached target Sockets.
[[0;32m  OK  [0m] Started OSTree Monitor Staged Deployment.
[[0;32m  OK  [0m] Reached target Paths.
[[0;32m  OK  [0m] Reached target Basic System.
         Starting Log RPM-OSTree Booted Deployment Status To Journal...
         Starting NTP client/server...
         Starting Login Service...
         Starting Kubernetes Kubelet...
[[0;32m  OK  [0m] Started irqbalance daemon.
[[0;32m  OK  [0m] Reached target sshd-keygen.target.
[[0;32m  OK  [0m] Started D-Bus System Message Bus.
         Starting Network Manager...
[[0;32m  OK  [0m] Started Daily Cleanup of Temporary Directories.
[[0;32m  OK  [0m] Reached target Timers.
[[0;32m  OK  [0m] Started Login Service.
[[0;32m  OK  [0m] Started NTP client/server.
[[0;32m  OK  [0m] Started Network Manager.
[[0;32m  OK  [0m] Reached target Network.
         Starting OpenSSH server daemon...
         Starting Network Manager Wait Online...
         Starting Hostname Service...
         Starting RPM-OSTree System Management Daemon...
[[0;32m  OK  [0m] Started OpenSSH server daemon.
         Starting Authorization Manager...
[[0;32m  OK  [0m] Started Hostname Service.
         Starting Network Manager Script Dispatcher Service...
[[0;32m  OK  [0m] Started Network Manager Script Dispatcher Service.
[    9.746404] IPv6: ADDRCONF(NETDEV_UP): ens3: link is not ready
[    9.763087] ixgbevf 0000:00:03.0: NIC Link is Up 10 Gbps
[    9.764628] IPv6: ADDRCONF(NETDEV_UP): ens3: link is not ready
[    9.770182] IPv6: ADDRCONF(NETDEV_CHANGE): ens3: link becomes ready
[[0;32m  OK  [0m] Started Authorization Manager.
[[0;32m  OK  [0m] Started RPM-OSTree System Management Daemon.
[    9.864602] ixgbevf 0000:00:03.0: NIC Link is Up 10 Gbps
[[0;32m  OK  [0m] Started Log RPM-OSTree Booted Deployment Status To Journal.
[[0;32m  OK  [0m] Started Kubernetes systemd probe.
[[0;1;31mFAILED[0m] Failed to start Kubernetes Kubelet.
See 'systemctl status kubelet.service' for details.
[[0;32m  OK  [0m] Started Network Manager Wait Online.
[[0;32m  OK  [0m] Reached target Network is Online.
         Starting Open Container Initiative Daemon...
         Starting Generate /run/issue.d/console-login-helper-messages.issue...
         Starting NFS status monitor for NFSv2/3 locking....
         Starting RPC Bind...
[[0;32m  OK  [0m] Started Generate /run/issue.d/console-login-helper-messages.issue.
         Starting Permit User Sessions...
[[0;32m  OK  [0m] Started RPC Bind.
[[0;32m  OK  [0m] Started Permit User Sessions.
[[0;32m  OK  [0m] Started Getty on tty1.
[[0;32m  OK  [0m] Started Serial Getty on ttyS0.
[[0;32m  OK  [0m] Reached target Login Prompts.
[[0;32m  OK  [0m] Started NFS status monitor for NFSv2/3 locking..
[[0;32m  OK  [0m] Started Open Container Initiative Daemon.
[[0;32m  OK  [0m] Reached target Multi-User System.
         Starting Update UTMP about System Runlevel Changes...
[[0;32m  OK  [0m] Started Update UTMP about System Runlevel Changes.


Red Hat Enterprise Linux CoreOS 410.8.20190517.0 (Ootpa) 4.1
SSH host key: SHA256:uiayX1u+Gtg6RVKrMNiJbHOHu7GOlwWh1v3elp4jW6Q (ECDSA)
SSH host key: SHA256:eoapZcF2vvktTFlgDGTVveSJyVYU4IsrgmZSHlFwIqA (ED25519)
SSH host key: SHA256:Mmo57tHAFYCJPEmMySPQPWMs3wwnnH2JWNhu4iM0060 (RSA)
ens3: 10.0.142.161 fe80::e7e9:43fc:270c:6cde

ip-10-0-142-161 login: [   21.696203] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   21.696503] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   21.772413] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   21.927566] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   21.965586] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   22.007091] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   22.550047] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   22.733663] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   22.908043] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   23.558098] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   23.597105] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   23.640805] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   23.792899] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   23.828476] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   24.572892] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   24.790159] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   44.870465] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
Comment 2 Qin Ping 2019-05-28 07:45:18 UTC 
Had a simalar options with 4.1.0-rc.7, seems no this issue.

The steps are:
1.Get openshift-install binary file with cmd:
oc adm release extract ---command='openshift-install' --command-os='linux' quay.io/openshift-release-dev/ocp-release:4.1.0-rc.7
2. Install an openshift cluster in ap-northeast-1 region
3. Wait about 1 or 2 hours
4. Stop all the instances in the aws web console
5. Then start all the instances in the aws web console
6. Check the cluster with cmd: oc get node, and succeeded
Comment 9 Ryan Phillips 2019-06-03 15:30:20 UTC 
Closing as a duplicate of BZ1711910.

*** This bug has been marked as a duplicate of bug 1711910 ***