Hide Forgot
Description of problem: CPU hotplug in host hmp leads to guest call trace, although cpu is added when `cat /proc/cpuinfo`. Version-Release number of selected component (if applicable): guest: 3.10.0-366.rt56.243.el7.x86_64 cmdline of guest: BOOT_IMAGE=/vmlinuz-3.10.0-366.rt56.243.el7.x86_64 root=/dev/mapper/rhel_dhcp--66--106--180-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-66-106-180/root rd.lvm.lv=rhel_dhcp-66-106-180/swap rhgb quiet LANG=en_US.UTF-8 isolcpus=1 nohz_full=1 intel_pstate=disable nosoftlockup console=tty0 console=ttyS0,115200 host: 3.10.0-364.rt56.241.el7.x86_64 qemu-kvm-rhev-2.5.0-2.el7.x86_64 cmdline of host: BOOT_IMAGE=/vmlinuz-3.10.0-364.rt56.241.el7.x86_64 root=/dev/mapper/rhel_dell--per730--11-root ro crashkernel=auto rd.lvm.lv=rhel_dell-per730-11/root intel_iommu=on console=ttyS0,115200n81 LANG=en_US.UTF-8 isolcpus=0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 nohz_full=0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 default_hugepagesz=1G hugepagesz=1G hugepages=16 How reproducible: 100% Steps to Reproduce: 1. boot a guest /usr/libexec/qemu-kvm -name rhel7.2-rt-355 -machine pc-i440fx-rhel7.2.0 -cpu IvyBridge \ -m 4096,slots=10,maxmem=10G -realtime mlock=off -smp 4,maxcpus=10 \ -drive file=/home/xfu/rhel7.2-z-host1.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,snapshot=off -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \ -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a1:d0:5f \ -monitor stdio -device qxl-vga,id=video0 -serial unix:/tmp/console,server,nowait -vnc :1 -spice port=5900,disable-ticketing -qmp tcp:0:4440,server,nowait 2. cpu hotplug in host (qemu) cpu-add 4 3. dmesg in guest, get call trace Actual results: # [ 59.996031] intel_rapl: no valid rapl domains found in package 0 [ 60.031423] intel_rapl: no valid rapl domains found in package 0 [root@guest ~]# dmesg dmesg [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-366.rt56.243.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP PREEMPT RT Mon Mar 21 11:34:32 EDT 2016 [ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.10.0-366.rt56.243.el7.x86_64 root=/dev/mapper/rhel_dhcp--66--106--180-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-66-106-180/root rd.lvm.lv=rhel_dhcp-66-106-180/swap rhgb quiet LANG=en_US.UTF-8 isolcpus=1 nohz_full=1 intel_pstate=disable nosoftlockup console=tty0 console=ttyS0,115200 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffdefff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffdf000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000013fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: Red Hat KVM, BIOS seabios-1.7.5-11.el7 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x140000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: write-back [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0000C0000000 mask 3FFFC0000000 uncachable [ 0.000000] 1 disabled [ 0.000000] 2 disabled [ 0.000000] 3 disabled [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: last_pfn = 0xbffdf max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f1fa0-0x000f1faf] mapped at [ffff8800000f1fa0] [ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576 [ 0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff] [ 0.000000] [mem 0x00000000-0x000fffff] page 4k [ 0.000000] BRK [0x02001000, 0x02001fff] PGTABLE [ 0.000000] BRK [0x02002000, 0x02002fff] PGTABLE [ 0.000000] BRK [0x02003000, 0x02003fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x13fe00000-0x13fffffff] [ 0.000000] [mem 0x13fe00000-0x13fffffff] page 2M [ 0.000000] BRK [0x02004000, 0x02004fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x13c000000-0x13fdfffff] [ 0.000000] [mem 0x13c000000-0x13fdfffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x100000000-0x13bffffff] [ 0.000000] [mem 0x100000000-0x13bffffff] page 2M [ 0.000000] init_memory_mapping: [mem 0x00100000-0xbffdefff] [ 0.000000] [mem 0x00100000-0x001fffff] page 4k [ 0.000000] [mem 0x00200000-0xbfdfffff] page 2M [ 0.000000] [mem 0xbfe00000-0xbffdefff] page 4k [ 0.000000] RAMDISK: [mem 0x3478d000-0x363befff] [ 0.000000] ACPI: RSDP 00000000000f1f50 00014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 00000000bffe1de0 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 00000000bffe08d4 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 00000000bffdfd00 00BD4 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 00000000bffdfcc0 00040 [ 0.000000] ACPI: SSDT 00000000bffe0948 013D8 (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) [ 0.000000] ACPI: APIC 00000000bffe1d20 000C0 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000013fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x13fffffff] [ 0.000000] NODE_DATA [mem 0x13ffd6000-0x13fffcfff] [ 0.000000] Reserving 161MB of memory at 672MB for crashkernel (System RAM: 4095MB) [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: cpu 0, msr 1:3ff86001, primary cpu clock [ 0.000000] kvm-clock: using sched offset of 146413833833 cycles [ 0.000000] [ffffea0000000000-ffffea0004ffffff] PMD -> [ffff88013b600000-ffff88013f5fffff] on node 0 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x13fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009efff] [ 0.000000] node 0: [mem 0x00100000-0xbffdefff] [ 0.000000] node 0: [mem 0x100000000-0x13fffffff] [ 0.000000] On node 0 totalpages: 1048445 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3998 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 12224 pages used for memmap [ 0.000000] DMA32 zone: 782303 pages, LIFO batch:31 [ 0.000000] Normal zone: 4096 pages used for memmap [ 0.000000] Normal zone: 262144 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ5 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] ACPI: IRQ10 used by override. [ 0.000000] ACPI: IRQ11 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] smpboot: Allowing 10 CPUs, 6 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffdf000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] e820: [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:10 nr_cpu_ids:10 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88013b000000 s97472 r8192 d29504 u524288 [ 0.000000] pcpu-alloc: s97472 r8192 d29504 u524288 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 00 01 02 03 [0] 04 05 06 07 [ 0.000000] pcpu-alloc: [0] 08 09 -- -- [ 0.000000] KVM setup async PF for cpu 0 [ 0.000000] kvm-stealtime: cpu 0, msr 13b00e1c0 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 1032040 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.10.0-366.rt56.243.el7.x86_64 root=/dev/mapper/rhel_dhcp--66--106--180-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-66-106-180/root rd.lvm.lv=rhel_dhcp-66-106-180/swap rhgb quiet LANG=en_US.UTF-8 isolcpus=1 nohz_full=1 intel_pstate=disable nosoftlockup console=tty0 console=ttyS0,115200 [ 0.000000] NOHZ: nohz enabled due to nohz_full encountered [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 3849420k/5242880k available (6443k kernel code, 1049100k absent, 344360k reserved, 4459k data, 1588k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=10, Nodes=1 [ 0.000000] Preemptible hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=10. [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-9. [ 0.000000] RCU kthread priority: 2. [ 0.000000] NO_HZ: Full dynticks CPUs: 1. [ 0.000000] NR_IRQS:327936 nr_irqs:504 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 67108864 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] tsc: Detected 2299.998 MHz processor [ 0.191765] Calibrating delay loop (skipped) preset value.. 4599.99 BogoMIPS (lpj=2299998) [ 0.191766] pid_max: default: 32768 minimum: 301 [ 0.191811] Security Framework initialized [ 0.191820] SELinux: Initializing. [ 0.191845] SELinux: Starting in permissive mode [ 0.192581] Dentry cache hash table entries: 524288 (order: 11, 8388608 bytes) [ 0.196896] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.197942] Mount-cache hash table entries: 4096 [ 0.198269] Initializing cgroup subsys memory [ 0.198279] Initializing cgroup subsys devices [ 0.198280] Initializing cgroup subsys freezer [ 0.198281] Initializing cgroup subsys net_cls [ 0.198282] Initializing cgroup subsys blkio [ 0.198283] Initializing cgroup subsys perf_event [ 0.198285] Initializing cgroup subsys hugetlb [ 0.198286] Initializing cgroup subsys net_prio [ 0.199193] mce: CPU supports 10 MCE banks [ 0.199234] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.199235] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.199235] tlb_flushall_shift: 2 [ 0.199506] Freeing SMP alternatives: 20k freed [ 0.201645] ACPI: Core revision 20130517 [ 0.202672] ACPI: All ACPI Tables successfully acquired [ 0.202725] ftrace: allocating 24211 entries in 95 pages [ 0.229812] Enabling x2apic [ 0.229818] Enabled x2apic [ 0.230070] Switched APIC routing to physical x2apic. [ 0.231127] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.231130] smpboot: CPU0: Intel Xeon E3-12xx v2 (Ivy Bridge) (fam: 06, model: 3a, stepping: 09) [ 0.231140] TSC deadline timer enabled [ 0.231166] Performance Events: unsupported p6 CPU model 58 no PMU driver, software events only. [ 0.231303] KVM setup paravirtual spinlock [ 0.257502] kvm-clock: cpu 1, msr 1:3ff86041, secondary cpu clock [ 0.278531] KVM setup async PF for cpu 1 [ 0.278535] kvm-stealtime: cpu 1, msr 13b08e1c0 [ 0.293733] kvm-clock: cpu 2, msr 1:3ff86081, secondary cpu clock [ 0.314755] KVM setup async PF for cpu 2 [ 0.314758] kvm-stealtime: cpu 2, msr 13b10e1c0 [ 0.329942] kvm-clock: cpu 3, msr 1:3ff860c1, secondary cpu clock [ 0.246426] smpboot: Booting Node 0, Processors #1 #2 #3 [ 0.350960] Brought up 4 CPUs [ 0.350964] smpboot: Total of 4 processors activated (18399.98 BogoMIPS) [ 0.350968] KVM setup async PF for cpu 3 [ 0.350972] kvm-stealtime: cpu 3, msr 13b18e1c0 [ 0.351828] devtmpfs: initialized [ 0.353988] EVM: security.selinux [ 0.353990] EVM: security.ima [ 0.353991] EVM: security.capability [ 0.355149] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.355306] NET: Registered protocol family 16 [ 0.355693] ACPI: bus type PCI registered [ 0.355696] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.355802] PCI: Using configuration type 1 for base access [ 0.364588] ACPI: Added _OSI(Module Device) [ 0.364591] ACPI: Added _OSI(Processor Device) [ 0.364592] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.364594] ACPI: Added _OSI(Processor Aggregator Device) [ 0.365391] ACPI: EC: Look up EC in DSDT [ 0.366551] ACPI: Interpreter enabled [ 0.366556] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20130517/hwxface-571) [ 0.366560] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20130517/hwxface-571) [ 0.366562] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20130517/hwxface-571) [ 0.366565] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20130517/hwxface-571) [ 0.366570] ACPI: (supports S0 S5) [ 0.366571] ACPI: Using IOAPIC for interrupt routing [ 0.366593] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.371150] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.371156] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 0.371162] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.371175] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.371426] acpiphp: Slot [2] registered [ 0.371447] acpiphp: Slot [3] registered [ 0.371478] acpiphp: Slot [5] registered [ 0.371496] acpiphp: Slot [6] registered [ 0.371514] acpiphp: Slot [7] registered [ 0.371532] acpiphp: Slot [8] registered [ 0.371549] acpiphp: Slot [9] registered [ 0.371574] acpiphp: Slot [10] registered [ 0.371594] acpiphp: Slot [11] registered [ 0.371612] acpiphp: Slot [12] registered [ 0.371630] acpiphp: Slot [13] registered [ 0.371648] acpiphp: Slot [14] registered [ 0.371665] acpiphp: Slot [15] registered [ 0.371683] acpiphp: Slot [16] registered [ 0.371712] acpiphp: Slot [17] registered [ 0.371731] acpiphp: Slot [18] registered [ 0.371749] acpiphp: Slot [19] registered [ 0.371767] acpiphp: Slot [20] registered [ 0.371785] acpiphp: Slot [21] registered [ 0.371803] acpiphp: Slot [22] registered [ 0.371821] acpiphp: Slot [23] registered [ 0.371845] acpiphp: Slot [24] registered [ 0.371865] acpiphp: Slot [25] registered [ 0.371883] acpiphp: Slot [26] registered [ 0.371901] acpiphp: Slot [27] registered [ 0.371925] acpiphp: Slot [28] registered [ 0.371944] acpiphp: Slot [29] registered [ 0.371962] acpiphp: Slot [30] registered [ 0.371991] acpiphp: Slot [31] registered [ 0.372015] PCI host bridge to bus 0000:00 [ 0.372019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.372021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7] [ 0.372023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff] [ 0.372024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] [ 0.372026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff] [ 0.372060] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.372420] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.372888] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376309] pci 0000:00:01.1: reg 0x20: [io 0xc080-0xc08f] [ 0.377781] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377784] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.377786] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377787] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377942] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378338] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.378348] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378578] pci 0000:00:02.0: [1af4:1001] type 00 class 0x010000 [ 0.380174] pci 0000:00:02.0: reg 0x10: [io 0xc000-0xc03f] [ 0.381673] pci 0000:00:02.0: reg 0x14: [mem 0xfc052000-0xfc052fff] [ 0.389264] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 [ 0.390584] pci 0000:00:03.0: reg 0x10: [io 0xc040-0xc05f] [ 0.391877] pci 0000:00:03.0: reg 0x14: [mem 0xfc053000-0xfc053fff] [ 0.399647] pci 0000:00:03.0: reg 0x30: [mem 0xfc000000-0xfc03ffff pref] [ 0.399943] pci 0000:00:04.0: [1b36:0100] type 00 class 0x030000 [ 0.409595] pci 0000:00:04.0: reg 0x10: [mem 0xf4000000-0xf7ffffff] [ 0.414162] pci 0000:00:04.0: reg 0x14: [mem 0xf8000000-0xfbffffff] [ 0.418503] pci 0000:00:04.0: reg 0x18: [mem 0xfc050000-0xfc051fff] [ 0.422477] pci 0000:00:04.0: reg 0x1c: [io 0xc060-0xc07f] [ 0.436061] pci 0000:00:04.0: reg 0x30: [mem 0xfc040000-0xfc04ffff pref] [ 0.437066] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) [ 0.437174] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) [ 0.437271] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) [ 0.437360] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) [ 0.437413] ACPI: PCI Interrupt Link [LNKS] (IRQs *9) [ 0.438348] ACPI: Enabled 16 GPEs in block 00 to 0F [ 0.438683] vgaarb: device added: PCI:0000:00:04.0,decodes=io+mem,owns=io+mem,locks=none [ 0.438685] vgaarb: loaded [ 0.438687] vgaarb: bridge control possible 0000:00:04.0 [ 0.438881] SCSI subsystem initialized [ 0.438928] ACPI: bus type USB registered [ 0.438977] usbcore: registered new interface driver usbfs [ 0.438988] usbcore: registered new interface driver hub [ 0.439094] usbcore: registered new device driver usb [ 0.439362] PCI: Using ACPI for IRQ routing [ 0.439365] PCI: pci_cache_line_size set to 64 bytes [ 0.439488] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff] [ 0.439493] e820: reserve RAM buffer [mem 0xbffdf000-0xbfffffff] [ 0.439726] NetLabel: Initializing [ 0.439729] NetLabel: domain hash size = 128 [ 0.439730] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.439750] NetLabel: unlabeled traffic allowed by default [ 0.439971] Switched to clocksource kvm-clock [ 0.452001] pnp: PnP ACPI init [ 0.452018] ACPI: bus type PNP registered [ 0.452101] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 0.452152] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active) [ 0.452200] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active) [ 0.452258] pnp 00:03: [dma 2] [ 0.452271] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active) [ 0.452367] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 0.452849] pnp: PnP ACPI: found 5 devices [ 0.452851] ACPI: bus type PNP unregistered [ 0.461142] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7] [ 0.461145] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff] [ 0.461146] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff] [ 0.461148] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff] [ 0.461206] NET: Registered protocol family 2 [ 0.461569] TCP established hash table entries: 32768 (order: 6, 262144 bytes) [ 0.461895] TCP bind hash table entries: 32768 (order: 8, 1835008 bytes) [ 0.463060] TCP: Hash tables configured (established 32768 bind 32768) [ 0.463148] TCP: reno registered [ 0.463173] UDP hash table entries: 2048 (order: 6, 262144 bytes) [ 0.463378] UDP-Lite hash table entries: 2048 (order: 6, 262144 bytes) [ 0.463736] NET: Registered protocol family 1 [ 0.463760] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.463781] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.463799] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.463849] pci 0000:00:04.0: Boot video device [ 0.463858] PCI: CLS 0 bytes, default 64 [ 0.463958] Unpacking initramfs... [ 0.952876] Freeing initrd memory: 28872k freed [ 0.960558] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 0.960565] software IO TLB [mem 0xbbfdf000-0xbffdf000] (64MB) mapped at [ffff8800bbfdf000-ffff8800bffdefff] [ 0.966773] microcode: CPU0 sig=0x306a9, pf=0x1, revision=0x1 [ 0.966787] microcode: CPU1 sig=0x306a9, pf=0x1, revision=0x1 [ 0.966792] microcode: CPU2 sig=0x306a9, pf=0x1, revision=0x1 [ 0.966806] microcode: CPU3 sig=0x306a9, pf=0x1, revision=0x1 [ 0.966836] microcode: Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba [ 0.967086] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 0.967229] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 0.967620] futex hash table entries: 4096 (order: 7, 524288 bytes) [ 0.967923] Initialise system trusted keyring [ 0.968000] audit: initializing netlink socket (disabled) [ 0.968034] type=2000 audit(1458722431.502:1): initialized [ 0.985990] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 0.987179] zpool: loaded [ 0.987183] zbud: loaded [ 0.987456] VFS: Disk quotas dquot_6.5.2 [ 0.987493] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.987728] msgmni has been set to 7574 [ 0.987804] Key type big_key registered [ 0.987807] SELinux: Registering netfilter hooks [ 0.989839] NET: Registered protocol family 38 [ 0.989850] Key type asymmetric registered [ 0.989855] Asymmetric key parser 'x509' registered [ 0.989890] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.989956] io scheduler noop registered [ 0.989960] io scheduler deadline registered (default) [ 0.990007] io scheduler cfq registered [ 0.990117] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.990124] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.990203] intel_idle: does not run on family 6 model 58 [ 0.990279] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 0.990284] ACPI: Power Button [PWRF] [ 0.990447] GHES: HEST is not enabled! [ 0.990517] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 1.014695] 00:04: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 1.015310] Non-volatile memory driver v1.3 [ 1.015314] Linux agpgart interface v0.103 [ 1.015465] crash memory driver: version 1.1 [ 1.015501] rdac: device handler registered [ 1.015558] hp_sw: device handler registered [ 1.015562] emc: device handler registered [ 1.015565] alua: device handler registered [ 1.015601] libphy: Fixed MDIO Bus: probed [ 1.015687] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 1.015696] ehci-pci: EHCI PCI platform driver [ 1.015709] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 1.015717] ohci-pci: OHCI PCI platform driver [ 1.015729] uhci_hcd: USB Universal Host Controller Interface driver [ 1.015788] usbcore: registered new interface driver usbserial [ 1.015799] usbcore: registered new interface driver usbserial_generic [ 1.015808] usbserial: USB Serial support registered for generic [ 1.015838] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 1.016739] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 1.016746] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 1.016823] mousedev: PS/2 mouse device common for all mice [ 1.017086] rtc_cmos 00:00: RTC can wake from S4 [ 1.017389] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 1.017578] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram [ 1.017628] cpuidle: using governor menu [ 1.017713] hidraw: raw HID events driver (C) Jiri Kosina [ 1.017812] usbcore: registered new interface driver usbhid [ 1.017814] usbhid: USB HID core driver [ 1.017843] drop_monitor: Initializing network drop monitor service [ 1.017935] TCP: cubic registered [ 1.017941] Initializing XFRM netlink socket [ 1.018083] NET: Registered protocol family 10 [ 1.018428] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 1.018472] NET: Registered protocol family 17 [ 1.019058] Loading compiled-in X.509 certificates [ 1.019845] Loaded X.509 cert 'Magrathea: Glacier signing key: 3396ce119650c1291ed2aef13e5964c4d274f35f' [ 1.019868] registered taskstats version 1 [ 1.022779] Key type trusted registered [ 1.024904] Key type encrypted registered [ 1.027287] IMA: No TPM chip found, activating TPM-bypass! [ 1.027687] rtc_cmos 00:00: setting system clock to 2016-03-23 08:40:31 UTC (1458722431) [ 1.028684] Freeing unused kernel memory: 1588k freed [ 1.033573] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 1.033594] systemd[1]: Detected virtualization kvm. [ 1.033598] systemd[1]: Detected architecture x86-64. [ 1.033601] systemd[1]: Running in initial RAM disk. [ 1.033632] systemd[1]: Set hostname to <guest>. [ 1.075849] systemd[1]: Reached target Timers. [ 1.075860] systemd[1]: Starting Timers. [ 1.075872] systemd[1]: Reached target Swap. [ 1.075878] systemd[1]: Starting Swap. [ 1.075888] systemd[1]: Reached target Local File Systems. [ 1.075894] systemd[1]: Starting Local File Systems. [ 1.076124] systemd[1]: Created slice -.slice. [ 1.076134] systemd[1]: Starting -.slice. [ 1.076188] systemd[1]: Listening on udev Control Socket. [ 1.076196] systemd[1]: Starting udev Control Socket. [ 1.076253] systemd[1]: Listening on Journal Socket. [ 1.076261] systemd[1]: Starting Journal Socket. [ 1.076334] systemd[1]: Created slice System Slice. [ 1.076343] systemd[1]: Starting System Slice. [ 1.076916] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 1.077429] systemd[1]: Starting Journal Service... [ 1.077461] systemd[1]: Reached target Slices. [ 1.077476] systemd[1]: Starting Slices. [ 1.077531] systemd[1]: Started dracut ask for additional cmdline parameters. [ 1.078175] systemd[1]: Starting dracut cmdline hook... [ 1.078348] systemd[1]: Started Load Kernel Modules. [ 1.078823] systemd[1]: Starting Apply Kernel Variables... [ 1.078888] systemd[1]: Listening on udev Kernel Socket. [ 1.078897] systemd[1]: Starting udev Kernel Socket. [ 1.078922] systemd[1]: Reached target Sockets. [ 1.078929] systemd[1]: Starting Sockets. [ 1.080350] systemd[1]: Starting Setup Virtual Console... [ 1.081041] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 1.081919] systemd[1]: Starting Create Static Device Nodes in /dev... [ 1.084247] systemd[1]: Started Apply Kernel Variables. [ 1.091103] systemd[1]: Started Create Static Device Nodes in /dev. [ 1.091723] systemd[1]: Started Journal Service. [ 1.220288] device-mapper: uevent: version 1.0.3 [ 1.220725] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel [ 1.386006] libata version 3.00 loaded. [ 1.422618] Floppy drive(s): fd0 is 1.44M [ 1.422739] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 [ 1.422849] virtio-pci 0000:00:02.0: virtio_pci: leaving for legacy driver [ 1.424694] ata_piix 0000:00:01.1: version 2.13 [ 1.433386] FDC 0 is a S82078B [ 1.440664] scsi host0: ata_piix [ 1.445682] scsi host1: ata_piix [ 1.449822] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc080 irq 14 [ 1.449825] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc088 irq 15 [ 1.477688] [drm] Initialized drm 1.1.0 20060810 [ 1.478821] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 [ 1.478901] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver [ 1.565073] virtio-pci 0000:00:02.0: irq 24 for MSI/MSI-X [ 1.565092] virtio-pci 0000:00:02.0: irq 25 for MSI/MSI-X [ 1.566911] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 [ 1.568622] vda: vda1 vda2 [ 1.569421] [drm] Device Version 0.0 [ 1.569423] [drm] Compression level 0 log level 0 [ 1.569425] [drm] Currently using mode #0, list at 0x488 [ 1.569431] [drm] 12286 io pages at offset 0x1000000 [ 1.569432] [drm] 16777216 byte draw area at offset 0x0 [ 1.569433] [drm] RAM header offset: 0x3ffe000 [ 1.569434] [drm] rom modes offset 0x488 for 128 modes [ 1.570006] [TTM] Zone kernel: Available graphics memory: 1939950 kiB [ 1.570008] [TTM] Initializing pool allocator [ 1.570177] [TTM] Initializing DMA pool allocator [ 1.570610] [drm] qxl: 16M of VRAM memory size [ 1.570612] [drm] qxl: 63M of IO pages memory ready (VRAM domain) [ 1.570613] [drm] qxl: 64M of Surface memory size [ 1.574208] [drm] main mem slot 1 [f4000000,3ffe000] [ 1.574210] [drm] surface mem slot 2 [f8000000,4000000] [ 1.574279] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 1.574281] [drm] No driver support for vblank timestamp query. [ 1.581767] [drm] fb mappable at 0xF4000000, size 3145728 [ 1.581770] [drm] fb: depth 24, pitch 4096, width 1024, height 768 [ 1.583802] fbcon: qxldrmfb (fb0) is primary device [ 1.601476] ata2.01: NODEV after polling detection [ 1.601728] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100 [ 1.602164] ata2.00: configured for MWDMA2 [ 1.602692] scsi 1:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5 [ 1.606367] Console: switching to colour frame buffer device 128x48 [ 1.608845] qxl 0000:00:04.0: fb0: qxldrmfb frame buffer device [ 1.608846] qxl 0000:00:04.0: registered panic notifier [ 1.612706] virtio-pci 0000:00:03.0: irq 26 for MSI/MSI-X [ 1.612726] virtio-pci 0000:00:03.0: irq 27 for MSI/MSI-X [ 1.612743] virtio-pci 0000:00:03.0: irq 28 for MSI/MSI-X [ 1.626038] [drm] Initialized qxl 0.1.0 20120117 for 0000:00:04.0 on minor 0 [ 1.674203] sr 1:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray [ 1.674205] cdrom: Uniform CD-ROM driver Revision: 3.20 [ 1.674540] sr 1:0:0:0: Attached scsi CD-ROM sr0 [ 1.853363] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2 [ 1.905122] SGI XFS with ACLs, security attributes, no debug enabled [ 1.909166] XFS (dm-0): Mounting V4 Filesystem [ 1.914940] XFS (dm-0): Ending clean mount [ 1.962029] tsc: Refined TSC clocksource calibration: 2299.986 MHz [ 2.170664] systemd-journald[133]: Received SIGTERM from PID 1 (systemd). [ 2.207466] type=1404 audit(1458722432.679:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295 [ 2.342164] SELinux: 2048 avtab hash slots, 112391 rules. [ 2.358732] SELinux: 2048 avtab hash slots, 112391 rules. [ 2.390469] SELinux: 8 users, 105 roles, 4954 types, 302 bools, 1 sens, 1024 cats [ 2.390472] SELinux: 83 classes, 112391 rules [ 2.395146] SELinux: Completing initialization. [ 2.395148] SELinux: Setting up existing superblocks. [ 2.395155] SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts [ 2.395160] SELinux: initialized (dev rootfs, type rootfs), uses genfs_contexts [ 2.395172] SELinux: initialized (dev bdev, type bdev), uses genfs_contexts [ 2.395177] SELinux: initialized (dev proc, type proc), uses genfs_contexts [ 2.395231] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs [ 2.395610] SELinux: initialized (dev devtmpfs, type devtmpfs), uses transition SIDs [ 2.396089] SELinux: initialized (dev sockfs, type sockfs), uses task SIDs [ 2.396094] SELinux: initialized (dev debugfs, type debugfs), uses genfs_contexts [ 2.397394] SELinux: initialized (dev pipefs, type pipefs), uses task SIDs [ 2.397399] SELinux: initialized (dev anon_inodefs, type anon_inodefs), uses genfs_contexts [ 2.397407] SELinux: initialized (dev aio, type aio), not configured for labeling [ 2.397410] SELinux: initialized (dev devpts, type devpts), uses transition SIDs [ 2.397422] SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses transition SIDs [ 2.397427] SELinux: initialized (dev mqueue, type mqueue), uses transition SIDs [ 2.397433] SELinux: initialized (dev selinuxfs, type selinuxfs), uses genfs_contexts [ 2.397447] SELinux: initialized (dev securityfs, type securityfs), uses genfs_contexts [ 2.397452] SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts [ 2.397831] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs [ 2.397836] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs [ 2.397910] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs [ 2.397936] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.397945] SELinux: initialized (dev pstore, type pstore), uses genfs_contexts [ 2.397947] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.397980] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.397986] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.397991] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.398002] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.398008] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.398012] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.398017] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.398021] SELinux: initialized (dev cgroup, type cgroup), uses genfs_contexts [ 2.398031] SELinux: initialized (dev configfs, type configfs), uses genfs_contexts [ 2.398034] SELinux: initialized (dev drm, type drm), not configured for labeling [ 2.398042] SELinux: initialized (dev dm-0, type xfs), uses xattr [ 2.408567] type=1403 audit(1458722432.880:3): policy loaded auid=4294967295 ses=4294967295 [ 2.414725] systemd[1]: Successfully loaded SELinux policy in 207.565ms. [ 2.433193] ip_tables: (C) 2000-2006 Netfilter Core Team [ 2.433264] systemd[1]: Inserted module 'ip_tables' [ 2.456460] systemd[1]: Relabelled /dev and /run in 20.143ms. [ 2.581898] SELinux: initialized (dev hugetlbfs, type hugetlbfs), uses transition SIDs [ 2.589912] SELinux: initialized (dev autofs, type autofs), uses genfs_contexts [ 2.614934] RPC: Registered named UNIX socket transport module. [ 2.614936] RPC: Registered udp transport module. [ 2.614938] RPC: Registered tcp transport module. [ 2.614939] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.615613] SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts [ 2.629849] Installing knfsd (copyright (C) 1996 okir.de). [ 2.630257] SELinux: initialized (dev nfsd, type nfsd), uses genfs_contexts [ 2.684102] systemd-journald[530]: Received request to flush runtime journal from PID 1 [ 3.076914] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 3.148641] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 3.180575] sr 1:0:0:0: Attached scsi generic sg0 type 5 [ 3.307851] AES CTR mode by8 optimization enabled [ 3.315729] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 3.316320] XFS (vda1): Mounting V4 Filesystem [ 3.340242] alg: No test for crc32 (crc32-pclmul) [ 3.385391] intel_rapl: no valid rapl domains found in package 0 [ 3.417998] intel_rapl: no valid rapl domains found in package 0 [ 3.425011] intel_rapl: no valid rapl domains found in package 0 [ 3.524258] ppdev: user-space parallel port driver [ 3.526255] XFS (vda1): Ending clean mount [ 3.526268] SELinux: initialized (dev vda1, type xfs), uses xattr [ 3.570458] Adding 2097148k swap on /dev/mapper/rhel_dhcp--66--106--180-swap. Priority:-1 extents:1 across:2097148k FS [ 3.715522] type=1305 audit(1458722434.187:4): audit_pid=682 old=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:auditd_t:s0 res=1 [ 4.193884] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) [ 4.210489] ip6_tables: (C) 2000-2006 Netfilter Core Team [ 4.258385] Ebtables v2.0 registered [ 4.272000] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this. [ 4.474751] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 33.531206] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs [ 58.942131] CPU4 has been hot-added [ 58.955880] smpboot: Booting Node 0 Processor 4 APIC 0x4 [ 58.967406] kvm-clock: cpu 4, msr 1:3ff86101, secondary cpu clock [ 58.988385] TSC synchronization [CPU#3 -> CPU#4]: [ 58.988386] Measured 139627348439 cycles TSC warp between CPUs, turning off TSC clock. [ 58.988389] tsc: Marking TSC unstable due to check_tsc_sync_source failed [ 58.988808] KVM setup async PF for cpu 4 [ 58.988812] kvm-stealtime: cpu 4, msr 13b20e1c0 [ 58.988840] microcode: CPU4 sig=0x306a9, pf=0x1, revision=0x1 [ 58.989151] Will online and init hotplugged CPU: 4 [ 58.691009] ------------[ cut here ]------------ [ 58.691017] WARNING: at kernel/time/tick-sched.c:196 can_stop_full_tick+0x169/0x280() [ 58.691018] NO_HZ FULL will not work with unstable sched clock [ 58.691052] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_blk qxl syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common drm_kms_helper crc32c_intel ttm drm serio_raw i2c_core [ 58.691055] floppy ata_piix virtio_pci libata dm_mirror dm_region_hash dm_log dm_mod [ 58.691057] CPU: 1 PID: 21 Comm: migration/1 Not tainted 3.10.0-366.rt56.243.el7.x86_64 #1 [ 58.691058] Hardware name: Red Hat KVM, BIOS seabios-1.7.5-11.el7 04/01/2014 [ 58.691061] ffff88013b083ee8 000000009e99d285 ffff88013b083ea0 ffffffff81637f9e [ 58.691062] ffff88013b083ed8 ffffffff81073a10 ffff88013b08fdc0 0000000000000001 [ 58.691063] ffff880131043c01 0000000000000282 ffff88013b08ff20 ffff88013b083f40 [ 58.691064] Call Trace: [ 58.691070] <IRQ> [<ffffffff81637f9e>] dump_stack+0x19/0x1b [ 58.691073] [<ffffffff81073a10>] warn_slowpath_common+0x70/0xc0 [ 58.691075] [<ffffffff81073abc>] warn_slowpath_fmt+0x5c/0x80 [ 58.691076] [<ffffffff810d6f19>] can_stop_full_tick+0x169/0x280 [ 58.691078] [<ffffffff810d7410>] tick_nohz_irq_exit+0x80/0xb0 [ 58.691080] [<ffffffff8107e470>] irq_exit+0xf0/0x1b0 [ 58.691083] [<ffffffff816487f5>] smp_apic_timer_interrupt+0x45/0x60 [ 58.691086] [<ffffffff81646fdd>] apic_timer_interrupt+0x6d/0x80 [ 58.691091] <EOI> [<ffffffff810faabf>] ? multi_cpu_stop+0x8f/0x120 [ 58.691093] [<ffffffff810faa30>] ? cpu_stop_park+0x40/0x40 [ 58.691094] [<ffffffff810fa81f>] cpu_stopper_thread+0xcf/0x200 [ 58.691097] [<ffffffff810ab20e>] ? finish_task_switch+0xfe/0x160 [ 58.691100] [<ffffffff8163bd54>] ? __schedule+0x2f4/0x850 [ 58.691103] [<ffffffff810a97dd>] smpboot_thread_fn+0x1cd/0x310 [ 58.691105] [<ffffffff8163c2e0>] ? schedule+0x30/0xa0 [ 58.691107] [<ffffffff810a9610>] ? lg_double_unlock+0x40/0x40 [ 58.691110] [<ffffffff810a0401>] kthread+0xc1/0xd0 [ 58.691112] [<ffffffff810a0340>] ? kthread_worker_fn+0x170/0x170 [ 58.691120] [<ffffffff81646298>] ret_from_fork+0x58/0x90 [ 58.691122] [<ffffffff810a0340>] ? kthread_worker_fn+0x170/0x170 [ 58.691123] ---[ end trace 0000000000000002 ]--- [ 59.996031] intel_rapl: no valid rapl domains found in package 0 [ 60.031423] intel_rapl: no valid rapl domains found in package 0 [ 67.463931] SELinux: initialized (dev binfmt_misc, type binfmt_misc), uses genfs_contexts [root@guest ~]# [ 69.382049] blk_update_request: I/O error, dev fd0, sector 0 Expected results: cpu hotplug succeed without error in both guest and host. Additional info: 1. tested this case with guest in non-rt kernel, cpu hotplug works fine.
Hello, I tried to look at stacktrace and the code. kernel cmdline: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-366.rt56.243.el7.x86_64 root=/dev/mapper/rhel_dhcp--66--106--180-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-66-106-180/root rd.lvm.lv=rhel_dhcp-66-106-180/swap rhgb quiet LANG=en_US.UTF-8 isolcpus=1 nohz_full=1 <-----here I found this error message: "NO_HZ FULL will not work with unstable sched clock"--------------| | static bool can_stop_full_tick(void) { ... if (!sched_clock_stable()) { trace_tick_stop(0, "unstable sched clock\n"); /* * Don't allow the user to think they can get * full NO_HZ with this machine. */ WARN_ONCE(tick_nohz_full_running, "NO_HZ FULL will not work with unstable sched clock"); return false; \------> looks like it is returning because of unstable clock source } ... } tick_nohz_irq_exit tick_nohz_full_stop_tick can_stop_full_tick It looks like because of unstable tsc and 'nohz_full' enabled system might not behave properly. Two options we have here: 1) Disable nohz_full and test the cpu hotplug. 2) Test this in other machine which have stable tsc. Best regards, Pankaj
(In reply to pagupta from comment #2) > Hello, > > I tried to look at stacktrace and the code. > > kernel cmdline: Command line: > BOOT_IMAGE=/vmlinuz-3.10.0-366.rt56.243.el7.x86_64 > root=/dev/mapper/rhel_dhcp--66--106--180-root ro crashkernel=auto > rd.lvm.lv=rhel_dhcp-66-106-180/root rd.lvm.lv=rhel_dhcp-66-106-180/swap rhgb > quiet LANG=en_US.UTF-8 isolcpus=1 nohz_full=1 <-----here > > I found this error message: > "NO_HZ FULL will not work with unstable sched clock"--------------| > | > static bool can_stop_full_tick(void) > { > ... > if (!sched_clock_stable()) { > trace_tick_stop(0, "unstable sched clock\n"); > /* > * Don't allow the user to think they can get > * full NO_HZ with this machine. > */ > WARN_ONCE(tick_nohz_full_running, > "NO_HZ FULL will not work with unstable sched > clock"); > return false; \------> looks like it is returning > > because of unstable clock source > } > > ... > } > > tick_nohz_irq_exit > tick_nohz_full_stop_tick > can_stop_full_tick > > It looks like because of unstable tsc and 'nohz_full' enabled system might > not behave properly. Two options we have here: > > 1) Disable nohz_full and test the cpu hotplug. > 2) Test this in other machine which have stable tsc. > > Best regards, > Pankaj I tried without nohz_full param, and cpu hotplug also have some issue, please check out below: host: kernel-rt: 3.10.0-513.rt56.419.el7.x86_64 cmdline: # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-513.rt56.419.el7.x86_64 root=/dev/mapper/rhel_hp--dl385pg8--01-root ro crashkernel=auto rd.lvm.lv=rhel_hp-dl385pg8-01/root rd.lvm.lv=rhel_hp-dl385pg8-01/swap rhgb quiet LANG=en_US.UTF-8 isolcpus=0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 default_hugepagesz=1G hugepagesz=1G hugepages=16 guest: kernel-rt: 3.10.0-513.rt56.419.el7.x86_64 cmdline: # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-513.rt56.419.el7.x86_64 root=/dev/mapper/rhel_dhcp--66--106--180-root ro crashkernel=auto rd.lvm.lv=rhel_dhcp-66-106-180/root rd.lvm.lv=rhel_dhcp-66-106-180/swap rhgb quiet LANG=en_US.UTF-8 console=tty0 console=ttyS0,115200 isolcpus=1 intel_pstate=disable nosoftlockup steps: 1. boot a guest with the same command line listed in comment 1 2. cat /proc/cpuinfo in guest # cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 1 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 2 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 3 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 3 siblings : 1 core id : 0 cpu cores : 1 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: 3. cpu hotplug in host (qemu) cpu-add 4 4. cat /proc/cpuinfo in guest, only one cpu added and no more error message displayed # cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 1 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 2 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 3 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 3 siblings : 1 core id : 0 cpu cores : 1 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 4 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 4 siblings : 1 core id : 0 cpu cores : 1 apicid : 4 initial apicid : 4 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: #dmesg ... [ 281.821259] smpboot: APIC(4) Converting physical 4 to logical package 4 [ 281.821372] CPU4 has been hot-added [ 281.840403] smpboot: Booting Node 0 Processor 4 APIC 0x4 [ 0.002000] kvm-clock: cpu 4, msr 1:3ff86101, secondary cpu clock [ 282.104112] KVM setup async PF for cpu 4 [ 282.104120] kvm-stealtime: cpu 4, msr 13b210200 [ 282.104173] Will online and init hotplugged CPU: 4
Hello Xiyue, Thanks for checking without 'nohz_full' parameter for unstable tsc. Looks like earlier issue passed and vCPU hotplug succeeded. New vCPU got hotplugged but some issue in its on-lining. I am trying to understand what's happening here. There are some known issues with vCPU hotplug with RT. There is re-structuring of CPU hotplug infrastructure in upstream based on known issues. Its big chunk of work which will take some time to stabilize. Meanwhile can you please check if this issue happens with non-rt kernel as well? Thanks, Pankaj
Hi Pankaj, Here's the result of cpu hotplug on non-rt kernel. It looks worked well. guest: 3.10.0-514.el7.x86_64 1. boot a guest with the command listed in comment 1 2. cpu hotplug in host hmp (qemu) cpu-add 4 (qemu) cpu-add 5 (qemu) cpu-add 6 (qemu) cpu-add 7 (qemu) cpu-add 8 (qemu) cpu-add 9 3. cat /proc/cpuinfo in guest # cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 1 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 apicid : 1 initial apicid : 1 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 2 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 3 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 3 siblings : 1 core id : 0 cpu cores : 1 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 4 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 4 siblings : 1 core id : 0 cpu cores : 1 apicid : 4 initial apicid : 4 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 5 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 5 siblings : 1 core id : 0 cpu cores : 1 apicid : 5 initial apicid : 5 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 6 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 6 siblings : 1 core id : 0 cpu cores : 1 apicid : 6 initial apicid : 6 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 7 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 7 siblings : 1 core id : 0 cpu cores : 1 apicid : 7 initial apicid : 7 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 8 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 8 siblings : 1 core id : 0 cpu cores : 1 apicid : 8 initial apicid : 8 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: processor : 9 vendor_id : AuthenticAMD cpu family : 6 model : 58 model name : Intel Xeon E3-12xx v2 (Ivy Bridge) stepping : 9 microcode : 0x1000065 cpu MHz : 2294.248 cache size : 512 KB physical id : 9 siblings : 1 core id : 0 cpu cores : 1 apicid : 9 initial apicid : 9 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm art nopl pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm bogomips : 4588.49 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 48 bits physical, 48 bits virtual power management: 4. dmesg in guest ... [ 56.600929] smpboot: APIC(4) Converting physical 4 to logical package 4 [ 56.601058] CPU4 has been hot-added [ 57.212088] smpboot: Booting Node 0 Processor 4 APIC 0x4 [ 0.002000] kvm-clock: cpu 4, msr 1:3ff86101, secondary cpu clock [ 57.470961] KVM setup async PF for cpu 4 [ 57.470970] kvm-stealtime: cpu 4, msr 13b20f3c0 [ 57.471045] Will online and init hotplugged CPU: 4 [ 124.731848] smpboot: APIC(5) Converting physical 5 to logical package 5 [ 124.731936] CPU5 has been hot-added [ 125.117716] smpboot: Booting Node 0 Processor 5 APIC 0x5 [ 0.002000] kvm-clock: cpu 5, msr 1:3ff86141, secondary cpu clock [ 125.375523] KVM setup async PF for cpu 5 [ 125.375533] kvm-stealtime: cpu 5, msr 13b28f3c0 [ 125.375575] Will online and init hotplugged CPU: 5 [ 143.525151] smpboot: APIC(6) Converting physical 6 to logical package 6 [ 143.525233] CPU6 has been hot-added [ 143.546873] smpboot: Booting Node 0 Processor 6 APIC 0x6 [ 0.002000] kvm-clock: cpu 6, msr 1:3ff86181, secondary cpu clock [ 143.804749] KVM setup async PF for cpu 6 [ 143.804757] kvm-stealtime: cpu 6, msr 13b30f3c0 [ 143.804795] Will online and init hotplugged CPU: 6 [ 147.908971] smpboot: APIC(7) Converting physical 7 to logical package 7 [ 147.909080] CPU7 has been hot-added [ 148.331485] smpboot: Booting Node 0 Processor 7 APIC 0x7 [ 0.002000] kvm-clock: cpu 7, msr 1:3ff861c1, secondary cpu clock [ 148.587843] KVM setup async PF for cpu 7 [ 148.587853] kvm-stealtime: cpu 7, msr 13b38f3c0 [ 148.587904] Will online and init hotplugged CPU: 7 [ 149.788467] smpboot: APIC(8) Converting physical 8 to logical package 8 [ 149.788545] CPU8 has been hot-added [ 149.806692] smpboot: Booting Node 0 Processor 8 APIC 0x8 [ 0.002000] kvm-clock: cpu 8, msr 1:3ff86201, secondary cpu clock [ 150.059104] KVM setup async PF for cpu 8 [ 150.059113] kvm-stealtime: cpu 8, msr 13b40f3c0 [ 150.059178] Will online and init hotplugged CPU: 8 [ 169.158180] smpboot: APIC(9) Converting physical 9 to logical package 9 [ 169.158253] CPU9 has been hot-added [ 169.617825] smpboot: Booting Node 0 Processor 9 APIC 0x9 [ 0.002000] kvm-clock: cpu 9, msr 1:3ff86241, secondary cpu clock [ 169.873596] KVM setup async PF for cpu 9 [ 169.873605] kvm-stealtime: cpu 9, msr 13b48f3c0 [ 169.873679] Will online and init hotplugged CPU: 9 The result with RT kernel after 'cpu-add 4' also looks fine without nohz_full param in cmdline, since 'cpu-add id' is used to add a vCPU id x, not to add x vCPUs. But is it OK to remove the nohz_full param? I mean, it's added to the cmdline by default after active tuned-profiles-realtime. Should we always manually remove this param when we testing RT kernel related functions? Thanks -Xiyue
Good, its working for RT as well. I think configuration which we use for KVM RT guest profile don't use 'nohz_full' for guest. As per my understanding reason for this as we run 'cyclictest'(sched_fifo) and 'stress'(sched_other) on same isolated CPU(which is realtime CPU). nohz_full' would be effective if there is only single task running. I also found below commit. https://github.com/jeremyeder/tuned-profiles-realtime/commit/bb0404695d608c884a9235a38215aa9a337af66e Could you please check which version of tuned guest profile you are using or maybe try using latest version. The reason for not using nohz_full in this BZ is **unstable tsc** which will fail or give undesired result. In normal system with stable tsc source, 'nohz_full' should make system to work but just latency will be on higher side. Best regards, Pankaj
Hi Pankaj, Checked the tuned guest profile I'm using and turns out it's quite old: [root@guest ~]# rpm -qa | grep tuned-profiles-realtime tuned-profiles-realtime-2.5.1-4.el7.noarch I updated to the newest profile tuned-profiles-realtime-2.7.1-3.el7.noarch. After active the new profile, nohz_full doesn't appeared in guest. So the issue mentioned above is resolved. I have another question: In what situation 'unstable tsc' will be detected? 'Cause the testing host I'm using is quite normal and I don't understand how to check whether the tsc time source is stable or not. Thanks, Xiyue
(In reply to xiywang from comment #7) > Hi Pankaj, > > Checked the tuned guest profile I'm using and turns out it's quite old: > > [root@guest ~]# rpm -qa | grep tuned-profiles-realtime > tuned-profiles-realtime-2.5.1-4.el7.noarch > > I updated to the newest profile tuned-profiles-realtime-2.7.1-3.el7.noarch. > After active the new profile, nohz_full doesn't appeared in guest. So the > issue mentioned above is resolved. ok, good. > > I have another question: In what situation 'unstable tsc' will be detected? It gets detected when tsc clock source is not stable means kernel is dependent on clock source for alot of its activities. At initial boot up and later as well kernel try to calibrate(complex algorithm) the clock source to take care of some delta but if kernel does not able to caliberate the source and think there might be something wrong with the hardware side and mark the clocksource 'unstable' > 'Cause the testing host I'm using is quite normal and I don't understand how > to check whether the tsc time source is stable or not. You can easily check in 'dmesg' logs > > Thanks, > Xiyue
As per comment 7, marking this BZ closed.