Bug 667340
| Summary: | kexec: Make sure to stop all CPUs before exiting the kernel | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Paolo Bonzini <pbonzini> |
| Component: | kernel | Assignee: | Paolo Bonzini <pbonzini> |
| Status: | CLOSED ERRATA | QA Contact: | Han Pingtian <phan> |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | 6.0 | CC: | clalance, drjones, qcai, xen-maint |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | kernel-2.6.32-117.el6 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2011-05-23 20:32:56 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Paolo Bonzini
2011-01-05 10:15:11 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. You can try doing kexec in a RHEL6 guest (either Xen HVM on a RHEL5 host or KVM) while the host is under heavy load. Separately, you can try rebooting a RHEL6 Xen PV guest on a RHEL5 host to ensure that there is no regression. I have tried several times. I used usex to generate some load on the host, then load the new kernel with kexec -l then jump to the new kernel with reboot on the rhel6 kvm guest. I cannot cause the panic, only saw one time the guest didn't bypass the bois, it performed a normal boot. Is this a reproduction? I think so, yes. You can try the brew build at https://brewweb.devel.redhat.com/taskinfo?taskID=3022266 if you want to "pre-verify" now. I have managed to let the testing kernel also doesn't bypass BIOS on kvm guest with a load average about 42~43 on the host, it can be achieved on my workstation using 'usex -e 34'. Patch(es) available on kernel-2.6.32-117.el6 Perhaps you can try the reproducer in bug 690419? It's for a different bug, but it could work here too. Start the CPU add-remove loop and a little after kexec. After some time, without the patch you may see a failure to boot the new kernel. With the patch, you will have to stop the loop for kexec to proceed. I saw two times this kind of warnings with -71.el6 kvm x86_64 guest, on host tyan-gt24-01.rhts.eng.bos.redhat.com.
The first:
...
Unmounting file systems: [ OK ]
init: Re-executing /sbin/init
Please stand by while rebooting the system...
md: stopping all md devices.
Starting new kernel
------------[ cut here ]------------
WARNING: at arch/x86/kernel/smp.c:117 native_smp_send_reschedule+0x5c/0x60() (Not tainted)
Hardware name: KVM
Modules linked in: sit tunnel4 sunrpc ipv6 dm_mirror dm_region_hash dm_log virtio_balloon virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: freq_table]
Pid: 2190, comm: kexec Not tainted 2.6.32-71.el6.x86_64 #1
Call Trace:
<IRQ> [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
[<ffffffff8106b8aa>] warn_slowpath_null+0x1a/0x20
[<ffffffff8102ea7c>] native_smp_send_reschedule+0x5c/0x60
[<ffffffff810507f8>] resched_task+0x68/0x80
[<ffffffff81057a6d>] resched_cpu+0x8d/0xa0
[<ffffffff8106636b>] scheduler_tick+0x26b/0x280
[<ffffffff810a0c90>] ? tick_sched_timer+0x0/0xc0
[<ffffffff8107d7e2>] update_process_times+0x52/0x70
[<ffffffff810a0cf6>] tick_sched_timer+0x66/0xc0
[<ffffffff8109564e>] __run_hrtimer+0x8e/0x1a0
[<ffffffff8103be39>] ? kvm_clock_get_cycles+0x9/0x10
[<ffffffff810959f6>] hrtimer_interrupt+0xe6/0x250
[<ffffffff814cf9fc>] smp_apic_timer_interrupt+0x6c/0x9c
[<ffffffff81013c93>] apic_timer_interrupt+0x13/0x20
<EOI> [<ffffffff8101af06>] ? native_read_tsc+0x6/0x20
[<ffffffff8126644c>] ? __bitmap_weight+0x8c/0xb0
[<ffffffff8126333a>] delay_tsc+0x4a/0x80
[<ffffffff812632e6>] __const_udelay+0x46/0x50
[<ffffffff8102ebab>] native_smp_send_stop+0x6b/0xb0
[<ffffffff8102e38f>] native_machine_shutdown+0x5f/0x80
[<ffffffff8103bd75>] kvm_shutdown+0x15/0x20
[<ffffffff8102df5f>] machine_shutdown+0xf/0x20
[<ffffffff810b8928>] kernel_kexec+0x158/0x160
[<ffffffff8108a5a4>] sys_reboot+0x144/0x220
[<ffffffff8105c394>] ? try_to_wake_up+0x284/0x380
[<ffffffff8106789a>] ? __cond_resched+0x2a/0x40
[<ffffffff810d40a2>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
---[ end trace 0f4c73c2233b3107 ]---
Second:
...
Starting new kernel
------------[ cut here ]------------
WARNING: at arch/x86/kernel/smp.c:117 native_smp_send_reschedule+0x5c/0x60() (Not tainted)
Hardware name: KVM
Modules linked in: sit tunnel4 sunrpc ipv6 dm_mirror dm_region_hash dm_log virtio_balloon virtio_net i2c_piix4 i2c_core sg ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: freq_table]
Pid: 22142, comm: kexec Not tainted 2.6.32-71.el6.x86_64 #1
Call Trace:
<IRQ> [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
[<ffffffff8106b8aa>] warn_slowpath_null+0x1a/0x20
[<ffffffff8102ea7c>] native_smp_send_reschedule+0x5c/0x60
[<ffffffff810507f8>] resched_task+0x68/0x80
[<ffffffff81057a6d>] resched_cpu+0x8d/0xa0
[<ffffffff8106636b>] scheduler_tick+0x26b/0x280
[<ffffffff810a0c90>] ? tick_sched_timer+0x0/0xc0
[<ffffffff8107d7e2>] update_process_times+0x52/0x70
[<ffffffff810a0cf6>] tick_sched_timer+0x66/0xc0
[<ffffffff8109564e>] __run_hrtimer+0x8e/0x1a0
[<ffffffff8103be39>] ? kvm_clock_get_cycles+0x9/0x10
[<ffffffff810959f6>] hrtimer_interrupt+0xe6/0x250
[<ffffffff814cf9fc>] smp_apic_timer_interrupt+0x6c/0x9c
[<ffffffff81013c93>] apic_timer_interrupt+0x13/0x20
<EOI> [<ffffffff81034721>] ? native_apic_mem_write+0x11/0x20
[<ffffffff8102f9ad>] disconnect_bsp_APIC+0x3d/0xc0
[<ffffffff810329b2>] disable_IO_APIC+0xa2/0x110
[<ffffffff8102f906>] ? disable_local_APIC+0x46/0x50
[<ffffffff8102e399>] native_machine_shutdown+0x69/0x80
[<ffffffff8103bd75>] kvm_shutdown+0x15/0x20
[<ffffffff8102df5f>] machine_shutdown+0xf/0x20
[<ffffffff810b8928>] kernel_kexec+0x158/0x160
[<ffffffff8108a5a4>] sys_reboot+0x144/0x220
[<ffffffff810507f8>] ? resched_task+0x68/0x80
[<ffffffff8105c394>] ? try_to_wake_up+0x284/0x380
[<ffffffff8105c4e5>] ? wake_up_process+0x15/0x20
[<ffffffff8119522d>] ? bdi_queue_work+0x7d/0x110
[<ffffffff810d40a2>] ? audit_syscall_entry+0x272/0x2a0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
---[ end trace c3b8a463f323d0c9 ]---
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-71.el6.x86_64 (mockbuild.bos.redhat.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Wed Sep 1 01:33:01 EDT 2010
Command line: ro root=/dev/mapper/vg_dhcp71107-lv_root rd_LVM_LV=vg_dhcp71107/lv_root rd_LVM_LV=vg_dhcp71107/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=ttyS0,115200
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
Centaur CentaurHauls
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000100 - 000000000009cc00 (usable)
BIOS-e820: 000000000009cc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000001fffb000 (usable)
BIOS-e820: 000000001fffb000 - 0000000020000000 (reserved)
BIOS-e820: 00000000fffbc000 - 0000000100000000 (reserved)
DMI 2.4 present.
last_pfn = 0x1fffb max_arch_pfn = 0x400000000
x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106
init_memory_mapping: 0000000000000000-000000001fffb000
RAMDISK: 1f31f000 - 1ffefd30
ACPI: RSDP 00000000000f80a0 00014 (v00 BOCHS )
ACPI: RSDT 000000001fffdda0 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
ACPI: FACP 000000001ffffdc0 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
ACPI: DSDT 000000001fffdf30 01E4B (v01 BXPC BXDSDT 00000001 INTL 20090123)
ACPI: FACS 000000001ffffd80 00040
ACPI: SSDT 000000001fffded0 0005E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001)
ACPI: APIC 000000001fffddd0 0008A (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
No NUMA configuration found
Faking a node at 0000000000000000-000000001fffb000
Bootmem setup node 0 0000000000000000-000000001fffb000
NODE_DATA [0000000000009000 - 000000000003cfff]
bootmap [000000000003d000 - 0000000000040fff] pages 4
(7 early reservations) ==> bootmem [0000000000 - 001fffb000]
#0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000]
#1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000]
#2 [0001000000 - 0001c9eff8] TEXT DATA BSS ==> [0001000000 - 0001c9eff8]
#3 [001f31f000 - 001ffefd30] RAMDISK ==> [001f31f000 - 001ffefd30]
#4 [000009cc00 - 0000100000] BIOS reserved ==> [000009cc00 - 0000100000]
#5 [0001c9f000 - 0001c9f079] BRK ==> [0001c9f000 - 0001c9f079]
#6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000]
found SMP MP-table at [ffff8800000f80f0] f80f0
kvm-clock: Using msrs 12 and 11
kvm-clock: cpu 0, msr 0:18bf901, boot clock
Zone PFN ranges:
DMA 0x00000001 -> 0x00001000
DMA32 0x00001000 -> 0x00100000
Normal 0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0x00000001 -> 0x0000009c
0: 0x00000100 -> 0x0001fffb
ACPI: PM-Timer IO Port: 0xb008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 4 CPUs, 0 hotplug CPUs
PM: Registered nosave memory: 000000000009c000 - 000000000009d000
PM: Registered nosave memory: 000000000009d000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000
PM: Registered nosave memory: 00000000000f0000 - 0000000000100000
Allocating PCI resources starting at 20000000 (gap: 20000000:dffbc000)
Booting paravirtualized kernel on KVM
NR_CPUS:4096 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
PERCPU: Embedded 31 pages/cpu @ffff880001e00000 s95064 r8192 d23720 u524288
pcpu-alloc: s95064 r8192 d23720 u524288 alloc=1*2097152
pcpu-alloc: [0] 0 1 2 3
kvm-clock: cpu 0, msr 0:1e16901, primary cpu clock
Built 1 zonelists in Node order, mobility grouping on. Total pages: 129071
Policy zone: DMA32
Kernel command line: ro root=/dev/mapper/vg_dhcp71107-lv_root rd_LVM_LV=vg_dhcp71107/lv_root rd_LVM_LV=vg_dhcp71107/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us console=ttyS0,115200
PID hash table entries: 2048 (order: 2, 16384 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
Memory: 488640k/524268k available (4935k kernel code, 404k absent, 35224k reserved, 3927k data, 1220k init)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:440
Console: colour VGA+ 80x25
console [ttyS0] enabled
allocated 5242880 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Detected 2394.012 MHz processor.
Calibrating delay loop (skipped) preset value.. 4788.02 BogoMIPS (lpj=2394012)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux: Initializing.
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
mce: CPU supports 10 MCE banks
Performance Events: AMD PMU driver.
... version: 0
... bit width: 48
... generic registers: 4
... value mask: 0000ffffffffffff
... max period: 00007fffffffffff
... fixed-purpose events: 0
... event mask: 000000000000000f
alternatives: switching to unfair spinlock
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20276 entries in 80 pages
Setting APIC routing to flat
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: AMD QEMU Virtual CPU version (cpu64-rhel6) stepping 03
Booting Node 0, Processors #1
kvm-clock: cpu 1, msr 0:1e96901, secondary cpu clock
#2
kvm-clock: cpu 2, msr 0:1f16901, secondary cpu clock
#3 Ok.
kvm-clock: cpu 3, msr 0:1f96901, secondary cpu clock
Brought up 4 CPUs
Total of 4 processors activated (19152.09 BogoMIPS).
Testing NMI watchdog ...
WARNING: CPU#0: NMI appears to be stuck (0->0)!
Please report this to bugzilla.kernel.org,
and attach the output of the 'dmesg' command.
WARNING: CPU#1: NMI appears to be stuck (0->0)!
Please report this to bugzilla.kernel.org,
and attach the output of the 'dmesg' command.
WARNING: CPU#2: NMI appears to be stuck (0->0)!
Please report this to bugzilla.kernel.org,
and attach the output of the 'dmesg' command.
WARNING: CPU#3: NMI appears to be stuck (0->0)!
Please report this to bugzilla.kernel.org,
and attach the output of the 'dmesg' command.
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: Interpreter enabled
ACPI: (supports S0 S3 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI
pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB
Unable to assume PCIe control: Disabling ASPM
ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKB] (IRQs 5 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 5 *10 11)
ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
Switching to clocksource kvm-clock
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 6 devices
ACPI: ACPI bus type pnp unregistered
NET: Registered protocol family 2
IP route cache hash table entries: 4096 (order: 3, 32768 bytes)
TCP established hash table entries: 16384 (order: 6, 262144 bytes)
TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
TCP reno registered
NET: Registered protocol family 1
pci 0000:00:00.0: Limiting direct PCI/PCI transfers
pci 0000:00:01.0: Activating ISA DMA hang workarounds
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (broken padding); looks like an initrd
Freeing initrd memory: 13123k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1301369751.257:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 980
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key 9D480283FFC05098
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
acpiphp: Slot [1] registered
acpiphp: Slot [2] registered
acpiphp: Slot [3] registered
acpiphp: Slot [4] registered
acpiphp: Slot [5] registered
acpiphp: Slot [6] registered
acpiphp: Slot [7] registered
acpiphp: Slot [8] registered
acpiphp: Slot [9] registered
acpiphp: Slot [10] registered
acpiphp: Slot [11] registered
acpiphp: Slot [12] registered
acpiphp: Slot [13] registered
acpiphp: Slot [14] registered
acpiphp: Slot [15] registered
acpiphp: Slot [16] registered
acpiphp: Slot [17] registered
acpiphp: Slot [18] registered
acpiphp: Slot [19] registered
acpiphp: Slot [20] registered
acpiphp: Slot [21] registered
acpiphp: Slot [22] registered
acpiphp: Slot [23] registered
acpiphp: Slot [24] registered
acpiphp: Slot [25] registered
acpiphp: Slot [26] registered
acpiphp: Slot [27] registered
acpiphp: Slot [28] registered
acpiphp: Slot [29] registered
acpiphp: Slot [30] registered
acpiphp: Slot [31] registered
pci-stub: invalid id string ""
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button [PWRF]
processor LNXCPU:00: registered as cooling_device0
processor LNXCPU:01: registered as cooling_device1
processor LNXCPU:02: registered as cooling_device2
processor LNXCPU:03: registered as cooling_device3
hpet_acpi_add: no address or irqs in _CRS
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.0
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
On the other hand, I cannot reproduce this warning with -127.el6 x86_64 kvm guest. Is this enough to verify this bug? I think so, yes. Due to comment 28-30, setting it verified. Thanks! An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0542.html |