Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 877292

Summary: RHEL5.9 guest kernel generates call trace after set cpu online/offline
Product: Red Hat Enterprise Linux 6 Reporter: Qingtang Zhou <qzhou>
Component: qemu-kvmAssignee: Igor Mammedov <imammedo>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.4CC: acathrow, areis, bsarathy, chayang, drjones, flang, juzhang, michen, mkenneth, qzhang, virt-maint
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-28 15:41:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
guest serial log none

Description Qingtang Zhou 2012-11-16 07:24:27 UTC 
Created attachment 646216 [details]
guest serial log

Description of problem:
RHEL5.9 guest generates some calltrace after setting one vcpu offline/online (1 of 2 vcpus).

Version-Release number of selected component (if applicable):
Host version:
$ /bin/grep -E "^kernel|qemu|bios" installed_packages.txt 
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
kernel-2.6.32-330.el6.x86_64
kernel-2.6.32-338.el6.x86_64
kernel-devel-2.6.32-330.el6.x86_64
kernel-firmware-2.6.32-338.el6.noarch
kernel-headers-2.6.32-330.el6.x86_64
qemu-guest-agent-0.12.1.2-2.334.el6.x86_64
qemu-img-0.12.1.2-2.334.el6.x86_64
qemu-kvm-0.12.1.2-2.334.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.334.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.334.el6.x86_64
seabios-0.6.1.2-25.el6.x86_64
sgabios-0-0.3.20110621svn.el6.x86_64
sgabios-bin-0-0.3.20110621svn.el6.noarch
vgabios-0.6b-3.6.el6.noarch

Guest version:
Kernel 2.6.18-345.el5

How reproducible:


Steps to Reproduce:
1) Log into a guest with kernel cli divider=10.
2) Take a time reading from the guest and host.
3) Set cpu offline in vm.
4) Take the time from the guest and host as given frequency.
5) Set cpu online, which was set offline before
5) Take the time from the guest and host as given frequency.
6) Check if the drift (in seconds) is higher than a user specified value.
  
Actual results:


Expected results:


Additional info:
Call trace: (This is the first calltrace, following calltrace is attached as attachment)
2012-11-12 17:42:02: Breaking affinity for irq 4
2012-11-12 17:42:02: Breaking affinity for irq 185
2012-11-12 17:42:02: CPU 1 is now offline
2012-11-12 17:42:02: Booting processor 1/1 eip 11000
2012-11-12 17:42:02: CPU 1 irqstacks, hard=c0778000 soft=c0758000
2012-11-12 17:42:03: CPU1: Intel Core 2 Duo P9xxx (Penryn Class Core 2) stepping 03
2012-11-12 17:42:03: mtrr: type mismatch for f0000000,100000 old: uncachable new: write-combining
2012-11-12 17:42:03: mtrr: type mismatch for f0000000,400000 old: uncachable new: write-combining
2012-11-12 17:44:58: INFO: task kjournald:430 blocked for more than 120 seconds.
2012-11-12 17:44:58: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2012-11-12 17:44:58: kjournald     D 0000002F  3060   430     11           456   415 (L-TLB)
2012-11-12 17:44:58:        f7c6ced4 00000046 19648741 0000002f c041d52b 00b3ff33 000009bd 0000000a
2012-11-12 17:44:58:        f7f57550 1964a232 0000002f 00001af1 00000000 f7f5765c c200c500 c21a6ac0
2012-11-12 17:44:58:        0000007d 0000364c 00000000 f7c6cf14 0000364c c042e48f c200c544 0000364c
2012-11-12 17:44:58: Call Trace:
2012-11-12 17:44:58:  [<c041d52b>] pvclock_clocksource_read+0x101/0x117
2012-11-12 17:44:58:  [<c042e48f>] getnstimeofday+0x30/0xb6
2012-11-12 17:44:58:  [<c0627500>] io_schedule+0x36/0x59
2012-11-12 17:44:58:  [<c047bc02>] sync_buffer+0x30/0x33
2012-11-12 17:44:58:  [<c06276d7>] __wait_on_bit+0x33/0x58
2012-11-12 17:44:58:  [<c047bbd2>] sync_buffer+0x0/0x33
2012-11-12 17:44:58:  [<c047bbd2>] sync_buffer+0x0/0x33
2012-11-12 17:44:58:  [<c062775e>] out_of_line_wait_on_bit+0x62/0x6a
2012-11-12 17:44:58:  [<c043820c>] wake_bit_function+0x0/0x3c
2012-11-12 17:44:58:  [<c047bb7f>] __wait_on_buffer+0x1c/0x1f
2012-11-12 17:44:58:  [<f88524c6>] journal_commit_transaction+0x4d6/0xf60 [jbd]
2012-11-12 17:44:58:  [<c042f2e9>] lock_timer_base+0x15/0x2f
2012-11-12 17:44:58:  [<c042f368>] try_to_del_timer_sync+0x65/0x6c
2012-11-12 17:44:58:  [<f8855d38>] kjournald+0xa1/0x1c2 [jbd]
2012-11-12 17:44:58:  [<c04381df>] autoremove_wake_function+0x0/0x2d
2012-11-12 17:44:58:  [<f8855c97>] kjournald+0x0/0x1c2 [jbd]
2012-11-12 17:44:58:  [<c043811a>] kthread+0xc0/0xee
2012-11-12 17:44:58:  [<c043805a>] kthread+0x0/0xee
2012-11-12 17:44:58:  [<c0405c87>] kernel_thread_helper+0x7/0x10
2012-11-12 17:44:58:  =======================

qemu-kvm cli:
11/12 17:38:44 INFO |    kvm_vm:1857| /usr/local/staf/test/RHEV/kvm/autotest-devel/client/tests/kvm/qemu 
11/12 17:38:44 INFO |    kvm_vm:1857|     -S 
11/12 17:38:44 INFO |    kvm_vm:1857|     -name 'vm1' 
11/12 17:38:44 INFO |    kvm_vm:1857|     -nodefaults 
11/12 17:38:44 INFO |    kvm_vm:1857|     -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20121112-165032-sRN82KnG,server,nowait 
11/12 17:38:44 INFO |    kvm_vm:1857|     -mon chardev=qmp_id_qmpmonitor1,mode=control 
11/12 17:38:44 INFO |    kvm_vm:1857|     -chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20121112-165032-sRN82KnG,server,nowait 
11/12 17:38:44 INFO |    kvm_vm:1857|     -device isa-serial,chardev=serial_id_serial1 
11/12 17:38:44 INFO |    kvm_vm:1857|     -chardev socket,id=seabioslog_id_20121112-165032-sRN82KnG,path=/tmp/seabios-20121112-165032-sRN82KnG,server,nowait 
11/12 17:38:44 INFO |    kvm_vm:1857|     -device isa-debugcon,chardev=seabioslog_id_20121112-165032-sRN82KnG,iobase=0x402 
11/12 17:38:44 INFO |    kvm_vm:1857|     -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 
11/12 17:38:44 INFO |    kvm_vm:1857|     -drive file='/usr/local/staf/test/RHEV/kvm/autotest-devel/client/tests/kvm/images/RHEL-Server-5.9-32-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native 
11/12 17:38:44 INFO |    kvm_vm:1857|     -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 
11/12 17:38:44 INFO |    kvm_vm:1857|     -device virtio-net-pci,netdev=idqmfI4N,mac=9a:49:4a:4b:4c:4d,bus=pci.0,addr=0x3,id='idUsHWTI' 
11/12 17:38:44 INFO |    kvm_vm:1857|     -netdev tap,id=idqmfI4N,vhost=on,fd=27 
11/12 17:38:44 INFO |    kvm_vm:1857|     -m 2048 
11/12 17:38:44 INFO |    kvm_vm:1857|     -smp 2,cores=1,threads=1,sockets=2 
11/12 17:38:44 INFO |    kvm_vm:1857|     -cpu 'Penryn' 
11/12 17:38:44 INFO |    kvm_vm:1857|     -M rhel6.4.0 
11/12 17:38:44 INFO |    kvm_vm:1857|     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 
11/12 17:38:44 INFO |    kvm_vm:1857|     -vnc :0 
11/12 17:38:44 INFO |    kvm_vm:1857|     -vga cirrus 
11/12 17:38:44 INFO |    kvm_vm:1857|     -rtc base=utc,clock=host,driftfix=slew  
11/12 17:38:44 INFO |    kvm_vm:1857|     -boot order=cdn,once=c,menu=off   
11/12 17:38:44 INFO |    kvm_vm:1857|     -no-kvm-pit-reinjection 
11/12 17:38:44 INFO |    kvm_vm:1857|     -enable-kvm
Comment 1 Qingtang Zhou 2012-11-16 07:27:23 UTC 
sorry, reformatted qemu-kvm cli:

/path/to/qemu 
    -S  
    -name 'vm1' 
    -nodefaults 
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20121112-165032-sRN82KnG,server,nowait 
    -mon chardev=qmp_id_qmpmonitor1,mode=control 
    -chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20121112-165032-sRN82KnG,server,nowait 
    -device isa-serial,chardev=serial_id_serial1 
    -chardev socket,id=seabioslog_id_20121112-165032-sRN82KnG,path=/tmp/seabios-20121112-165032-sRN82KnG,server,nowait 
    -device isa-debugcon,chardev=seabioslog_id_20121112-165032-sRN82KnG,iobase=0x402 
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 
    -drive file='RHEL-Server-5.9-32-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native 
    -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 
    -device virtio-net-pci,netdev=idqmfI4N,mac=9a:49:4a:4b:4c:4d,bus=pci.0,addr=0x3,id='idUsHWTI' 
    -netdev tap,id=idqmfI4N,vhost=on,fd=27 
    -m 2048 
    -smp 2,cores=1,threads=1,sockets=2 
    -cpu 'Penryn' 
    -M rhel6.4.0 
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 
    -vnc :0  
    -vga cirrus 
    -rtc base=utc,clock=host,driftfix=slew  
    -boot order=cdn,once=c,menu=off   
    -no-kvm-pit-reinjection 
    -enable-kvm
Comment 5 Igor Mammedov 2013-05-10 14:02:06 UTC 
(In reply to comment #0)
> Created attachment 646216 [details]
> guest serial log
> 
> Description of problem:
> RHEL5.9 guest generates some calltrace after setting one vcpu offline/online
> (1 of 2 vcpus).
> 
> Version-Release number of selected component (if applicable):
> Host version:
> $ /bin/grep -E "^kernel|qemu|bios" installed_packages.txt 
> gpxe-roms-qemu-0.9.7-6.9.el6.noarch
> kernel-2.6.32-330.el6.x86_64
> kernel-2.6.32-338.el6.x86_64
> kernel-devel-2.6.32-330.el6.x86_64
> kernel-firmware-2.6.32-338.el6.noarch
> kernel-headers-2.6.32-330.el6.x86_64
> qemu-guest-agent-0.12.1.2-2.334.el6.x86_64
> qemu-img-0.12.1.2-2.334.el6.x86_64
> qemu-kvm-0.12.1.2-2.334.el6.x86_64
> qemu-kvm-debuginfo-0.12.1.2-2.334.el6.x86_64
> qemu-kvm-tools-0.12.1.2-2.334.el6.x86_64
> seabios-0.6.1.2-25.el6.x86_64
> sgabios-0-0.3.20110621svn.el6.x86_64
> sgabios-bin-0-0.3.20110621svn.el6.noarch
> vgabios-0.6b-3.6.el6.noarch
> 
> Guest version:
> Kernel 2.6.18-345.el5
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1) Log into a guest with kernel cli divider=10.
> 2) Take a time reading from the guest and host.
> 3) Set cpu offline in vm.
> 4) Take the time from the guest and host as given frequency.
> 5) Set cpu online, which was set offline before
> 5) Take the time from the guest and host as given frequency.
> 6) Check if the drift (in seconds) is higher than a user specified value.
>   

I've tried to reproduce it locally on intel system without success so far.
Could you reproduce call trace with released 5.9 kernel with ignore_loglevel parameter on guest kernel cli? If yes, please provide access to system where it reproduces.
Comment 6 langfang 2013-08-22 07:44:12 UTC 
Seem hit the same problem on rhel6.4.z host

Version:
Host:
# uname -r
2.6.32-414.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.397.el6.x86_64

Guest:RHEL6.4.z-32

 kernel-2.6.32-358.19.1.el6.i686

steps:
1.Boot guest
# /usr/libexec/qemu-kvm -name rhel6.4.z -cpu Opteron_G4 -M rhel6.5.0 -m 4096 -realtime mlock=off -smp 4,maxcpus=7 -uuid `uuidgen` -nodefconfig -nodefaults  -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/root/rhel6.4.z-newinstall.raw,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,id=hostnet0,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:9e:3e:ea,bus=pci.0,addr=0x3 -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -drive file=/home/RHEL6.4-20130130.0-Server-i386-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -boot menu=on -vnc :10 -vga std -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -monitor unix:/tmp/monitor3,server,nowait -monitor stdio

2.Hotplug/Hotunplug cpu
QEMU 0.12.1 monitor - type 'help' for more information
(qemu) info cpus
* CPU #0: pc=0x00000000c0434f42 (halted) thread_id=16530 
  CPU #1: pc=0x00000000c0434f42 (halted) thread_id=16531 
  CPU #2: pc=0x00000000c0434f42 (halted) thread_id=16532 
  CPU #3: pc=0x00000000c0434f42 (halted) thread_id=16533 
(qemu) cpu_set 4 online
(qemu) info cpus
* CPU #0: pc=0x00000000c0434f42 (halted) thread_id=16530 
  CPU #1: pc=0x00000000c0434f42 (halted) thread_id=16531 
  CPU #2: pc=0x00000000c0434f42 (halted) thread_id=16532 
  CPU #3: pc=0x00000000c0434f42 (halted) thread_id=16533 
  CPU #4: pc=0x00000000c0434f42 (halted) thread_id=16595 
(qemu) cpu_set 4 offline
(qemu) info cpus
* CPU #0: pc=0x00000000c0434f42 (halted) thread_id=16530 
  CPU #1: pc=0x00000000c0434f42 (halted) thread_id=16531 
  CPU #2: pc=0x00000000c0434f42 (halted) thread_id=16532 
  CPU #3: pc=0x00000000c0434f42 (halted) thread_id=16533 
  CPU #4: pc=0x00000000c0434f42 (halted) thread_id=16595 

3.Check "dmesg" in guest

Results:Give the info  CPU Hot Add not support.But guest should not call trace
...

SELinux: initialized (dev rpc_pipefs, type rpc_pipefs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
SELinux: initialized (dev autofs, type autofs), uses genfs_contexts
UNSUPPORTED HARDWARE DEVICE: CPU Hot Add is not supported for x86 32-bit.

------------[ cut here ]------------
WARNING: at kernel/rh_taint.c:13 mark_hardware_unsupported+0x37/0x40() (serial8250: too much work for irq4
Not tainted)
Hardware name: KVM
Your hardware is unsupported.  Please do not report bugs, panics, oopses, etc., on this hardware.
Modules linked in: autofs4 sunrpc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode virtio_console virtio_balloon virtio_net sg i2c_piix4 i2c_core ext4 jbd2 mbcache virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mperf]
Pid: 39, comm: kacpi_notify Not tainted 2.6.32-358.19.1.el6.i686 #1
Call Trace:
 [<c04585d1>] ? warn_slowpath_common+0x81/0xc0
 [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
 [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
 [<c065b07d>] ? acpi_os_execute_deferred+0x0/0x23
 [<c0458663>] ? warn_slowpath_fmt_taint+0x33/0x40
 [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
 [<c0834d12>] ? acpi_processor_hotplug_notify+0x1b/0xb9
 [<c065ccee>] ? acpi_bus_notify+0x35/0x63
 [<c0667c13>] ? acpi_ev_notify_dispatch+0x4c/0x55
 [<c065b097>] ? acpi_os_execute_deferred+0x1a/0x23
 [<c04759fb>] ? worker_thread+0x11b/0x230
 [<c047a890>] ? autoremove_wake_function+0x0/0x40
 [<c04758e0>] ? worker_thread+0x0/0x230
 [<c047a654>] ? kthread+0x74/0x80
 [<c047a5e0>] ? kthread+0x0/0x80
 [<c0409f1f>] ? kernel_thread_helper+0x7/0x10
---[ end trace 37b79ca82c124aa6 ]---
Booting Node 0 Processor 4 APIC 0x4
Initializing CPU#4
kvm-clock: cpu 4, msr 0:bf086e1, secondary cpu clock
kvm-stealtime: cpu 4, msr bf03800
microcode: CPU4: patch_level=0x1000065
microcode: CPU4: update failed (for patch_level=0x6000629)
UNSUPPORTED HARDWARE DEVICE: CPU Hot Add is not supported for x86 32-bit.

------------[ cut here ]------------
WARNING: at kernel/rh_taint.c:13 mark_hardware_unsupported+0x37/0x40() (Tainted: G           --------------- H )
Hardware name: KVM
Your hardware is unsupported.  Please do not report bugs, panics, oopses, etc., on this hardware.
Modules linked in: autofs4 sunrpc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode virtio_console virtio_balloon virtio_net sg i2c_piix4 i2c_core ext4 jbd2 mbcache virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mperf]
Pid: 39, comm: kacpi_notify Tainted: G           --------------- H  2.6.32-358.19.1.el6.i686 #1
Call Trace:
 [<c04585d1>] ? warn_slowpath_common+0x81/0xc0
 [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
 [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
 [<c065b07d>] ? acpi_os_execute_deferred+0x0/0x23
 [<c0458663>] ? warn_slowpath_fmt_taint+0x33/0x40
 [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
 [<c0834d12>] ? acpi_processor_hotplug_notify+0x1b/0xb9
 [<c065ccee>] ? acpi_bus_notify+0x35/0x63
 [<c0667c13>] ? acpi_ev_notify_dispatch+0x4c/0x55
 [<c065b097>] ? acpi_os_execute_deferred+0x1a/0x23
 [<c04759fb>] ? worker_thread+0x11b/0x230
 [<c047a890>] ? autoremove_wake_function+0x0/0x40
 [<c04758e0>] ? worker_thread+0x0/0x230
 [<c047a654>] ? kthread+0x74/0x80
 [<c047a5e0>] ? kthread+0x0/0x80
 [<c0409f1f>] ? kernel_thread_helper+0x7/0x10
---[ end trace 37b79ca82c124aa7 ]---


Addtional info:
1)After step3,try shutdown  guest ,maybe will hit cannot shutdown guest normally,and hang in the end.(i only hit once)
...
In guest:
#shutdown -h now
..
(qemu) info status--->cannot shutdown guest normally
VM status: running
(qemu) system_reset
(qemu) KVM internal error. Suberror: 1
rax 000000000000f736 rbx 000000000000c000 rcx 000000000000aa00 rdx 0000000000606d12
rsi 0000000000000018 rdi 000000000000001a rsp 000000000000aff1 rbp 00000000000000cb
r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11 0000000000000000
r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15 0000000000000000
rip 000000000000e2c3 rflags 00000086
cs aff7 (000aff70/0000ffff p 1 dpl 0 db 0 s 1 type b l 0 g 0 avl 0)
ds 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
es 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
ss 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 2 l 0 g 0 avl 0)
fs 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
gs 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
tr 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type 3 l 0 g 0 avl 0)
ldt 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
gdt 0/ffff
idt 0/ffff
cr0 10 cr2 0 cr3 0 cr4 0 cr8 0 efer 0
emulation failure, check dmesg for details

(qemu) 
(qemu) 
(qemu) info status
VM status: paused (internal-error)
Comment 7 langfang 2013-08-22 07:51:46 UTC 
Correct comment 6.

Seem hit the same problem on rhel6.4.z **guest**

Version:
Host:
# uname -r
2.6.32-414.el6.x86_64
# rpm -q qemu-kvm
qemu-kvm-0.12.1.2-2.397.el6.x86_64

Guest:RHEL6.4.z-32

 kernel-2.6.32-358.19.1.el6.i686

...

Detail info please see comment6.
Comment 8 Andrew Jones 2013-08-22 09:50:37 UTC 
This bug is messy because it was originally reported for a hung task backtrace in a rhel5 guest, but is now addressing a different backtrace in a rhel6 guest. Please keep in mind that even if the host is the same that many times each guest kernel needs its own BZ, because the fix will be in the guest kernel, not the host kernel.

For the rhel5 guest:
getting these backtraces after a cpu has been offlined is likely not a regression, and, while not ideal, likely not going to be fixed.

For the rhel6 guest:
these backtraces are a due to bug 600435. We have to decide if 32-bit cpu hot-plug is supported for rhel6 kvm guests or not. If so, then the implementation for bug 600435 needs to be "fixed" in order to detect running over kvm, and only taint the kernel when not.
Comment 9 Igor Mammedov 2013-08-28 15:21:10 UTC 
(In reply to langfang from comment #6)
> Seem hit the same problem on rhel6.4.z host
> 
> Version:
> Host:
> # uname -r
> 2.6.32-414.el6.x86_64
> # rpm -q qemu-kvm
> qemu-kvm-0.12.1.2-2.397.el6.x86_64
> 
> Guest:RHEL6.4.z-32
> 
>  kernel-2.6.32-358.19.1.el6.i686
[...]

> UNSUPPORTED HARDWARE DEVICE: CPU Hot Add is not supported for x86 32-bit.
> 
> ------------[ cut here ]------------
> WARNING: at kernel/rh_taint.c:13 mark_hardware_unsupported+0x37/0x40()
> (serial8250: too much work for irq4
> Not tainted)
> Hardware name: KVM
> Your hardware is unsupported.  Please do not report bugs, panics, oopses,
> etc., on this hardware.
> Modules linked in: autofs4 sunrpc ipt_REJECT nf_conntrack_ipv4
> nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6
> nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6
> microcode virtio_console virtio_balloon virtio_net sg i2c_piix4 i2c_core
> ext4 jbd2 mbcache virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio
> pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last
> unloaded: mperf]
> Pid: 39, comm: kacpi_notify Not tainted 2.6.32-358.19.1.el6.i686 #1
> Call Trace:
>  [<c04585d1>] ? warn_slowpath_common+0x81/0xc0
>  [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
>  [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
>  [<c065b07d>] ? acpi_os_execute_deferred+0x0/0x23
>  [<c0458663>] ? warn_slowpath_fmt_taint+0x33/0x40
>  [<c0482587>] ? mark_hardware_unsupported+0x37/0x40
>  [<c0834d12>] ? acpi_processor_hotplug_notify+0x1b/0xb9
>  [<c065ccee>] ? acpi_bus_notify+0x35/0x63
>  [<c0667c13>] ? acpi_ev_notify_dispatch+0x4c/0x55
>  [<c065b097>] ? acpi_os_execute_deferred+0x1a/0x23
>  [<c04759fb>] ? worker_thread+0x11b/0x230
>  [<c047a890>] ? autoremove_wake_function+0x0/0x40
>  [<c04758e0>] ? worker_thread+0x0/0x230
>  [<c047a654>] ? kthread+0x74/0x80
>  [<c047a5e0>] ? kthread+0x0/0x80
>  [<c0409f1f>] ? kernel_thread_helper+0x7/0x10
> ---[ end trace 37b79ca82c124aa6 ]---
> Booting Node 0 Processor 4 APIC 0x4
> Initializing CPU#4
> kvm-clock: cpu 4, msr 0:bf086e1, secondary cpu clock
> kvm-stealtime: cpu 4, msr bf03800
> microcode: CPU4: patch_level=0x1000065
> microcode: CPU4: update failed (for patch_level=0x6000629)
> UNSUPPORTED HARDWARE DEVICE: CPU Hot Add is not supported for x86 32-bit.
> 
> ------------[ cut here ]------------
> WARNING: at kernel/rh_taint.c:13 mark_hardware_unsupported+0x37/0x40()
> (Tainted: G           --------------- H )
> Hardware name: KVM
> Your hardware is unsupported.  Please do not report bugs, panics, oopses,
> etc., on this hardware.

that statement clearly says not to report bugs for unsupported case. i.e. (CPU hotplug on x32). If x32 CPU hot-plug is needed please reopen bug 600435, provided there is strong enough business justification to back it up.

Following back-trace is for developer convenience only, I doubt it will be ever removed.

[...]


 
> Addtional info:
> 1)After step3,try shutdown  guest ,maybe will hit cannot shutdown guest
> normally,and hang in the end.(i only hit once)
> ...
> In guest:
> #shutdown -h now
> ..
> (qemu) info status--->cannot shutdown guest normally
> VM status: running
> (qemu) system_reset
> (qemu) KVM internal error. Suberror: 1
> rax 000000000000f736 rbx 000000000000c000 rcx 000000000000aa00 rdx
> 0000000000606d12
> rsi 0000000000000018 rdi 000000000000001a rsp 000000000000aff1 rbp
> 00000000000000cb
> r8  0000000000000000 r9  0000000000000000 r10 0000000000000000 r11
> 0000000000000000
> r12 0000000000000000 r13 0000000000000000 r14 0000000000000000 r15
> 0000000000000000
> rip 000000000000e2c3 rflags 00000086
> cs aff7 (000aff70/0000ffff p 1 dpl 0 db 0 s 1 type b l 0 g 0 avl 0)
> ds 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
> es 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
> ss 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 2 l 0 g 0 avl 0)
> fs 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
> gs 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 1 type 3 l 0 g 0 avl 0)
> tr 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type 3 l 0 g 0 avl 0)
> ldt 0000 (00000000/0000ffff p 1 dpl 0 db 0 s 0 type 2 l 0 g 0 avl 0)
> gdt 0/ffff
> idt 0/ffff
> cr0 10 cr2 0 cr3 0 cr4 0 cr8 0 efer 0
> emulation failure, check dmesg for details
> 
> (qemu) 
> (qemu) 
> (qemu) info status
> VM status: paused (internal-error)
CPU hot-add in RHEL6 is tech-preview (i.e. unsupported), and unplug isn't even declared as existing at all.

However if you will be able to reproduce shutdown issue for x64 guest, please report it as separate bug, it might be interesting/worth to look at it from the guest side.
Comment 10 Igor Mammedov 2013-08-28 15:41:10 UTC 
I've looked in guest log attached to comment 1. It's 32-bit guest as well and RHEL5 i386 CPU online/offline code is in much worse state than REHL6's (i.e. basically un-maintained). Bringing it to sync with x64 part of it is pretty big and risky effort, not worth implementing this late in RHEL5 life cycle.