Bug 901751

Summary: [abrt]: BUG: soft lockup - CPU#6 stuck for 22s! [qemu-kvm:872]
Product: [Fedora] Fedora Reporter: Lubo <lubomir.carik>
Component: kernelAssignee: Marcelo Tosatti <mtosatti>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: gansalmon, itamar, jforbes, jonathan, kernel-maint, madhu.chinakonda, sales
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:d4f4efaac00f8b4aa61c5b7a7d4083b69f9dff5a
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-08 21:45:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Image capture of a kernel panic that started wit CPU soft lockup none

Description Lubo 2013-01-19 00:14:52 UTC 
Additional info:
BUG: soft lockup - CPU#6 stuck for 22s! [qemu-kvm:872]
Modules linked in: bluetooth cpufreq_stats fuse ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle bridge stp llc ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i cxgb3 mdio libcxgbi ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi snd_hda_codec_hdmi snd_hda_codec_realtek uvcvideo videobuf2_vmalloc videobuf2_memops snd_hda_intel snd_hda_codec arc4 iwldvm videobuf2_core mac80211 videodev media snd_hwdep snd_seq coretemp iwlwifi snd_seq_device snd_pcm r8169 cfg80211 iTCO_wdt iTCO_vendor_support lpc_ich snd_page_alloc mii dell_wmi snd_timer mfd_core microcode dell_laptop dcdbas i2c_i801 mei sparse_keymap snd rfkill soundcore joydev vhost_net tun macvtap macvlan nfsd auth_rpcgss kvm_intel nfs_acl kvm lockd sunrpc binfmt_misc uinput i915 nouveau mxm_wmi i2c_algo_bit drm_kms_helper ttm crc32c_intel drm ghash_clmulni_intel i2c_core wmi video
CPU 6 
Pid: 872, comm: qemu-kvm Not tainted 3.7.2-201.fc18.x86_64 #1 Dell Inc.          Dell System XPS L702X/0XN71K
RIP: 0010:[<ffffffffa0312979>]  [<ffffffffa0312979>] kvm_arch_vcpu_ioctl_run+0x3d9/0x1010 [kvm]
RSP: 0018:ffff8801f38fdd58  EFLAGS: 00000203
RAX: 00000893136d0d0d RBX: 000000000000000d RCX: ffff88019f94bdd0
RDX: 0000000000002010 RSI: 0000000000000000 RDI: ffff88019f94bdd0
RBP: ffff8801f38fddf8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa02b3ac3
R13: ffff8801f38fdcd8 R14: ffffffffa02afb4c R15: 0000000000000000
FS:  00007f642efe1700(0000) GS:ffff88023e780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000001ad2b5000 CR4: 00000000000427e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu-kvm (pid: 872, threadinfo ffff8801f38fc000, task ffff8802308d2e40)
Stack:
 0000000000000000 0000000000000000 ffff8801f38fdd98 ffffffffa02b08d5
 ffff8801f38fddb8 ffffffffa02b273e ffff8801f38fdfd8 ffff8801f38fddf0
 ffff8802308d2e40 ffff8802308d2e40 ffff8801c1585100 fffffffe7ffbfeff
Call Trace:
 [<ffffffffa02b08d5>] ? __vmx_load_host_state+0x105/0x150 [kvm_intel]
 [<ffffffffa02b273e>] ? vmx_vcpu_load+0x3e/0x1a0 [kvm_intel]
 [<ffffffffa030dc57>] ? kvm_arch_vcpu_load+0x57/0x1b0 [kvm]
 [<ffffffffa02fb202>] kvm_vcpu_ioctl+0x472/0x610 [kvm]
 [<ffffffff81195d3e>] ? do_readv_writev+0x18e/0x1e0
 [<ffffffff811a64e9>] do_vfs_ioctl+0x99/0x580
 [<ffffffff811a6a61>] sys_ioctl+0x91/0xb0
 [<ffffffff810dc7dc>] ? __audit_syscall_exit+0x3ec/0x450
 [<ffffffff8163dad9>] system_call_fastpath+0x16/0x1b
Code: 00 a8 aa 0f 85 31 08 00 00 48 8b 05 62 77 03 00 48 89 df ff 90 38 02 00 00 c7 43 2c 00 00 00 00 48 89 83 58 20 00 00 fb 66 66 90 <66> 66 90 83 83 b8 00 00 00 01 48 8b 7d a0 e8 14 3a d8 e0 48 8b
Comment 1 Alves 2013-01-31 14:03:39 UTC 
Created attachment 690915 [details]
Image capture of a kernel panic that started wit CPU soft lockup

It happens very often, to the point of making my KVM installation useless.
uname -r
2.6.32-279.19.1.el6.x86_64

I tried upgrading the kernel to 3.5, but the issue remains.

If this is not fixed soon, KVM is history.
Comment 2 Alves 2013-01-31 14:55:17 UTC 
In my case I think the issue is a Windows 8 virtual machine. The disk is IDE, and the network interface is Virtio. In 2.6.32-279.19.1.el6.x86_64, it does not even start, shows an error saying "Your CPU does not support .." some type of CPU feature, but it is incorrect, since this is a Dell R900 server. Also, while it shows this error, it eats up 2 cores 100%. Paradoxically, this virtual machines starts just fine in Kernel 3.5.0, but then we fall into this bug after some time.
If somebody analyzes my Windows 8 Virtual machine, you may find the problem. The kernel and kvm seem to be stable if I only use Linux virtual machines. I also noticed that if I change my Windows disk type to Virtio, it is several times slower to boot than when using IDE.
Comment 3 Josh Boyer 2013-04-01 15:56:51 UTC 
Alves, you should file a bug against RHEL (or CentOS if you're getting it from there) for the 3.6.32 kernel issues.

Lubo, are you still seeing this with 3.8.x?
Comment 4 Lubo 2013-04-08 21:18:26 UTC 
Hi Josh, in last weeks - no. May it was a problem only on that kernel/sw combination (?),