Bug 606589

Summary: 32 bit guest kernel panic when boot up in RHEL 5.6 with multi storage
Product: Red Hat Enterprise Linux 6 Reporter: Joy Pu <ypu>
Component: kernelAssignee: chellwig <chellwig>
Status: CLOSED WORKSFORME QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.1CC: chellwig, ddutile, mkenneth, tburke, virt-maint, ykaul
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-11 11:09:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 580948    
Attachments:
Description Flags
The whole serial log
none
debug hack none

Description Joy Pu 2010-06-22 02:34:22 UTC 
Description:
In the RHEL5.6 host, 32 bit guest kernel panic while boot up with multi storage. This is more easier to reproduce in Intel machine.

Version-Release number of selected component (if applicable):
host kernel:2.6.18-203.el5.x86_64 
guest kernel: 2.6.32-36.el6.i686
#rpm -qa | grep qemu
kvm-qemu-img-83-183.el5
gpxe-roms-qemu-0.9.7-8.el5

How reproducible:
twice of fifty

Steps to Reproduce:
1. boot up a RHEL-6.0 guest with multi storage
2. listen to serial by nc
# /tmp/serial-20100621-185331-rkJ3

Actual results:
guest kernel panic when boot up

Expected results:
guest can boot up successfully

Additional info:
1. The command line:
# /usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/qemu -name 'vm1' -monitor tcp:0:6001,server,nowait -drive file=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/images/RHEL-Server-6.0-32-virtio.qcow2,if=virtio,cache=none,boot=on -drive file=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/images/storage_4k.qcow2,if=virtio,cache=none -drive file=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/images/storage_64k.qcow2,if=virtio,cache=none -net nic,vlan=0,model=virtio,macaddr=02:70:05:8F:dc:27 -net tap,vlan=0,ifname=virtio_0_6001,script=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 1024 -smp 1 -soundhw ac97 -redir tcp:5000::22 -vnc :0 -spice port=8000,disable-ticketing -usbdevice tablet -rtc-td-hack -cpu qemu64,+sse2 -no-kvm-pit-reinjection -serial unix:/tmp/serial-20100621-185331-rkJ3,server,nowait -no-hpet

2.Host cpuinfo
processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3520  @ 2.67GHz
stepping	: 5
cpu MHz		: 1596.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 5319.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management: [8]


3.Kernel panic info
  vda: vda1 vda2
 vdb: unknown partition table
 vdc: unknown partition table
------------[ cut here ]------------
WARNING: at fs/sysfs/dir.c:491 sysfs_add_one+0xb1/0x110() (Not tainted)
Hardware name: KVM
sysfs: cannot create duplicate filename '/bus/virtio/drivers/virtio_blk/virtio3'
Modules linked in: virtio_blk(U) virtio_net(U) virtio_pci(+)(U) virtio_ring(U) virtio(U) ata_piix(U) dm_mod(U)
Pid: 140, comm: modprobe Not tainted 2.6.32-36.el6.i686 #1
Call Trace:
 [<c044f917>] ? warn_slowpath_common+0x77/0xb0
 [<c0576841>] ? sysfs_add_one+0xb1/0x110
 [<c044f99b>] ? warn_slowpath_fmt+0x2b/0x30
 [<c0576841>] ? sysfs_add_one+0xb1/0x110
 [<c0575b0f>] ? sysfs_addrm_start+0x3f/0xb0
 [<c0576c29>] ? sysfs_do_create_link+0xe9/0x130
 [<c0576c9f>] ? sysfs_create_link+0xf/0x20
 [<c06a2c13>] ? driver_sysfs_add+0x23/0x70
 [<c06a3081>] ? device_bind_driver+0x11/0x30
 [<c06a30ce>] ? device_attach+0x2e/0x90
 [<c06a1c9d>] ? bus_probe_device+0x1d/0x40
 [<c06a0456>] ? device_add+0x4d6/0x5f0
 [<c040a335>] ? apic_timer_interrupt+0x31/0x38
 [<f7e683e5>] ? register_virtio_device+0x65/0x80 [virtio]
 [<f7e84df2>] ? virtio_pci_probe+0x147/0x182 [virtio_pci]
 [<c0576be7>] ? sysfs_do_create_link+0xa7/0x130
 [<c05f6b66>] ? pci_match_device+0x16/0xc0
 [<c05f6a8b>] ? local_pci_probe+0xb/0x10
 [<c05f7891>] ? pci_device_probe+0x61/0x80
 [<c06a2d97>] ? driver_probe_device+0x87/0x290
 [<c05f6b66>] ? pci_match_device+0x16/0xc0
 [<c06a3019>] ? __driver_attach+0x79/0x80
 [<c06a2fa0>] ? __driver_attach+0x0/0x80
 [<c06a2192>] ? bus_for_each_dev+0x52/0x80
 [<c06a2b86>] ? driver_attach+0x16/0x20
 [<c06a2fa0>] ? __driver_attach+0x0/0x80
 [<c06a253f>] ? bus_add_driver+0x1cf/0x320
 [<c05ded73>] ? kset_find_obj+0x33/0x60
 [<c05f77d0>] ? pci_device_remove+0x0/0x40
 [<c06a32a3>] ? driver_register+0x63/0x120
 [<c06a05f2>] ? __root_device_register+0x72/0xf0
 [<f7e88000>] ? virtio_pci_init+0x0/0x46 [virtio_pci]
 [<c05f7aad>] ? __pci_register_driver+0x3d/0xb0
 [<f7e88032>] ? virtio_pci_init+0x32/0x46 [virtio_pci]
 [<c040303f>] ? do_one_initcall+0x2f/0x1c0
 [<c048b4c4>] ? sys_init_module+0xb4/0x220
 [<c051bfc1>] ? sys_read+0x41/0x70
 [<c04098fb>] ? sysenter_do_call+0x12/0x28
---[ end trace 64f62824637b4e16 ]---
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
BUG: unable to handle kernel NULL pointer dereference at 0000003c
IP: [<f7e68094>] virtio_check_driver_offered_feature+0x4/0x30 [virtio]
*pdpt = 0000000035cba001 *pde = 000000003ece1067 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/virtio-pci/virtio3/block/vdc/removable
Modules linked in: virtio_blk(U) virtio_net(U) virtio_pci(+)(U) virtio_ring(U) virtio(U) ata_piix(U) dm_mod(U)

Pid: 219, comm: blkid Tainted: G        W  (2.6.32-36.el6.i686 #1) KVM
EIP: 0060:[<f7e68094>] EFLAGS: 00010286 CPU: 0
EIP is at virtio_check_driver_offered_feature+0x4/0x30 [virtio]
EAX: 00000000 EBX: f622be00 ECX: 00005331 EDX: 00000007
ESI: f622ee00 EDI: 00005331 EBP: 0000101d ESP: f5cdfe58
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process blkid (pid: 219, ti=f5cde000 task=f63ba560 task.ti=f5cde000)
Stack:
 f622be00 f7eae49f 00000000 00000000 f7eaeb00 f7eae470 00000000 f622be00
<0> c05cf0ad 00000000 00005331 0000101d f6710880 0000101d f6710880 00005331
<0> f5cdfee8 c05cf509 00000000 00110000 00000000 c04f73ad f5cba008 00000148
Call Trace:
 [<f7eae49f>] ? virtblk_ioctl+0x2f/0x70 [virtio_blk]
 [<f7eae470>] ? virtblk_ioctl+0x0/0x70 [virtio_blk]
 [<c05cf0ad>] ? __blkdev_driver_ioctl+0x7d/0x90
 [<c05cf509>] ? blkdev_ioctl+0x1b9/0x720
 [<c04f73ad>] ? handle_pte_fault+0xcd/0xc90
 [<c0547815>] ? block_ioctl+0x35/0x50
 [<c05477e0>] ? block_ioctl+0x0/0x50
 [<c052b8fb>] ? vfs_ioctl+0x1b/0xa0
 [<c052baec>] ? do_vfs_ioctl+0x6c/0x5d0
 [<c052c0c6>] ? sys_ioctl+0x76/0x90
 [<c05e6da3>] ? copy_to_user+0x33/0x110
 [<c04098fb>] ? sysenter_do_call+0x12/0x28
Code: 71 0c ff 51 08 09 f8 0f b6 d0 89 d8 ff d6 8b 1c 24 8b 74 24 04 8b 7c 24 08 83 c4 0c c3 8d 76 00 8d bc 27 00 00 00 00 53 8b 40 4c <8b> 48 3c 85 c9 74 19 8b 58 38 31 c0 39 13 75 09 eb 12 66 90 39 
EIP: [<f7e68094>] virtio_check_driver_offered_feature+0x4/0x30 [virtio] SS:ESP 0068:f5cdfe58
CR2: 000000000000003c
---[ end trace 64f62824637b4e17 ]---
Kernel panic - not syncing: Fatal exception
Pid: 219, comm: blkid Tainted: G      D W  2.6.32-36.el6.i686 #1
Call Trace:
 [<c081644f>] ? panic+0x42/0xed
 [<c0819b8c>] ? oops_end+0xbc/0xd0
 [<c04303f2>] ? no_context+0xc2/0x190
 [<c043067b>] ? bad_area+0x3b/0x50
 [<c081ae60>] ? do_page_fault+0x0/0x480
 [<c081b2d3>] ? do_page_fault+0x473/0x480
 [<c05953e4>] ? avc_has_perm+0x64/0x80
 [<c081ae60>] ? do_page_fault+0x0/0x480
 [<c0818f6b>] ? error_code+0x73/0x78
 [<c059007b>] ? cap_inode_mknod+0xb/0x10
 [<f7e68094>] ? virtio_check_driver_offered_feature+0x4/0x30 [virtio]
 [<f7eae49f>] ? virtblk_ioctl+0x2f/0x70 [virtio_blk]
 [<f7eae470>] ? virtblk_ioctl+0x0/0x70 [virtio_blk]
 [<c05cf0ad>] ? __blkdev_driver_ioctl+0x7d/0x90
 [<c05cf509>] ? blkdev_ioctl+0x1b9/0x720
 [<c04f73ad>] ? handle_pte_fault+0xcd/0xc90
 [<c0547815>] ? block_ioctl+0x35/0x50
 [<c05477e0>] ? block_ioctl+0x0/0x50
 [<c052b8fb>] ? vfs_ioctl+0x1b/0xa0
 [<c052baec>] ? do_vfs_ioctl+0x6c/0x5d0
 [<c052c0c6>] ? sys_ioctl+0x76/0x90
 [<c05e6da3>] ? copy_to_user+0x33/0x110
 [<c04098fb>] ? sysenter_do_call+0x12/0x28
Comment 1 Joy Pu 2010-06-22 02:35:40 UTC 
Created attachment 425800 [details]
The whole serial log
Comment 2 chellwig@redhat.com 2010-10-15 20:34:15 UTC 
What is multi storage?  Does it still happen with the latest rhel6 snapshot?
Comment 3 Joy Pu 2010-10-18 12:04:12 UTC 
(In reply to comment #2)
> What is multi storage?  Does it still happen with the latest rhel6 snapshot?

Multi storage means that the guest boot up with more than one harddisk device. In the report command line, I tried to boot up a guest with 3 storage. I only find this problem in RHEL 5 host. Don't meet it in the RHEL 6 host(Boot up the guest more than 50 times with multi-disk).
Comment 6 RHEL Program Management 2011-01-11 21:15:39 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.
Comment 7 RHEL Program Management 2011-01-11 22:52:19 UTC 
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 8 chellwig@redhat.com 2011-01-17 16:38:28 UTC 
Created attachment 473889 [details]
debug hack

The duplicate name and oops in the ioctl path make it look like this might be related to udev renaming things based on serial numbers.  Can you try botting with the attached debug patch?

Also does this only happen when booting with three virtio devices, or also with two?
Comment 9 chellwig@redhat.com 2011-01-17 16:40:08 UTC 
This is most likely a virtio-blk guest driver issue, updating component and release.
Comment 11 Dor Laor 2011-02-17 13:32:40 UTC 
QE, please test with #8 and with latest rhel6.1 candidate guest
Comment 12 Joy Pu 2011-03-02 09:55:59 UTC 
(In reply to comment #11)
> QE, please test with #8 and with latest rhel6.1 candidate guest

Test the rhel6.1 guest for more the 50 times, didn't trig this problem again.
kernel version: 2.6.32-118.el6.x86_64
Comment 13 Dor Laor 2011-03-02 11:08:12 UTC 
Can you close the bug?
Comment 14 Joy Pu 2011-03-11 11:09:46 UTC 
Base the comment 12, close this bug. Will reopen it when it is trigged again.