RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 606589 - 32 bit guest kernel panic when boot up in RHEL 5.6 with multi storage
Summary: 32 bit guest kernel panic when boot up in RHEL 5.6 with multi storage
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.1
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: chellwig@redhat.com
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: Rhel5KvmTier2
TreeView+ depends on / blocked
 
Reported: 2010-06-22 02:34 UTC by Joy Pu
Modified: 2013-01-09 22:46 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-03-11 11:09:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
The whole serial log (20.19 KB, application/octet-stream)
2010-06-22 02:35 UTC, Joy Pu
no flags Details
debug hack (595 bytes, patch)
2011-01-17 16:38 UTC, chellwig@redhat.com
no flags Details | Diff

Description Joy Pu 2010-06-22 02:34:22 UTC
Description:
In the RHEL5.6 host, 32 bit guest kernel panic while boot up with multi storage. This is more easier to reproduce in Intel machine.

Version-Release number of selected component (if applicable):
host kernel:2.6.18-203.el5.x86_64 
guest kernel: 2.6.32-36.el6.i686
#rpm -qa | grep qemu
kvm-qemu-img-83-183.el5
gpxe-roms-qemu-0.9.7-8.el5

How reproducible:
twice of fifty

Steps to Reproduce:
1. boot up a RHEL-6.0 guest with multi storage
2. listen to serial by nc
# /tmp/serial-20100621-185331-rkJ3

Actual results:
guest kernel panic when boot up

Expected results:
guest can boot up successfully

Additional info:
1. The command line:
# /usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/qemu -name 'vm1' -monitor tcp:0:6001,server,nowait -drive file=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/images/RHEL-Server-6.0-32-virtio.qcow2,if=virtio,cache=none,boot=on -drive file=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/images/storage_4k.qcow2,if=virtio,cache=none -drive file=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/images/storage_64k.qcow2,if=virtio,cache=none -net nic,vlan=0,model=virtio,macaddr=02:70:05:8F:dc:27 -net tap,vlan=0,ifname=virtio_0_6001,script=/usr/local/staf/test/RHEV/kvm-new/autotest/client/tests/kvm/scripts/qemu-ifup-switch,downscript=no -m 1024 -smp 1 -soundhw ac97 -redir tcp:5000::22 -vnc :0 -spice port=8000,disable-ticketing -usbdevice tablet -rtc-td-hack -cpu qemu64,+sse2 -no-kvm-pit-reinjection -serial unix:/tmp/serial-20100621-185331-rkJ3,server,nowait -no-hpet

2.Host cpuinfo
processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3520  @ 2.67GHz
stepping	: 5
cpu MHz		: 1596.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 5319.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management: [8]


3.Kernel panic info
  vda: vda1 vda2
 vdb: unknown partition table
 vdc: unknown partition table
------------[ cut here ]------------
WARNING: at fs/sysfs/dir.c:491 sysfs_add_one+0xb1/0x110() (Not tainted)
Hardware name: KVM
sysfs: cannot create duplicate filename '/bus/virtio/drivers/virtio_blk/virtio3'
Modules linked in: virtio_blk(U) virtio_net(U) virtio_pci(+)(U) virtio_ring(U) virtio(U) ata_piix(U) dm_mod(U)
Pid: 140, comm: modprobe Not tainted 2.6.32-36.el6.i686 #1
Call Trace:
 [<c044f917>] ? warn_slowpath_common+0x77/0xb0
 [<c0576841>] ? sysfs_add_one+0xb1/0x110
 [<c044f99b>] ? warn_slowpath_fmt+0x2b/0x30
 [<c0576841>] ? sysfs_add_one+0xb1/0x110
 [<c0575b0f>] ? sysfs_addrm_start+0x3f/0xb0
 [<c0576c29>] ? sysfs_do_create_link+0xe9/0x130
 [<c0576c9f>] ? sysfs_create_link+0xf/0x20
 [<c06a2c13>] ? driver_sysfs_add+0x23/0x70
 [<c06a3081>] ? device_bind_driver+0x11/0x30
 [<c06a30ce>] ? device_attach+0x2e/0x90
 [<c06a1c9d>] ? bus_probe_device+0x1d/0x40
 [<c06a0456>] ? device_add+0x4d6/0x5f0
 [<c040a335>] ? apic_timer_interrupt+0x31/0x38
 [<f7e683e5>] ? register_virtio_device+0x65/0x80 [virtio]
 [<f7e84df2>] ? virtio_pci_probe+0x147/0x182 [virtio_pci]
 [<c0576be7>] ? sysfs_do_create_link+0xa7/0x130
 [<c05f6b66>] ? pci_match_device+0x16/0xc0
 [<c05f6a8b>] ? local_pci_probe+0xb/0x10
 [<c05f7891>] ? pci_device_probe+0x61/0x80
 [<c06a2d97>] ? driver_probe_device+0x87/0x290
 [<c05f6b66>] ? pci_match_device+0x16/0xc0
 [<c06a3019>] ? __driver_attach+0x79/0x80
 [<c06a2fa0>] ? __driver_attach+0x0/0x80
 [<c06a2192>] ? bus_for_each_dev+0x52/0x80
 [<c06a2b86>] ? driver_attach+0x16/0x20
 [<c06a2fa0>] ? __driver_attach+0x0/0x80
 [<c06a253f>] ? bus_add_driver+0x1cf/0x320
 [<c05ded73>] ? kset_find_obj+0x33/0x60
 [<c05f77d0>] ? pci_device_remove+0x0/0x40
 [<c06a32a3>] ? driver_register+0x63/0x120
 [<c06a05f2>] ? __root_device_register+0x72/0xf0
 [<f7e88000>] ? virtio_pci_init+0x0/0x46 [virtio_pci]
 [<c05f7aad>] ? __pci_register_driver+0x3d/0xb0
 [<f7e88032>] ? virtio_pci_init+0x32/0x46 [virtio_pci]
 [<c040303f>] ? do_one_initcall+0x2f/0x1c0
 [<c048b4c4>] ? sys_init_module+0xb4/0x220
 [<c051bfc1>] ? sys_read+0x41/0x70
 [<c04098fb>] ? sysenter_do_call+0x12/0x28
---[ end trace 64f62824637b4e16 ]---
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
BUG: unable to handle kernel NULL pointer dereference at 0000003c
IP: [<f7e68094>] virtio_check_driver_offered_feature+0x4/0x30 [virtio]
*pdpt = 0000000035cba001 *pde = 000000003ece1067 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/virtio-pci/virtio3/block/vdc/removable
Modules linked in: virtio_blk(U) virtio_net(U) virtio_pci(+)(U) virtio_ring(U) virtio(U) ata_piix(U) dm_mod(U)

Pid: 219, comm: blkid Tainted: G        W  (2.6.32-36.el6.i686 #1) KVM
EIP: 0060:[<f7e68094>] EFLAGS: 00010286 CPU: 0
EIP is at virtio_check_driver_offered_feature+0x4/0x30 [virtio]
EAX: 00000000 EBX: f622be00 ECX: 00005331 EDX: 00000007
ESI: f622ee00 EDI: 00005331 EBP: 0000101d ESP: f5cdfe58
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process blkid (pid: 219, ti=f5cde000 task=f63ba560 task.ti=f5cde000)
Stack:
 f622be00 f7eae49f 00000000 00000000 f7eaeb00 f7eae470 00000000 f622be00
<0> c05cf0ad 00000000 00005331 0000101d f6710880 0000101d f6710880 00005331
<0> f5cdfee8 c05cf509 00000000 00110000 00000000 c04f73ad f5cba008 00000148
Call Trace:
 [<f7eae49f>] ? virtblk_ioctl+0x2f/0x70 [virtio_blk]
 [<f7eae470>] ? virtblk_ioctl+0x0/0x70 [virtio_blk]
 [<c05cf0ad>] ? __blkdev_driver_ioctl+0x7d/0x90
 [<c05cf509>] ? blkdev_ioctl+0x1b9/0x720
 [<c04f73ad>] ? handle_pte_fault+0xcd/0xc90
 [<c0547815>] ? block_ioctl+0x35/0x50
 [<c05477e0>] ? block_ioctl+0x0/0x50
 [<c052b8fb>] ? vfs_ioctl+0x1b/0xa0
 [<c052baec>] ? do_vfs_ioctl+0x6c/0x5d0
 [<c052c0c6>] ? sys_ioctl+0x76/0x90
 [<c05e6da3>] ? copy_to_user+0x33/0x110
 [<c04098fb>] ? sysenter_do_call+0x12/0x28
Code: 71 0c ff 51 08 09 f8 0f b6 d0 89 d8 ff d6 8b 1c 24 8b 74 24 04 8b 7c 24 08 83 c4 0c c3 8d 76 00 8d bc 27 00 00 00 00 53 8b 40 4c <8b> 48 3c 85 c9 74 19 8b 58 38 31 c0 39 13 75 09 eb 12 66 90 39 
EIP: [<f7e68094>] virtio_check_driver_offered_feature+0x4/0x30 [virtio] SS:ESP 0068:f5cdfe58
CR2: 000000000000003c
---[ end trace 64f62824637b4e17 ]---
Kernel panic - not syncing: Fatal exception
Pid: 219, comm: blkid Tainted: G      D W  2.6.32-36.el6.i686 #1
Call Trace:
 [<c081644f>] ? panic+0x42/0xed
 [<c0819b8c>] ? oops_end+0xbc/0xd0
 [<c04303f2>] ? no_context+0xc2/0x190
 [<c043067b>] ? bad_area+0x3b/0x50
 [<c081ae60>] ? do_page_fault+0x0/0x480
 [<c081b2d3>] ? do_page_fault+0x473/0x480
 [<c05953e4>] ? avc_has_perm+0x64/0x80
 [<c081ae60>] ? do_page_fault+0x0/0x480
 [<c0818f6b>] ? error_code+0x73/0x78
 [<c059007b>] ? cap_inode_mknod+0xb/0x10
 [<f7e68094>] ? virtio_check_driver_offered_feature+0x4/0x30 [virtio]
 [<f7eae49f>] ? virtblk_ioctl+0x2f/0x70 [virtio_blk]
 [<f7eae470>] ? virtblk_ioctl+0x0/0x70 [virtio_blk]
 [<c05cf0ad>] ? __blkdev_driver_ioctl+0x7d/0x90
 [<c05cf509>] ? blkdev_ioctl+0x1b9/0x720
 [<c04f73ad>] ? handle_pte_fault+0xcd/0xc90
 [<c0547815>] ? block_ioctl+0x35/0x50
 [<c05477e0>] ? block_ioctl+0x0/0x50
 [<c052b8fb>] ? vfs_ioctl+0x1b/0xa0
 [<c052baec>] ? do_vfs_ioctl+0x6c/0x5d0
 [<c052c0c6>] ? sys_ioctl+0x76/0x90
 [<c05e6da3>] ? copy_to_user+0x33/0x110
 [<c04098fb>] ? sysenter_do_call+0x12/0x28

Comment 1 Joy Pu 2010-06-22 02:35:40 UTC
Created attachment 425800 [details]
The whole serial log

Comment 2 chellwig@redhat.com 2010-10-15 20:34:15 UTC
What is multi storage?  Does it still happen with the latest rhel6 snapshot?

Comment 3 Joy Pu 2010-10-18 12:04:12 UTC
(In reply to comment #2)
> What is multi storage?  Does it still happen with the latest rhel6 snapshot?

Multi storage means that the guest boot up with more than one harddisk device. In the report command line, I tried to boot up a guest with 3 storage. I only find this problem in RHEL 5 host. Don't meet it in the RHEL 6 host(Boot up the guest more than 50 times with multi-disk).

Comment 6 RHEL Program Management 2011-01-11 21:15:39 UTC
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated in the
current release, Red Hat is unfortunately unable to address this
request at this time. Red Hat invites you to ask your support
representative to propose this request, if appropriate and relevant,
in the next release of Red Hat Enterprise Linux.

Comment 7 RHEL Program Management 2011-01-11 22:52:19 UTC
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.

Comment 8 chellwig@redhat.com 2011-01-17 16:38:28 UTC
Created attachment 473889 [details]
debug hack

The duplicate name and oops in the ioctl path make it look like this might be related to udev renaming things based on serial numbers.  Can you try botting with the attached debug patch?

Also does this only happen when booting with three virtio devices, or also with two?

Comment 9 chellwig@redhat.com 2011-01-17 16:40:08 UTC
This is most likely a virtio-blk guest driver issue, updating component and release.

Comment 11 Dor Laor 2011-02-17 13:32:40 UTC
QE, please test with #8 and with latest rhel6.1 candidate guest

Comment 12 Joy Pu 2011-03-02 09:55:59 UTC
(In reply to comment #11)
> QE, please test with #8 and with latest rhel6.1 candidate guest

Test the rhel6.1 guest for more the 50 times, didn't trig this problem again.
kernel version: 2.6.32-118.el6.x86_64

Comment 13 Dor Laor 2011-03-02 11:08:12 UTC
Can you close the bug?

Comment 14 Joy Pu 2011-03-11 11:09:46 UTC
Base the comment 12, close this bug. Will reopen it when it is trigged again.


Note You need to log in before you can comment on or make changes to this bug.