Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 886828

Summary: win2012-64 guest hang in boot up process with 224 usb storage
Product: Red Hat Enterprise Linux 6 Reporter: mazhang <mazhang>
Component: qemu-kvmAssignee: Gerd Hoffmann <kraxel>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 6.4CC: acathrow, areis, bsarathy, juzhang, mazhang, michen, mkenneth, qzhang, rhod, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-27 09:30:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
qemu crash (start win2012 with 224 usb storage) none

Description mazhang 2012-12-13 09:37:32 UTC
Description of problem:
Scalability test, try boot up win2012-64 guest with 224 usb storage, but guest hung in boot process.
tcms link: https://tcms.engineering.redhat.com/case/201033/?from_plan=6381\

Version-Release number of selected component (if applicable):
kernel:2.6.32-347.el6.x86_64
qemu-kvm:qemu-kvm-0.12.1.2-2.340.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1.use qemu-img create 224 images which format is qcow2
2.boot up win2012 with those usb storage with below command line:
#/usr/libexec/qemu-kvm -M rhel6.4.0 -monitor stdio -enable-kvm -m 16G -smp 8,sockets=2,cores=4,threads=1 -name rhel6 -uuid 745fe449-aac8-29f1-0c2d-5042a707263b -boot menu=on -drive file=/opt/windows/win2012_x.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none,aio=threads -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -spice disable-ticketing,port=5900 -vga qxl -k en-us -rtc base=localtime,clock=host,driftfix=slew -no-kvm-pit-reinjection -qmp tcp:0:6666,server,nowait -chardev socket,path=/tmp/isa-serial,server,nowait,id=isa1 -device isa-serial,chardev=isa1,id=isa-serial1 \
-device usb-ehci,id=ehci8,multifunction=on,addr=0x04.0  -device usb-storage,drive=usbstick8 -drive if=none,id=usbstick8,file=usb/u8 \
-device usb-ehci,id=ehci9,multifunction=on,addr=0x04.1  -device usb-storage,drive=usbstick9 -drive if=none,id=usbstick9,file=usb/u9 \
-device usb-ehci,id=ehci10,multifunction=on,addr=0x04.2  -device usb-storage,drive=usbstick10 -drive if=none,id=usbstick10,file=usb/u10 \
-device usb-ehci,id=ehci11,multifunction=on,addr=0x04.3  -device usb-storage,drive=usbstick11 -drive if=none,id=usbstick11,file=usb/u11 \
-device usb-ehci,id=ehci12,multifunction=on,addr=0x04.4  -device usb-storage,drive=usbstick12 -drive if=none,id=usbstick12,file=usb/u12 \
-device usb-ehci,id=ehci13,multifunction=on,addr=0x04.5  -device usb-storage,drive=usbstick13 -drive if=none,id=usbstick13,file=usb/u13 \
-device usb-ehci,id=ehci14,multifunction=on,addr=0x04.6  -device usb-storage,drive=usbstick14 -drive if=none,id=usbstick14,file=usb/u14 \
-device usb-ehci,id=ehci15,multifunction=on,addr=0x04.7  -device usb-storage,drive=usbstick15 -drive if=none,id=usbstick15,file=usb/u15 \
-device usb-ehci,id=ehci16,multifunction=on,addr=0x05.0  -device usb-storage,drive=usbstick16 -drive if=none,id=usbstick16,file=usb/u16 
...
...
-device usb-ehci,id=ehci230,multifunction=on,addr=0x1f.6  -device usb-storage,drive=usbstick230 -drive if=none,id=usbstick230,file=usb/u230 \
-device usb-ehci,id=ehci231,multifunction=on,addr=0x1f.7  -device usb-storage,drive=usbstick231 -drive if=none,id=usbstick231,file=usb/u231

3.
  
Actual results:
guest hung in boot up process

Expected results:
boot up ok

Additional info:

Comment 2 mazhang 2012-12-15 09:42:20 UTC
also boot up win2012-64 guest with 224 virtual nic by enable multifunction got this issue, assume it's pci device multifunction support not very well.

Comment 3 FuXiangChun 2013-03-07 11:34:19 UTC
For rhl6.4-64 guest. If boot rhel6.4 guest with 232 usb controller as comment1 command line. Then only 64 usb devices are found via 'fdisk -l' in guest.

Comment 5 Ronen Hod 2013-05-26 17:05:50 UTC
Looks like a generic scalability issue. May be related to MultiFunction, but not sure. There are issues with Win2012 USB storage, NICs, and Linux guests storage.
Probably not related to another scalability issue in bug 882050.
For Windows, worth trying with "hv_relaxed" flag, although I doubt.
Deferring to 6.6 for now.

Comment 6 Amos Kong 2013-05-27 08:20:09 UTC
In Linux kernel, the USB buses are limited to 64 when we register the USB host controller.

This why only 64 usb disks were identified in comment #3.


*** drivers/usb/core/hcd.c:
USB_MAXBUS[94]                 #define USB_MAXBUS 64

*** drivers/usb/core/hcd.c:
static int usb_register_bus(struct usb_bus *bus)
{
	int result = -E2BIG;
	int busnum;

	mutex_lock(&usb_bus_list_lock);
	busnum = find_next_zero_bit (busmap.busmap, USB_MAXBUS, 1);
	if (busnum >= USB_MAXBUS) {
		printk (KERN_ERR "%s: too many buses\n", usbcore_name);
		goto error_find_busnum;
	}

Comment 7 Amos Kong 2013-05-27 08:22:42 UTC
Maosheng, how about other Windows guests?

Comment 8 Amos Kong 2013-05-28 01:58:36 UTC
Created attachment 753701 [details]
qemu crash (start win2012 with 224 usb storage)

Started win2012 with 224 usb storage, it's very slow. passed more than 30 mins then crashed.
qemu-kvm-0.12.1.2-2.369.el6

#0  0x00007ffff5e008a5 in raise () from /lib64/libc.so.6
#1  0x00007ffff5e02085 in abort () from /lib64/libc.so.6
#2  0x00007ffff5df9a1e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff5df9ae0 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000004adb7e in usb_msd_handle_data (dev=0x14ba9c10, p=0x22973e88) at /home/devel/qemu-kvm-rhel6/hw/usb-msd.c:407
#5  0x00000000004a4557 in do_token_out (s=0x14ba9c10, p=0x22973e88) at /home/devel/qemu-kvm-rhel6/hw/usb.c:176
#6  0x00000000004a47e4 in usb_generic_handle_packet (s=0x14ba9c10, p=0x22973e88) at /home/devel/qemu-kvm-rhel6/hw/usb.c:250
#7  0x00000000004a49e2 in usb_handle_packet (dev=0x14ba9c10, p=0x22973e88) at /home/devel/qemu-kvm-rhel6/hw/usb.c:325
#8  0x0000000000576c23 in ehci_execute (q=0x22973e10) at /home/devel/qemu-kvm-rhel6/hw/usb-ehci.c:1441
#9  0x00000000005779f3 in ehci_state_execute (q=0x22973e10, async=1) at /home/devel/qemu-kvm-rhel6/hw/usb-ehci.c:1887
#10 0x0000000000577d1d in ehci_advance_state (ehci=0x14ba3680, async=1) at /home/devel/qemu-kvm-rhel6/hw/usb-ehci.c:2004
#11 0x0000000000577f21 in ehci_advance_async_state (ehci=0x14ba3680) at /home/devel/qemu-kvm-rhel6/hw/usb-ehci.c:2067
#12 0x00000000005783ef in ehci_frame_timer (opaque=0x14ba3680) at /home/devel/qemu-kvm-rhel6/hw/usb-ehci.c:2213
#13 0x000000000040c451 in qemu_run_timers (ptimer_head=0x984f18, current_time=15963010267070) at /home/devel/qemu-kvm-rhel6/vl.c:1328
#14 0x00000000004105e2 in main_loop_wait (timeout=1000) at /home/devel/qemu-kvm-rhel6/vl.c:4051
#15 0x000000000043ebeb in kvm_main_loop () at /home/devel/qemu-kvm-rhel6/qemu-kvm.c:2244
#16 0x0000000000410af2 in main_loop () at /home/devel/qemu-kvm-rhel6/vl.c:4234
#17 0x000000000041532b in main (argc=1308, argv=0x7fffffff47e8, envp=0x7fffffff70d0) at /home/devel/qemu-kvm-rhel6/vl.c:6590

Comment 9 Amos Kong 2013-05-28 02:07:01 UTC
Got another crash with upstream qemu very quickly, will investigate it.

#1  0x00007ffff4e32085 in abort () from /lib64/libc.so.6
#2  0x00007ffff4e29a1e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff4e29ae0 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff7dd3d65 in register_subpage (d=0x7ffffe80e610, section=0x7fff6bc5dd20) at /home/devel/qemu/exec.c:748
#5  0x00007ffff7dd4175 in mem_add (listener=0x7ffffe80e618, section=0x7fff6bc5de20) at /home/devel/qemu/exec.c:808
#6  0x00007ffff7e4410d in address_space_update_topology_pass (as=0x7fff70b4e000, old_view=..., new_view=..., adding=true) at /home/devel/qemu/memory.c:711
#7  0x00007ffff7e4421e in address_space_update_topology (as=0x7fff70b4e000) at /home/devel/qemu/memory.c:726
#8  0x00007ffff7e44374 in memory_region_transaction_commit () at /home/devel/qemu/memory.c:750
#9  0x00007ffff7e46d3c in memory_region_set_enabled (mr=0x7fff70b4e040, enabled=true) at /home/devel/qemu/memory.c:1392
#10 0x00007ffff7cbf92b in pci_default_write_config (d=0x7fff70b4dde0, addr=4, val=0, l=2) at hw/pci/pci.c:1162
#11 0x00007ffff7d061c0 in usb_ehci_pci_write_config (dev=0x7fff70b4dde0, addr=4, val=7, l=2) at hw/usb/hcd-ehci-pci.c:83
#12 0x00007ffff7cc40f5 in pci_host_config_write_common (pci_dev=0x7fff70b4dde0, addr=4, limit=256, val=7, len=2) at hw/pci/pci_host.c:54
#13 0x00007ffff7cc420d in pci_data_write (s=0x7fff7385ae20, addr=2147498756, val=7, len=2) at hw/pci/pci_host.c:75
#14 0x00007ffff7cc43dc in pci_host_data_write (opaque=0x7fff71251390, addr=0, val=7, len=2) at hw/pci/pci_host.c:128
#15 0x00007ffff7e41e2b in memory_region_write_accessor (opaque=0x7fff71253768, addr=0, value=0x7fff6bc5e580, size=2, shift=0, mask=65535) at /home/devel/qemu/memory.c:334
#16 0x00007ffff7e41f0d in access_with_adjusted_size (addr=0, value=0x7fff6bc5e580, size=2, access_size_min=1, access_size_max=4, access=
    0x7ffff7e41d9f <memory_region_write_accessor>, opaque=0x7fff71253768) at /home/devel/qemu/memory.c:364
#17 0x00007ffff7e42395 in memory_region_iorange_write (iorange=0x7ffff9850ea0, offset=0, width=2, data=7) at /home/devel/qemu/memory.c:439
#18 0x00007ffff7e3a42b in ioport_writew_thunk (opaque=0x7ffff9850ea0, addr=3324, data=7) at /home/devel/qemu/ioport.c:219
#19 0x00007ffff7e39d6b in ioport_write (index=1, address=3324, data=7) at /home/devel/qemu/ioport.c:83
#20 0x00007ffff7e3a9c7 in cpu_outw (addr=3324, val=7) at /home/devel/qemu/ioport.c:296
#21 0x00007ffff7e91e61 in helper_outw (port=3324, data=2147483655) at /home/devel/qemu/target-i386/misc_helper.c:82
#22 0x00007fffc41052dd in code_gen_buffer ()
#23 0x00007ffff7dc7271 in cpu_tb_exec (cpu=0x7fff6bc60010, tb_ptr=0x7fffc4105180 "A\213nČ…\355\017\205\275\002") at /home/devel/qemu/cpu-exec.c:56
#24 0x00007ffff7dc7f91 in cpu_x86_exec (env=0x7fff6bc60120) at /home/devel/qemu/cpu-exec.c:625
#25 0x00007ffff7dcb270 in tcg_cpu_exec (env=0x7fff6bc60120) at /home/devel/qemu/cpus.c:1144
#26 0x00007ffff7dcb3bb in tcg_exec_all () at /home/devel/qemu/cpus.c:1177
#27 0x00007ffff7dca539 in qemu_tcg_cpu_thread_fn (arg=0x7fff6bc60010) at /home/devel/qemu/cpus.c:844
#28 0x00007ffff5df8851 in start_thread () from /lib64/libpthread.so.0

Comment 10 mazhang 2013-05-29 09:56:34 UTC
amos, Sorry for reply so late, haven't test other window guest yet.

Comment 11 Amos Kong 2013-06-03 09:07:58 UTC
The calltraces are all related with usb, not pci multi-func. Reassigned to Gerd.

Comment 14 Qunfang Zhang 2014-08-05 08:29:02 UTC
*** Bug 1126748 has been marked as a duplicate of this bug. ***