RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1168213 - main-loop: WARNING: I/O thread spun for 1000 iterations while doing stream block device.
Summary: main-loop: WARNING: I/O thread spun for 1000 iterations while doing stream bl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: Gu Nini
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-26 12:17 UTC by Yang Xue
Modified: 2018-11-01 11:01 UTC (History)
11 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-5.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-01 11:01:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
iometer.icf (9.01 KB, text/plain)
2016-05-12 10:27 UTC, Gu Nini 		no flags Details
View All

Description Yang Xue 2014-11-26 12:17:40 UTC 
Description of problem:
win2012r2 guest main-loop: WARNING: I/O thread spun for 1000 iterations ,while doing stream block device.

Version-Release number of selected component (if applicable):
Host:
kernel-3.10.0-205.el7.x86_64
qemu-kvm-rhev-2.1.2-10.el7.x86_64

How reproducible:
50%

Steps to Reproduce:
1.start win2012r2 guest
/bin/qemu-kvm \
    -S  \
    -name 'virt-tests-vm1'  \
    -sandbox off  \
    -M pc  \
    -nodefaults  \
    -vga qxl  \
    -global qxl-vga.vram_size=33554432 \
    -device intel-hda,bus=pci.0,addr=03 \
    -device hda-duplex  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20141122-031435-M0GYyrum,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20141122-031435-M0GYyrum,server,nowait \
    -device isa-serial,chardev=serial_id_serial0 \
    -device virtio-serial-pci,id=virtio_serial_pci0,bus=pci.0,addr=04  \
    -chardev socket,id=devvs,path=/tmp/virtio_port-vs-20141122-031435-M0GYyrum,server,nowait \
    -device virtserialport,chardev=devvs,name=vs,id=vs,bus=virtio_serial_pci0.0  \
    -chardev socket,id=seabioslog_id_20141122-031435-M0GYyrum,path=/tmp/seabios-20141122-031435-M0GYyrum,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20141122-031435-M0GYyrum,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=05 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=06 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/home/autotest-devel/client/tests/virt/shared/data/images/win2012-64r2-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:8e:8f:90:91:92,id=idxToyso,vectors=4,netdev=idU9oayT,bus=pci.0,addr=07  \
    -netdev tap,id=idU9oayT,vhost=on,vhostfd=23,fd=22  \
    -m 65536  \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
    -cpu 'Penryn',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/home/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \
    -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.0,unit=0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,password=123456,addr=0,tls-port=3200,x509-dir=/tmp/spice_x509d,tls-channel=main,tls-channel=inputs,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off \
    -enable-kvm
2.{'execute': 'blockdev-snapshot-sync', 'arguments': {'device': u'drive_image1', 'snapshot-file': '/home/autotest-devel/client/tests/virt/shared/data/images/sn1', 'format': 'qcow2'}, 'id': 'UdasHoGF'}
3.start to stream block device
{'execute': 'block-stream', 'arguments': {'device': u'drive_image1', 'speed': 1048576}, 'id': 'RKxrPfkM'}
4.{"execute": "block-job-set-speed", "arguments": {"device": "drive_image1", "speed": 10485760}, "id": "m7HB8euI"}
5.{'execute': 'query-block-jobs', 'id': 'NaHuYYlt'}
when block stream job is done.
6.reboot guest

Actual results:
[qemu output] main-loop: WARNING: I/O thread spun for 1000 iterations
guest reboot and can't get IP address.

Expected results:
[qemu output] no warning 
guest reboot in normal status.

Additional info:
Comment 2 Yang Xue 2014-11-27 02:05:50 UTC 
Reproduce this problem again and 100% reproducible. 
This problem causes the vcpu not running.
Comment 3 Joy Pu 2014-12-10 08:50:57 UTC 
Met this in ppc host with ppc64le guest too:
qemu-kvm related package:
ipxe-roms-qemu-20130517-6.gitc4bce43.el7.noarch
qemu-img-rhev-2.1.2-14.el7.ppc64
qemu-kvm-tools-rhev-2.1.2-14.el7.ppc64
libvirt-daemon-driver-qemu-1.2.8-9.el7.ppc64
qemu-kvm-common-rhev-2.1.2-14.el7.ppc64
qemu-kvm-rhev-2.1.2-14.el7.ppc64
qemu-kvm-rhev-debuginfo-2.1.2-14.el7.ppc64

host kernel verion:3.10.0-201.el7.ppc64
guest kernel version: 3.10.0-210
Comment 4 Joy Pu 2014-12-10 08:57:19 UTC 
This problem in ppc host have some different results:

Met this once the qemu process is terminated after 8 minutes that this message shows up.
main-loop: WARNING: I/O thread spun for 1000 iterations
qemu: terminating on signal 1

And normally it will blocked the guest boot up normally after the block stream guest shut down.
Comment 8 Gu Nini 2016-05-12 10:27:23 UTC 
Created attachment 1156629 [details]
iometer.icf

Found the same log in the bug on following sw versions with Win10.x86_64 guest started with aio=native disks; while for the same guest started with aio=threads disks, there is no the problem.

Host kernel: 3.10.0-370.el7.x86_64
Qemu-kvm: qemu-kvm-rhev-2.5.0-4.el7.x86_64
virtio-win: virtio-win-1.8.0-4


Steps:

1) Start an installed Win10.x86_64 guest with following qemu command:

/usr/libexec/qemu-kvm \
     -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga std  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_1,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_2,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idZelHT0  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_3,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20160511-031111-dD14QDp6,path=/var/tmp/avocado_4,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20160511-031111-dD14QDp6,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=1d.4,firstport=4,bus=pci.0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=03,disable-legacy=off,disable-modern=on \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/usr/share/avocado/data/avocado-vt/images/win10-64-virtio-scsi.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -drive id=drive_disk1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/usr/share/avocado/data/avocado-vt/images/storage.qcow2 \
    -device scsi-hd,id=disk1,drive=drive_disk1 \
    -device virtio-net-pci,mac=9a:1a:1b:1c:1d:1e,id=idTv7QY9,vectors=4,netdev=idgrlUJn,bus=pci.0,addr=04,disable-legacy=off,disable-modern=on  \
    -netdev tap,id=idgrlUJn,vhost=on \
    -m 16384  \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2  \
    -cpu 'Westmere',+kvm_pv_unhalt,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/usr/share/avocado/data/avocado-vt/isos/windows/winutils.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio

2). After the guest boots up, run iometer read/write operation on data drive drive_disk1 inside the guest with iometer.icf(attached).


Results:
Log 'main-loop: WARNING: I/O thread spun for 1000 iterations' was sent out in hmp, the guest hang AND the cpu usage rate of qemu-kvm process would increase to 300% in the host.
Comment 9 Gu Nini 2016-05-12 10:33:16 UTC 
Jeff,

Would you help to check if comment #8 is caused for the same reason of the bug? If not, I will report a new one.

BTW, there is some discussion on the same log at https://bugs.launchpad.net/qemu/+bug/1481272
Comment 10 Gu Nini 2016-05-13 05:55:50 UTC 
(In reply to Gu Nini from comment #9)
> Jeff,
> 
> Would you help to check if comment #8 is caused for the same reason of the
> bug? If not, I will report a new one.
> 
> BTW, there is some discussion on the same log at
> https://bugs.launchpad.net/qemu/+bug/1481272

Could not reproduce comment #8 problem with following newest 7.3 sw:

Host Kernel: 3.10.0-396.el7.x86_64
Qemu-kvm-rhev: qemu-kvm-rhev-2.6.0-1.el7.x86_64
virtio-win: virtio-win-1.8.0-4

So please ignore the needinfo in comment #9.
Comment 12 Longxiang Lyu 2017-07-28 08:07:05 UTC 
Could reproduce the warning message with the following version selected:
Host:
kernel-3.10.0-693.el7.x86_64
qemu-kvm-rhev-2.9.0-17.el7a.x86_64

Guest:
kernel-3.10.0-693.el7.x86_64
Comment 13 Jeff Cody 2018-05-30 20:20:18 UTC 
Commit upstream from Stefan that addresses this BZ:  https://lists.gnu.org/archive/html/qemu-devel/2018-05/msg06889.html


However, the dependency for this commit is non-trivial; it depends on changes made to thread lock optimizations for the BQL, in Pavel Dovgalyuk's "replay additions" patch series.  This is probably too invasive to backport into 7.6, unless we just want to suppress the output anyways: https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06572.html
Comment 14 Jeff Cody 2018-06-13 16:21:47 UTC 
Submitted backport to go ahead and suppress the output anyway; regardless of the BQL, it has never been a very useful debug message and produces a lot of spurious noise.
Comment 16 Miroslav Rezanina 2018-06-25 14:01:02 UTC 
Fix included in qemu-kvm-rhev-2.12.0-5.el7
Comment 18 Gu Nini 2018-07-02 04:00:02 UTC 
Reproduce the bug on 'qemu-kvm-rhev-2.12.0-4.el7.x86_64' with the same steps as that in the bz description.

Verify the bug on 'qemu-kvm-rhev-2.12.0-5.el7.x86_64', i.e. no 'main-loop: WARNING: I/O thread spun for 1000 iterations' is sent out from hmp.
Comment 21 errata-xmlrpc 2018-11-01 11:01:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443
Note You need to log in before you can comment on or make changes to this bug.