Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
boot guest with a iscsi data disk, then use firewall to block this iscsi data disk. After this, qemu-kvm will hang.
Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.3.0-1.el7.x86_64
kernel-3.10.0-246.el7.x86_64
How reproducible:
100%
Steps to Reproduce:
1.boot guest with libiscsi storage as a data disk.
#/usr/libexec/qemu-kvm -m 2G -smp 4 -boot menu=on -drive file=/home/system-img.qcow2,snapshot=off,if=none,id=img,cache=none,format=qcow2 -device virtio-blk-pci,drive=img,id=sys-img,bootindex=1 -drive file=iscsi://10.66.8.139:3260/storage.disk1.xqu.xyz/3,if=none,id=disk1,cache=none,format=raw -device virtio-blk-pci,drive=disk1,id=data-disk1,scsi=on -monitor stdio -spice port=5943,disable-ticketing -netdev tap,id=tap0 -device virtio-net-pci,netdev=tap0,mac=24:be:05:14:95:22 -qmp tcp::8884,server,nowait
2 after guets boot up, block iscsi server using firewall on host.
# iptables -A OUTPUT -s 10.66.8.139 -p tcp --dport 3260 -j REJECT
3. wait for 1 minute, check the status of qemu-kvm.
Actual results:
After step 3, qemu-kvm will hang. And hmp will print following info:
(qemu)
qemu-kvm: iSCSI: NOP timeout. Reconnecting...
Expected results:
1, qemu-kvm works well, no hang.
2, give some more reasonable hints, as following:
qemu:
block I/O error in device 'drive-virtio-disk1': Input/output error (5)
Additional info:
The callback shows it is the synchronous iscsi_reconnect() that blocks QEMU when the connection is down:
#0 0x00007fde2a7462dd in nanosleep () from /lib64/libc.so.6
#1 0x00007fde2a746174 in sleep () from /lib64/libc.so.6
#2 0x00007fde2cda773e in iscsi_reconnect () from /usr/lib64/iscsi/libiscsi.so.4
#3 0x00007fde2dc06122 in iscsi_nop_timed_event (opaque=0x7fde305a4960) at /home/fam/qemu/block/iscsi.c:1105
#4 0x00007fde2dbb3f1d in timerlist_run_timers (timer_list=0x7fde30557c60) at /home/fam/qemu/qemu-timer.c:502
#5 0x00007fde2dbb4016 in timerlistgroup_run_timers (tlg=0x7fde305577c8) at /home/fam/qemu/qemu-timer.c:538
#6 0x00007fde2dbb472d in aio_dispatch (ctx=0x7fde30557690) at /home/fam/qemu/aio-posix.c:182
#7 0x00007fde2dba4b7d in aio_ctx_dispatch (source=0x7fde30557690, callback=0x0, user_data=0x0) at /home/fam/qemu/async.c:219
#8 0x00007fde2b328a8a in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#9 0x00007fde2dbb2a47 in glib_pollfds_poll () at /home/fam/qemu/main-loop.c:199
#10 0x00007fde2dbb2b22 in os_host_main_loop_wait (timeout=27000000) at /home/fam/qemu/main-loop.c:244
#11 0x00007fde2dbb2bde in main_loop_wait (nonblocking=0) at /home/fam/qemu/main-loop.c:493
#12 0x00007fde2d9e9dc3 in main_loop () at /home/fam/qemu/vl.c:1887
#13 0x00007fde2d9f1a78 in main (argc=29, argv=0x7ffd82760ef8, envp=0x7ffd82760fe8) at /home/fam/qemu/vl.c:4604
Related change from upstream libiscsi:
commit b152d26eb948520dbc7e1a1078a590d71804021b
Author: Peter Lieven <pl>
Date: Tue Mar 31 16:20:19 2015 +0200
connect: make the reconnect async
Signed-off-by: Peter Lieven <pl>