Bug 1083413
| Summary: | qemu-kvm: iSCSI: Failure. SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Sibiao Luo <sluo> |
| Component: | qemu-kvm | Assignee: | Fam Zheng <famz> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.0 | CC: | chayang, famz, hhuang, jherrman, juli, juzhang, knoel, michen, pbonzini, qzhang, rbalakri, sluo, virt-maint, xfu |
| Target Milestone: | rc | Keywords: | Regression, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-1.5.3-61.el7 | Doc Type: | Bug Fix |
| Doc Text: |
Due to a bug in the Internet Small Computer System Interface (iSCSI) driver, a qemu-kvm process terminated unexpectedly with a segmentation fault when the "write same" command was executed in guest mode under the iSCSI protocol. This update fixes the regression and the "write same" command now functions in guest mode under iSCSI as intended.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-03-05 08:05:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1076184, 1090978 | ||
|
Description
Sibiao Luo
2014-04-02 08:17:30 UTC
Core was generated by `/usr/libexec/qemu-kvm -M pc -cpu Conroe -enable-kvm -m 2048 -smp 2,sockets=2,co'.
Program terminated with signal 11, Segmentation fault.
#0 iscsi_co_write_zeroes (bs=<optimized out>, sector_num=<optimized out>, nb_sectors=<optimized out>,
flags=<optimized out>) at block/iscsi.c:1023
1023 if (iTask.status == SCSI_STATUS_CHECK_CONDITION &&
(gdb) bt
#0 iscsi_co_write_zeroes (bs=<optimized out>, sector_num=<optimized out>, nb_sectors=<optimized out>,
flags=<optimized out>) at block/iscsi.c:1023
#1 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a5120, sector_num=sector_num@entry=1572864,
nb_sectors=nb_sectors@entry=32768, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964
#2 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=16777216, offset=<optimized out>,
req=0x7f66003d1480, bs=0x7f66001a5120) at block.c:3019
#3 bdrv_co_do_pwritev (bs=0x7f66001a5120, offset=<optimized out>, bytes=16777216, qiov=<optimized out>,
flags=BDRV_REQ_ZERO_WRITE) at block.c:3151
#4 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a0ed0, sector_num=sector_num@entry=1572864,
nb_sectors=nb_sectors@entry=524272, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964
#5 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=268427264, offset=<optimized out>,
req=0x7f66003d1660, bs=0x7f66001a0ed0) at block.c:3019
#6 bdrv_co_do_pwritev (bs=0x7f66001a0ed0, offset=<optimized out>, bytes=268427264, qiov=<optimized out>,
flags=BDRV_REQ_ZERO_WRITE) at block.c:3151
#7 0x00007f65fecc6654 in bdrv_co_do_writev (flags=<optimized out>, qiov=<optimized out>, nb_sectors=<optimized out>,
sector_num=<optimized out>, bs=<optimized out>) at block.c:3175
#8 bdrv_co_do_rw (opaque=0x7f66005b9ab0) at block.c:4474
#9 0x00007f65fecfed2a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:118
#10 0x00007f65f98ac570 in ?? () from /lib64/libc.so.6
#11 0x00007fff266fcd50 in ?? ()
#12 0x0000000000000000 in ?? ()
(gdb) bt full
#0 iscsi_co_write_zeroes (bs=<optimized out>, sector_num=<optimized out>, nb_sectors=<optimized out>,
flags=<optimized out>) at block/iscsi.c:1023
iscsilun = 0x7f66001a8350
iTask = {status = 2, complete = 1, retries = 4, do_retry = 0, task = 0x0, co = 0x7f660028f950}
lba = 1572864
nb_blocks = <optimized out>
#1 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a5120, sector_num=sector_num@entry=1572864,
nb_sectors=nb_sectors@entry=32768, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964
num = 32768
drv = 0x7f65ff2bc740 <bdrv_iscsi>
qiov = {iov = 0x11f883000, niov = -19928276, nalloc = 32613, size = 140076067394688}
iov = {iov_base = 0x0, iov_len = 0}
ret = -95
max_write_zeroes = 32768
#2 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=16777216, offset=<optimized out>,
req=0x7f66003d1480, bs=0x7f66001a5120) at block.c:3019
waited = <optimized out>
ret = <optimized out>
nb_sectors = 32768
drv = 0x7f65ff2bc740 <bdrv_iscsi>
sector_num = 1572864
#3 bdrv_co_do_pwritev (bs=0x7f66001a5120, offset=<optimized out>, bytes=16777216, qiov=<optimized out>,
flags=BDRV_REQ_ZERO_WRITE) at block.c:3151
req = {bs = 0x7f66001a5120, offset = 805306368, bytes = 16777216, is_write = true, serialising = false,
overlap_offset = 805306368, overlap_bytes = 16777216, list = {le_next = 0x0, le_prev = 0x7f66001a7310},
co = 0x7f660028f950, wait_queue = {entries = {tqh_first = 0x0, tqh_last = 0x7f66003d14c0},
ctx = 0x7f660000d090}, waiting_for = 0x0}
align = <optimized out>
head_buf = 0x0
tail_buf = <optimized out>
local_qiov = {iov = 0x7f660003d340, niov = 529010688, nalloc = 1, size = 4294975488}
use_local_qiov = <optimized out>
ret = <optimized out>
__PRETTY_FUNCTION__ = "bdrv_co_do_pwritev"
#4 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a0ed0, sector_num=sector_num@entry=1572864,
nb_sectors=nb_sectors@entry=524272, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964
num = 32768
drv = 0x7f65ff2be560 <bdrv_raw>
qiov = {iov = 0x0, niov = -19928276, nalloc = 32613, size = 140076067395168}
iov = {iov_base = 0x0, iov_len = 0}
ret = -95
max_write_zeroes = 32768
#5 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=268427264, offset=<optimized out>,
req=0x7f66003d1660, bs=0x7f66001a0ed0) at block.c:3019
waited = <optimized out>
ret = <optimized out>
nb_sectors = 524272
drv = 0x7f65ff2be560 <bdrv_raw>
sector_num = 1572864
#6 bdrv_co_do_pwritev (bs=0x7f66001a0ed0, offset=<optimized out>, bytes=268427264, qiov=<optimized out>,
flags=BDRV_REQ_ZERO_WRITE) at block.c:3151
req = {bs = 0x7f66001a0ed0, offset = 805306368, bytes = 268427264, is_write = true, serialising = false,
overlap_offset = 805306368, overlap_bytes = 268427264, list = {le_next = 0x0, le_prev = 0x7f66001a30c0},
co = 0x7f660028f950, wait_queue = {entries = {tqh_first = 0x0, tqh_last = 0x7f66003d16a0},
ctx = 0x7f660000d090}, waiting_for = 0x0}
align = <optimized out>
head_buf = 0x0
tail_buf = <optimized out>
local_qiov = {iov = 0x0, niov = 0, nalloc = 0, size = 0}
use_local_qiov = <optimized out>
ret = <optimized out>
__PRETTY_FUNCTION__ = "bdrv_co_do_pwritev"
#7 0x00007f65fecc6654 in bdrv_co_do_writev (flags=<optimized out>, qiov=<optimized out>, nb_sectors=<optimized out>,
sector_num=<optimized out>, bs=<optimized out>) at block.c:3175
No locals.
#8 bdrv_co_do_rw (opaque=0x7f66005b9ab0) at block.c:4474
acb = 0x7f66005b9ab0
bs = <optimized out>
#9 0x00007f65fecfed2a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:118
self = 0x7f660028f950
co = 0x7f660028f950
#10 0x00007f65f98ac570 in ?? () from /lib64/libc.so.6
No symbol table info available.
#11 0x00007fff266fcd50 in ?? ()
No symbol table info available.
#12 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) q
# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 42 Model name: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz Stepping: 7 CPU MHz: 1978.242 BogoMIPS: 6784.57 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K NUMA node0 CPU(s): 0-7 we support the fix WRITE SAME support in qemu-kvm-1.5.3-31.el7 according to bug 1007815. If i use the qemu-kvm-1.5.3-31.el7.x86_64 which cann't execute the sg_write_same with the error message 'bad field in Write same(10) cdb, option probably not supported'. qemu-kvm-1.5.3-31.el7.x86_64 guest ]# yes | head -n2048 > buf guest ]# sg_write_same --in buf --num=32 --lba=80 /dev/sda bad field in Write same(10) cdb, option probably not supported qemu-kvm-1.5.3-32.el7.x86_64 ------- hit this issue qemu-kvm-1.5.3-38.el7.x86_64 ------- hit this issue qemu-kvm-1.5.3-59.el7.x86_64 ------- hit this issue qemu-kvm-1.5.3-60.el7.x86_64 ------- hit this issue Base on above, this issue is a regression issue caused by qemu-kvm-1.5.3-32.el7.x86_64. Best Regards, sluo There are actually other problems. One is the crash, which is fixed by upstream commit d9738fd2463f71530d8d92fbb52ebdd1d78074fc. The second is that QEMU does not recognize INVALID_FIELD_IN_CDB (reported also by dmesg on the target as "WRITE_SAME sectors: 32768 exceeds max_write_same_len: 4096") and fall back to regular writes. This will be fixed by commit d7b09f19fd85456ec282b03e736ce9244ff9acec (from scsi-next). The third is that QEMU does not properly split the WRITE SAME requests because LBPME=0. This is fixed by commit 24d3bd67aca958c8ea103646d9d326de00056e4d, which doesn't apply cleanly to RHEL but the conflict are easily solved, commit 5d259fc7da83249a4f78fe32de2bc2874a997a9f and commit e4942f2df8f9969231823fc51cedf8f042fe7a31 (this one from scsi-next). There is also a fourth problem which is reported on the iSCSI target as "SCSI OP 8ah with too big sectors 32768 exceeds fabric_max_sectors: 8192". I will open a separate bug for this once I figure out where the fix lies. The above five patches are enough to fix this completely for WRITE SAME. *** Bug 1025342 has been marked as a duplicate of this bug. *** Fix included in qemu-kvm-1.5.3-61.el7 Verify this issue on qemu-kvm-1.5.3-75.el7.x86_64 which has no any core dumped. host info: # uname -r && rpm -q qemu-kvm 3.10.0-171.el7.x86_64 qemu-kvm-1.5.3-75.el7.x86_64 Steps: the same to comment #0. after step 3, execute the time sg_write_same cause QEMU core dumped, i will attach the core dumped log later. guest ]1.# yes | head -n2048 > buf guest ]2.# sg_write_same --in buf --num=32 --lba=80 /dev/sdb guest ]3.# sg_write_same --in /dev/zero --num=96 --lba=0 /dev/sdb guest ]4.# sg_write_same -U --in /dev/zero --num=16 --lba=0 /dev/sdb guest ]5.# time sg_write_same --in buf --num=65536 --lba=131074 /dev/sdb real 0m0.399s user 0m0.000s sys 0m0.001s guest ]6.# time sg_write_same --in /dev/zero --num=65534 --lba=196608 /dev/sdb real 0m0.388s user 0m0.000s sys 0m0.003s guest ]7.# sg_write_same --in /dev/zero --num=0 --lba=128 /dev/sdb bad field in Write same(10) cdb, option probably not supported (qemu) qemu-kvm: iSCSI: Failure. SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_OPERATION_CODE(0x2000) (qemu) info status VM status: running ############################################# Also tried on qemu-kvm-rhev-2.1.2-1.el7.x86_64 which has no any core dumped. host info: # uname -r && rpm -q qemu-kvm-rhev 3.10.0-183.el7.x86_64 qemu-kvm-rhev-2.1.2-1.el7.x86_64 Results: # yes | head -n2048 > buf # sg_write_same --in buf --num=32 --lba=80 /dev/sdb # sg_write_same --in /dev/zero --num=96 --lba=0 /dev/sdb # sg_write_same -U --in /dev/zero --num=16 --lba=0 /dev/sdb # time sg_write_same --in buf --num=65536 --lba=131074 /dev/sdb real 0m0.386s user 0m0.000s sys 0m0.002s # time sg_write_same --in /dev/zero --num=65534 --lba=196608 /dev/sdb real 0m0.371s user 0m0.000s sys 0m0.002s # # sg_write_same --in /dev/zero --num=0 --lba=128 /dev/sdb bad field in Write same(10) cdb, option probably not supported (qemu) qemu-kvm: iSCSI Failure: SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_OPERATION_CODE(0x2000) (qemu) info status VM status: running Base on above, this issue has been fixed correctly, move to VERIFIED status. Best Regards, sluo Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0349.html |