Red Hat Bugzilla – Bug 1083413
qemu-kvm: iSCSI: Failure. SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400)
Last modified: 2015-03-05 03:05:21 EST
Description of problem: Execute the WRITE SAME command in guest to the libiscsi LUN cause QEMU core dumped. Version-Release number of selected component (if applicable): host info: # uname -r && rpm -q qemu-kvm 3.10.0-115.el7.x86_64 qemu-kvm-1.5.3-60.el7.x86_64 guest info: # uname -r 3.10.0-115.el7.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.prepare a libiscsi LUN. # qemu-img info iscsi://10.66.8.202:3261/iqn.2003-01.org.linux-iscsi.dhcp-8-202.x8664:sn.10b139e7a94e/0 image: iscsi://10.66.8.202:3261/iqn.2003-01.org.linux-iscsi.dhcp-8-202.x8664:sn.10b139e7a94e/0 file format: raw virtual size: 10G (10737418240 bytes) disk size: unavailable 2.start qemu with this libiscsi disk. # /usr/libexec/qemu-kvm -M pc -cpu Conroe -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection...-drive file=iscsi://10.66.8.202:3261/iqn.2003-01.org.linux-iscsi.dhcp-8-202.x8664:sn.10b139e7a94e/0,if=none,media=disk,format=raw,id=drive-data-disk,cache=none,aio=native,discard=on -device virtio-scsi-pci,bus=pci.0,addr=0x7,id=scsi0,multifunction=on -device scsi-hd,drive=drive-data-disk,id=data-disk,bus=scsi0.0,logical_block_size=4096 3. Execute the following commands in guest: # yes | head -n2048 > buf # sg_write_same --in buf --num=32 --lba=80 /dev/sda # sg_write_same --in /dev/zero --num=96 --lba=0 /dev/sda # sg_write_same -U --in /dev/zero --num=16 --lba=0 /dev/sda # time sg_write_same --in buf --num=65536 --lba=131074 /dev/sda # time sg_write_same --in /dev/zero --num=65534 --lba=196608 /dev/sda # sg_write_same --in /dev/zero --num=0 --lba=128 /dev/sda Actual results: after step 3, execute the time sg_write_same cause QEMU core dumped, i will attach the core dumped log later. guest ]1.# yes | head -n2048 > buf guest ]2.# sg_write_same --in buf --num=32 --lba=80 /dev/sda guest ]3.# sg_write_same --in /dev/zero --num=96 --lba=0 /dev/sda guest ]4.# sg_write_same -U --in /dev/zero --num=16 --lba=0 /dev/sda guest ]5.# time sg_write_same --in buf --num=65536 --lba=131074 /dev/sda real 0m0.225s user 0m0.000s sys 0m0.001s guest ]6.# time sg_write_same --in /dev/zero --num=65534 --lba=196608 /dev/sda Write failed: Broken pipe (qemu) qemu-kvm: iSCSI: Failure. SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_FIELD_IN_CDB(0x2400) Segmentation fault (core dumped) Expected results: It should no any core dumped occurs. Additional info: # /usr/libexec/qemu-kvm -M pc -cpu Conroe -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-47b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -drive file=/home/RHEL-Server-7.0-64-virtio.qcow2bk,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-system-disk,id=system-disk,bus=pci.0,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:21,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -k en-us -boot menu=on -qmp tcp:0:5555,server,nowait -serial unix:/tmp/ttyS0,server,nowait -vnc :2 -spice disable-ticketing,port=5932 -monitor stdio -drive file=iscsi://10.66.8.202:3261/iqn.2003-01.org.linux-iscsi.dhcp-8-202.x8664:sn.10b139e7a94e/0,if=none,media=disk,format=raw,id=drive-data-disk,cache=none,aio=native,discard=on -device virtio-scsi-pci,bus=pci.0,addr=0x7,id=scsi0,multifunction=on -device scsi-hd,drive=drive-data-disk,id=data-disk,bus=scsi0.0,logical_block_size=4096 Warning: option deprecated, use lost_tick_policy property of kvm-pit instead.
Core was generated by `/usr/libexec/qemu-kvm -M pc -cpu Conroe -enable-kvm -m 2048 -smp 2,sockets=2,co'. Program terminated with signal 11, Segmentation fault. #0 iscsi_co_write_zeroes (bs=<optimized out>, sector_num=<optimized out>, nb_sectors=<optimized out>, flags=<optimized out>) at block/iscsi.c:1023 1023 if (iTask.status == SCSI_STATUS_CHECK_CONDITION && (gdb) bt #0 iscsi_co_write_zeroes (bs=<optimized out>, sector_num=<optimized out>, nb_sectors=<optimized out>, flags=<optimized out>) at block/iscsi.c:1023 #1 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a5120, sector_num=sector_num@entry=1572864, nb_sectors=nb_sectors@entry=32768, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964 #2 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=16777216, offset=<optimized out>, req=0x7f66003d1480, bs=0x7f66001a5120) at block.c:3019 #3 bdrv_co_do_pwritev (bs=0x7f66001a5120, offset=<optimized out>, bytes=16777216, qiov=<optimized out>, flags=BDRV_REQ_ZERO_WRITE) at block.c:3151 #4 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a0ed0, sector_num=sector_num@entry=1572864, nb_sectors=nb_sectors@entry=524272, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964 #5 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=268427264, offset=<optimized out>, req=0x7f66003d1660, bs=0x7f66001a0ed0) at block.c:3019 #6 bdrv_co_do_pwritev (bs=0x7f66001a0ed0, offset=<optimized out>, bytes=268427264, qiov=<optimized out>, flags=BDRV_REQ_ZERO_WRITE) at block.c:3151 #7 0x00007f65fecc6654 in bdrv_co_do_writev (flags=<optimized out>, qiov=<optimized out>, nb_sectors=<optimized out>, sector_num=<optimized out>, bs=<optimized out>) at block.c:3175 #8 bdrv_co_do_rw (opaque=0x7f66005b9ab0) at block.c:4474 #9 0x00007f65fecfed2a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:118 #10 0x00007f65f98ac570 in ?? () from /lib64/libc.so.6 #11 0x00007fff266fcd50 in ?? () #12 0x0000000000000000 in ?? () (gdb) bt full #0 iscsi_co_write_zeroes (bs=<optimized out>, sector_num=<optimized out>, nb_sectors=<optimized out>, flags=<optimized out>) at block/iscsi.c:1023 iscsilun = 0x7f66001a8350 iTask = {status = 2, complete = 1, retries = 4, do_retry = 0, task = 0x0, co = 0x7f660028f950} lba = 1572864 nb_blocks = <optimized out> #1 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a5120, sector_num=sector_num@entry=1572864, nb_sectors=nb_sectors@entry=32768, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964 num = 32768 drv = 0x7f65ff2bc740 <bdrv_iscsi> qiov = {iov = 0x11f883000, niov = -19928276, nalloc = 32613, size = 140076067394688} iov = {iov_base = 0x0, iov_len = 0} ret = -95 max_write_zeroes = 32768 #2 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=16777216, offset=<optimized out>, req=0x7f66003d1480, bs=0x7f66001a5120) at block.c:3019 waited = <optimized out> ret = <optimized out> nb_sectors = 32768 drv = 0x7f65ff2bc740 <bdrv_iscsi> sector_num = 1572864 #3 bdrv_co_do_pwritev (bs=0x7f66001a5120, offset=<optimized out>, bytes=16777216, qiov=<optimized out>, flags=BDRV_REQ_ZERO_WRITE) at block.c:3151 req = {bs = 0x7f66001a5120, offset = 805306368, bytes = 16777216, is_write = true, serialising = false, overlap_offset = 805306368, overlap_bytes = 16777216, list = {le_next = 0x0, le_prev = 0x7f66001a7310}, co = 0x7f660028f950, wait_queue = {entries = {tqh_first = 0x0, tqh_last = 0x7f66003d14c0}, ctx = 0x7f660000d090}, waiting_for = 0x0} align = <optimized out> head_buf = 0x0 tail_buf = <optimized out> local_qiov = {iov = 0x7f660003d340, niov = 529010688, nalloc = 1, size = 4294975488} use_local_qiov = <optimized out> ret = <optimized out> __PRETTY_FUNCTION__ = "bdrv_co_do_pwritev" #4 0x00007f65fecc1aff in bdrv_co_do_write_zeroes (bs=bs@entry=0x7f66001a0ed0, sector_num=sector_num@entry=1572864, nb_sectors=nb_sectors@entry=524272, flags=flags@entry=BDRV_REQ_ZERO_WRITE) at block.c:2964 num = 32768 drv = 0x7f65ff2be560 <bdrv_raw> qiov = {iov = 0x0, niov = -19928276, nalloc = 32613, size = 140076067395168} iov = {iov_base = 0x0, iov_len = 0} ret = -95 max_write_zeroes = 32768 #5 0x00007f65fecc5d05 in bdrv_aligned_pwritev (flags=2, qiov=0x0, bytes=268427264, offset=<optimized out>, req=0x7f66003d1660, bs=0x7f66001a0ed0) at block.c:3019 waited = <optimized out> ret = <optimized out> nb_sectors = 524272 drv = 0x7f65ff2be560 <bdrv_raw> sector_num = 1572864 #6 bdrv_co_do_pwritev (bs=0x7f66001a0ed0, offset=<optimized out>, bytes=268427264, qiov=<optimized out>, flags=BDRV_REQ_ZERO_WRITE) at block.c:3151 req = {bs = 0x7f66001a0ed0, offset = 805306368, bytes = 268427264, is_write = true, serialising = false, overlap_offset = 805306368, overlap_bytes = 268427264, list = {le_next = 0x0, le_prev = 0x7f66001a30c0}, co = 0x7f660028f950, wait_queue = {entries = {tqh_first = 0x0, tqh_last = 0x7f66003d16a0}, ctx = 0x7f660000d090}, waiting_for = 0x0} align = <optimized out> head_buf = 0x0 tail_buf = <optimized out> local_qiov = {iov = 0x0, niov = 0, nalloc = 0, size = 0} use_local_qiov = <optimized out> ret = <optimized out> __PRETTY_FUNCTION__ = "bdrv_co_do_pwritev" #7 0x00007f65fecc6654 in bdrv_co_do_writev (flags=<optimized out>, qiov=<optimized out>, nb_sectors=<optimized out>, sector_num=<optimized out>, bs=<optimized out>) at block.c:3175 No locals. #8 bdrv_co_do_rw (opaque=0x7f66005b9ab0) at block.c:4474 acb = 0x7f66005b9ab0 bs = <optimized out> #9 0x00007f65fecfed2a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:118 self = 0x7f660028f950 co = 0x7f660028f950 #10 0x00007f65f98ac570 in ?? () from /lib64/libc.so.6 No symbol table info available. #11 0x00007fff266fcd50 in ?? () No symbol table info available. #12 0x0000000000000000 in ?? () No symbol table info available. (gdb) q
# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 42 Model name: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz Stepping: 7 CPU MHz: 1978.242 BogoMIPS: 6784.57 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K NUMA node0 CPU(s): 0-7
we support the fix WRITE SAME support in qemu-kvm-1.5.3-31.el7 according to bug 1007815. If i use the qemu-kvm-1.5.3-31.el7.x86_64 which cann't execute the sg_write_same with the error message 'bad field in Write same(10) cdb, option probably not supported'. qemu-kvm-1.5.3-31.el7.x86_64 guest ]# yes | head -n2048 > buf guest ]# sg_write_same --in buf --num=32 --lba=80 /dev/sda bad field in Write same(10) cdb, option probably not supported qemu-kvm-1.5.3-32.el7.x86_64 ------- hit this issue qemu-kvm-1.5.3-38.el7.x86_64 ------- hit this issue qemu-kvm-1.5.3-59.el7.x86_64 ------- hit this issue qemu-kvm-1.5.3-60.el7.x86_64 ------- hit this issue Base on above, this issue is a regression issue caused by qemu-kvm-1.5.3-32.el7.x86_64. Best Regards, sluo
There are actually other problems. One is the crash, which is fixed by upstream commit d9738fd2463f71530d8d92fbb52ebdd1d78074fc. The second is that QEMU does not recognize INVALID_FIELD_IN_CDB (reported also by dmesg on the target as "WRITE_SAME sectors: 32768 exceeds max_write_same_len: 4096") and fall back to regular writes. This will be fixed by commit d7b09f19fd85456ec282b03e736ce9244ff9acec (from scsi-next). The third is that QEMU does not properly split the WRITE SAME requests because LBPME=0. This is fixed by commit 24d3bd67aca958c8ea103646d9d326de00056e4d, which doesn't apply cleanly to RHEL but the conflict are easily solved, commit 5d259fc7da83249a4f78fe32de2bc2874a997a9f and commit e4942f2df8f9969231823fc51cedf8f042fe7a31 (this one from scsi-next). There is also a fourth problem which is reported on the iSCSI target as "SCSI OP 8ah with too big sectors 32768 exceeds fabric_max_sectors: 8192". I will open a separate bug for this once I figure out where the fix lies. The above five patches are enough to fix this completely for WRITE SAME.
*** Bug 1025342 has been marked as a duplicate of this bug. ***
Fix included in qemu-kvm-1.5.3-61.el7
Verify this issue on qemu-kvm-1.5.3-75.el7.x86_64 which has no any core dumped. host info: # uname -r && rpm -q qemu-kvm 3.10.0-171.el7.x86_64 qemu-kvm-1.5.3-75.el7.x86_64 Steps: the same to comment #0. after step 3, execute the time sg_write_same cause QEMU core dumped, i will attach the core dumped log later. guest ]1.# yes | head -n2048 > buf guest ]2.# sg_write_same --in buf --num=32 --lba=80 /dev/sdb guest ]3.# sg_write_same --in /dev/zero --num=96 --lba=0 /dev/sdb guest ]4.# sg_write_same -U --in /dev/zero --num=16 --lba=0 /dev/sdb guest ]5.# time sg_write_same --in buf --num=65536 --lba=131074 /dev/sdb real 0m0.399s user 0m0.000s sys 0m0.001s guest ]6.# time sg_write_same --in /dev/zero --num=65534 --lba=196608 /dev/sdb real 0m0.388s user 0m0.000s sys 0m0.003s guest ]7.# sg_write_same --in /dev/zero --num=0 --lba=128 /dev/sdb bad field in Write same(10) cdb, option probably not supported (qemu) qemu-kvm: iSCSI: Failure. SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_OPERATION_CODE(0x2000) (qemu) info status VM status: running ############################################# Also tried on qemu-kvm-rhev-2.1.2-1.el7.x86_64 which has no any core dumped. host info: # uname -r && rpm -q qemu-kvm-rhev 3.10.0-183.el7.x86_64 qemu-kvm-rhev-2.1.2-1.el7.x86_64 Results: # yes | head -n2048 > buf # sg_write_same --in buf --num=32 --lba=80 /dev/sdb # sg_write_same --in /dev/zero --num=96 --lba=0 /dev/sdb # sg_write_same -U --in /dev/zero --num=16 --lba=0 /dev/sdb # time sg_write_same --in buf --num=65536 --lba=131074 /dev/sdb real 0m0.386s user 0m0.000s sys 0m0.002s # time sg_write_same --in /dev/zero --num=65534 --lba=196608 /dev/sdb real 0m0.371s user 0m0.000s sys 0m0.002s # # sg_write_same --in /dev/zero --num=0 --lba=128 /dev/sdb bad field in Write same(10) cdb, option probably not supported (qemu) qemu-kvm: iSCSI Failure: SENSE KEY:ILLEGAL_REQUEST(5) ASCQ:INVALID_OPERATION_CODE(0x2000) (qemu) info status VM status: running Base on above, this issue has been fixed correctly, move to VERIFIED status. Best Regards, sluo
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0349.html