Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: 3.19 kernels make iscsi not work against QNAP device (suspect will effect other vendors but can test this). Lots of this with ANT writes: [44277.671889] sd 18:0:0:0: [sde] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [44277.671904] sd 18:0:0:0: [sde] Sense Key : Illegal Request [current] [44277.671909] sd 18:0:0:0: [sde] Add. Sense: Invalid field in cdb [44277.671912] sd 18:0:0:0: [sde] CDB: [44277.671915] Read(10): 28 00 00 01 71 80 00 30 a0 00 [44277.671924] blk_update_request: critical target error, dev sde, sector 94592 On the QNAP device lots of this: [ 786.600208] SCSI OP 88h with too big sectors 9536 [ 787.251458] SCSI OP 88h with too big sectors 11296 [ 787.952351] SCSI OP 88h with too big sectors 9280 Version-Release number of selected component (if applicable): Up to date Fedora 21 3.19.3-200.fc21.x86_64 How reproducible: All this time with any 3.19 kernel. Steps to Reproduce: 1. Boot to a 3.19 kernel 2. Do some I/O to the iscsi device. 3. File system becomes unclean and lots of errors. Actual results: I/O issues. Expected results: Works as it should as with <= 3.18 kernels. Additional info: Seems to be this issue: https://lkml.org/lkml/2015/2/19/91
This is on a host using jumbo frames (9000) on the storage network with bonded interface with two gigbit interfaces.
I reported this issue to Linux SCSI mailing list and had a discussion with Ewan Milne and Mike Christie: "iSCSI regression with linux 3.9 and 4.0" http://comments.gmane.org/gmane.linux.scsi/99902 We switched to off-list discussion after some sensitive data was included. Mike told me he has enough information to fix the issue, and promised to send patches after he returned from PTO. I will keep you updated.
WARNING: This is just a hack and has bad impact on performance, but makes iscsi work with qnap devices. Use on your own risk, and for testing purpose only. Do not apply in production environment. You have been warned! --- a/drivers/scsi/iscsi_tcp.c +++ b/drivers/scsi/iscsi_tcp.c @@ -955,7 +955,7 @@ static struct scsi_host_template iscsi_sw_tcp_sht = { .change_queue_depth = scsi_change_queue_depth, .can_queue = ISCSI_DEF_XMIT_CMDS_MAX - 1, .sg_tablesize = 4096, - .max_sectors = 0xFFFF, + .max_sectors = 0x400, .cmd_per_lun = ISCSI_DEF_CMD_PER_LUN, .eh_abort_handler = iscsi_eh_abort, .eh_device_reset_handler= iscsi_eh_device_reset,
Created attachment 1016683 [details] SCSI: add 1024 max sectors black list flag Mike Christie sent a patch to linux-scsi [0] that fixes the issue. [0] http://marc.info/?l=linux-scsi&m=142958776223667&w=2
Dear Fedora Kernel maintainers can we get this patch applied to the Fedora kernels?
Yes, I'll do so today. Thank you very much for the pointers.
Added in Fedora git. It will be in the next build on all the various branches.
I got burned by this too with my entire oVirt datacenter going down since the qemu disks use werror=stop, rerror=stop. Fortunately that's just my home IT, but it sure hurt. However, I mostly wanted to chime in to mention that my QNAP (TS-569 Pro) shows a bunch of: [269167.463722] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION. [269167.470709] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION. [269744.226822] TARGET_CORE[iSCSI]: Unsupported SCSI Opcode 0x85, sending CHECK_CONDITION. It doesn't seem to be making those any more now I'm back running under 3.18. I can't swear their related, but I'm quite certain they must be. I couldn't access my QNAP during the outage because my passwords were inaccessible with my datacenter down. :-( Me needs to print and hide passwords and come up with some cloned home directory on local storage for times like this. So glad to have escaped out of Zombieville, the dark village of the unliving.
kernel-4.0.1-300.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/kernel-4.0.1-300.fc22
I have installed the 3.19.6-200.fc21.x86_64 Kernel for the build server (http://koji.fedoraproject.org/koji/buildinfo?buildID=632302) and it indeed fixes this issue.
I also seems to be getting better I/O performance to my QNAP iscsi LUNs as a side note. I have carried out quite a large amount of I/O to test it all out and no issues.
kernel-3.19.6-200.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/kernel-3.19.6-200.fc21
kernel-3.19.6-100.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/kernel-3.19.6-100.fc20
Package kernel-4.0.1-300.fc22: * should fix your issue, * was pushed to the Fedora 22 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing kernel-4.0.1-300.fc22' as soon as you are able to, then reboot. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-7371/kernel-4.0.1-300.fc22 then log in and leave karma (feedback).
kernel-4.0.1-300.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
kernel-3.19.7-100.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/kernel-3.19.7-100.fc20
kernel-3.19.7-200.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/kernel-3.19.7-200.fc21
kernel-3.19.7-200.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.
kernel-3.19.8-100.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/kernel-3.19.8-100.fc20
kernel-3.19.8-100.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
I'm still getting a lot of these with a Thecus N5550 NAS, and F23 - has there been any further work in this area?