Bug 1218882

Summary: liotarget: "fd_do_rw() write returned -22", iSCSI not functioning
Product: [Fedora] Fedora Reporter: Tomasz Torcz <tomek>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: agrover, gansalmon, green, itamar, jonathan, kernel-maint, madhu.chinakonda, mchehab, panormitis, watanabe.yu
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-4.0.5-300.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1298281 (view as bug list) Environment:
Last Closed: 2015-06-20 23:58:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1298281    
Attachments:
Description Flags
saveconfig.json none

Description Tomasz Torcz 2015-05-06 06:44:15 UTC
Description of problem:
Since some time iSCSI target is not functioning. After some initial I/O, target throws "fd_do_rw() write returned -22" and initiator throws IO error.
Detailed configuration steps follow.

TARGET (iqn.2003-01.org.linux-iscsi.seurat.x8664, 192.168.22.1) :
Rootfs is btrfs (rw,relatime,seclabel,compress=zlib,ssd,space_cache,autodefrag)
1. truncate -s 800M /iscsi.bs0
2. targetcli:
  cd backstores/fileio
  create file_or_dev=/iscsi.bs0 name=fbs0
  cd /iscsi
  create
  cd iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7/tpg1 \
  cd luns
  create /backstores/fileio/fbs0
  cd /iscsi/iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7/tpg1/acls
  create iqn.2003-01.org.linux-iscsi.ginjo.x8664:sn.37648a3d9cbe

  ls /
o- / ............................................................................ [...]
  o- backstores ................................................................. [...]
  | o- block ..................................................... [Storage Objects: 0]
  | o- fileio .................................................... [Storage Objects: 1]
  | | o- fbs0 ............................ [/iscsi.bs0 (800.0MiB) write-back activated]
  | o- pscsi ..................................................... [Storage Objects: 0]
  | o- ramdisk ................................................... [Storage Objects: 0]
  | o- user ...................................................... [Storage Objects: 0]
  o- iscsi ............................................................... [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7 ............. [TPGs: 1]
  |   o- tpg1 .................................................. [no-gen-acls, no-auth]
  |     o- acls ............................................................. [ACLs: 1]
  |     | o- iqn.2003-01.org.linux-iscsi.ginjo.x8664:sn.37648a3d9cbe . [Mapped LUNs: 1]
  |     |   o- mapped_lun0 .................................... [lun0 fileio/fbs0 (rw)]
  |     o- luns ............................................................. [LUNs: 1]
  |     | o- lun0 .......................................... [fileio/fbs0 (/iscsi.bs0)]
  |     o- portals ....................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ........................................................ [OK]




INITIATOR (iqn.2003-01.org.linux-iscsi.ginjo.x8664:sn.37648a3d9cbe, 192.168.22.3) :
1. iscsiadm -m discovery -t st -p 192.168.22.1
   192.168.22.1:3260,1 iqn.2003-01.org.linux-iscsi.seurat.x8664:sn.513490d29dc7
2. iscsiadm -m node -l
[May 6 08:05] scsi host4: iSCSI Initiator over TCP/IP
[  +0.041362] scsi 4:0:0:0: Direct-Access     LIO-ORG  fbs0             4.0  PQ: 0 ANSI: 5
[  +0.028042] sd 4:0:0:0: Attached scsi generic sg2 type 0
[  +0.014333] sd 4:0:0:0: [sdb] 1638400 512-byte logical blocks: (838 MB/800 MiB)
[  +0.010611] sd 4:0:0:0: [sdb] Write Protect is off
[  +0.003731] sd 4:0:0:0: [sdb] Mode Sense: 43 00 10 08
[  +0.013365] sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.083011] sd 4:0:0:0: [sdb] Attached SCSI disk

3. mkfs.btrfs -M -L seurat-iscsi /dev/sdb
4. btrfstune -x /dev/sdb

5. mount /dev/sdb /mnt/tmp -o autodefrag,compress=lzo
[May 6 08:06] BTRFS: device label seurat-iscsi devid 1 transid 3 /dev/sdb
[ +22.736061] BTRFS info (device sdb): disk space caching is enabled
[  +0.003400] BTRFS: has skinny extents
[  +0.010614] BTRFS: creating UUID tree
[May 6 08:07] BTRFS info (device sdb): enabling auto defrag
[  +0.005219] BTRFS info (device sdb): setting 8 feature flag
[  +0.003252] BTRFS info (device sdb): disk space caching is enabled
(there are two mounts in the log snippet above)


NOW START THE I/O. Soon:

INITIATOR:
[May 6 08:08] fd_do_rw() write returned -22
[  +0.107407] fd_do_rw() write returned -22


TARGET:
[May 6 08:08] sd 4:0:0:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  +0.004159] sd 4:0:0:0: [sdb] Sense Key : Not Ready [current] 
[  +0.001032] sd 4:0:0:0: [sdb] Add. Sense: Logical unit communication failure
[  +0.003389] sd 4:0:0:0: [sdb] CDB: Write(10) 2a 00 00 01 6b 20 00 2b e0 00
[  +0.003223] blk_update_request: I/O error, dev sdb, sector 92960
[  +0.001409] BTRFS: bdev /dev/sdb errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[  +0.002002] BTRFS: bdev /dev/sdb errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[  +0.000759] BTRFS: bdev /dev/sdb errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
[  +0.001236] BTRFS: bdev /dev/sdb errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
[  +0.000801] BTRFS: bdev /dev/sdb errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
[  +0.000644] BTRFS: bdev /dev/sdb errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
[  +0.088333] sd 4:0:0:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  +0.000003] sd 4:0:0:0: [sdb] Sense Key : Not Ready [current] 
[  +0.000003] sd 4:0:0:0: [sdb] Add. Sense: Logical unit communication failure
[  +0.000003] sd 4:0:0:0: [sdb] CDB: Write(10) 2a 00 00 00 a0 00 00 3b 20 00
[  +0.000003] blk_update_request: I/O error, dev sdb, sector 40960
[  +0.000006] BTRFS: bdev /dev/sdb errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
[  +0.000195] BTRFS: bdev /dev/sdb errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
[  +0.000085] BTRFS: bdev /dev/sdb errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
[  +0.000082] BTRFS: bdev /dev/sdb errs: wr 10, rd 0, flush 0, corrupt 0, gen 0

Application exits with I/O errors, btrfs sometimes switches to read-only.



Version-Release number of selected component (if applicable):
TARGET:
kernel-core-4.0.1-300.fc22.x86_64
targetcli-2.1.fb40-1.fc21.noarch

INITIATOR:
kernel-core-4.1.0-0.rc1.git1.1.fc23.x86_64
iscsi-initiator-utils-6.2.0.873-25.gitc9d830b.fc22.x86_64

How reproducible:
I've reproduced it twice on different hosts/VMs. In both cases initiator/target are directly connected: a) target on physical host, initiator is a VM hosted on this host - KVM is used; b) target and initiator are VMs hosted on the same machine, VirtualBox is used.

Comment 1 Tomasz Torcz 2015-05-06 06:47:10 UTC
Created attachment 1022488 [details]
saveconfig.json

Comment 2 Oleg Drokin 2015-06-01 22:08:33 UTC
I think I met the same problem.

So,e digging around and I decided to give a try to this patch: https://git.kernel.org/cgit/linux/kernel/git/gregkh/staging.git/commit/?h=staging-testing&id=6b9a44d0939ca6235c72b811bd55b462d6a0a553 

With this in-place my problems seems to be all gone now, or so it seems.

If you wnat to give it a try, but don't want to build the RPM yourself, here is the one I built: http://crimea.edu/~green/kernel/

Comment 3 Oleg Drokin 2015-06-01 22:14:32 UTC
I just want to add that the patch is to the target, not to initiator. so the patched rpm is that of the patched current F22 kernel.

Comment 4 Fedora Update System 2015-06-09 12:57:28 UTC
kernel-4.0.5-300.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/kernel-4.0.5-300.fc22

Comment 5 Fedora Update System 2015-06-09 12:59:32 UTC
kernel-4.0.5-200.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/kernel-4.0.5-200.fc21

Comment 6 Fedora Update System 2015-06-10 19:09:05 UTC
Package kernel-4.0.5-200.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing kernel-4.0.5-200.fc21'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-9704/kernel-4.0.5-200.fc21
then log in and leave karma (feedback).

Comment 7 Fedora Update System 2015-06-20 23:58:34 UTC
kernel-4.0.5-200.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 8 Fedora Update System 2015-06-21 00:31:28 UTC
kernel-4.0.5-300.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.