Bug 1266311 - Buffer IO error when using usb flash disk as SCSI generic-device by hostdev
Buffer IO error when using usb flash disk as SCSI generic-device by hostdev
Status: CLOSED DUPLICATE of bug 1318199
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Fam Zheng
Virtualization Bugs
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-24 22:59 EDT by Han Han
Modified: 2016-07-08 03:22 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-10 04:47:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Han Han 2015-09-24 22:59:34 EDT
Description of problem:
Assign usb flash disk as guest's scsi device by hostdev, meet Buffer
IO error in guest.

Version-Release number of selected component (if applicable):
In host:
libvirt-1.2.17-10.el7.x86_64
qemu-kvm-rhev-2.3.0-26.el7.x86_64
kernel-3.10.0-319.el7.x86_64
In guest:
kernel-3.10.0-319.el7.x86_64
parted-3.1-23.el7.x86_64
libgudev1-219-16.el7.x86_64


How reproducible:
90%

Steps to Reproduce:
1. Plug in a usb flash disk in host and find it in node device.
# virsh nodedev-list --tree|grep scsi                                                                     
  |                   +- scsi_host6
  |                       +- scsi_target6_0_0
  |                           +- scsi_6_0_0_0
  |                               +- scsi_generic_sg2
  |   +- scsi_host0
  |   |   +- scsi_target0_0_0
  |   |       +- scsi_0_0_0_0
  |   |           +- scsi_generic_sg0
  |   +- scsi_host1
  |   +- scsi_host2
  |   |   +- scsi_target2_0_0
  |   |       +- scsi_2_0_0_0
  |   |           +- scsi_generic_sg1
  |   +- scsi_host3
  |   +- scsi_host4
  |   +- scsi_host5
The scsi_host6 is the usb disk device.
Add following xml in guest:
<hostdev mode='subsystem' type='scsi' managed='no'>
      <source>
        <adapter name='scsi_host6'/>
        <address bus='0' target='0' unit='0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </hostdev>

2. Start guest add do some disk operations to the usb disk(in guest):
# lsblk  /dev/sda
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    1   999M  0 disk 
├─sda1   8:1    1 476.9M  0 part 
└─sda2   8:2    1   522M  0 part 

# parted -s /dev/sda mklabel msdos
[  415.259863] blk_update_request: I/O error, dev sda, sector 128
[  415.400855] blk_update_request: I/O error, dev sda, sector 128

# parted -s /dev/sda mkpart primary ext4 512B 500M                                                                                                       
Warning: The resulting partition is not properly aligned for best performance.
[  551.508832] blk_update_request: I/O error, dev sda, sector 129
[  551.705838] blk_update_request: I/O error, dev sda, sector 128
[  551.826796] blk_update_request: I/O error, dev sda, sector 129

# mkfs.ext4 /dev/sda1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
122400 inodes, 488280 blocks
24414 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=34078720
60 block groups
8192 blocks per group, 8192 fragments per group
2040 inodes per group
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: [  609.322287] blk_update_request: I/O error, dev sda, sector 1
[  609.324735] Buffer I/O error on device sda1, logical block 0
[  609.326387] Buffer I/O error on device sda1, logical block 1
[  609.327940] Buffer I/O error on device sda1, logical block 2
[  609.329472] Buffer I/O error on device sda1, logical block 3
[  609.331008] Buffer I/O error on device sda1, logical block 4
[  609.332548] Buffer I/O error on device sda1, logical block 5
[  609.334074] Buffer I/O error on device sda1, logical block 6
[  609.335599] Buffer I/O error on device sda1, logical block 7
[  609.337089] Buffer I/O error on device sda1, logical block 8
[  609.338536] Buffer I/O error on device sda1, logical block 9
[  609.340092] blk_update_request: I/O error, dev sda, sector 253
[  609.342465] blk_update_request: I/O error, dev sda, sector 279035
[  609.344161] blk_update_request: I/O error, dev sda, sector 278783
[  609.350674] blk_update_request: I/O error, dev sda, sector 278531
[  609.352210] blk_update_request: I/O error, dev sda, sector 282311
[  609.353767] blk_update_request: I/O error, dev sda, sector 286091
[  609.355326] blk_update_request: I/O error, dev sda, sector 285839
[  609.358776] blk_update_request: I/O error, dev sda, sector 283571
[  609.360295] blk_update_request: I/O error, dev sda, sector 283319
done

# mount /dev/sda1 /mnt
[  664.747833] blk_update_request: I/O error, dev sda, sector 129
[  664.957132] EXT4-fs (sda1): ext4_check_descriptors: Block bitmap for group 0 not in group (block 0)!
[  664.959343] EXT4-fs (sda1): group descriptors corrupted!
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

# dmesg |grep -E '(Buffer|blk_update_request)'                                                                                                           
[    2.520507] blk_update_request: I/O error, dev sda, sector 128
[    2.617925] blk_update_request: I/O error, dev sda, sector 129
[    2.621404] blk_update_request: I/O error, dev sda, sector 977024
[  415.259863] blk_update_request: I/O error, dev sda, sector 128
[  415.400855] blk_update_request: I/O error, dev sda, sector 128
[  551.508832] blk_update_request: I/O error, dev sda, sector 129
[  551.705838] blk_update_request: I/O error, dev sda, sector 128
[  551.826796] blk_update_request: I/O error, dev sda, sector 129
[  609.322287] blk_update_request: I/O error, dev sda, sector 1
[  609.324735] Buffer I/O error on device sda1, logical block 0
[  609.326387] Buffer I/O error on device sda1, logical block 1
[  609.327940] Buffer I/O error on device sda1, logical block 2
[  609.329472] Buffer I/O error on device sda1, logical block 3
[  609.331008] Buffer I/O error on device sda1, logical block 4
[  609.332548] Buffer I/O error on device sda1, logical block 5
[  609.334074] Buffer I/O error on device sda1, logical block 6
[  609.335599] Buffer I/O error on device sda1, logical block 7
[  609.337089] Buffer I/O error on device sda1, logical block 8
[  609.338536] Buffer I/O error on device sda1, logical block 9
[  609.340092] blk_update_request: I/O error, dev sda, sector 253
[  609.342465] blk_update_request: I/O error, dev sda, sector 279035
[  609.344161] blk_update_request: I/O error, dev sda, sector 278783
[  609.350674] blk_update_request: I/O error, dev sda, sector 278531
[  609.352210] blk_update_request: I/O error, dev sda, sector 282311
[  609.353767] blk_update_request: I/O error, dev sda, sector 286091
[  609.355326] blk_update_request: I/O error, dev sda, sector 285839
[  609.358776] blk_update_request: I/O error, dev sda, sector 283571
[  609.360295] blk_update_request: I/O error, dev sda, sector 283319
[  664.747825] blk_update_request: 59 callbacks suppressed
[  664.747833] blk_update_request: I/O error, dev sda, sector 129

Actual Results:
As above.

Expect Results:
Doing disk operation has no error. The scsi hostdev usb disk can be mounted&read&write normally in guest.

Additional Info:
Do same operations on iscsi disk and scsi hard disk, it works well.
Comment 3 Han Han 2015-09-25 05:52:07 EDT
One more error may have relationship with the bug(the sdc is a usb disk):
1. Start a guest with following xml(the):
 <disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/sdc'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
2. In guest, do disk opterations:
# parted -s /dev/sda mklabel gpt                                                                                                                         
[  102.514706] blk_update_request: critical target error, dev sda, sector 0
[  102.516566] Buffer I/O error on device sda, logical block 0
[  102.519592] blk_update_request: critical target error, dev sda, sector 0
[  102.521252] Buffer I/O error on device sda, logical block 0
[  102.524023] blk_update_request: critical target error, dev sda, sector 0
[  102.526591] blk_update_request: critical target error, dev sda, sector 0
[  102.528274] Buffer I/O error on device sda, logical block 0
[  102.530647] blk_update_request: critical target error, dev sda, sector 0
[  102.532230] Buffer I/O error on device sda, logical block 0
[  102.534501] blk_update_request: critical target error, dev sda, sector 0
[  102.536094] Buffer I/O error on device sda, logical block 0
[  102.538344] blk_update_request: critical target error, dev sda, sector 0
[  102.539894] Buffer I/O error on device sda, logical block 0
[  102.541276] blk_update_request: critical target error, dev sda, sector 0
[  102.542820] Buffer I/O error on device sda, logical block 0
[  102.545454] blk_update_request: critical target error, dev sda, sector 0
[  102.546978] Buffer I/O error on device sda, logical block 0
[  102.548432] blk_update_request: critical target error, dev sda, sector 0
[  102.549954] Buffer I/O error on device sda, logical block 0
[  102.552065] Buffer I/O error on device sda, logical block 0
Error: Input/output error during read on /dev/sda
Error: Input/output error during write on /dev/sda
Warning: Error fsyncing/closing /dev/sda: Input/output error

# parted -s /dev/sda print                                                                                                                        
[  378.090490] blk_update_request: critical target error, dev sda, sector 0
[  378.092543] Buffer I/O error on device sda, logical block 0
[  378.095709] blk_update_request: critical target error, dev sda, sector 0
[  378.097440] Buffer I/O error on device sda, logical block 0
[  378.100452] blk_update_request: critical target error, dev sda, sector 0
[  378.103084] blk_update_request: critical target error, dev sda, sector 0
[  378.104828] Buffer I/O error on device sda, logical block 0
[  378.107182] blk_update_request: critical target error, dev sda, sector 0
[  378.108819] Buffer I/O error on device sda, logical block 0
[  378.111092] blk_update_request: critical target error, dev sda, sector 0
[  378.112739] Buffer I/O error on device sda, logical block 0
[  378.115025] blk_update_request: critical target error, dev sda, sector 0
[  378.116591] Buffer I/O error on device sda, logical block 0
[  378.118774] blk_update_request: critical target error, dev sda, sector 0
[  378.120338] Buffer I/O error on device sda, logical block 0
[  378.121799] blk_update_request: critical target error, dev sda, sector 0
[  378.123357] Buffer I/O error on device sda, logical block 0
[  378.125953] blk_update_request: critical target error, dev sda, sector 0
[  378.127435] Buffer I/O error on device sda, logical block 0
[  378.128792] Buffer I/O error on device sda, logical block 0
Error: /dev/sda: unrecognised disk label
Model: Alcor Flash Disk (scsi)
Disk /dev/sda: 1048MB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags: 

It seems more serious than what comment0 says. The mklabel operation isn't effective  while the comment0 is. 
Just as comment0, when replace the source dev to iscsi disk, it's ok to read&write.
Comment 5 Fam Zheng 2016-04-14 20:51:34 EDT
Can you try sg3_utils (e.g. sg_read, sg_write) on the scsi_host6's corresponding device from the host /dev/sdc?
Comment 6 Han Han 2016-04-18 23:33:09 EDT
On host:
# sg_read bs=512 count=2 if=/dev/sg2 
2+0 records in, SCSI READ commands issued: 1
# sg_write_buffer --in=/dev/urandom --length=512 /dev/sg2
Write buffer command not supported
# sg_write_long --in=/dev/urandom /dev/sg2               
SCSI WRITE LONG (10) command not supported

For qemu-kvm-rhev-2.5.0-4.el7.x86_64, format the hostdev scsi disk in guest:
# mkfs.ext4 /dev/sda1
The dmesg:
[   22.625681] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.631576] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.632316] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.632979] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 c4 33 e8 00 03 f0 00
[   22.633738] blk_update_request: I/O error, dev sda, sector 12858344
[   22.634429] Buffer I/O error on device sda1, logical block 1607037
[   22.635108] lost page write due to I/O error on sda1
[   22.635661] Buffer I/O error on device sda1, logical block 1607038
[   22.636348] lost page write due to I/O error on sda1
[   22.636898] Buffer I/O error on device sda1, logical block 1607039
[   22.637576] lost page write due to I/O error on sda1
[   22.638127] Buffer I/O error on device sda1, logical block 1607040
[   22.638805] lost page write due to I/O error on sda1
[   22.639356] Buffer I/O error on device sda1, logical block 1607041
[   22.640035] lost page write due to I/O error on sda1
[   22.640585] Buffer I/O error on device sda1, logical block 1607042
[   22.641267] lost page write due to I/O error on sda1
[   22.641817] Buffer I/O error on device sda1, logical block 1607043
[   22.642494] lost page write due to I/O error on sda1
[   22.643040] Buffer I/O error on device sda1, logical block 1607044
[   22.643730] lost page write due to I/O error on sda1
[   22.644284] Buffer I/O error on device sda1, logical block 1607045
[   22.644963] lost page write due to I/O error on sda1
[   22.645518] Buffer I/O error on device sda1, logical block 1607046
[   22.646200] lost page write due to I/O error on sda1
[   22.646929] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.648686] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.649388] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.650056] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 c4 2f f8 00 03 f0 00
[   22.650823] blk_update_request: I/O error, dev sda, sector 12857336
[   22.651572] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.652685] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.653383] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.654044] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 c4 2c 08 00 03 f0 00
[   22.654801] blk_update_request: I/O error, dev sda, sector 12856328
[   22.655546] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.656522] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.657362] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.658211] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 c4 28 18 00 03 f0 00
[   22.659184] blk_update_request: I/O error, dev sda, sector 12855320
[   22.660431] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.662172] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.663422] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.664592] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 23 a8 00 03 f0 00
[   22.665909] blk_update_request: I/O error, dev sda, sector 9128
[   22.667136] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.668015] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.668552] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.669044] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 1f b8 00 03 f0 00
[   22.669623] blk_update_request: I/O error, dev sda, sector 8120
[   22.670160] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.671993] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.673289] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.674488] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 1b c8 00 03 f0 00
[   22.675885] blk_update_request: I/O error, dev sda, sector 7112
[   22.677167] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.678937] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.680229] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.681465] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 17 d8 00 03 f0 00
[   22.682861] blk_update_request: I/O error, dev sda, sector 6104
[   22.684143] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.685948] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.687234] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.688434] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 13 e8 00 03 f0 00
[   22.689815] blk_update_request: I/O error, dev sda, sector 5096
[   22.691074] sd 2:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[   22.692878] sd 2:0:0:0: [sda] Sense Key : Aborted Command [current] 
[   22.694174] sd 2:0:0:0: [sda] Add. Sense: I/O process terminated
[   22.695378] sd 2:0:0:0: [sda] CDB: Write(10) 2a 00 00 00 0f f8 00 03 f0 00
[   22.696763] blk_update_request: I/O error, dev sda, sector 4088
[   22.989380] ------------[ cut here ]------------
[   22.990146] WARNING: at fs/block_dev.c:67 bdev_inode_switch_bdi+0x7a/0x90()
[   22.991054] Modules linked in: ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device ppdev snd_pcm sg snd_timer pcspkr virtio_balloon snd soundcore i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi virtio_blk virtio_console virtio_scsi 8139too qxl syscopyarea
[   22.999614]  sysfillrect sysimgblt drm_kms_helper ttm ata_piix drm libata serio_raw 8139cp mii virtio_pci virtio_ring virtio i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
[   23.001471] CPU: 0 PID: 2377 Comm: mkfs.ext4 Not tainted 3.10.0-366.el7.x86_64 #1
[   23.002322] Hardware name: Red Hat KVM, BIOS 1.9.1-2.el7 04/01/2014
[   23.003005]  0000000000000000 000000006aca0131 ffff88003ba27db0 ffffffff81646497
[   23.003866]  ffff88003ba27de8 ffffffff8107f3e0 ffff88003dc25130 ffff88003dc251b8
[   23.004736]  ffffffff819dc940 000000000000009f 0000000000000000 ffff88003ba27df8
[   23.005592] Call Trace:
[   23.005879]  [<ffffffff81646497>] dump_stack+0x19/0x1b
[   23.006445]  [<ffffffff8107f3e0>] warn_slowpath_common+0x70/0xb0
[   23.007114]  [<ffffffff8107f52a>] warn_slowpath_null+0x1a/0x20
[   23.007753]  [<ffffffff812226ba>] bdev_inode_switch_bdi+0x7a/0x90
[   23.008425]  [<ffffffff81223284>] __blkdev_put+0x74/0x1a0
[   23.009016]  [<ffffffff81223cfe>] blkdev_put+0x4e/0x140
[   23.009598]  [<ffffffff81223ea5>] blkdev_close+0x25/0x30
[   23.010189]  [<ffffffff811e9319>] __fput+0xe9/0x260
[   23.010726]  [<ffffffff811e95ce>] ____fput+0xe/0x10
[   23.011271]  [<ffffffff810a64c7>] task_work_run+0xa7/0xe0
[   23.011867]  [<ffffffff81014b12>] do_notify_resume+0x92/0xb0
[   23.012499]  [<ffffffff81656b7d>] int_signal+0x12/0x17
[   23.013064] ---[ end trace efccf90fd91908e5 ]---
[   36.895488]  sda:
[   49.632851]  sda:
Comment 7 Fam Zheng 2016-04-19 09:04:07 EDT
This means the host device is incapable of handling scsi requests. Please test with disks that at least _work_ with sg3_utils.
Comment 8 Gerd Hoffmann 2016-04-21 10:36:24 EDT
(In reply to Fam Zheng from comment #7)
> This means the host device is incapable of handling scsi requests. Please
> test with disks that at least _work_ with sg3_utils.

Most likely the usb scsi host has a pretty crappy scsi implementation and possibly supports only READ(6) and WRITE(6).

/me suggests to pass through the usb flash drive using usb pass-through instead of scsi pass-through.  Has the advantage that the guest will actually see it's backed by usb (and which usb scsi host it is) and can apply quirks accordingly.

[ explicitly not vetoing the closed/notabug here ]
Comment 9 Fam Zheng 2016-04-21 11:36:39 EDT
I checked the strace on qemu-kvm which said the issued WRITE (10) request is invalid in size (namely 0x3f0 blocks). I still need to check why guest decides to issue this size (or whether it makes sense. I can only tell sg_raw is refusing such a size), or why it's rejected by usb host scsi.
Comment 10 yisun 2016-04-24 22:35:27 EDT
Hi Fam, 
Per we talked in IRC, you may still need sometime to debug this issue. Do you think it's worth to reopen this bug for now? since we need to track it somewhere.
Comment 11 Fam Zheng 2016-05-10 04:47:27 EDT

*** This bug has been marked as a duplicate of bug 1318199 ***

Note You need to log in before you can comment on or make changes to this bug.