Hide Forgot
Description of problem: Add one shareable sg device in 2 shutoff guests, start 1st one, then it can be started up normally, the read and write are working well. Start 2nd guest, then the shareable sg device didn't work well in the 1st guest. Version-Release number of selected component (if applicable): libvirt-1.1.1-29.el7.x86_64 qemu-kvm-rhev-1.5.3-59.el7ev.x86_64 kernel-3.10.0-115.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. make sure the sg module is probed. # lsmod|grep sg sg 36533 0 2. Add the shareable sg device in 2 shutoff guests. ...... <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <shareable> </hostdev> ...... 3. start the 1st geust # virsh start rhel7-2 Domain rhel7-2 started 4. login the guest, the read and write to the sg device is working well. In guest: 4.1 can find the sg device. Disk /dev/sda: 500.1 GB, 500107862016 bytes, 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x56d97d69 Device Boot Start End Blocks Id System /dev/sda1 2048 2099199 1048576 83 Linux 4.2. mount, then read and write to the sg device. # mount /dev/sda1 /mnt/ # ls /mnt/ 1 2 lost+found # touch /mnt/new # ls /mnt/ 1 2 lost+found new 5. start the 2nd guest. # virsh start rhel7-xuzhang Domain rhel7-xuzhang started 6. then the 1st guest report error "fdisk: cannot open /dev/sda: Input/output error". In 1st guest: # fdisk -l Disk /dev/sdb: 8589 MB, 8589934592 bytes, 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x0004f78e Device Boot Start End Blocks Id System /dev/sdb1 * 2048 1026047 512000 83 Linux /dev/sdb2 1026048 16777215 7875584 8e Linux LVM fdisk: cannot open /dev/sda: Input/output error Actual results: As step 6 Expected results: the shareable sg device should be working well on both the 2 guests. Additional info: If using the whole sg device in the 2 guests, this error can not appear. Only will meet this issue while using the partition in the 2 guests.
Some other info for your reference. 1. This can not be reproduce while disable the selinux. # getenforce Permissive 2. the selinux label will be changed after start the 2nd guest. 2.1 original selinux label of the shareable sg device. # ll -Z /dev/sg0 crw-rw----. root root system_u:object_r:scsi_generic_device_t:s0 /dev/sg0 2.2 start 1st guest, check the shareable sg device selinux label. # virsh start rhel7-2 Domain rhel7-2 started # ll -Z /dev/sg0 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c727,c793 /dev/sg0 2.3 start 2nd guest, the selinux label is changed. # virsh start rhel7-xuzhang Domain rhel7-xuzhang started # ll -Z /dev/sg0 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c219,c388 /dev/sg0
(In reply to Zhang Xuesong from comment #1) > Some other info for your reference. > 1. This can not be reproduce while disable the selinux. Correct one typo, not "disable the selinux", it should be "permissive mode" > # getenforce > Permissive > > 2. the selinux label will be changed after start the 2nd guest. > 2.1 original selinux label of the shareable sg device. > # ll -Z /dev/sg0 > crw-rw----. root root system_u:object_r:scsi_generic_device_t:s0 /dev/sg0 > > 2.2 start 1st guest, check the shareable sg device selinux label. > # virsh start rhel7-2 > Domain rhel7-2 started > > # ll -Z /dev/sg0 > crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c727,c793 /dev/sg0 > > 2.3 start 2nd guest, the selinux label is changed. > # virsh start rhel7-xuzhang > Domain rhel7-xuzhang started > > # ll -Z /dev/sg0 > crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c219,c388 /dev/sg0
In order to help more quickly reproduce, set things up, etc. - providing a few more details on the configuration would be helpful. The following: Steps to Reproduce: 1. make sure the sg module is probed. # lsmod|grep sg sg 36533 0 2. Add the shareable sg device in 2 shutoff guests. ...... <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <shareable> </hostdev> ...... Is very vague. Missing is what you did to make sure 'sg' is loaded for lsmod to find it, how you went about creating the shareable device that 'scsi_host0' is using. Is it an iSCSI device? Is it a /dev/sdxn (eg, /dev/sda3)? Also I assume by the 'hostdev' XML that this only fails for host devices? Do you know if it works when the 'shared' disk is defined within a <disk...> element? Being 7 months after first reported - I have to wonder if this works now as there have been many changes. Getting a similar configuration knowing details of how you set things up just makes it easier for me to recreate.
(In reply to John Ferlan from comment #3) > In order to help more quickly reproduce, set things up, etc. - providing a > few more details on the configuration would be helpful. The following: > > Steps to Reproduce: > 1. make sure the sg module is probed. > # lsmod|grep sg > sg 36533 0 > > 2. Add the shareable sg device in 2 shutoff guests. > ...... > <hostdev mode='subsystem' type='scsi' managed='no'> > <source> > <adapter name='scsi_host0'/> > <address bus='0' target='0' unit='0'/> > </source> > <shareable> > </hostdev> > ...... > > > Is very vague. Missing is what you did to make sure 'sg' is loaded for > lsmod to find it, how you went about creating the shareable device that > 'scsi_host0' is using. Is it an iSCSI device? Is it a /dev/sdxn (eg, > /dev/sda3)? Here is the details prepared steps for your reference: 1. probe the sg module # modprobe sg # lsmod |grep sg sg 40721 0 2. checking the sg devices on my host, '/dev/sda' is my boot disk, the '/dev/sdb' is my data disk. So, I'd like to use the data disk "/dev/sg2" for the shared sg device in the following steps. # lsscsi -g [0:0:0:0] disk ATA WDC WD5000AAKS-7 05.0 /dev/sda /dev/sg0 [1:0:0:0] cd/dvd PLDS DVD+-RW DH-16AAS JD12 /dev/sr0 /dev/sg1 [2:0:0:0] disk ATA WDC WD3200AAKS-7 01.0 /dev/sdb /dev/sg2 [7:0:0:0] storage IET Controller 0001 - /dev/sg3 [7:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdd /dev/sg4 3. checking the adapter name of sg device /dev/sg2, it's "scsi_host2". # virsh nodedev-list --tree ...... | +- scsi_host2 | | | | | +- scsi_target2_0_0 | | | | | +- scsi_2_0_0_0 | | | | | +- block_sdb_WDC_WD3200AAKS_75L9A0_WD_WMAV26572317 | | +- scsi_generic_sg2 ...... 4. add the following shareable sg disk to 2 shutoff guests. <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host0'/> <address bus='0' target='0' unit='0'/> </source> <shareable/> </hostdev> So, the following steps, you can reference the bug description. > > Also I assume by the 'hostdev' XML that this only fails for host devices? > Do you know if it works when the 'shared' disk is defined within a <disk...> The shared disk is working well with the following disk element: <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='none' io='native'/> <source dev='/dev/sdb'/> <backingStore/> <target dev='hdb' bus='ide'/> <shareable/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> > element? Being 7 months after first reported - I have to wonder if this > works now as there have been many changes. Getting a similar configuration I tried just now, this issue can be produced with the latest libvirt + qemu-kvm-rhev + kernel. > knowing details of how you set things up just makes it easier for me to > recreate.
(In reply to Zhang Xuesong from comment #4) > (In reply to John Ferlan from comment #3) > > In order to help more quickly reproduce, set things up, etc. - providing a > > few more details on the configuration would be helpful. The following: > > > > Steps to Reproduce: > > 1. make sure the sg module is probed. > > # lsmod|grep sg > > sg 36533 0 > > > > 2. Add the shareable sg device in 2 shutoff guests. > > ...... > > <hostdev mode='subsystem' type='scsi' managed='no'> > > <source> > > <adapter name='scsi_host0'/> Sorry, the adapter name should be the following one. <adapter name='scsi_host2'/> > > <address bus='0' target='0' unit='0'/> > > </source> > > <shareable> > > </hostdev> > > ...... > > > > > > Is very vague. Missing is what you did to make sure 'sg' is loaded for > > lsmod to find it, how you went about creating the shareable device that > > 'scsi_host0' is using. Is it an iSCSI device? Is it a /dev/sdxn (eg, > > /dev/sda3)? > > Here is the details prepared steps for your reference: > 1. probe the sg module > # modprobe sg > # lsmod |grep sg > sg 40721 0 > > 2. checking the sg devices on my host, '/dev/sda' is my boot disk, the > '/dev/sdb' is my data disk. So, I'd like to use the data disk "/dev/sg2" for > the shared sg device in the following steps. > # lsscsi -g > [0:0:0:0] disk ATA WDC WD5000AAKS-7 05.0 /dev/sda /dev/sg0 > [1:0:0:0] cd/dvd PLDS DVD+-RW DH-16AAS JD12 /dev/sr0 /dev/sg1 > [2:0:0:0] disk ATA WDC WD3200AAKS-7 01.0 /dev/sdb /dev/sg2 > [7:0:0:0] storage IET Controller 0001 - /dev/sg3 > [7:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdd /dev/sg4 > > 3. checking the adapter name of sg device /dev/sg2, it's "scsi_host2". > # virsh nodedev-list --tree > ...... > | +- scsi_host2 > | | | > | | +- scsi_target2_0_0 > | | | > | | +- scsi_2_0_0_0 > | | | > | | +- block_sdb_WDC_WD3200AAKS_75L9A0_WD_WMAV26572317 > | | +- scsi_generic_sg2 > ...... > > 4. add the following shareable sg disk to 2 shutoff guests. > <hostdev mode='subsystem' type='scsi' managed='no'> > <source> > <adapter name='scsi_host0'/> > <address bus='0' target='0' unit='0'/> > </source> > <shareable/> > </hostdev> > > > So, the following steps, you can reference the bug description. > > > > > Also I assume by the 'hostdev' XML that this only fails for host devices? > > Do you know if it works when the 'shared' disk is defined within a <disk...> > > The shared disk is working well with the following disk element: > <disk type='block' device='disk'> > <driver name='qemu' type='raw' cache='none' io='native'/> > <source dev='/dev/sdb'/> > <backingStore/> > <target dev='hdb' bus='ide'/> > <shareable/> > <alias name='ide0-0-1'/> > <address type='drive' controller='0' bus='0' target='0' unit='1'/> > </disk> > > > > element? Being 7 months after first reported - I have to wonder if this > > works now as there have been many changes. Getting a similar configuration > > I tried just now, this issue can be produced with the latest libvirt + > qemu-kvm-rhev + kernel. > > > knowing details of how you set things up just makes it easier for me to > > recreate.
OK - great thanks for the details and ensuring that the <disk> works just fine. That narrows the frame of reference for the issue to just when a disk/lun is being used for hostdev passthru. Of course I don't have a second physical disk, but that shouldn't matter since this is more of a hostdev for sg type issue...
(In reply to John Ferlan from comment #6) > OK - great thanks for the details and ensuring that the <disk> works just > fine. That narrows the frame of reference for the issue to just when a > disk/lun is being used for hostdev passthru. > > Of course I don't have a second physical disk, but that shouldn't matter > since this is more of a hostdev for sg type issue... I think you can use the USB for the second physical disk, it's also ok. Such as the the "/dev/sg3" in following example, it is one USB device on my host. # lsscsi -g [0:0:0:0] disk ATA ST3250310AS 3.AD /dev/sda /dev/sg0 [1:0:0:0] cd/dvd PLDS DVD+-RW DH-16AAS JD12 /dev/sr0 /dev/sg1 [2:0:0:0] disk ATA WDC WD5000AAKS-7 05.0 /dev/sdb /dev/sg2 [6:0:0:0] disk SanDisk Cruzer Force 1.26 /dev/sdc /dev/sg3
I have the following: # lsscsi -g [0:0:0:0] disk ATA HGST HTS725032A7 B550 /dev/sda /dev/sg0 [1:0:0:0] cd/dvd PLDS DVD-RW DS8A8SH KU51 /dev/sr0 /dev/sg1 [7:0:0:0] storage IET Controller 0001 - /dev/sg2 [7:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdb /dev/sg5 [7:0:0:2] disk IET VIRTUAL-DISK 0001 /dev/sde /dev/sg8 [10:0:0:0] storage IET Controller 0001 - /dev/sg3 [10:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdc /dev/sg6 [10:0:0:2] disk IET VIRTUAL-DISK 0001 /dev/sdg /dev/sg10 [11:0:0:0] storage IET Controller 0001 - /dev/sg4 [11:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdd /dev/sg7 [11:0:0:2] disk IET VIRTUAL-DISK 0001 /dev/sdf /dev/sg9 What I did to test was "use" one of the iSCSI virtual disks that I created, but created a scsi hostdev to reference it <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host7'/> <address bus='0' target='0' unit='1'/> </source> <shareable/> </hostdev> I didn't see the same issue as described in this bug with "Permissive" mode, but if I go with "Enforcing", then I saw the issue. I've created a potential fix which keeps the volume visible for both guests, but I'm troubled with another aspect over the contents of the volume. Creating files has some odd interactions, but I think that just could be because the backend is an iSCSI rather than what it "should" be. I didn't restart iSCSI server in between changing selinux enforcement... One thing at a time I guess. I'll be putting together and posting a patch today...
(In reply to John Ferlan from comment #8) > I have the following: > > # lsscsi -g > [0:0:0:0] disk ATA HGST HTS725032A7 B550 /dev/sda /dev/sg0 > [1:0:0:0] cd/dvd PLDS DVD-RW DS8A8SH KU51 /dev/sr0 /dev/sg1 > [7:0:0:0] storage IET Controller 0001 - /dev/sg2 > [7:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdb /dev/sg5 > [7:0:0:2] disk IET VIRTUAL-DISK 0001 /dev/sde /dev/sg8 > [10:0:0:0] storage IET Controller 0001 - /dev/sg3 > [10:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdc /dev/sg6 > [10:0:0:2] disk IET VIRTUAL-DISK 0001 /dev/sdg /dev/sg10 > [11:0:0:0] storage IET Controller 0001 - /dev/sg4 > [11:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdd /dev/sg7 > [11:0:0:2] disk IET VIRTUAL-DISK 0001 /dev/sdf /dev/sg9 > > > What I did to test was "use" one of the iSCSI virtual disks that I created, > but created a scsi hostdev to reference it > > <hostdev mode='subsystem' type='scsi' managed='no'> > <source> > <adapter name='scsi_host7'/> > <address bus='0' target='0' unit='1'/> > </source> > <shareable/> > </hostdev> > > > I didn't see the same issue as described in this bug with "Permissive" mode, > but if I go with "Enforcing", then I saw the issue. Yes, this issue only occurred while selinux is "Enforcing", the selinux context is changed after start the 2nd guests. Note: the 2 guests use the same sg disk. Step 1: the original selinux context while the 2 guest are shutoff status. # ll -Z /dev/sg3 crw-rw----. root disk system_u:object_r:scsi_generic_device_t:s0 /dev/sg3 Step 2: start the 1st guest, note the selinux context of sg disk. # virsh start rhel7.1 Domain rhel7.1 started # ll -Z /dev/sg3 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c82,c240 /dev/sg3 Step 3: start the 2nd guest, the selinux contest is changed. # virsh start r7.0-2 Domain r7.0-2 started # ll -Z /dev/sg3 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c492,c673 /dev/sg3 > > I've created a potential fix which keeps the volume visible for both guests, > but I'm troubled with another aspect over the contents of the volume. > Creating files has some odd interactions, but I think that just could be > because the backend is an iSCSI rather than what it "should" be. I didn't > restart iSCSI server in between changing selinux enforcement... One thing > at a time I guess. > > I'll be putting together and posting a patch today...
I can reproduce it using libvirt-1.2.8-10.el7.x86_64. Steps: 1. On host. [root@intel-q9550-4-1 ~]# rpm -q libvirt qemu-kvm-rhev libvirt-1.2.8-10.el7.x86_64 qemu-kvm-rhev-2.1.2-17.el7.x86_64 [root@intel-q9550-4-1 ~]# getenforce Enforcing [root@intel-q9550-4-1 ~]# lsscsi [0:0:0:0] disk ATA Hitachi HDP72503 GM3O /dev/sda [7:0:0:0] storage IET Controller 0001 - [7:0:0:1] tape IET VIRTUAL-TAPE 0001 /dev/st0 [10:0:0:0] disk EQLOGIC 100E-00 5.0 /dev/sdb [13:0:0:0] disk EQLOGIC 100E-00 5.0 /dev/sdc [14:0:0:0] disk EQLOGIC 100E-00 5.0 /dev/sdd [24:0:0:0] storage IET Controller 0001 - [24:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sde [25:0:0:0] storage IET Controller 0001 - [25:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdf <== using this iSCSI virtual disk [root@intel-q9550-4-1 ~]# cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: Hitachi HDP72503 Rev: GM3O Type: Direct-Access ANSI SCSI revision: 05 Host: scsi7 Channel: 00 Id: 00 Lun: 00 Vendor: IET Model: Controller Rev: 0001 Type: RAID ANSI SCSI revision: 05 Host: scsi7 Channel: 00 Id: 00 Lun: 01 Vendor: IET Model: VIRTUAL-TAPE Rev: 0001 Type: Sequential-Access ANSI SCSI revision: 05 Host: scsi10 Channel: 00 Id: 00 Lun: 00 Vendor: EQLOGIC Model: 100E-00 Rev: 5.0 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi13 Channel: 00 Id: 00 Lun: 00 Vendor: EQLOGIC Model: 100E-00 Rev: 5.0 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi14 Channel: 00 Id: 00 Lun: 00 Vendor: EQLOGIC Model: 100E-00 Rev: 5.0 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi24 Channel: 00 Id: 00 Lun: 00 Vendor: IET Model: Controller Rev: 0001 Type: RAID ANSI SCSI revision: 05 Host: scsi25 Channel: 00 Id: 00 Lun: 00 Vendor: IET Model: Controller Rev: 0001 Type: RAID ANSI SCSI revision: 05 Host: scsi24 Channel: 00 Id: 00 Lun: 01 Vendor: IET Model: VIRTUAL-DISK Rev: 0001 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi25 Channel: 00 Id: 00 Lun: 01 Vendor: IET Model: VIRTUAL-DISK Rev: 0001 Type: Direct-Access ANSI SCSI revision: 05 2. Starting guest 'a'. [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root disk system_u:object_r:scsi_generic_device_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh start a Domain a started [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c112,c318 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh dumpxml a | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <shareable/> <alias name='hostdev0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c112,c318</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c112,c318</imagelabel> </seclabel> </domain> 2. Login guest 'a'. The read/write to the sg device is working well. [root@localhost ~]# fdisk -l ... Disk /dev/sdb: 104 MB, 104857600 bytes, 204800 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xa8663ade Device Boot Start End Blocks Id System /dev/sdb1 2048 204799 101376 83 Linux [root@localhost ~]# mount /dev/sdb1 /mnt mount: /dev/sdb1 is write-protected, mounting read-only mount: unknown filesystem type '(null)' [root@localhost ~]# cd /mnt/ [root@localhost mnt]# ls -al total 4 drwxr-xr-x. 2 root root 6 Mar 13 2014 . dr-xr-xr-x. 17 root root 4096 Nov 24 12:02 .. [root@localhost mnt]# touch newfile [root@localhost mnt]# ls -al total 4 drwxr-xr-x. 2 root root 20 Dec 29 15:19 . dr-xr-xr-x. 17 root root 4096 Nov 24 12:02 .. -rw-r--r--. 1 root root 0 Dec 29 15:19 newfile [root@localhost mnt]# echo "12345" > newfile [root@localhost mnt]# cat newfile 12345 3. Starting guest 'r7'. [root@intel-q9550-4-1 ~]# virsh start r7 Domain r7 started [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c654,c779 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh dumpxml r7 | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <shareable/> <alias name='hostdev0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c654,c779</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c654,c779</imagelabel> </seclabel> </domain> 4. The guest 'a' report error. [root@localhost mnt]# fdisk -l Disk /dev/sda: 8589 MB, 8589934592 bytes, 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x00094fa7 Device Boot Start End Blocks Id System /dev/sda1 * 2048 1026047 512000 83 Linux /dev/sda2 1026048 16777215 7875584 8e Linux LVM Disk /dev/mapper/rhel-root: 7159 MB, 7159676928 bytes, 13983744 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/rhel-swap: 859 MB, 859832320 bytes, 1679360 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes [ 519.664201] end_request: I/O error, dev sdb, sector 0 [ 519.665050] Buffer I/O error on device sdb, logical block 0 [ 519.698167] end_request: I/O error, dev sdb, sector 0 [ 519.698980] Buffer I/O error on device sdb, logical block 0 [ 519.731179] end_request: I/O error, dev sdb, sector 0 [ 519.731998] Buffer I/O error on device sdb, logical block 0 [ 519.764193] end_request: I/O error, dev sdb, sector 0 [ 519.765001] Buffer I/O error on device sdb, logical block 0 [ 519.797173] end_request: I/O error, dev sdb, sector 0 [ 519.797992] Buffer I/O error on device sdb, logical block 0 fdisk: cannot open /dev/sdb: Input/output error
Verify it as follows. Steps: [root@intel-q9550-4-1 ~]# rpm -q libvirt qemu-kvm-rhev libvirt-1.2.8-11.el7.x86_64 qemu-kvm-rhev-2.1.2-17.el7.x86_64 [root@intel-q9550-4-1 ~]# getenforce Enforcing [root@intel-q9550-4-1 ~]# lsscsi ... [25:0:0:1] disk IET VIRTUAL-DISK 0001 /dev/sdf Scenario 1: Add 'shareable' sg device in 2 guests. 1. Start guest 'a'. [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:scsi_generic_device_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh start a Domain a started [root@intel-q9550-4-1 ~]# virsh dumpxml a | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <shareable/> <alias name='hostdev0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c129,c351</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c129,c351</imagelabel> </seclabel> </domain> [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0 /dev/sg9 2. Login guest 'a'. The read/write to the sg device is working well. [root@localhost ~]# fdisk -l ... Disk /dev/sdb: 104 MB, 104857600 bytes, 204800 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xa8663ade Device Boot Start End Blocks Id System /dev/sdb1 2048 204799 101376 83 Linux [root@localhost ~]# mount /dev/sdb1 /mnt mount: /dev/sdb1 is write-protected, mounting read-only mount: unknown filesystem type '(null)' [root@localhost ~]# cd /mnt/ [root@localhost mnt]# ls newfile [root@localhost mnt]# ls -al total 8 drwxr-xr-x. 2 root root 20 Dec 29 15:19 . dr-xr-xr-x. 17 root root 4096 Nov 24 12:02 .. -rw-r--r--. 1 root root 6 Dec 29 15:19 newfile [root@localhost mnt]# echo "67890" >> newfile [root@localhost mnt]# cat newfile 12345 67890 3. Start guest 'r7'. [root@intel-q9550-4-1 ~]# virsh start r7 Domain r7 started [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh dumpxml r7 | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <shareable/> <alias name='hostdev0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c114,c669</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c114,c669</imagelabel> </seclabel> </domain> 4. Login guest 'a'. Don't report error. The read/write to the sg device is working well. [root@localhost mnt]# fdisk -l ... Disk /dev/sdb: 104 MB, 104857600 bytes, 204800 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xa8663ade Device Boot Start End Blocks Id System /dev/sdb1 2048 204799 101376 83 Linu 5. Login guest 'r7'. [root@localhost ~]# fdisk -l ... Disk /dev/sdb: 104 MB, 104857600 bytes, 204800 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xa8663ade Device Boot Start End Blocks Id System /dev/sdb1 2048 204799 101376 83 Linux 6. Destroy guest 'a' and 'r7'. [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh destroy a Domain a destroyed [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:svirt_image_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh destroy r7 Domain r7 destroyed [root@intel-q9550-4-1 ~]# virsh list --all Id Name State ---------------------------------------------------- - a shut off - r7 shut off [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:svirt_image_t:s0 /dev/sg9 ----------------------------------------------------------------- Scenario 2: Add 'readonly' sg device in 2 guests. [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:scsi_generic_device_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh start a error: Failed to start domain a error: internal error: process exited while connecting to monitor: 2014-12-29T08:52:15.631208Z qemu-kvm: -drive file=/dev/sg9,if=none,id=drive-hostdev0,readonly=on: could not open disk image /dev/sg9: Could not open '/dev/sg9': Permission denied [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:virt_content_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh dumpxml a | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... [root@intel-q9550-4-1 ~]# virsh start r7 error: Failed to start domain r7 error: internal error: process exited while connecting to monitor: 2014-12-29T08:57:27.170771Z qemu-kvm: -drive file=/dev/sg9,if=none,id=drive-hostdev0,readonly=on: could not open disk image /dev/sg9: Could not open '/dev/sg9': Permission denied [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:virt_content_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh dumpxml r7 | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... ----------------------------------------------------------- Scenario 3: Don't add 'readonly' or 'shareable' sg device in 2 guests. [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:scsi_generic_device_t:s0 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh start a Domain a started [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. qemu qemu system_u:object_r:svirt_image_t:s0:c260,c302 /dev/sg9 [root@intel-q9550-4-1 ~]# virsh dumpxml a | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <alias name='hostdev0'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c260,c302</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c260,c302</imagelabel> </seclabel> </domain> [root@intel-q9550-4-1 ~]# virsh dumpxml r7 | grep hostdev -A10 <hostdev mode='subsystem' type='scsi' managed='no'> <source> <adapter name='scsi_host25'/> <address bus='0' target='0' unit='1'/> </source> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </hostdev> ... [root@intel-q9550-4-1 ~]# virsh start r7 error: Failed to start domain r7 error: Requested operation is not valid: SCSI device 25:0:0:1 is already in use by other domain(s) as 'non-shareable'
Hi John, I am going to verify this bug. The comment 13 is my verification steps. I found two issues. In scenario 1, for step 6, after the two guests are destroyed, the sg device's selinux label should be as following: [root@intel-q9550-4-1 ~]# ll -Z /dev/sg9 crw-rw----. root root system_u:object_r:scsi_generic_device_t:s0 /dev/sg9 In scenario 2, the guest 'a' should be started successfully. Are these correct? Thanks, Liping
With respect to what I think is the first question - "After destroying both the guests should the label revert back to 'scsi_generic_device_t' instead of keeping the 'svirt_image_t' label"? The answer is 'no' (for at least the SELinux case) because it's a 'shared' device. The following is a snippet of the relative code: virSecuritySELinuxRestoreSecurityImageLabelInt() ... /* Don't restore labels on readonly/shared disks, because other VMs may * still be accessing these. Alternatively we could iterate over all * running domains and try to figure out if it is in use, but this would * not work for clustered filesystems, since we can't see running VMs using * the file on other nodes. Safest bet is thus to skip the restore step. */ if (src->readonly || src->shared) return 0; ... With respect to using readonly on the hostdev instead of shareable - I'm not able to reproduce in a quick test. Did you have 'Enforcing' mode set or 'Permissive' (I'm using Permissive). I don't have enough information to give a definative answer, but since it's not specifically related to this particular issue I'd rather see it handled separately.
> With respect to using readonly on the hostdev instead of shareable - I'm not > able to reproduce in a quick test. Did you have 'Enforcing' mode set or > 'Permissive' (I'm using Permissive). I don't have enough information to give > a definative answer, but since it's not specifically related to this > particular issue I'd rather see it handled separately. Hi John, Thanks for your reply. I'm using 'Enforcing' mode to execute Scenario 2. When I executed Scenario 2 with libvirt-1.2.8-10.el7.x86_64, the guest can be started successfully. But when I used libvirt-1.2.8-11.el7.x86_64 to test, the guest started to fail. The error information as follow: [root@intel-q9550-4-1 ~]# virsh start a error: Failed to start domain a error: internal error: process exited while connecting to monitor: 2014-12-29T08:52:15.631208Z qemu-kvm: -drive file=/dev/sg9,if=none,id=drive-hostdev0,readonly=on: could not open disk image /dev/sg9: Could not open '/dev/sg9': Permission denied So, I think this issue is a regression. Is it correct? Thanks, lcheng
In general, if you think you have a regression, then filing a bug is fine. Not sure if the issue reaches the level of "need" in order to get the change into rhel71 though. At the very least filing a bug allows for better tracking and prioritization rather than trying to continue using needinfo through this bug. In any case, with Enforcing enabled - I can reproduce with <readonly/> instead of <shareable/> set. However, if both <readonly/> and <shareable/> are provided, then the guest starts. So I think this perhaps has more to do with some sort of selinux/label issue - although I'm not quite sure yet exactly what the issue could be and don't have the cycles to dig deeper yet since I'm working on something else right now. To help "narrow down" what one would have to look at while working through a new bug - could you also provide what happens if you take the same target and try to add it as a "<disk>... <readonly/>... </disk>"... I mention this because the hostdev changes I made were designed to mimic the disk code, so it could be an important datapoint to know whether this also fails or works for a disk. When filing the new bug be sure to mention that the problem happens when Enforcing is set...
(In reply to John Ferlan from comment #17) > In general, if you think you have a regression, then filing a bug is fine. > Not sure if the issue reaches the level of "need" in order to get the change > into rhel71 though. At the very least filing a bug allows for better > tracking and prioritization rather than trying to continue using needinfo > through this bug. > > In any case, with Enforcing enabled - I can reproduce with <readonly/> > instead of <shareable/> set. Hi John, Thanks for your response. I am file a new bug to tracking this issue. Bug 1179564 - When selinux is 'Enforcing', guest with a readonly attribute sg disk can not be started. > However, if both <readonly/> and <shareable/> are provided, then the guest > starts. So I think this perhaps has more to do with some sort of > selinux/label issue - although I'm not quite sure yet exactly what the issue > could be and don't have the cycles to dig deeper yet since I'm working on > something else right now. > To help "narrow down" what one would have to look at while working through a > new bug - could you also provide what happens if you take the same target > and try to add it as a "<disk>... <readonly/>... </disk>"... I mention this > because the hostdev changes I made were designed to mimic the disk code, so > it could be an important datapoint to know whether this also fails or works > for a disk. I tested the 'readonly' sg disk with the following disk element. The guest started to fail too. For test steps, you can reference bug 1179564 description. <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/sg11'/> <target dev='sdb' bus='scsi'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> Thanks, lcheng > When filing the new bug be sure to mention that the problem happens when > Enforcing is set...
According to comment 13 and comment 15, this bug is fixed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html