Bug 1410506

Summary: hotplug - Attaching a virtio-blk direct lun disk to VM that is up fails (only virtio-scsi is now supported with virtio-1.0 - machine type -7.3)
Product: [oVirt] ovirt-engine Reporter: Avihai <aefrat>
Component: BLL.StorageAssignee: Tal Nisan <tnisan>
Status: CLOSED CURRENTRELEASE QA Contact: Avihai <aefrat>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.1.0CC: ahadas, amureini, bugs, gklein, jsuchane, michal.skrivanek, tnisan
Target Milestone: ovirt-4.1.1Keywords: Regression
Target Release: 4.1.1.4Flags: rule-engine: ovirt-4.1+
rule-engine: blocker+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-21 09:35:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine and vdsm logs none

Description Avihai 2017-01-05 15:56:28 UTC 
Created attachment 1237736 [details]
engine and vdsm logs

Description of problem:
Attaching a virtio direct lun disk to VM fails

Version-Release number of selected component (if applicable):
ovirt-engine-4.1.0-0.4.master.20170105101830.gitebfce39.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1.create a VM with a bootable disk (I created VM from template with rhel7.2)
2.create a direct lun disk with default interface virtio_scsi
3.attach direct lun disk to VM - no problem (virtio_scsi)  !
4.change direct lun disk interface to virtio
5.activate direct lun disk  

Actual results:
No issue expected


Expected results:
activation failed .
In engine log errors appear starting with "Failed in 'HotPlugDiskVDS' method"

Additional info:
From engine log:
2017-01-05 17:23:01,468+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Failed in 'HotPlugDiskVDS' method
2017-01-05 17:23:01,478+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host_mixed_1 command failed: internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0
2017-01-05 17:23:01,479+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand' return value 'StatusOnlyReturn [status=Status [code=45, message=internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0]]'
2017-01-05 17:23:01,479+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] HostName = host_mixed_1
2017-01-05 17:23:01,479+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Command 'HotPlugDiskVDSCommand(HostName = host_mixed_1, HotPlugDiskVDSParameters:{runAsync='true', hostId='eb9942e0-cdca-4be6-9a20-4d6553798376', vmId='dd1522d3-d49d-4cd1-8b61-38713c957225', diskId='04592bc0-fe28-4c9e-a3ec-b0d416e887b1', addressMap='null'})' execution failed: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0, code = 45
2017-01-05 17:23:01,479+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] FINISH, HotPlugDiskVDSCommand, log id: 365eccf8
2017-01-05 17:23:01,480+02 ERROR [org.ovirt.engine.core.bll.storage.disk.HotPlugDiskToVmCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Command 'org.ovirt.engine.core.bll.storage.disk.HotPlugDiskToVmCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0, code = 45 (Failed with error FailedToPlugDisk and code 45)
2017-01-05 17:23:01,500+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Correlation ID: 0dbb64de-933e-4da5-b978-a9ab73ab0229, Call Stack: null, Custom Event ID: -1, Message: Failed to plug disk 2 to VM golden_env_mixed_virtio_1_0 (User: admin@internal-authz).
2017-01-05 17:23:01,504+02 INFO  [org.ovirt.engine.core.bll.storage.disk.HotPlugDiskToVmCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Lock freed to object 'EngineLock:{exclusiveLocks='[04592bc0-fe28-4c9e-a3ec-b0d416e887b1=<DISK, ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases 2>]', sharedLocks='[dd1522d3-d49d-4cd1-8b61-38713c957225=<VM, ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}'
Comment 1 Avihai 2017-01-05 16:02:44 UTC 
Clarification :
VM was up before I attached the direct lun disk , so this is a hot plug issue .
Comment 2 Idan Shaby 2017-01-09 15:21:16 UTC 
It seems like libvirt stopped to support direct luns when connected via a VirtIO interface, as noted by BZ 1365823 and BZ 1245453.

Jaroslav, can you please clarify if it is supported or not?

This is an example for a disk's xml that we try to send:

<disk address="" device="lun" snapshot="no" type="block">
    <source dev="/dev/mapper/360014056f49d86377c740a99504e7b9e" />
    <target bus="virtio" dev="vdc" />
    <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw" />
</disk>

Thanks!
Comment 3 Jaroslav Suchanek 2017-01-12 09:29:32 UTC 
(In reply to Idan Shaby from comment #2)
> It seems like libvirt stopped to support direct luns when connected via a
> VirtIO interface, as noted by BZ 1365823 and BZ 1245453.
> 
> Jaroslav, can you please clarify if it is supported or not?
> 
> This is an example for a disk's xml that we try to send:
> 
> <disk address="" device="lun" snapshot="no" type="block">
>     <source dev="/dev/mapper/360014056f49d86377c740a99504e7b9e" />
>     <target bus="virtio" dev="vdc" />
>     <driver cache="none" error_policy="stop" io="native" name="qemu"
> type="raw" />
> </disk>
> 
> Thanks!

Uf, I thought I already responded to this needinfo. Anyway, you linked the correct bzs and they contain all necessary information. With new machine type pc-i440fx-rhel7.3.0 the virtio-1.0 is switched on by default. In this case the virtio-blk with scsi=on is not supported. Instead the virtio-scsi should be used.
Comment 4 Idan Shaby 2017-01-15 13:33:39 UTC 
Thanks Jaroslav!
Tal, what do we do in such a case?
It's a regression that we cannot fix.
Comment 5 Tal Nisan 2017-01-15 15:46:45 UTC 
It is indeed a change in the infra and we will need to change the xml we send to Libvirt, upon our discussion we've figured it to be in the virt team responsibility so moving it to them
Comment 6 Tomas Jelinek 2017-01-16 15:17:01 UTC 
An issue in hotplugging direct lun disk is not really a virt area. Moving to storage since there is the expertise on how to set this up and simulate the problem.

If you will need some help with the machine type related code please feel free to contact us.
Comment 7 Michal Skrivanek 2017-02-07 16:24:04 UTC 
(In reply to Tal Nisan from comment #5)
> It is indeed a change in the infra and we will need to change the xml we
> send to Libvirt, upon our discussion we've figured it to be in the virt team
> responsibility so moving it to them

Tal, what do you want to do? As per libvirt and qemu people virtio-blk direct lun disk is no longer supported. Do we support it?

Underlying bugs are closed as there is currently no way how to represent the previous configuration. We can do dirty hacks, but I do not know if we want/have to
Comment 8 Tal Nisan 2017-02-12 14:13:24 UTC 
The change should be in the Libvirt xml, from the discussion with the guys there it should be something like:

if ((direct attached lun) and (machine type==rhel-7.3) and
(driver==virtio-blk)) {
   device=disk
} else {
   device=lun
}
Comment 9 Michal Skrivanek 2017-02-20 11:40:15 UTC 
(In reply to Tal Nisan from comment #8)
> The change should be in the Libvirt xml, from the discussion with the guys
> there it should be something like:
> 
> if ((direct attached lun) and (machine type==rhel-7.3) and
> (driver==virtio-blk)) {
>    device=disk
> } else {
>    device=lun
> }

sounds reasonable. The machine type differentiation is possible in vdsm (unlike cluster level), though it's not so simple to make it future proof - an engine driven solution might work better but needs a bit more code. 
This is buried deep in storage code in vdsm so we may need to expose some properties in the right way...not sure exactly, a storage vdsm person should know best
Comment 10 Tal Nisan 2017-03-07 12:54:15 UTC 
Leaving only the releveant tags.

Before the fix:
        <disk device="lun" sgio="filtered" snapshot="no" type="block">
            <target bus="scsi" dev="sda" />
        </disk>
        <disk device="lun" snapshot="no" type="block">
            <target bus="virtio" dev="vdb" />
        </disk>

After the fix:
        <disk device="lun" sgio="filtered" snapshot="no" type="block">
            <target bus="scsi" dev="sda" />
        </disk>
        <disk device="disk" snapshot="no" type="block">
            <target bus="virtio" dev="vdb" />
        </disk>
Comment 11 Avihai 2017-03-15 14:35:05 UTC 
Verified.

Engine:
ovirt-engine-4.1.1.4-0.1.el7.noarch
VDSM:
4.19.6-1