Bug 1410506 - hotplug - Attaching a virtio-blk direct lun disk to VM that is up fails (only virtio-scsi is now supported with virtio-1.0 - machine type -7.3)
Summary: hotplug - Attaching a virtio-blk direct lun disk to VM that is up fails (only...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ovirt-4.1.1
: 4.1.1.4
Assignee: Tal Nisan
QA Contact: Avihai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-05 15:56 UTC by Avihai
Modified: 2017-04-21 09:35 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-21 09:35:52 UTC
oVirt Team: Storage
rule-engine: ovirt-4.1+
rule-engine: blocker+


Attachments (Terms of Use)
engine and vdsm logs (321.41 KB, application/x-gzip)
2017-01-05 15:56 UTC, Avihai
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 73391 0 master MERGED core: Send direct LUN devices with virtio interface as disk 2020-12-08 22:55:35 UTC
oVirt gerrit 73739 0 ovirt-engine-4.1.1.z MERGED core: Send direct LUN devices with virtio interface as disk 2020-12-08 22:55:36 UTC
oVirt gerrit 73740 0 ovirt-engine-4.1 MERGED core: Send direct LUN devices with virtio interface as disk 2020-12-08 22:55:36 UTC

Description Avihai 2017-01-05 15:56:28 UTC
Created attachment 1237736 [details]
engine and vdsm logs

Description of problem:
Attaching a virtio direct lun disk to VM fails

Version-Release number of selected component (if applicable):
ovirt-engine-4.1.0-0.4.master.20170105101830.gitebfce39.el7.centos.noarch

How reproducible:
100%

Steps to Reproduce:
1.create a VM with a bootable disk (I created VM from template with rhel7.2)
2.create a direct lun disk with default interface virtio_scsi
3.attach direct lun disk to VM - no problem (virtio_scsi)  !
4.change direct lun disk interface to virtio
5.activate direct lun disk  

Actual results:
No issue expected


Expected results:
activation failed .
In engine log errors appear starting with "Failed in 'HotPlugDiskVDS' method"

Additional info:
From engine log:
2017-01-05 17:23:01,468+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Failed in 'HotPlugDiskVDS' method
2017-01-05 17:23:01,478+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM host_mixed_1 command failed: internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0
2017-01-05 17:23:01,479+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand' return value 'StatusOnlyReturn [status=Status [code=45, message=internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0]]'
2017-01-05 17:23:01,479+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] HostName = host_mixed_1
2017-01-05 17:23:01,479+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Command 'HotPlugDiskVDSCommand(HostName = host_mixed_1, HotPlugDiskVDSParameters:{runAsync='true', hostId='eb9942e0-cdca-4be6-9a20-4d6553798376', vmId='dd1522d3-d49d-4cd1-8b61-38713c957225', diskId='04592bc0-fe28-4c9e-a3ec-b0d416e887b1', addressMap='null'})' execution failed: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0, code = 45
2017-01-05 17:23:01,479+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] FINISH, HotPlugDiskVDSCommand, log id: 365eccf8
2017-01-05 17:23:01,480+02 ERROR [org.ovirt.engine.core.bll.storage.disk.HotPlugDiskToVmCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Command 'org.ovirt.engine.core.bll.storage.disk.HotPlugDiskToVmCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = internal error: unable to execute QEMU command 'device_add': Please set scsi=off for virtio-blk devices in order to use virtio 1.0, code = 45 (Failed with error FailedToPlugDisk and code 45)
2017-01-05 17:23:01,500+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Correlation ID: 0dbb64de-933e-4da5-b978-a9ab73ab0229, Call Stack: null, Custom Event ID: -1, Message: Failed to plug disk 2 to VM golden_env_mixed_virtio_1_0 (User: admin@internal-authz).
2017-01-05 17:23:01,504+02 INFO  [org.ovirt.engine.core.bll.storage.disk.HotPlugDiskToVmCommand] (org.ovirt.thread.pool-6-thread-38) [0dbb64de-933e-4da5-b978-a9ab73ab0229] Lock freed to object 'EngineLock:{exclusiveLocks='[04592bc0-fe28-4c9e-a3ec-b0d416e887b1=<DISK, ACTION_TYPE_FAILED_DISKS_LOCKED$diskAliases 2>]', sharedLocks='[dd1522d3-d49d-4cd1-8b61-38713c957225=<VM, ACTION_TYPE_FAILED_VM_IS_LOCKED>]'}'

Comment 1 Avihai 2017-01-05 16:02:44 UTC
Clarification :
VM was up before I attached the direct lun disk , so this is a hot plug issue .

Comment 2 Idan Shaby 2017-01-09 15:21:16 UTC
It seems like libvirt stopped to support direct luns when connected via a VirtIO interface, as noted by BZ 1365823 and BZ 1245453.

Jaroslav, can you please clarify if it is supported or not?

This is an example for a disk's xml that we try to send:

<disk address="" device="lun" snapshot="no" type="block">
    <source dev="/dev/mapper/360014056f49d86377c740a99504e7b9e" />
    <target bus="virtio" dev="vdc" />
    <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw" />
</disk>

Thanks!

Comment 3 Jaroslav Suchanek 2017-01-12 09:29:32 UTC
(In reply to Idan Shaby from comment #2)
> It seems like libvirt stopped to support direct luns when connected via a
> VirtIO interface, as noted by BZ 1365823 and BZ 1245453.
> 
> Jaroslav, can you please clarify if it is supported or not?
> 
> This is an example for a disk's xml that we try to send:
> 
> <disk address="" device="lun" snapshot="no" type="block">
>     <source dev="/dev/mapper/360014056f49d86377c740a99504e7b9e" />
>     <target bus="virtio" dev="vdc" />
>     <driver cache="none" error_policy="stop" io="native" name="qemu"
> type="raw" />
> </disk>
> 
> Thanks!

Uf, I thought I already responded to this needinfo. Anyway, you linked the correct bzs and they contain all necessary information. With new machine type pc-i440fx-rhel7.3.0 the virtio-1.0 is switched on by default. In this case the virtio-blk with scsi=on is not supported. Instead the virtio-scsi should be used.

Comment 4 Idan Shaby 2017-01-15 13:33:39 UTC
Thanks Jaroslav!
Tal, what do we do in such a case?
It's a regression that we cannot fix.

Comment 5 Tal Nisan 2017-01-15 15:46:45 UTC
It is indeed a change in the infra and we will need to change the xml we send to Libvirt, upon our discussion we've figured it to be in the virt team responsibility so moving it to them

Comment 6 Tomas Jelinek 2017-01-16 15:17:01 UTC
An issue in hotplugging direct lun disk is not really a virt area. Moving to storage since there is the expertise on how to set this up and simulate the problem.

If you will need some help with the machine type related code please feel free to contact us.

Comment 7 Michal Skrivanek 2017-02-07 16:24:04 UTC
(In reply to Tal Nisan from comment #5)
> It is indeed a change in the infra and we will need to change the xml we
> send to Libvirt, upon our discussion we've figured it to be in the virt team
> responsibility so moving it to them

Tal, what do you want to do? As per libvirt and qemu people virtio-blk direct lun disk is no longer supported. Do we support it?

Underlying bugs are closed as there is currently no way how to represent the previous configuration. We can do dirty hacks, but I do not know if we want/have to

Comment 8 Tal Nisan 2017-02-12 14:13:24 UTC
The change should be in the Libvirt xml, from the discussion with the guys there it should be something like:

if ((direct attached lun) and (machine type==rhel-7.3) and
(driver==virtio-blk)) {
   device=disk
} else {
   device=lun
}

Comment 9 Michal Skrivanek 2017-02-20 11:40:15 UTC
(In reply to Tal Nisan from comment #8)
> The change should be in the Libvirt xml, from the discussion with the guys
> there it should be something like:
> 
> if ((direct attached lun) and (machine type==rhel-7.3) and
> (driver==virtio-blk)) {
>    device=disk
> } else {
>    device=lun
> }

sounds reasonable. The machine type differentiation is possible in vdsm (unlike cluster level), though it's not so simple to make it future proof - an engine driven solution might work better but needs a bit more code. 
This is buried deep in storage code in vdsm so we may need to expose some properties in the right way...not sure exactly, a storage vdsm person should know best

Comment 10 Tal Nisan 2017-03-07 12:54:15 UTC
Leaving only the releveant tags.

Before the fix:
        <disk device="lun" sgio="filtered" snapshot="no" type="block">
            <target bus="scsi" dev="sda" />
        </disk>
        <disk device="lun" snapshot="no" type="block">
            <target bus="virtio" dev="vdb" />
        </disk>

After the fix:
        <disk device="lun" sgio="filtered" snapshot="no" type="block">
            <target bus="scsi" dev="sda" />
        </disk>
        <disk device="disk" snapshot="no" type="block">
            <target bus="virtio" dev="vdb" />
        </disk>

Comment 11 Avihai 2017-03-15 14:35:05 UTC
Verified.

Engine:
ovirt-engine-4.1.1.4-0.1.el7.noarch
VDSM:
4.19.6-1


Note You need to log in before you can comment on or make changes to this bug.