Bug 1630744

Isn't this first of all a gluster issue that should be fixed in gluster primarily?

Having a workaround in RHV might be an interim solution, but BZ #1616270 should be handled with high prio as well, esp. as it probably also affects "normal" gluster deployments.

feel free to make the change as you see fit for gluster.
It's not configurable in any way right now, but the setting can be flipped easily if it works for all workloads in https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/builder/vminfo/LibvirtVmXmlBuilder.java#L1766

if it is supposed to be configurable by end user then this becomes a bit more complicated and probably should be tracked as a feature

(In reply to Martin Tessun from comment #1)
> Isn't this first of all a gluster issue that should be fixed in gluster
> primarily?
> 
> Having a workaround in RHV might be an interim solution, but BZ #1616270
> should be handled with high prio as well, esp. as it probably also affects
> "normal" gluster deployments.

The bug 1616270 is dependent on 2 bugs, a gluster bug which is being fixed and the change of aio option (this is not a workaround)

(In reply to Michal Skrivanek from comment #2)
> feel free to make the change as you see fit for gluster.
> It's not configurable in any way right now, but the setting can be flipped
> easily if it works for all workloads in
> https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/
> vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/builder/vminfo/
> LibvirtVmXmlBuilder.java#L1766
> 
> if it is supposed to be configurable by end user then this becomes a bit
> more complicated and probably should be tracked as a feature

I agree with Michal here, having native io set for all disks residing on a gluster storage domain is doable quite easily (although the storage domain info is missing in LibvirtVmXmlBuilder but that can be handled.
Letting the user choosing whether io will be native or threads is an RFE which will require more work

(In reply to Tal Nisan from comment #4)
> (In reply to Michal Skrivanek from comment #2)
> > feel free to make the change as you see fit for gluster.
> > It's not configurable in any way right now, but the setting can be flipped
> > easily if it works for all workloads in
> > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/
> > vdsbroker/src/main/java/org/ovirt/engine/core/vdsbroker/builder/vminfo/
> > LibvirtVmXmlBuilder.java#L1766
> > 
> > if it is supposed to be configurable by end user then this becomes a bit
> > more complicated and probably should be tracked as a feature
> 
> I agree with Michal here, having native io set for all disks residing on a
> gluster storage domain is doable quite easily (although the storage domain
> info is missing in LibvirtVmXmlBuilder but that can be handled.
> Letting the user choosing whether io will be native or threads is an RFE
> which will require more work

Ok, we'll run further tests before making the switch. In the meanwhile, do we know why aio=threads was used for file based storage domains and not aio=native? Any data to suggest threads is better?

> Ok, we'll run further tests before making the switch. In the meanwhile, do
> we know why aio=threads was used for file based storage domains and not
> aio=native? Any data to suggest threads is better?

I recall Yaniv wrote something about that a while ago, Yaniv can you please advise?

(In reply to Tal Nisan from comment #7)
> > Ok, we'll run further tests before making the switch. In the meanwhile, do
> > we know why aio=threads was used for file based storage domains and not
> > aio=native? Any data to suggest threads is better?
> 
> I recall Yaniv wrote something about that a while ago, Yaniv can you please
> advise?

See https://bugzilla.redhat.com/show_bug.cgi?id=1305886 for a long discussion.
Specifically, https://bugzilla.redhat.com/show_bug.cgi?id=1305886#c4 .

I'm not sure if anything has changed since.

Thanks, Yaniv and Tal. I've submitted a patch that switches to aio=native for gluster storage domains from cluster version 4.2. The config option also provides flexibility for users to switch if needed

    
 <disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source file='/rhev/data-center/mnt/glusterSD/gluster01.scl.lab.tlv.redhat.com:_storage__local__ge11__volume__0/2ad4743c-7ef3-4797-942f-8b6ac75f8d6c/images/7a3b5fae-56ee-4069-b77f-9f46795cc59a/29230098-79b
0-4f39-b7f7-97946462f11e'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/rhev/data-center/mnt/glusterSD/gluster01.scl.lab.tlv.redhat.com:_storage__local__ge11__volume__0/2ad4743c-7ef3-4797-942f-8b6ac75f8d6c/images/7a3b5fae-56ee-4069-b77f-9f46795cc59a/f89d5ab0-b
9a8-4643-940f-820151762b18'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>7a3b5fae-56ee-4069-b77f-9f46795cc59a</serial>
      <boot order='1'/>
      <alias name='ua-7a3b5fae-56ee-4069-b77f-9f46795cc59a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

Is aio=native needs to be that way? (io='native')

I checked by looking at 
#ps -ef | grep qemu | grep native
and saw:
file=gluster://rhsdev-grafton2.lab.eng.blr.redhat.com:24007/vmstore/288cfe58-913e-48f9-82a3-bb966e00ba15/images/8f9b1c65-e782-4e70-bc9a-9b7595b7fbd3/c0b08836-1b1a-496f-878e-bc4295728827,file.debug=4,format=qcow2,if=none,id=drive-ua-8f9b1c65-e782-4e70-bc9a-9b7595b7fbd3,serial=8f9b1c65-e782-4e70-bc9a-9b7595b7fbd3,cache=none,werror=stop,rerror=stop,aio=native


and the domain xml has:
 <disk type='network' device='disk' snapshot='no'>
        <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' discard='unmap'/>
        <source protocol='gluster' name='vmstore/288cfe58-913e-48f9-82a3-bb966e00ba15/images/cc684867-a217-42ac-a25b-a671a272c12a/50946f49-bf26-4e9e-9686-2c8aae4c63da' tlsFromConfig='0'>
          <host name='rhsdev-grafton2.lab.eng.blr.redhat.com' port='24007'/>
        </source>


The above was for network disk type - seems similar to the output seen for disks for gluster fuse mount

Verified
file=/rhev/datacenter/mnt/glusterSD/gluster01.scl.lab.tlv.redhat.com:_storage__local__ge11__volume__0/2ad4743c-7ef3-4797-942f-8b6ac75f8d6c/images/7a3b5fae-56ee-4069-b77f-9f46795cc59a/29230098-79b0-4f39-b7f7-97946462f11e,format=qcow2,if=none,id=drive-ua-7a3b5fae-56ee-4069-b77f-9f46795cc59a,serial=7a3b5fae-56ee-4069-b77f-9f46795cc59a,werror=stop,rerror=stop,cache=none,aio=native

Why is this bug still in MODIFIED state, but the fix is already available and verified ?

Should have been moved to VERIFIED by a script (since it has 'verified_upstream' in the QA whiteboard).

This bugzilla is included in oVirt 4.2.7 release, published on November 2nd 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.7 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.