Bug 1535961

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

*** Bug 1535953 has been marked as a duplicate of this bug. ***

seems someone somewhere forgot to port pinToIoThreads() :)

How correct xml definition should look like?
Should it be like in the documentation, e.g. <driver> sub element with iothread=iothread_id attribute?

<controller type='scsi' index='0' model='virtio-scsi'>
  <driver iothread='4'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
</controller>

If so, I don't see iothreadids in the vm xml, this is the only info I have

virsh -r dumpxml 8 | grep iothread
  <name>vm_iothread_virtio</name>
  <iothreads>4</iothreads>

Also, how <disk> should look like?

I'm fixing our autotest for this feature now, and it's quite hard for me to guess how correct vm XML should look like, should every disk be assigned a controller>

Fix for SCSI disks is missing.

(In reply to Vladimir from comment #4)
> How correct xml definition should look like?
> Should it be like in the documentation, e.g. <driver> sub element with
> iothread=iothread_id attribute?
> 
> <controller type='scsi' index='0' model='virtio-scsi'>
>   <driver iothread='4'/>
>   <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
> </controller>

Hi Vladimir,
Yes, it should look like that and the reason you couldn't see that is since it was not handled properly in the new domain xml.

(In reply to Vladimir from comment #5)
> Also, how <disk> should look like?

So for a VM configured to run with 2 iothreads and has 3 disks for example, the domain xml should include the follows:

<domain type='kvm' id='137'>
  <name>disk_vm</name>
  ....
   <iothreads>2</iothreads>
   <devices>
      <disk type='file' device='disk' snapshot='no'>
        <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
        <source file='/rhev/data-center/mnt/......'/>
        <backingStore/>
        <target dev='sda' bus='scsi'/>
        <serial>38ce501f-9bc8-4611-b73a-a9495464dff3</serial>
        <boot order='1'/>
        <alias name='scsi0-0-0-0'/>
        <address type='drive' controller='0' bus='0' target='0' unit='0'/>   <== this scsi disk is addressed to controller with index=0 and unit=0
      </disk>

      <disk type='file' device='disk' snapshot='no'>
        <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
        <source file='/rhev/data-center/mnt/...'/>
        <backingStore/>
        <target dev='sdb' bus='scsi'/>
        <serial>74e0add5-6b7b-4646-bd9f-f20a7a3a1de1</serial>
        <alias name='scsi1-0-0-0'/>
        <address type='drive' controller='1' bus='0' target='0' unit='0'/> <== this scsi disk is addressed to controller with index=1 and unit=0
      </disk>

      <controller type='scsi' index='0' model='virtio-scsi'>
        <driver iothread='1'/>   <== this scsi controller is pinned to io thread 1
        <alias name='scsi0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
      </controller>

      <controller type='scsi' index='1' model='virtio-scsi'>
        <driver iothread='2'/>  <== this scsi controller is pinned to io thread 2
        <alias name='scsi1'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      </controller>
     ....
   </devices>
</domain>

Hi Vladimir,

Regarding your questions:
- a SCSI controller is created per each io thread and pinned to it.
- the SCSI disks are addressed to SCSI controller(s) using round-robin algo.
- by using the engine admin - the IO threads number can be either 0 (disabled) or   
  1. Setting them to 2 or more can be done only via rest api or ansible.
- I remind you that it usually requires activate/deactivate of existing disks in 
  case of changing io threads num (for new disk-controller allocations to be 
  calculated again).

I checked it on RHEV 4.2.2.2-0.1.el7

In the UI I see that no new controllers created, and it has {IoThreadId=1}


But VM XML looks like this: 

    <disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='threads' iothread='1'/>
      <source file='/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_GE_42_ENV__9_nfs__0/5a31f74b-709d-4b6b-945b-c5e95bb95790/images/0f2d0278-1530-4275-bccf-6802dbece9ee/f4703fff-6972-4896-8b85-a9933a7cef29'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_GE_42_ENV__9_nfs__0/5a31f74b-709d-4b6b-945b-c5e95bb95790/images/0f2d0278-1530-4275-bccf-6802dbece9ee/f2463a1b-e9b8-487a-9070-91640e661464'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>0f2d0278-1530-4275-bccf-6802dbece9ee</serial>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='virtio-serial' index='0' ports='16'>
      <alias name='ua-211d378d-be2c-4511-8315-c99324e94ccd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='ua-9ac2654c-81f2-4687-95e7-cab4c75f528d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='ua-e7e2e12c-f93f-4ab3-9e0d-986d225872f7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>

I don't see neither assigned controller for disk nor pinned iothread for controller

The fix didn't make it to the latest build.

(In reply to Arik from comment #10)
> The fix didn't make it to the latest build.

In which build it will be fixed then?

(In reply to Vladimir from comment #11)
> (In reply to Arik from comment #10)
> > The fix didn't make it to the latest build.
> 
> In which build it will be fixed then?

If there was a build after the one you checked, then it's there already.
Otherwise, in the next build.

With fix VM xml now looks like this 

Disk:
<disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='threads' iothread='1'/>
      <source file='/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Compute__NFS_GE_42_ENV__9_nfs__0/5a31f74b-709d-4b6b-945b-c5e95bb95790/images/a60a7131-3355-4e62-a532-7fef65b66c2e/0d7fba3a-8197-46d3-9d25-e4556fc3d17f'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>a60a7131-3355-4e62-a532-7fef65b66c2e</serial>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>

Controller:
<controller type='scsi' index='0' model='virtio-scsi'>
      <driver iothread='1'/>
      <alias name='ua-25c008fb-824a-4759-86d4-30d9d87e9ef1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>

I still don't see assigned controller in the <address> is this correct?

(In reply to Vladimir from comment #13) 
> Disk:
> <disk type='file' device='disk' snapshot='no'>
>       <driver name='qemu' type='qcow2' cache='none' error_policy='stop'
> io='threads' iothread='1'/>
>       <source
> file='/rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:
> _Compute__NFS_GE_42_ENV__9_nfs__0/5a31f74b-709d-4b6b-945b-c5e95bb95790/
> images/a60a7131-3355-4e62-a532-7fef65b66c2e/0d7fba3a-8197-46d3-9d25-
> e4556fc3d17f'/>
>       <backingStore/>
>       <target dev='vda' bus='virtio'/>
>       <serial>a60a7131-3355-4e62-a532-7fef65b66c2e</serial>
>       <boot order='1'/>
>       <alias name='virtio-disk0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
>     </disk>

This is not a SCSI disk but rather a VIRTIO-block disk and for those the iothread pinning is done by adding the following attribute to the driver element:
iothread='1'.

Since disks with VIRTIO interface are not addressed to SCSI controllers. they are pinned directly to the specific iothread.

> Controller:
> <controller type='scsi' index='0' model='virtio-scsi'>
>       <driver iothread='1'/>
>       <alias name='ua-25c008fb-824a-4759-86d4-30d9d87e9ef1'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
> </controller>

Each SCSI controller has an index and number of SCSI controllers = number of io threads.
In your example, the SCSI controller has an index=0 and it is assigned to iothread=1.

Each SCSI disk (not VIRTIO disks) which will be assigned to this controller, will be pinned to iothread=1 also.

An example of a SCSI disk assigned to this SCSI controller:

<disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
      <source file='/rhev/data-center/mnt/10.35.0.20:_exports_sgratch_data__domain14/5fa30f72-e287-4466-bca2-d1f97807c023/images/a644264e-b5a0-4ef5-990e-f2d1036807eb/3315b94c-f263-4409-a90f-659cb5d47d4b'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <serial>a644264e-b5a0-4ef5-990e-f2d1036807eb</serial>
      <boot order='1'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>

Specifically note the address element:
 <address type='drive' controller='0' bus='0' target='0' unit='0'/>
which means that this disk is addressed to SCSI controller with index=0.

> 
> I still don't see assigned controller in the <address> is this correct?
So as explained above, since you had a VIRTIO disk and not SCSI then the disk won't be assigned to a SCSI controller. It has a pci address and it is pinned to the iothread by adding the iothread attribute to the driver element:
<driver ...... iothread='1'/>

Thanks for the info, but I cannot create Vm with SCSI disk somehow, what I've done so far:
I create new VM, check iothreads enabled, virtio-isci enabled, select iscsi storage domain, and what I'm getting is this:

 <disk type='block' device='disk' snapshot='no'>
      <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='native' iothread='1'/>
      <source dev='/rhev/data-center/mnt/blockSD/7624cec9-1ec5-4feb-a04b-8362bc501206/images/c30ec452-e94a-4167-a9ba-2f0a284a5009/4a366c5f-6640-4e89-9675-f695b95fc64c'/>
      <backingStore type='block' index='1'>
        <format type='qcow2'/>
        <source dev='/rhev/data-center/mnt/blockSD/7624cec9-1ec5-4feb-a04b-8362bc501206/images/c30ec452-e94a-4167-a9ba-2f0a284a5009/70b5f77b-2c7b-4f85-87d5-d6b8af61b8d7'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <serial>c30ec452-e94a-4167-a9ba-2f0a284a5009</serial>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver iothread='1'/>
      <alias name='ua-719cfdee-64df-4b00-982a-1e1d1eb45d3b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>

Also in UI VM disk is created as a virtio one, if I later add new disk to the VM it's created as SCSI. Is it just me, or something is wrong here?

(In reply to Vladimir from comment #15)
> Also in UI VM disk is created as a virtio one, if I later add new disk to
> the VM it's created as SCSI. Is it just me, or something is wrong here?

Select the VM in the VMs tab, then go to its 'Disks' tab, choose the disk, open the 'Edit' dialog and change the 'Interface' to VirtIO-SCSI

It works this way, but why it's change to virtio on initial creation? do we have corresponding BZ for it?

(In reply to Vladimir from comment #17)
> It works this way, but why it's change to virtio on initial creation? do we
> have corresponding BZ for it?

The selection of the disk interface on VM creation is based on the interface of the disk of the template you created the VM from. I don't see a way to change it before the VM is created, if it's missing and you find it important, it should be filed as a separate bug on storage.

Verified on 4.2.2.4-0.1.el7

Scenarios:
1. Add virtio disk, check that iothread was pinned successfully - PASS
2. Add iscsi disk, check that iothread was pinned successfully - PASS
3. Add both virtio & iscsi disks to the VM, check that iothread was pinned successfully - PASS

This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Verified on RHEVM 4.2.2.6-0.1.el7

Scenarios:

Create VM with IOthreads enabled on all storage domains types for:

1. Block storage
2. File storage with virtio-scsi
3. File storage with virtio

*** Bug 1563769 has been marked as a duplicate of this bug. ***