Bug 2245646

Summary: Failed to start VM with scsi disk with io='threads'
Product: Container Native Virtualization (CNV) Reporter: chhu
Component: StorageAssignee: Álvaro Romero <alromero>
Status: CLOSED MIGRATED QA Contact: Natalie Gavrielov <ngavrilo>
Severity: medium Docs Contact:
Priority: high    
Version: 4.14.0CC: dafrank, kgoldbla, mprivozn, pkrempa
Target Milestone: ---   
Target Release: 4.15.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: libvirt_CNV_INT
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-14 16:12:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description chhu 2023-10-23 12:15:47 UTC
Description of problem:
Start VM with disk setting dedicatedIOThread:true, hotplug disk and make persistent, restart VM failed with "LibvirtError:'unsupported configuration: IOThreads not available for bus scsi target sda')"

Version-Release number of selected component (if applicable):
CNV 4.14

How reproducible:
100%

Steps to Reproduce:
1. Create VM with dedicatedIOThread in yaml file: asb-vm-dv-nfs-rhel.yaml,
Login to VM console, check there is one disk:/dev/vda

# oc create -f asb-pv-dv-nfs-rhel.yaml
persistentvolume/asb-pv-dv-nfs-rhel created
# oc create -f asb-vm-dv-nfs-rhel.yaml
virtualmachine.kubevirt.io/asb-vm-dv-nfs-rhel created
# oc get vm
NAME                 AGE   STATUS    READY
asb-vm-dv-nfs-rhel   74s   Running   True

2. Create PV and PVC
# oc create -f asb-hotplug-pv-disk.yaml
persistentvolume/asb-hotplug-pv-disk created
# oc create -f asb-hotplug-pvc-disk.yaml
persistentvolumeclaim/asb-hotplug-pvc-disk created

# oc get pv|grep asb
asb-hotplug-pv-disk                        4Gi        RWX            Retain           Bound       default/asb-hotplug-pvc-disk                                     nfs                     4m25s
asb-pv-dv-nfs-rhel                         12Gi       RWX            Retain           Bound       default/asb-dv-nfs-rhel                                          nfs                     4m35s
# oc get pvc|grep asb
asb-dv-nfs-rhel        Bound    asb-pv-dv-nfs-rhel    12Gi       RWX            nfs            4m19s
asb-hotplug-pvc-disk   Bound    asb-hotplug-pv-disk   4Gi        RWX            nfs            4m24s

3. Hotplug PVC: asb-hotplug-pvc-disk to VM: asb-vm-dv-nfs-rhel in webconsole:
VirtualMachines -> Configuration -> Disks -> Add disk:
Name: hotplug-disk   Source: Use an existing PVC   PVC project: default
PVC name: asb-hotplug-pvc-disk   Type: Disk   Interface: SCSI
Click Save. Check the Events, there is no error, check in VM Console: there are two disks: /dev/vda, /dev/sda.

4. VirtualMachines -> Configuration -> Disks -> Name: hotplug-disk: Make persistent, click Save. Check the disk xml, the hotpluged scsi disk with io='threads'

# oc get vm
NAME                 AGE     STATUS    READY
asb-vm-dv-nfs-rhel   6m34s   Running   True

# oc get pod|grep virt-launcher
virt-launcher-asb-vm-dv-nfs-rhel-6d8jx   1/1     Running   0          5m41s

# oc rsh virt-launcher-asb-vm-dv-nfs-rhel-6d8jx
sh-5.1$ virsh list --all
 Id   Name                         State
--------------------------------------------
 1    default_asb-vm-dv-nfs-rhel   running

sh-5.1$ virsh dumpxml default_asb-vm-dv-nfs-rhel| grep disk -A 15
    <disk type='file' device='disk' model='virtio-non-transitional'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' discard='unmap' iothread='1' queues='2'/>
      <source file='/var/run/kubevirt-private/vmi-disks/rootdisk/disk.img' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <alias name='ua-rootdisk'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads' discard='unmap'/>
      <source file='/var/run/kubevirt/hotplug-disks/hotplug-disk.img' index='2'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='ua-hotplug-disk'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

5. Restart the VM by click Actions: Restart, VM failed to start with error message:
   "LibvirtError(Code=67, Domain=10, Message='unsupported configuration: IOThreads not available for bus scsi target sda')"

# oc get vm
NAME                 AGE   STATUS     READY
asb-vm-dv-nfs-rhel   90m   Starting   False

# oc get vm
NAME                 AGE   STATUS             READY
asb-vm-dv-nfs-rhel   37m   CrashLoopBackOff   False


6. Collect the VM yaml file and virt-launcher pod logs
# oc describe vm asb-vm-dv-nfs-rhel > describe-vm-asb-vm-dv-nfs-rhel
# oc get vm asb-vm-dv-nfs-rhel -o yaml > get-vm-asb-vm-dv-nfs-rhel-yaml
# oc get pod|grep virt-launcher
virt-launcher-asb-vm-dv-nfs-rhel-v56kb   1/1     Running   0          100s
# oc logs virt-launcher-asb-vm-dv-nfs-rhel-v56kb > virt-launcher-asb-vm-dv-nfs-rhel-v56kb.log

Actual results:
In step 4, the hotpluged scsi disk xml with io='threads', in step 5, VM failed to start

Expected results:
After hotplug scsi disk In step 4, In step5, Restart VM successfully.

Additional info:
- Attached files:
  - VM, PV yaml files: asb-pv-dv-nfs-rhel.yaml, asb-vm-dv-nfs-rhel.yaml, asb-hotplug-pvc-disk.yaml, asb-hotplug-pv-disk.yaml
  - VM, Pod logs: describe-vm-asb-vm-dv-nfs-rhel, get-vm-asb-vm-dv-nfs-rhel-yaml, virt-launcher-asb-vm-dv-nfs-rhel-v56kb.log

Comment 6 Kedar Bidarkar 2023-10-25 12:20:41 UTC
Discussed about this bug with Virt Devs and we think, this could be related to the Storage component.
Could you plz take a look at this ?

Please feel free to revert back to the Virt component if you feel otherwise.

Comment 7 Kevin Alon Goldblatt 2024-05-15 20:20:35 UTC
@chhu 

In comment 2 of the description of the scenario you mention a second pv and pvc but did not provide the yamls for those. Could you please provide them:

2. Create PV and PVC
# oc create -f asb-hotplug-pv-disk.yaml
persistentvolume/asb-hotplug-pv-disk created
# oc create -f asb-hotplug-pvc-disk.yaml
persistentvolumeclaim/asb-hotplug-pvc-disk created

Comment 8 Red Hat Bugzilla 2024-09-13 04:25:09 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days