Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2214147

Summary: s390x guest can not be started on rhel9.3 after executing dracut --force --add-drivers 'virtio_scsi' in guest internal
Product: Red Hat Enterprise Linux 9 Reporter: chunfu wen <chwen>
Component: libvirtAssignee: Thomas Huth <thuth>
libvirt sub component: General QA Contact: virt-qe-z
Status: CLOSED NOTABUG Docs Contact:
Severity: medium    
Priority: medium CC: bfu, clegoate, lcong, meili, pkrempa, smitterl, thuth, virt-maint
Version: 9.3Keywords: Automation, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-06-29 12:13:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description chunfu wen 2023-06-12 05:34:20 UTC
Description of problem:
s390x guest can not be started on rhel9.3 after executing dracut --force --add-drivers 'virtio_scsi' in guest internal 

Version-Release number of selected component (if applicable):

libvirt-9.4.0-1.el9.s390x
qemu-kvm-8.0.0-4.el9.s390x

How reproducible:
100%

Steps to Reproduce:
1. Start a vm with configuration as below:
 <domain type='kvm'>
  <name>avocado-vt-vm1</name>
  <uuid>c1d24967-efcf-4ef1-9fe0-4cf7188826cf</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='s390x' machine='s390-ccw-virtio-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu mode='host-model' check='partial'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-s390x.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/>
    </disk>
    <controller type='virtio-serial' index='0'>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0004'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='network'>
      <mac address='52:54:00:72:66:93'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/>
    </interface>
    <serial type='pty'>
      <target type='sclp-serial' port='0'>
        <model name='sclpconsole'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <console type='pty'>
      <target type='virtio' port='1'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='keyboard' bus='virtio'>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0005'/>
    </input>
    <input type='mouse' bus='virtio'>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0006'/>
    </input>
    <graphics type='vnc' port='-1' autoport='yes'>
      <listen type='address'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/>
    </video>
    <memballoon model='virtio'>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0007'/>
    </memballoon>
    <panic model='s390'/>
    <vsock model='virtio'>
      <cid auto='yes'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0008'/>
    </vsock>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>

2. Start and login into the vm, and execute dracut --force --add-drivers 'virtio_scsi' and shutdown commands in guest internal 
[root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh console avocado-vt-vm1
Connected to domain 'avocado-vt-vm1'
Escape character is ^] (Ctrl + ])

localhost login: root
Password: 
[root@localhost ~]# dracut --force --add-drivers 'virtio_scsi'
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# echo $?
0
[root@localhost ~]# shutdown
Shutdown scheduled for Sun 2023-06-11 23:32:59 EDT, use 'shutdown -c' to cancel.
[root@localhost ~]# dracut Warning: Killing all remaining processes
[  217.472019] dracut Warning: Killing all remaining processes
dracut Warning: Unmounted /oldroot.
[  217.505014] dracut Warning: Unmounted /oldroot.
Powering off.

[root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh list --all
 Id   Name             State
---------------------------------
 -    avocado-vt-vm1   shut off
 -    vm2              shut off

3. Start Vm again:


[root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh start avocado-vt-vm1
Domain 'avocado-vt-vm1' started

[root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh list --all
 Id   Name             State
---------------------------------
 -    avocado-vt-vm1   shut off
 -    vm2              shut off

4. Actual results:
VM can not be started 

5. Expected results:
VM expect to be started successfully

Comment 3 Meina Li 2023-06-12 07:01:18 UTC
I think the following simple test scenario is same with this one. 

Test Version:
libvirt-9.4.0-1.el9.s390x
qemu-kvm-8.0.0-4.el9.s390x


Test Steps:
1. Prepare a block disk by using scsi_debug or targetcli.
# modprobe scsi_debug dev_size_mb=500
# lsscsi
[0:0:0:0]    disk    Linux    scsi_debug       0191  /dev/sda 
2. Edit the guest only with block disk.
# virsh dumpxml avocado-vt-vm1 --xpath //disk
<disk type="block" device="disk">
  <driver name="qemu" type="raw"/>
  <source dev="/dev/sda"/>
  <target dev="vda" bus="virtio"/>
  <address type="ccw" cssid="0xfe" ssid="0x0" devno="0x0000"/>
</disk>
3. Start the guest.
# virsh start avocado-vt-vm1
Domain 'avocado-vt-vm1' started
# virsh list --all
 Id   Name             State
---------------------------------
 -    avocado-vt-vm1   shut off
4. Check the guest log.
# vim /var/log/libvirt/qemu/avocado-vt-vm1.log
......
char device redirected to /dev/pts/1 (label charserial0)
char device redirected to /dev/pts/2 (label charconsole1)
2023-06-12 06:26:09.891+0000: panic s390: core='0' psw-mask='0x0002000180000000' psw-addr='0x0000000000000000' reason='disabled-wait'
2023-06-12 06:26:09.892+0000: shutting down, reason=crashed
2023-06-12T06:26:09.892902Z qemu-kvm: terminating on signal 15 from pid 311581 (<unknown process>)

Comment 4 liang cong 2023-06-12 09:29:04 UTC
Another similar scenario:
Version:
# rpm -q libvirt qemu-kvm
libvirt-9.3.0-2.el9.s390x
qemu-kvm-8.0.0-2.el9.s390x


1. Create a qcow2 file
qemu-img create -f qcow2 /var/lib/avocado/data/avocado-vt/images/vdb.qcow2 100M

2. Define the guest with below disk config xml:
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/avocado/data/avocado-vt/images/vdb.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/>
</disk>

3. Start the guest
# virsh start avocado-vt-vm1
Domain 'avocado-vt-vm1' started

4. Check the guest status
# virsh list --all
 Id   Name             State
---------------------------------
 -    avocado-vt-vm1   shut off

5. Check the guest log:
2023-06-12 09:27:02.036+0000: starting up libvirt version: 9.3.0, package: 2.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-05-16-10:13:15, ), qemu version: 8.0.0qemu-kvm-8.0.0-2.el9, kernel: 5.14.0-316.el9.s390x, hostname: s390x-kvm-virtqez6.lab.eng.rdu2.redhat.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/.config \
/usr/libexec/qemu-kvm \
-name guest=avocado-vt-vm1,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/master-key.aes"}' \
-machine s390-ccw-virtio-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=s390.ram \
-accel kvm \
-cpu gen15a-base,aen=on,vxpdeh=on,aefsi=on,diag318=on,csske=on,mepoch=on,msa9=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,deflate=on,edat2=on,etoken=on,vx=on,ipter=on,mepochptff=on,ap=on,vxeh=on,vxpd=on,esop=on,msa9_pckmo=on,vxeh2=on,esort=on,apft=on,els=on,iep=on,apqci=on,cte=on,ais=on,bpb=on,gs=on,ppa15=on,zpci=on,sea_esop2=on,te=on \
-m 1024 \
-object '{"qom-type":"memory-backend-ram","id":"s390.ram","size":1073741824}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid 350542d5-1e72-4f1e-89d2-30e599a9ba88 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=25,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device '{"driver":"virtio-scsi-ccw","id":"scsi0","devno":"fe.0.0003"}' \
-device '{"driver":"virtio-serial-ccw","id":"virtio-serial0","devno":"fe.0.0004"}' \
-blockdev '{"driver":"file","filename":"/var/lib/avocado/data/avocado-vt/images/vdb.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}' \
-netdev '{"type":"tap","fd":"26","vhost":true,"vhostfd":"28","id":"hostnet0"}' \
-device '{"driver":"virtio-net-ccw","netdev":"hostnet0","id":"net0","mac":"52:54:00:9f:d9:06","devno":"fe.0.0001"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"sclpconsole","chardev":"charserial0","id":"serial0"}' \
-chardev socket,id=charchannel0,fd=24,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev pty,id=charconsole1 \
-device '{"driver":"virtconsole","chardev":"charconsole1","id":"console1"}' \
-device '{"driver":"virtio-keyboard-ccw","id":"input0","devno":"fe.0.0005"}' \
-device '{"driver":"virtio-mouse-ccw","id":"input1","devno":"fe.0.0006"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 127.0.0.1:0,audiodev=audio1 \
-device '{"driver":"virtio-gpu-ccw","id":"video0","max_outputs":1,"devno":"fe.0.0002"}' \
-device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0007"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-device '{"driver":"vhost-vsock-ccw","id":"vsock0","guest-cid":3,"vhostfd":"21","devno":"fe.0.0008"}' \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
char device redirected to /dev/pts/2 (label charconsole1)
2023-06-12 09:27:02.663+0000: panic s390: core='0' psw-mask='0x0002000180000000' psw-addr='0x0000000000000000' reason='disabled-wait'
2023-06-12 09:27:02.665+0000: shutting down, reason=crashed
2023-06-12T09:27:02.666824Z qemu-kvm: terminating on signal 15 from pid 1137767 (<unknown process>)

Comment 5 Peter Krempa 2023-06-20 14:22:56 UTC
The guest OS panic-ked after the initrd was rebuilt. Could you please try to see what the console prints?

This doesn't really look like either a libvirt or a qemu problem. Perhaps dracut didn't pick up a required module for the root filesystem?

Comment 11 Thomas Huth 2023-06-22 08:04:26 UTC
Hi @chwen , did you re-install the bootloader after updating the initrd? i.e. did you run "zipl" after running "dracut ..." ? On s390x, we don't use grub, so you have to make sure that "zipl" has been run each time you changed something with regards to the kernel or initrd.

Comment 15 Meina Li 2023-07-04 08:14:35 UTC
According to comment 14, file a new bug to track comment 3 and comment 4: https://bugzilla.redhat.com/show_bug.cgi?id=2219538