Bug 2214147
| Summary: | s390x guest can not be started on rhel9.3 after executing dracut --force --add-drivers 'virtio_scsi' in guest internal | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | chunfu wen <chwen> |
| Component: | libvirt | Assignee: | Thomas Huth <thuth> |
| libvirt sub component: | General | QA Contact: | virt-qe-z |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | bfu, clegoate, lcong, meili, pkrempa, smitterl, thuth, virt-maint |
| Version: | 9.3 | Keywords: | Automation, Triaged |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | s390x | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-06-29 12:13:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I think the following simple test scenario is same with this one. Test Version: libvirt-9.4.0-1.el9.s390x qemu-kvm-8.0.0-4.el9.s390x Test Steps: 1. Prepare a block disk by using scsi_debug or targetcli. # modprobe scsi_debug dev_size_mb=500 # lsscsi [0:0:0:0] disk Linux scsi_debug 0191 /dev/sda 2. Edit the guest only with block disk. # virsh dumpxml avocado-vt-vm1 --xpath //disk <disk type="block" device="disk"> <driver name="qemu" type="raw"/> <source dev="/dev/sda"/> <target dev="vda" bus="virtio"/> <address type="ccw" cssid="0xfe" ssid="0x0" devno="0x0000"/> </disk> 3. Start the guest. # virsh start avocado-vt-vm1 Domain 'avocado-vt-vm1' started # virsh list --all Id Name State --------------------------------- - avocado-vt-vm1 shut off 4. Check the guest log. # vim /var/log/libvirt/qemu/avocado-vt-vm1.log ...... char device redirected to /dev/pts/1 (label charserial0) char device redirected to /dev/pts/2 (label charconsole1) 2023-06-12 06:26:09.891+0000: panic s390: core='0' psw-mask='0x0002000180000000' psw-addr='0x0000000000000000' reason='disabled-wait' 2023-06-12 06:26:09.892+0000: shutting down, reason=crashed 2023-06-12T06:26:09.892902Z qemu-kvm: terminating on signal 15 from pid 311581 (<unknown process>) Another similar scenario:
Version:
# rpm -q libvirt qemu-kvm
libvirt-9.3.0-2.el9.s390x
qemu-kvm-8.0.0-2.el9.s390x
1. Create a qcow2 file
qemu-img create -f qcow2 /var/lib/avocado/data/avocado-vt/images/vdb.qcow2 100M
2. Define the guest with below disk config xml:
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/avocado/data/avocado-vt/images/vdb.qcow2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/>
</disk>
3. Start the guest
# virsh start avocado-vt-vm1
Domain 'avocado-vt-vm1' started
4. Check the guest status
# virsh list --all
Id Name State
---------------------------------
- avocado-vt-vm1 shut off
5. Check the guest log:
2023-06-12 09:27:02.036+0000: starting up libvirt version: 9.3.0, package: 2.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-05-16-10:13:15, ), qemu version: 8.0.0qemu-kvm-8.0.0-2.el9, kernel: 5.14.0-316.el9.s390x, hostname: s390x-kvm-virtqez6.lab.eng.rdu2.redhat.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/.config \
/usr/libexec/qemu-kvm \
-name guest=avocado-vt-vm1,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-avocado-vt-vm1/master-key.aes"}' \
-machine s390-ccw-virtio-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=s390.ram \
-accel kvm \
-cpu gen15a-base,aen=on,vxpdeh=on,aefsi=on,diag318=on,csske=on,mepoch=on,msa9=on,msa8=on,msa7=on,msa6=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,deflate=on,edat2=on,etoken=on,vx=on,ipter=on,mepochptff=on,ap=on,vxeh=on,vxpd=on,esop=on,msa9_pckmo=on,vxeh2=on,esort=on,apft=on,els=on,iep=on,apqci=on,cte=on,ais=on,bpb=on,gs=on,ppa15=on,zpci=on,sea_esop2=on,te=on \
-m 1024 \
-object '{"qom-type":"memory-backend-ram","id":"s390.ram","size":1073741824}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid 350542d5-1e72-4f1e-89d2-30e599a9ba88 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=25,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-device '{"driver":"virtio-scsi-ccw","id":"scsi0","devno":"fe.0.0003"}' \
-device '{"driver":"virtio-serial-ccw","id":"virtio-serial0","devno":"fe.0.0004"}' \
-blockdev '{"driver":"file","filename":"/var/lib/avocado/data/avocado-vt/images/vdb.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-ccw","devno":"fe.0.0000","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}' \
-netdev '{"type":"tap","fd":"26","vhost":true,"vhostfd":"28","id":"hostnet0"}' \
-device '{"driver":"virtio-net-ccw","netdev":"hostnet0","id":"net0","mac":"52:54:00:9f:d9:06","devno":"fe.0.0001"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"sclpconsole","chardev":"charserial0","id":"serial0"}' \
-chardev socket,id=charchannel0,fd=24,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev pty,id=charconsole1 \
-device '{"driver":"virtconsole","chardev":"charconsole1","id":"console1"}' \
-device '{"driver":"virtio-keyboard-ccw","id":"input0","devno":"fe.0.0005"}' \
-device '{"driver":"virtio-mouse-ccw","id":"input1","devno":"fe.0.0006"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 127.0.0.1:0,audiodev=audio1 \
-device '{"driver":"virtio-gpu-ccw","id":"video0","max_outputs":1,"devno":"fe.0.0002"}' \
-device '{"driver":"virtio-balloon-ccw","id":"balloon0","devno":"fe.0.0007"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-device '{"driver":"vhost-vsock-ccw","id":"vsock0","guest-cid":3,"vhostfd":"21","devno":"fe.0.0008"}' \
-msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
char device redirected to /dev/pts/2 (label charconsole1)
2023-06-12 09:27:02.663+0000: panic s390: core='0' psw-mask='0x0002000180000000' psw-addr='0x0000000000000000' reason='disabled-wait'
2023-06-12 09:27:02.665+0000: shutting down, reason=crashed
2023-06-12T09:27:02.666824Z qemu-kvm: terminating on signal 15 from pid 1137767 (<unknown process>)
The guest OS panic-ked after the initrd was rebuilt. Could you please try to see what the console prints? This doesn't really look like either a libvirt or a qemu problem. Perhaps dracut didn't pick up a required module for the root filesystem? Hi @chwen , did you re-install the bootloader after updating the initrd? i.e. did you run "zipl" after running "dracut ..." ? On s390x, we don't use grub, so you have to make sure that "zipl" has been run each time you changed something with regards to the kernel or initrd. According to comment 14, file a new bug to track comment 3 and comment 4: https://bugzilla.redhat.com/show_bug.cgi?id=2219538 |
Description of problem: s390x guest can not be started on rhel9.3 after executing dracut --force --add-drivers 'virtio_scsi' in guest internal Version-Release number of selected component (if applicable): libvirt-9.4.0-1.el9.s390x qemu-kvm-8.0.0-4.el9.s390x How reproducible: 100% Steps to Reproduce: 1. Start a vm with configuration as below: <domain type='kvm'> <name>avocado-vt-vm1</name> <uuid>c1d24967-efcf-4ef1-9fe0-4cf7188826cf</uuid> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>2</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='s390x' machine='s390-ccw-virtio-rhel9.2.0'>hvm</type> <boot dev='hd'/> </os> <cpu mode='host-model' check='partial'/> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-s390x.qcow2'/> <backingStore/> <target dev='vda' bus='virtio'/> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/> </disk> <controller type='virtio-serial' index='0'> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0004'/> </controller> <controller type='pci' index='0' model='pci-root'/> <interface type='network'> <mac address='52:54:00:72:66:93'/> <source network='default'/> <model type='virtio'/> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/> </interface> <serial type='pty'> <target type='sclp-serial' port='0'> <model name='sclpconsole'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <console type='pty'> <target type='virtio' port='1'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='keyboard' bus='virtio'> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0005'/> </input> <input type='mouse' bus='virtio'> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0006'/> </input> <graphics type='vnc' port='-1' autoport='yes'> <listen type='address'/> </graphics> <audio id='1' type='none'/> <video> <model type='virtio' heads='1' primary='yes'/> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/> </video> <memballoon model='virtio'> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0007'/> </memballoon> <panic model='s390'/> <vsock model='virtio'> <cid auto='yes'/> <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0008'/> </vsock> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'/> </domain> 2. Start and login into the vm, and execute dracut --force --add-drivers 'virtio_scsi' and shutdown commands in guest internal [root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh console avocado-vt-vm1 Connected to domain 'avocado-vt-vm1' Escape character is ^] (Ctrl + ]) localhost login: root Password: [root@localhost ~]# dracut --force --add-drivers 'virtio_scsi' [root@localhost ~]# [root@localhost ~]# [root@localhost ~]# echo $? 0 [root@localhost ~]# shutdown Shutdown scheduled for Sun 2023-06-11 23:32:59 EDT, use 'shutdown -c' to cancel. [root@localhost ~]# dracut Warning: Killing all remaining processes [ 217.472019] dracut Warning: Killing all remaining processes dracut Warning: Unmounted /oldroot. [ 217.505014] dracut Warning: Unmounted /oldroot. Powering off. [root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh list --all Id Name State --------------------------------- - avocado-vt-vm1 shut off - vm2 shut off 3. Start Vm again: [root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh start avocado-vt-vm1 Domain 'avocado-vt-vm1' started [root@s390x-kvm-virtqez1 .libvirt-ci-venv-ci-runtest-SG680a]# virsh list --all Id Name State --------------------------------- - avocado-vt-vm1 shut off - vm2 shut off 4. Actual results: VM can not be started 5. Expected results: VM expect to be started successfully