Bug 1279320 - ibm s390x kvm VM gets stuck in a crash loop
ibm s390x kvm VM gets stuck in a crash loop
Status: CLOSED DEFERRED
Product: Virtualization Tools
Classification: Community
Component: virt-manager (Show other bugs)
unspecified
x86_64 Linux
unspecified Severity high
: ---
: ---
Assigned To: Cole Robinson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-09 02:42 EST by kevinzs
Modified: 2015-11-11 21:12 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-11 10:03:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description kevinzs 2015-11-09 02:42:39 EST
Description of problem:
Virt-manager in amd64, connect libvirtd in s390x, install a guest OS redhat 7.0 in s390x.
If do not specify kernel and initrd path, We can create the new guest (with minimal capacity and default NAT for network), but as soon as the guest starts, virt-manager tries to open a console connection to it, and hangs. The --debug output from virt-manager looks like this:

[Fri, 30 Oct 2015 09:43:50 virt-manager 11927] DEBUG (connection:753) domain lifecycle event: domain=zamsaad_guest event=4 reason=0
[Fri, 30 Oct 2015 09:43:50 virt-manager 11927] DEBUG (connection:753) domain lifecycle event: domain=zamsaad_guest event=8 reason=0
[Fri, 30 Oct 2015 09:43:50 virt-manager 11927] DEBUG (connection:753) domain lifecycle event: domain=zamsaad_guest event=4 reason=0
[Fri, 30 Oct 2015 09:43:50 virt-manager 11927] DEBUG (connection:753) domain lifecycle event: domain=zamsaad_guest event=8 reason=0

Also , when we specify the kernel and initrd image path ,enable direct kernel boot, virt-manager will open guest install interface . But in this time, force off the guest ,disable direct kernel boot , virt-manager will go into loop like above.

Version-Release number of selected component (if applicable):
git clone from virt-tools . newest version

How reproducible:


Steps to Reproduce:
1.new connect to remote libvirtd using ssh
2.select Rhel7.0 iso,select image size 8G ,select virtual cpu and memory.
3.Do not specify kernel and initrd path, click finish to begin guest install

Actual results:
[一, 09 11月 2015 14:16:31 virt-manager 32128] DEBUG (guest:396) Generated install XML: 
<domain type="kvm">
  <name>generic</name>
  <uuid>d7ed24c8-d7e7-4025-a1d2-2abcc9161599</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch="s390x" machine="s390-ccw-virtio">hvm</type>
    <boot dev="cdrom"/>
    <boot dev="hd"/>
  </os>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/generic-1.qcow2"/>
      <target dev="vda" bus="virtio"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/var/lib/libvirt/images/RHEL-7.0-20140507.0-Server-s390x-dvd1.iso"/>
      <target dev="sda" bus="scsi"/>
      <readonly/>
    </disk>
    <interface type="network">
      <source network="default"/>
      <mac address="52:54:00:c5:5b:ee"/>
    </interface>
    <console type="pty"/>
  </devices>
</domain>

[一, 09 11月 2015 14:16:31 virt-manager 32128] DEBUG (guest:397) Generated boot XML: 
<domain type="kvm">
  <name>generic</name>
  <uuid>d7ed24c8-d7e7-4025-a1d2-2abcc9161599</uuid>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch="s390x" machine="s390-ccw-virtio">hvm</type>
    <boot dev="hd"/>
  </os>
  <clock offset="utc"/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/generic-1.qcow2"/>
      <target dev="vda" bus="virtio"/>
    </disk>
    <disk type="block" device="cdrom">
      <target dev="sda" bus="scsi"/>
      <readonly/>
    </disk>
    <interface type="network">
      <source network="default"/>
      <mac address="52:54:00:c5:5b:ee"/>
    </interface>
    <console type="pty"/>
  </devices>
</domain>

[一, 09 11月 2015 14:16:32 virt-manager 32128] DEBUG (connection:754) domain lifecycle event: domain=generic event=4 reason=0
[一, 09 11月 2015 14:16:32 virt-manager 32128] DEBUG (connection:754) domain lifecycle event: domain=generic event=2 reason=0
[一, 09 11月 2015 14:16:32 virt-manager 32128] DEBUG (connection:754) domain lifecycle event: domain=generic event=8 reason=0
[一, 09 11月 2015 14:16:32 virt-manager 32128] DEBUG (connection:754) domain lifecycle event: domain=generic event=5 reason=2
[一, 09 11月 2015 14:16:32 virt-manager 32128] DEBUG (connection:754) domain lifecycle event: domain=generic event=0 reason=0
[一, 09 11月 2015 14:16:33 virt-manager 32128] DEBUG (guest:427) XML fetched from libvirt object:
<domain type='kvm'>
  <name>generic</name>
  <uuid>d7ed24c8-d7e7-4025-a1d2-2abcc9161599</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='s390x' machine='s390-ccw-kvmibm-1.1.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/generic-1.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='sda' bus='scsi'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0'>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0002'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0003'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:c5:5b:ee'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/>
    </interface>
    <console type='pty'>
      <target type='virtio' port='0'/>
    </console>
    <memballoon model='virtio'>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0004'/>
    </memballoon>
  </devices>
</domain>

[一, 09 11月 2015 14:16:33 virt-manager 32128] DEBUG (create:2291) Install completed
[一, 09 11月 2015 14:16:36 virt-manager 32128] DEBUG (connection:1050) domain=generic status=Shutoff added
[一, 09 11月 2015 14:16:36 virt-manager 32128] DEBUG (connection:1044) New domain=generic requested, but it's already tracked.
[一, 09 11月 2015 14:16:36 virt-manager 32128] DEBUG (connection:1044) New domain=generic requested, but it's already tracked.
[一, 09 11月 2015 14:16:36 virt-manager 32128] DEBUG (connection:1044) New domain=generic requested, but it's already tracked.


Expected results:
Do not try to restart again and again ,or try restart several times then stop.

Additional info:
Comment 1 Cole Robinson 2015-11-10 16:16:27 EST
My guess is that something is causing qemu to repeatedly reset. Can you provide /var/log/libvirt/qemu/$vmname.log from the libvirtd server?
Comment 2 kevinzs 2015-11-10 21:45:06 EST
Hi Cole,
     Here is the log information : 
     /var/log/libvirt/qemu/generic.log, and virt-manager shows that the generic has the states "crashed"

2015-11-11 02:37:08.963+0000: starting up libvirt version: 1.2.20, package: 1.el7_1.kvmibm1_1_1.2 (Koji, 2015-11-02-15:14:28, lnx2), qemu version: 2.4.0
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name generic -S -machine s390-ccw-kvmibm-1.1.1,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 47653a1e-6726-4ef7-83ab-28acd68800b4 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -boot strict=on -device virtio-scsi-ccw,id=scsi0,devno=fe.0.0002 -drive file=/var/lib/libvirt/images/generic.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/var/lib/libvirt/images/RHEL-7.0-20140507.0-Server-s390x-dvd1.iso,if=none,id=drive-scsi0-0-0-0,readonly=on,format=raw -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:71:77:4d,devno=fe.0.0001 -chardev pty,id=charconsole0 -device sclpconsole,chardev=charconsole0,id=console0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 -msg timestamp=on
char device redirected to /dev/pts/1 (label charconsole0)
qemu: terminating on signal 15 from pid 1382
2015-11-11 02:37:09.399+0000: shutting down
2015-11-11 02:37:11.368+0000: starting up libvirt version: 1.2.20, package: 1.el7_1.kvmibm1_1_1.2 (Koji, 2015-11-02-15:14:28, lnx2), qemu version: 2.4.0
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name generic -S -machine s390-ccw-kvmibm-1.1.1,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 47653a1e-6726-4ef7-83ab-28acd68800b4 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device virtio-scsi-ccw,id=scsi0,devno=fe.0.0002 -drive file=/var/lib/libvirt/images/generic.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-scsi0-0-0-0,readonly=on,format=raw -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:71:77:4d,devno=fe.0.0001 -chardev pty,id=charconsole0 -device sclpconsole,chardev=charconsole0,id=console0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 -msg timestamp=on
char device redirected to /dev/pts/1 (label charconsole0)
Comment 3 kevinzs 2015-11-10 21:45:50 EST
(In reply to Cole Robinson from comment #1)
> My guess is that something is causing qemu to repeatedly reset. Can you
> provide /var/log/libvirt/qemu/$vmname.log from the libvirtd server?

Hi Cole,
     Thanks for your  help~
     Here is the log information : 
     /var/log/libvirt/qemu/generic.log, and virt-manager shows that the generic has the states "crashed"

2015-11-11 02:37:08.963+0000: starting up libvirt version: 1.2.20, package: 1.el7_1.kvmibm1_1_1.2 (Koji, 2015-11-02-15:14:28, lnx2), qemu version: 2.4.0
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name generic -S -machine s390-ccw-kvmibm-1.1.1,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 47653a1e-6726-4ef7-83ab-28acd68800b4 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -boot strict=on -device virtio-scsi-ccw,id=scsi0,devno=fe.0.0002 -drive file=/var/lib/libvirt/images/generic.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/var/lib/libvirt/images/RHEL-7.0-20140507.0-Server-s390x-dvd1.iso,if=none,id=drive-scsi0-0-0-0,readonly=on,format=raw -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:71:77:4d,devno=fe.0.0001 -chardev pty,id=charconsole0 -device sclpconsole,chardev=charconsole0,id=console0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 -msg timestamp=on
char device redirected to /dev/pts/1 (label charconsole0)
qemu: terminating on signal 15 from pid 1382
2015-11-11 02:37:09.399+0000: shutting down
2015-11-11 02:37:11.368+0000: starting up libvirt version: 1.2.20, package: 1.el7_1.kvmibm1_1_1.2 (Koji, 2015-11-02-15:14:28, lnx2), qemu version: 2.4.0
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name generic -S -machine s390-ccw-kvmibm-1.1.1,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 47653a1e-6726-4ef7-83ab-28acd68800b4 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-generic/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device virtio-scsi-ccw,id=scsi0,devno=fe.0.0002 -drive file=/var/lib/libvirt/images/generic.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-scsi0-0-0-0,readonly=on,format=raw -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0 -netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:71:77:4d,devno=fe.0.0001 -chardev pty,id=charconsole0 -device sclpconsole,chardev=charconsole0,id=console0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 -msg timestamp=on
char device redirected to /dev/pts/1 (label charconsole0)
Comment 4 Cole Robinson 2015-11-11 10:03:01 EST
If the VM is crashing then you need to dig a bit deeper to figure out why. Could be a bug there, or a virt-manager XML issue, or libvirt qemu command line issue, or user error

I'd suggest asking the KVM IBM folks, or take that qemu command line and try to manually make it work by hand. Closing as DEFERRED for now but please reopen if you determine virt-manager needs changes
Comment 5 kevinzs 2015-11-11 21:11:09 EST
(In reply to Cole Robinson from comment #4)
> If the VM is crashing then you need to dig a bit deeper to figure out why.
> Could be a bug there, or a virt-manager XML issue, or libvirt qemu command
> line issue, or user error
> 
> I'd suggest asking the KVM IBM folks, or take that qemu command line and try
> to manually make it work by hand. Closing as DEFERRED for now but please
> reopen if you determine virt-manager needs changes

Hi Cole,
     Thanks for your advice. I find that , when I don't specify the kernel and initrd ,  installation process has been skipped , and be treated successful. Then according to the boot xml, virt-manager will try to restart it again and again.
     Do you think I can debug this by getting the debug information of libvirtd in the server? 
     Also ,I don't know the domain lifecycle  event means , is it a event that sent by libvirtd in the server?
     Sincerely Thanks~
Comment 6 kevinzs 2015-11-11 21:12:00 EST
(In reply to Cole Robinson from comment #4)
> If the VM is crashing then you need to dig a bit deeper to figure out why.
> Could be a bug there, or a virt-manager XML issue, or libvirt qemu command
> line issue, or user error
> 
> I'd suggest asking the KVM IBM folks, or take that qemu command line and try
> to manually make it work by hand. Closing as DEFERRED for now but please
> reopen if you determine virt-manager needs changes

Hi Cole,
     Thanks for your advice. I find that , when I don't specify the kernel and initrd ,  installation process has been skipped , and be treated successful. Then according to the boot xml, virt-manager will try to restart it again and again.
     Do you think I can debug this by getting the debug information of libvirtd in the server? 
     Also ,I don't know the domain lifecycle  event means , is it a event that sent by libvirtd in the server?
     Sincerely Thanks~

Note You need to log in before you can comment on or make changes to this bug.