Bug 1746404
| Summary: | [s390x] Importing qcow2 image fails with libvirt.libvirtError: operation failed: domain 'rhel8.1' is already being removed | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Alexander Todorov <atodorov> | ||||||
| Component: | virt-manager | Assignee: | Pavel Hrdina <phrdina> | ||||||
| Status: | CLOSED WORKSFORME | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | low | ||||||||
| Version: | 8.1 | CC: | dzheng, juzhou, mzhan, phrdina, rbalakri, smitterl, thuth, tzheng, virt-maint, wchadwic | ||||||
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
||||||
| Target Release: | 8.1 | ||||||||
| Hardware: | s390x | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-02-21 22:06:05 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 1746399 | ||||||||
| Bug Blocks: | |||||||||
| Attachments: |
|
||||||||
|
Description
Alexander Todorov
2019-08-28 11:52:46 UTC
Hi all, since I cannot reproduce this issue on x86_64 platform with packages' version: virt-manager-2.2.1-2.el8.noarch libvirt-4.5.0-33.module+el8.1.0+4066+0f1aadab.x86_64 qemu-kvm-2.12.0-85.module+el8.1.0+4066+0f1aadab.x86_64 I change hardware to s390x, thanks. BR, juzhou. Reproduced: lorax-composer-28.14.30-1.el8.s390x libvirt-4.5.0-32.module+el8.1.0+4010+d6842f29.s390x virt-install-2.2.1-2.el8.noarch Steps 1. composer-cli compose start example-atlas qcow2 2. Copy image to libvirt storage pool 3. virt-install --name rhel8.1 --memory 2048 --vcpus 1 --disk /var/lib/libvirt/images/examples.atlas.s390x.lorax-composed.qcow2 --import --os-variant rhel8.1 >>> ERROR operation failed: domain 'rhel8.1' is already being removed virst-install.log [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (cli:208) Launched with command line: /usr/share/virt-manager/virt-install --name rhel8.1 --memory 2048 --vcpus 1 --disk /var/lib/libvirt/images/examples.atlas.s390x.lorax-composed.qcow2 --import --os-variant rhel8.1 [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (virt-install:207) Distilled --network options: ['default'] [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (virt-install:139) Distilled --disk options: ['/var/lib/libvirt/images/examples.atlas.s390x.lorax-composed.qcow2'] [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (cli:224) Requesting libvirt URI default [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (cli:227) Received libvirt URI qemu:///system [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (guest:463) Setting Guest osinfo name <_OsVariant name=generic> [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (installer:396) No media for distro detection. [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (installer:398) installer.detect_distro returned=None [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (guest:463) Setting Guest osinfo name <_OsVariant name=rhel8.1> [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (osdict:326) No recommended value found for key='n-cpus', using minimum=1 * 2 [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (virt-install:648) Guest.has_install_phase: False [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (cli:272) Starting install... [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (installer:442) Generated install XML: None required [Thu, 29 Aug 2019 04:18:24 virt-install 56875] DEBUG (installer:443) Generated boot XML: <domain type="kvm"> <name>rhel8.1</name> <uuid>725b7e88-6ecb-4c6a-bdf7-ad0e236b918c</uuid> <metadata> <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <libosinfo:os id="http://redhat.com/rhel/8.1"/> </libosinfo:libosinfo> </metadata> <memory>2097152</memory> <currentMemory>2097152</currentMemory> <vcpu>1</vcpu> <os> <type arch="s390x" machine="s390-ccw-virtio">hvm</type> <boot dev="hd"/> </os> <clock offset="utc"/> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2"/> <source file="/var/lib/libvirt/images/examples.atlas.s390x.lorax-composed.qcow2"/> <target dev="vda" bus="virtio"/> </disk> <interface type="network"> <source network="default"/> <mac address="52:54:00:2e:a4:be"/> <model type="virtio"/> </interface> <console type="pty"> <target type="sclp"/> </console> <channel type="unix"> <source mode="bind"/> <target type="virtio" name="org.qemu.guest_agent.0"/> </channel> <memballoon model="virtio"/> <rng model="virtio"> <backend model="random">/dev/urandom</backend> </rng> </devices> </domain> [Thu, 29 Aug 2019 04:18:25 virt-install 56875] DEBUG (cli:263) File "/usr/share/virt-manager/virt-install", line 1005, in <module> sys.exit(main()) File "/usr/share/virt-manager/virt-install", line 999, in main start_install(guest, installer, options) File "/usr/share/virt-manager/virt-install", line 686, in start_install fail(e, do_exit=False) File "/usr/share/virt-manager/virtinst/cli.py", line 263, in fail log.debug("".join(traceback.format_stack())) [Thu, 29 Aug 2019 04:18:25 virt-install 56875] ERROR (cli:264) operation failed: domain 'rhel8.1' is already being removed [Thu, 29 Aug 2019 04:18:25 virt-install 56875] DEBUG (cli:266) Traceback (most recent call last): File "/usr/share/virt-manager/virt-install", line 659, in start_install transient=options.transient) File "/usr/share/virt-manager/virtinst/install/installer.py", line 544, in start_install doboot, transient) File "/usr/share/virt-manager/virtinst/install/installer.py", line 493, in _create_guest domain = self.conn.defineXML(final_xml) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3743, in defineXML if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self) libvirt.libvirtError: operation failed: domain 'rhel8.1' is already being removed [Thu, 29 Aug 2019 04:18:25 virt-install 56875] DEBUG (cli:278) Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start rhel8.1 otherwise, please restart your installation. Anaconda compose log attached. Created attachment 1609302 [details] lorax-composer.log for qcow2 image build Adding compose log as requested on related issue https://bugzilla.redhat.com/show_bug.cgi?id=1746399 Created attachment 1609303 [details]
lorax-program.log for qcow2 image build
I've debugged this issue and the conclusion is that it's not virt-manager nor libvirt bug. What happens is that the guest crashes, libvirt receives panic from QEMU. Now the question is whether it's a QEMU bug or lorax-composer bug. When I downloaded the rhel8 qcow image [1] and used that image instead of the lorax qcow2 image everything worked fine. But when I tried to use empty qcow2 image created using qemu-img create -f qcow2 test.qcow2 100M it also panicked the same way as with the lorax image. I'm not familiar with s390 architecture and I have no idea why it would panic with empty or possibly incorrectly build image so I'm moving this BZ to QEMU. Same thing happens if I run /usr/libexec/qemu-kvm /path/to/image where the rhel image [1] works fine but empty qcow2 image and the lorax one exits immediately. [1] <http://download.devel.redhat.com/rhel-8/rel-eng/RHEL-8/latest-RHEL-8/compose/BaseOS/s390x/images/rhel-guest-image-8.1-167.s390x.qcow2> (In reply to Pavel Hrdina from comment #11) > I've debugged this issue and the conclusion is that it's not virt-manager > nor libvirt > bug. What happens is that the guest crashes, libvirt receives panic from > QEMU. > > Now the question is whether it's a QEMU bug or lorax-composer bug. > > When I downloaded the rhel8 qcow image [1] and used that image instead of > the lorax qcow2 > image everything worked fine. But when I tried to use empty qcow2 image > created using > > qemu-img create -f qcow2 test.qcow2 100M > > it also panicked the same way as with the lorax image. I'm not familiar > with s390 architecture > and I have no idea why it would panic with empty or possibly incorrectly > build image so I'm moving this BZ to QEMU. It's the "expected" behavior of the guest firmware on s390x that it shuts down the guest (with a panic message IIRC). Normal x86 or ppc64 firmware simply drops the user of the guest to the firmware prompt or menu when there is no way to boot, but the guest firmware on s390x is non-interactive and thus simply shuts down the guest when it fails to boot. So the main question is: Why is the guest image here not bootable here? ... thus we should have a closer look at BZ 1746399, I guess... Anyway, the error messages from libvirt / virt-install do not really sound helpful here ... could that be improved somehow? After having a look at an image created with "composer-cli", it's indeed the zipl bootloader that is missing there. So this bug here is just a subsequent error of BZ 1746399. There is nothing we can do here from the qemu-kvm side. The only difference on s390x is really that the guest firmware shuts down the guest if the disk is unbootable, while the x86 bios simply waits forever after printing out a "No bootable device" message. I think virt-install should be able to handle the guest shutdown in a more graceful way and print a more appropriate error message instead, so I'm assigning the BZ back to virt-manager. FWIW, there also seems to be a race here, I do not always get the strange error message. Sometimes I get:
# qemu-img create -f qcow2 /var/lib/libvirt/images/empty.qcow2 8G
Formatting '/var/lib/libvirt/images/empty.qcow2', fmt=qcow2 size=8589934592 cluster_size=65536 lazy_refcounts=off refcount_bits=16
# virt-install --name testguest --memory 2048 --vcpus 1 --disk /var/lib/libvirt/images/empty.qcow2 --import --os-variant rhel8.1 --debug
[...]
[Mon, 02 Sep 2019 06:56:10 virt-install 180715] DEBUG (cli:404) Connecting to text console
[Mon, 02 Sep 2019 06:56:10 virt-install 180715] DEBUG (cli:370) Running: virsh --connect qemu:///system console testguest
error: The domain is not running
[Mon, 02 Sep 2019 06:56:10 virt-install 180715] DEBUG (virt-install:705) Domain state after install: 5
[Mon, 02 Sep 2019 06:56:10 virt-install 180715] DEBUG (cli:272) Domain creation completed.
Domain creation completed.
[Mon, 02 Sep 2019 06:56:10 virt-install 180715] DEBUG (cli:272) You can restart your domain by running:
virsh --connect qemu:///system start testguest
You can restart your domain by running:
virsh --connect qemu:///system start testguest
But often I also get that confusing error message (that IMHO should be fixed):
# virt-install --name testguest --memory 2048 --vcpus 1 --disk /var/lib/libvirt/images/empty.qcow2 --import --os-variant rhel8.1 --debug
[...]
[Mon, 02 Sep 2019 06:58:36 virt-install 180932] DEBUG (cli:263) File "/usr/share/virt-manager/virt-install", line 1005, in <module>
sys.exit(main())
File "/usr/share/virt-manager/virt-install", line 999, in main
start_install(guest, installer, options)
File "/usr/share/virt-manager/virt-install", line 686, in start_install
fail(e, do_exit=False)
File "/usr/share/virt-manager/virtinst/cli.py", line 263, in fail
log.debug("".join(traceback.format_stack()))
[Mon, 02 Sep 2019 06:58:36 virt-install 180932] ERROR (cli:264) operation failed: domain 'testguest' is already being removed
[Mon, 02 Sep 2019 06:58:36 virt-install 180932] DEBUG (cli:266)
Traceback (most recent call last):
File "/usr/share/virt-manager/virt-install", line 659, in start_install
transient=options.transient)
File "/usr/share/virt-manager/virtinst/install/installer.py", line 544, in start_install
doboot, transient)
File "/usr/share/virt-manager/virtinst/install/installer.py", line 493, in _create_guest
domain = self.conn.defineXML(final_xml)
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3743, in defineXML
if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
libvirt.libvirtError: operation failed: domain 'testguest' is already being removed
[Mon, 02 Sep 2019 06:58:36 virt-install 180932] DEBUG (cli:278) Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
virsh --connect qemu:///system start testguest
otherwise, please restart your installation.
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
virsh --connect qemu:///system start testguest
otherwise, please restart your installation.
Thanks Thomas for debugging it from QEMU POV. Removing blocker from this BZ because even if we fix the error reported by virt-manager it will still fail to install the VM since there is different lorax bug that is the root cause of the issue. If the issue needs to be fixed and the qcow2 image should boot successfully please escalate as blocker in the referenced lorax BZ 1746399. I've retested with a recent snapshot and the zipl error is gone. I am able to boot the VM from the newly built image and there is no traceback in virt-manager. The VM itself fails with a dracut timeout but that is unrelated. Closing. |