Bug 1701710 - VM turns on, uses a lot of CPU, uses almost no memory, indefinite black console screen
Summary: VM turns on, uses a lot of CPU, uses almost no memory, indefinite black conso...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: edk2
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Paolo Bonzini
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-21 00:28 UTC by Chris Murphy
Modified: 2019-08-15 18:08 UTC (History)
7 users (show)

Fixed In Version: edk2-20190501stable-2.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-15 18:08:34 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
vm dumpxml (7.92 KB, text/plain)
2019-04-21 00:29 UTC, Chris Murphy
no flags Details
ps aux (3.43 KB, text/plain)
2019-04-21 00:30 UTC, Chris Murphy
no flags Details
journal excerpt (15.43 KB, text/plain)
2019-04-21 00:34 UTC, Chris Murphy
no flags Details
"Red Hat Secure Boot (PK/KEK key 1)", for comment 12 (1.29 KB, text/plain)
2019-05-16 20:18 UTC, Laszlo Ersek
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1666941 0 low CLOSED UEFI guest cannot boot into os when setting some special memory size 2021-02-22 00:41:40 UTC
TianoCore 1814 0 None None None 2019-07-12 22:38:27 UTC
TianoCore 1859 0 None None None 2019-07-12 22:38:27 UTC

Description Chris Murphy 2019-04-21 00:28:14 UTC
Description of problem:

Short version is it doesn't work out of the box. I'm only getting a black console screen on virt-manager with high CPU but almost no memory usage, no errors, and nothing else happens as in no progression beyond just high CPU usage and a black console.

This is all happening on a single computer.

Version-Release number of selected component (if applicable):

virt-manager-2.1.0-2.fc30.noarch

edk2-ovmf-20190308stable-1.fc30.noarch
ipxe-roms-qemu-20190125-1.git36a4c85f.fc30.noarch
libvirt-5.1.0-4.fc30.x86_64
libvirt-bash-completion-5.1.0-4.fc30.x86_64
libvirt-client-5.1.0-4.fc30.x86_64
libvirt-daemon-5.1.0-4.fc30.x86_64
libvirt-daemon-config-network-5.1.0-4.fc30.x86_64
libvirt-daemon-config-nwfilter-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-interface-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-libxl-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-lxc-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-network-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-nodedev-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-nwfilter-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-qemu-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-secret-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-core-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-disk-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-gluster-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-iscsi-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-iscsi-direct-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-logical-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-mpath-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-rbd-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-scsi-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-sheepdog-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-storage-zfs-5.1.0-4.fc30.x86_64
libvirt-daemon-driver-vbox-5.1.0-4.fc30.x86_64
libvirt-daemon-kvm-5.1.0-4.fc30.x86_64
libvirt-dbus-1.3.0-2.fc30.x86_64
libvirt-gconfig-2.0.0-3.fc30.x86_64
libvirt-glib-2.0.0-3.fc30.x86_64
libvirt-gobject-2.0.0-3.fc30.x86_64
libvirt-libs-5.1.0-4.fc30.x86_64
python3-libvirt-5.1.0-1.fc30.x86_64
qemu-audio-alsa-3.1.0-7.fc30.x86_64
qemu-audio-oss-3.1.0-7.fc30.x86_64
qemu-audio-pa-3.1.0-7.fc30.x86_64
qemu-audio-sdl-3.1.0-7.fc30.x86_64
qemu-block-curl-3.1.0-7.fc30.x86_64
qemu-block-dmg-3.1.0-7.fc30.x86_64
qemu-block-iscsi-3.1.0-7.fc30.x86_64
qemu-block-nfs-3.1.0-7.fc30.x86_64
qemu-block-rbd-3.1.0-7.fc30.x86_64
qemu-block-ssh-3.1.0-7.fc30.x86_64
qemu-common-3.1.0-7.fc30.x86_64
qemu-guest-agent-3.1.0-7.fc30.x86_64
qemu-img-3.1.0-7.fc30.x86_64
qemu-kvm-3.1.0-7.fc30.x86_64
qemu-system-x86-3.1.0-7.fc30.x86_64
qemu-system-x86-core-3.1.0-7.fc30.x86_64
qemu-ui-curses-3.1.0-7.fc30.x86_64
qemu-ui-gtk-3.1.0-7.fc30.x86_64
qemu-ui-sdl-3.1.0-7.fc30.x86_64


How reproducible:
Always


Steps to Reproduce:
1. Clean install Fedora-Workstation-Live-x86_64-30-20190415.n.2.iso
2. Update with latest u-t
3. `dnf group install Virtualization`
4. `dnf install libvirt libvirt-dbus` because those are installed on two unrelated Fedora 29->30 systems.
5. Create a new machine using the wizard, picking a local ISO for F30server, and a precreated raw file using fallocate, both are in /var/lib/libvirt/images/ are are selected in the wizard without problem.
6. Choose the checkbox to customize
7. Change from BIOS to UEFI firmware
8. Start the VM

Actual results:

Indefinite black screen, no error.


Expected results:

I should see Tianocore splash screen and then GRUB menu, or an error.

Additional info:

Comment 1 Chris Murphy 2019-04-21 00:29:27 UTC
Created attachment 1556789 [details]
vm dumpxml

Comment 2 Chris Murphy 2019-04-21 00:30:44 UTC
Created attachment 1556790 [details]
ps aux

show the qemu command virt-manager issued to start the VM

Comment 3 Chris Murphy 2019-04-21 00:34:40 UTC
Created attachment 1556791 [details]
journal excerpt

journalctl -f, and then turn on the VM (button with the play icon) this is everything recorded over 5 minutes of black screen

Comment 4 Chris Murphy 2019-04-21 00:37:20 UTC
Curiously, this exact same machine was running virt-manager just fine with both Fedora 28 and Fedora 29 and each of those upgraded (dnf system-upgrade) to Fedora 30.

Only the clean install of Fedora 30 is failing to work.

Comment 5 Chris Murphy 2019-04-21 01:28:00 UTC
Whenever I set memory to 2560, this happens. If I change it to 2048 or 3072 the problem doesn't happen. A value of 2560 works fine on a Fedora 29 Server.

Anyway, it's either a regression bug in qemu, I guess? Or if it's now the correct behavior, then it's a bug in virt-manager for letting me enter an invalid value.

Comment 6 Cole Robinson 2019-04-23 14:41:07 UTC
Thanks for the report. Sounds strange indeed having a specific memory value tickling this. I'll try to reproduce

Comment 7 Cole Robinson 2019-04-23 16:03:53 UTC
I can reproduce. It seems to be the combo of machine=q35 and edk2/uefi and the memory value. I tested both edk2-20190308stable-1.fc30 and older edk2-20180815gitcb5f4f45ce-5.fc30

With memory=2560, q35 + edk2 fails (no tianocore splash screen). q35+seabios and pc+edk2 work. With memory=2048 I don't see any failure. Other non-round memory values seem to tickle it too but I didn't investigate thoroughly

I was testing with qemu-4.0.0-0.4.rc1.fc30.x86_64 which is newer, so 3.1 and 4.0 are affected. VM config was roughly similar to what Chris posted above.

Laszlo have you heard of an issue like this?

Comment 8 Laszlo Ersek 2019-04-24 08:53:36 UTC
Yes, this is a known issue, and as much as I dislike it, it is
unfixable. Please use memory sizes that work.

For details, please refer to bug 1666941 (up to and including
bug 1666941 comment 13). The short version is that there are two
problems with arbitrary memory sizes:
(a) PCIEXBAR placement disagreement between QEMU and OVMF,
(b) MTRR configuration.

Problem (a) is fixable (you can find the patches in
bug 1666941 comment 14), but fixing just (a) is insufficient, to remove
the symptom.

Problem (b) is unfixable (see bug 1666941 comment 13). If a small RAM
size is needed, please pick one that works, e.g. from the following
list:
-  512 MB
-  768 MB
- 1024 MB
- 1536 MB
- 2048 MB
- 3072 MB
- 4096 MB

My apologies.

Comment 9 Cole Robinson 2019-04-24 21:18:05 UTC
Thanks Laszlo, that's interesting. If this is CANTFIX at the edk2 level then IMO we need to do something at a higher level, the failure scenario here of 'black screen VM that doesn't work' is horrible and going to generate lots of bug reports I imagine. Maybe libvirt or qemu can catch this situation and error for starters. I'll start a thread internally, I'm sure someone can come up with some clever idea.

Comment 11 Laszlo Ersek 2019-05-16 19:36:11 UTC
Upstream commits:

$ git log --oneline --reverse 3b7a897cd8e3..39b9a5ffe661 | cat -n
     1  60e95bf5094f OvmfPkg/PlatformPei: assign PciSize on both i440fx/q35 branches explicitly
     2  9a2e8d7c65ef OvmfPkg/PlatformPei: hoist PciBase assignment above the i440fx/q35 branching
     3  75136b29541b OvmfPkg/PlatformPei: reorder the 32-bit PCI window vs. the PCIEXBAR on q35
     4  39b9a5ffe661 OvmfPkg/PlatformPei: fix MTRR for low-RAM sizes that have many bits clear

Comment 12 Laszlo Ersek 2019-05-16 20:16:08 UTC
Cole, Gerd, Paolo, in place of picking the above patches, you might want
to rebase edk2 to the upcoming edk2-stable201905 release.

  Date (00:00:00 UTC-8)  Description
  ---------------------  ------------------------
  2018-03-08 (12PM)      Beginning of development
  2019-05-17             Soft Feature Freeze
  2019-05-24             Hard Feature Freeze
  2019-05-31             Release

Whenever you rebase to edk2-stable201905, please remember that the
following "-D" build flags have been renamed:

- HTTP_BOOT_ENABLE --> NETWORK_HTTP_BOOT_ENABLE
- TLS_ENABLE       --> NETWORK_TLS_ENABLE

Due to <https://bugzilla.tianocore.org/show_bug.cgi?id=1293>.


Other upstream BZs fixed in this release that are important for
ArmVirtQemu and/or OVMF are listed below. (Note that the upstream list
of proposed BZs that I'm picking the below from is not final yet --
check at <https://github.com/tianocore/edk2/releases/> once the upstream
release has been cut.)

* Update OpenSSL version to upcoming 1.1.1
  https://bugzilla.tianocore.org/show_bug.cgi?id=1089

  Downstream TODO: use Fedora's "OpenSSL-1.1.1b".

  (This upstream BZ might not make the edk2-stable201905 release.)

* Replace BSD 2-Clause License with BSD + Patent Licence
  https://bugzilla.tianocore.org/show_bug.cgi?id=1373

  Downstream TODO: update License tags in the spec file. The short name
  "BSD" should be replaced with "BSD-2-Clause-Patent" -- see at
  <https://fedoraproject.org/wiki/Licensing:Main?rd=Licensing#Good_Licenses>.
  Edk2 code covered by different licenses such as MIT is not affected by
  this re-licensing.

* Upstream the EnrollDefaultKeys application to OvmfPkg
  https://bugzilla.tianocore.org/show_bug.cgi?id=1747

  Downstream TODO:

  - Drop the downstream-only patch adding EnrollDefaultKeys.

  - Open a ticket with <secalert> so that they send you the
    self-signed X509 certificate for the following entity, in PEM
    format:

    "Red Hat Secure Boot (PK/KEK key 1)/emailAddress=secalert"
    SHA1: fdfc7f3c7ef3e05776add79878216c9be0e19597
    SHA256: 7f122fab825041c2b0c67696bb151db6f39fed7df2d1104107f5b892b354ef5c

    (I'm going to attach the same for convenience, but verify with
    ProdSec, just to be sure.)

    Package this certificate as another source file in the SRPM.

  - Refer to the "sed" command in
    <https://bugzilla.tianocore.org/show_bug.cgi?id=1747#c2> for
    extracting the raw base64 string from the PEM-encoded certificate,
    in the spec file.

  - Refer to <https://bugzilla.tianocore.org/show_bug.cgi?id=1747#c7>
    for passing the string to QEMU, when you launch
    "EnrollDefaultKeys.efi", also in the spec file.

Thanks.

Comment 13 Laszlo Ersek 2019-05-16 20:18:39 UTC
Created attachment 1569795 [details]
"Red Hat Secure Boot (PK/KEK key 1)", for comment 12

Comment 14 Cole Robinson 2019-05-16 20:33:48 UTC
Thanks for the work Laszlo! I'm fine with waiting for the stable release

Comment 15 Laszlo Ersek 2019-05-21 21:43:02 UTC
(In reply to Laszlo Ersek from comment #12)

> * Upstream the EnrollDefaultKeys application to OvmfPkg
>   https://bugzilla.tianocore.org/show_bug.cgi?id=1747
> 
>   Downstream TODO:
> 
>   - Drop the downstream-only patch adding EnrollDefaultKeys.
> 
>   - Open a ticket with <secalert> so that they send you the
>     self-signed X509 certificate for the following entity, in PEM
>     format:
> 
>     "Red Hat Secure Boot (PK/KEK key 1)/emailAddress=secalert"
>     SHA1: fdfc7f3c7ef3e05776add79878216c9be0e19597
>     SHA256: 7f122fab825041c2b0c67696bb151db6f39fed7df2d1104107f5b892b354ef5c
> 
>     (I'm going to attach the same for convenience, but verify with
>     ProdSec, just to be sure.)
> 
>     Package this certificate as another source file in the SRPM.
> 
>   - Refer to the "sed" command in
>     <https://bugzilla.tianocore.org/show_bug.cgi?id=1747#c2> for
>     extracting the raw base64 string from the PEM-encoded certificate,
>     in the spec file.
> 
>   - Refer to <https://bugzilla.tianocore.org/show_bug.cgi?id=1747#c7>
>     for passing the string to QEMU, when you launch
>     "EnrollDefaultKeys.efi", also in the spec file.

It's also necessary to use qemu-ovmf-secureboot @ commit f158f12be754 ("Pass OEM Strings to the guest", 2019-05-21) or later, for this.

https://github.com/puiterwijk/qemu-ovmf-secureboot/commit/f158f12be75482489b64330f7c5ee8b8138f04c7

Thanks.

Comment 16 Laszlo Ersek 2019-06-03 16:56:39 UTC
Another note on the rebase to edk2-stable201905: edk2 has gained a new git submodule, namely "SoftFloat" (<https://github.com/ucb-bar/berkeley-softfloat-3.git>). This should be added as another tarball (= source file) in the SRPM, most likely.

Comment 17 Laszlo Ersek 2019-06-06 13:37:04 UTC
edk2-stable201905 has been released today.

https://github.com/tianocore/edk2/releases/tag/edk2-stable201905

Comment 18 Fedora Update System 2019-07-15 19:58:21 UTC
FEDORA-2019-d47a9d4b8b has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-d47a9d4b8b

Comment 19 Fedora Update System 2019-07-16 00:54:30 UTC
edk2-20190501stable-2.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d47a9d4b8b

Comment 20 Fedora Update System 2019-08-15 18:08:34 UTC
edk2-20190501stable-2.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.