Bug 1097772 - Fedora cannot be booted in secure mode on Hyper-V
Summary: Fedora cannot be booted in secure mode on Hyper-V
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: shim
Version: 21
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Matthew Garrett
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-14 13:46 UTC by Sitsofe Wheeler
Modified: 2015-10-10 08:26 UTC (History)
13 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-02-12 19:35:27 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
/sys/firmware/efi/efivars/db-d719b2cb-3d3a-4596-a3bc-dad00e67656f (1.51 KB, application/octet-stream)
2014-05-14 13:46 UTC, Sitsofe Wheeler
no flags Details
Screenshot of failed boot - Fedora 21 Workstation Beta on Windows 8.1 Hyper-V (34.20 KB, image/png)
2014-11-20 00:37 UTC, M. Edward (Ed) Borasky
no flags Details
Error shown after Workstation ISO timeout. (16.70 KB, image/png)
2014-12-05 22:38 UTC, Carwyn Edwards
no flags Details
Error during setup - failed to set new efi boot target (105.82 KB, image/png)
2015-08-21 16:02 UTC, Alex
no flags Details

Description Sitsofe Wheeler 2014-05-14 13:46:15 UTC
Created attachment 895499 [details]
/sys/firmware/efi/efivars/db-d719b2cb-3d3a-4596-a3bc-dad00e67656f

Description of problem:
If you are trying to use a Microsoft Hyper-V 2012 R2 Generation 2 guest you will be unable to boot Fedora while secure mode is present.

Version-Release number of selected component (if applicable):
shim-0.7-1.fc20.x86_64

How reproducible:
Reproducible every time.

Steps to Reproduce:
In Hyper-V Manager 2012 R2:
1. At the top right select New -> Virtual Machine.
2. Click Next.
3. Tick Generation 2. Click Next.
4. Type 2048 in the Startup memory box. Click Next.
5. Click Next.
6. Click Next.
7. Tick Install an operating system from a bootable image file and select Fedora-Live-Desktop-x86_64-20-1 . Click Next.
8. Click Finish.
9. Press the right mouse button over the newly created VM and select Start.
10. Double click the newly made VMs name so that the Virtual Machine Connection window appears.

Actual results:
A message saying:
Hyper-V™

Boot Failed. EFI SCSI Device. Failed Secure Boot Verification.
Boot Failed. EFI Network.
Boot Failed. EFI SCSI Device.
No Operating System was Loaded. Press a key to retry the boot sequence...

Expected results:
ISO to be booted.

Additional info:
If you press the right mouse button over the VM and go to Settings... Click on Firmware at the left and untick Enable Secure Boot at the right, Click OK, right mouse button on the VM and click Start. You will see
Hyper-V™
Booting `Test this media & start Fedora Live'
_

and will sit there indefinitely.

In the extreme scenario, if you take the Fedora cloud image in a Generation 1 VM, update it, install various utilities, convert the disk from MBR to GPT using gdisk, create a FAT32 partition on the end with and EFI id, format the partition, mount it on /boot/efi, install grub2-efi and shim, run something like grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg etc., hack up /boot/efi/EFI/fedora/grub.cfg so linux -> linuxefi etc, attach the disk to a Generation 2 VM and boot it then you will have a Fedora booting in a Hyper-V 2012 Generation 2 VM. However the moment you turn on Secure Boot you will no longer be able to boot the VM.

The Ubuntu 14.04 ISO boots directly in a Generation 2 VM.

Attaching /sys/firmware/efi/efivars/db-d719b2cb-3d3a-4596-a3bc-dad00e67656f .

Comment 1 M. Edward (Ed) Borasky 2014-11-20 00:34:08 UTC
Same thing is happening on Fedora 21. What's worse, if you turn Secure Boot off, it *still* doesn't boot! It comes up in a dracut recovery screen - can't find the devices or disks or something. I'll grab a screenshot and attach it in a few minutes.

Comment 2 M. Edward (Ed) Borasky 2014-11-20 00:37:42 UTC
Created attachment 959206 [details]
Screenshot of failed boot - Fedora 21 Workstation Beta on Windows 8.1 Hyper-V

Comment 3 M. Edward (Ed) Borasky 2014-11-22 23:16:38 UTC
Just tried with Fedora 21 Workstation TC3 - still broken. Sigh - it's hard to call this a Fedora bug since the same ISO works fine in VirtualBox and Virtual Machine Manager. 

I don't have a secure boot box to test this on bare metal. My guess is that it works fine too.

Comment 4 M. Edward (Ed) Borasky 2014-11-22 23:46:17 UTC
A little more info:

1. If you create the Hyper-V machine generation 1, a Fedora 21 Workstation TC3 ISO will boot. Of course, you won't have EFI / Secure Boot with a Generation 1 machine, but at least you'll be able to make a Fedora VM!

2. I tried a CentOS net install CD with a generation 2 machine. It also refuses to work with EFI / Secure Boot, but it does boot up to the installer!

There are a lot of problems with Hyper-V - it's hardly "point click and ship" on a laptop in a coffee shop. I'm going to do a blog post on running Project Atomic with it, but it's hardly user-friendly enough for non-geeks.

Comment 5 Sitsofe Wheeler 2014-11-23 07:55:48 UTC
CC'ing some Red Hat and Microsoft Hyper-V folks.

Comment 6 M. Edward (Ed) Borasky 2014-11-23 08:29:31 UTC
(In reply to Sitsofe Wheeler from comment #5)
> CC'ing some Red Hat and Microsoft Hyper-V folks.

Thanks!!

Comment 7 K. Y. Srinivasan 2014-11-23 17:53:47 UTC
Gen 2 VMs work correctly on many Distros today (Ubuntu, sles 12, RHEL7 etc.) Is the synthetic keyboard driver included in the initrd. On Gen2 firmware, we don't emulate the legacy keyboard.

Comment 8 M. Edward (Ed) Borasky 2014-11-23 18:49:34 UTC
(In reply to K. Y. Srinivasan from comment #7)
> Gen 2 VMs work correctly on many Distros today (Ubuntu, sles 12, RHEL7 etc.)
> Is the synthetic keyboard driver included in the initrd. On Gen2 firmware,
> we don't emulate the legacy keyboard.

If it works on RHEL7 shouldn't it work on a CentOS 7 netinstall CD? I got a failure with one of those in a Gen 1 machine. https://bugzilla.redhat.com/show_bug.cgi?id=1097772#c4

Comment 9 M. Edward (Ed) Borasky 2014-11-23 18:51:02 UTC
(In reply to M. Edward (Ed) Borasky from comment #8)
> (In reply to K. Y. Srinivasan from comment #7)
> > Gen 2 VMs work correctly on many Distros today (Ubuntu, sles 12, RHEL7 etc.)
> > Is the synthetic keyboard driver included in the initrd. On Gen2 firmware,
> > we don't emulate the legacy keyboard.
> 
> If it works on RHEL7 shouldn't it work on a CentOS 7 netinstall CD? I got a
> failure with one of those in a Gen 1 machine.
> https://bugzilla.redhat.com/show_bug.cgi?id=1097772#c4

Oops - CentOS 7 netinstall failed in Gen 2 and worked in Gen 1. Sorry.

Comment 10 K. Y. Srinivasan 2014-11-24 21:36:11 UTC
Have you tried installation from the media?

Comment 11 M. Edward (Ed) Borasky 2014-11-24 22:45:54 UTC
(In reply to K. Y. Srinivasan from comment #10)
> Have you tried installation from the media?

Not CentOS - Fedora 21 install works on Gen 1 however

Comment 12 Carwyn Edwards 2014-12-05 22:35:42 UTC
Interestingly the Fedora-Server-DVD-x86_64-21.iso from:

https://dl.fedoraproject.org/pub/alt/stage/21_RC5/Server/x86_64/iso/

boots into anaconda on a hyper v gen 2 VM with mouse and keyboard working but:

https://dl.fedoraproject.org/pub/alt/stage/21_RC5/Workstation/x86_64/iso/

.. has a long pause at the Fedora boot logo then times out showing hv-gen2-error.png (attached). Keyboard input at the dracut:/# prompt after the timeout does not work.

At any point attempting to send Action -> Ctrl+Alt+Delete generates a dialog message:

"Could not send keys to the virtual machine. Interacting with the virtual machine's keyboard device failed."

Could it be that the virtual keyboard driver mentioned in comment #7 is in the server initrd but not in the workstation initrd?

It seems reasonable that a server would be more expected to be installed on HyperV than a workstation.

In the time it's taken to write this I've successfully installed Fedora 21 Server onto a Gen2 HyperV VM on a Windows 8.1.1 host (i.e. EFI only).

Comment 13 Carwyn Edwards 2014-12-05 22:38:21 UTC
Created attachment 965253 [details]
Error shown after Workstation ISO timeout.

This is what is shown on the VM console after the Fedora 21 Workstation eventually times out attempting to boot. Until this happens the white background of the Fedora boot logo "F" is stuck at around 80%.

Comment 14 Carwyn Edwards 2014-12-05 22:40:50 UTC
Oh, Fedora Server only works with Secure Boot off. Definitely Gen 2 though.

Comment 15 Sitsofe Wheeler 2014-12-10 04:00:32 UTC
I can confirm Carwyn's comment with the final release of Fedora 21 install media.

Main Issue:
When using a Generation 2 Hyper-V VM neither the Fedora 21 Server nor Workstation ISOs will boot if Secure Boot is enabled - the UEFI "BIOS" gives the result described in the Actual results of comment #1.

Secondary Issue:
If Secure Boot is disabled then the Fedora 21 Server ISO boots to a GUI. The Fedora 21 Workstation ISO also boots but does not respond to the keyboard and bails out with an error message of comment #13 (sending ctrl-alt-del didn't work for me either). Sadly it looks as if the serial console has gone away with Generation 2 VMs removing a useful debugging tool :-(

Further, there seems to be no way of displaying the bootloader menu before the kernel starts (I tried holding down shift, alt, escape and tab but none of these had any effect). initrd0.img contains next to nothing in terms other than microcode and I couldn't find out what the kernel was configured with because there are no .config files lying around nor does scripts/extract-ikconfig vmlinuz0 work.

Booting the Workstation ISO on a Gen 1 VM to bash and poking around shows
/boot/config-3.17.4-301.fc21.x86_64 and looking in that shows
CONFIG_HYPERV_KEYBOARD=m
CONFIG_HYPERV_STORAGE=m

My guess is that we have no legacy bus controllers and no way to install the enlightened modules that would access the controller we do have so we wind up dead in the water. But why does the Server image work?

Let's try reassigning this to the anaconda package but I can't change the person this is assigned to though...

Comment 16 David Shea 2014-12-10 14:12:18 UTC
(In reply to Sitsofe Wheeler from comment #15)
> Let's try reassigning this to the anaconda package

No. If there's an issue with secure boot, then it could be a problem with the secure boot shim, or the signatures on the kernel, or the Hyper-V firmware itself. Don't just reassign bugs to any old component fishing for help.

Comment 17 Carwyn Edwards 2014-12-10 19:14:45 UTC
I'd say secure mode is the lesser issue here. The bigger issue is that Fedora Workstation can't work in a HyperV Gen2 VM as the required drivers are missing.

Whether the Fedora Workstation product owners consider this a bug or not is another matter. There are two workarounds as is stands:

1. Use a Gen 1 VM
2. Install the server product and "convert" although this isn't an officially supported path.

Ownership wise this should belong to the Fedora Workstation product people.

Arguably it's a design choice. What we don't know (here at least) is if this is accidental or deliberate.

Comment 18 Sitsofe Wheeler 2014-12-10 19:45:06 UTC
Apologies David.

Carwyn:
I think we should leave this particular issue as purely "Can use secure boot with Gen2 Hyper-V" and spin off "Can't boot Workstation on Gen2 Hyper-V" into its own separate bug with a fresh start. There is the question of what component it should go under though...

Comment 19 M. Edward (Ed) Borasky 2014-12-10 21:31:16 UTC
(In reply to Carwyn Edwards from comment #17)
> I'd say secure mode is the lesser issue here. The bigger issue is that
> Fedora Workstation can't work in a HyperV Gen2 VM as the required drivers
> are missing.
> 
> Whether the Fedora Workstation product owners consider this a bug or not is
> another matter. There are two workarounds as is stands:
> 
> 1. Use a Gen 1 VM
> 2. Install the server product and "convert" although this isn't an
> officially supported path.
> 
> Ownership wise this should belong to the Fedora Workstation product people.
> 
> Arguably it's a design choice. What we don't know (here at least) is if this
> is accidental or deliberate.

What packages provide the required drivers? I can add them easily to my remix!

Comment 20 Sitsofe Wheeler 2015-02-12 12:39:12 UTC
With respect to the lack of keyboard driver on Gen2:

It looks like Vaughan Cao of Oracle also spotted this and added a change to dracut (http://git.kernel.org/cgit/boot/dracut/dracut.git/commit/modules.d/90kernel-modules/module-setup.sh?id=7a3e1a0e4a86681fab3baf9662f2d40733876ea2 ) but unfortunately it doesn't look like it made it into Fedora 21 (my current dracut is dracut-038-32.git20141216.fc21.x86_64 and doesn't have the change).

Comment 21 Sitsofe Wheeler 2015-02-12 13:04:19 UTC
I've spun of the non-secure boot portions of this issue into Bug 1192030 .

Comment 22 Sitsofe Wheeler 2015-02-12 13:08:20 UTC
Reassigning this back to shim and reopening.

pjones:
Any ideas?

Comment 23 M. Edward (Ed) Borasky 2015-02-12 19:20:35 UTC
FWIW I've had length discussions on Twitter with Ben Armstrong of Microsoft about Hyper-V, especially cloud and Docker issues. https://twitter.com/VirtualPCGuy. In particular:

@znmeb - Linux and Secure Boot is a whole different ball game. Windows 10 Hyper-V includes secure boot certificates for some Linux Distros.

https://twitter.com/VirtualPCGuy/status/561433139839795200

So someone probably needs a Windows 10 Tech Preview on bare metal to test this - it may not be showing up in Windows 8.1 or Windows 8. I have a Windows 10 virtual machine but Hyper-V is disabled in virtual machines. :-(

Comment 24 Matthew Garrett 2015-02-12 19:35:27 UTC
The db file posted only includes the Windows root CA, not the Microsoft UEFI root CA. The only thing that Hyper-V will boot in secure mode is Windows. This is a Hyper-V issue, not anything that can be fixed in Fedora.

Comment 25 Alex 2015-08-21 15:58:01 UTC
Hi,

Regarding the ability to use secure boot for a Fedora 22 guest on Hyper-V, the following steps worked for me on Windows 10 Pro x64:

1. Create a new Gen 2 VM (I used 1024MB ram, external switch, 8GB hard disk); don't specify any ISO yet (don't start the VM)
2. Go to Settings -> SCSI Controller, and add a DVD drive; specify the netinst Workstation iso instead of the Live Workstation iso, to workaround #1192030
3. Go to Settings -> Firmware, and put the DVD drive at top of boot order.
4. Go to Settings -> Firmware, and check secure boot is enabled.
5. Follow the 'Linux secure boot' step here: https://technet.microsoft.com/en-us/library/dn765471.aspx#BKMK_linux
6. Now start the VM and complete the Fedora 22 setup; I used the 'standard' partitioning scheme (no LVM, simple ext4; separate EFI boot partition was automatically created)
7. During the setup I got "The following error occurred while installing the boot loader. The system will not be bootable. Would you like to ignore this and continue the installation? Failed to set new efi boot target. This is most likely a kernel or firmware bug." However, I selected 'yes' to continue, and the Fedora setup continued and completed.
8. Go to Settings -> SCSI Controller -> DVD Drive and remove the iso from the DVD drive
9. Now start the VM (it should boot from the hard disk); despite the error in step 7, the VM does boot successfully with secure boot enabled in the Hyper-V VM settings

Attempting the above steps without step 5 prevents secure boot from working at step 6, so the step provided by Microsoft would appear to be required for secure boot to function.

Thanks,

Comment 26 Alex 2015-08-21 16:02:25 UTC
Created attachment 1065635 [details]
Error during setup - failed to set new efi boot target

Comment 27 Sitsofe Wheeler 2015-10-10 06:13:35 UTC
Alex: Thanks for the extra information! Looks like Windows 10+ and Windows Server 2016+ will be able to use secure boot on Hyper-V and Fedora with the MicrosoftUEFICertificateAuthority tweak you mentioned (sadly the SecureBootTemplate  option doesn't exist on my Windows Server 2012 R2 install).

It's probably worth spinning off  your "failed to set new efi boot target. This is most likely a kernel or firmware bug." issue into it's own bug report though - that way it can be closed separately to this and if it is a firmware bug we can point Microsoft directly to that issue alone.

Comment 28 Sitsofe Wheeler 2015-10-10 08:26:09 UTC
M. Edward (Ed) Borasky:
I forgot to thank you for pointing out it might be possible to make secure boot work with Windows 10's Hyper-V (which is what Alex went on to demonstrate). FWIW: I've heard that Hyper-V 2016 will support exposing nested virtualization (http://www.thomasmaurer.ch/2015/05/hyper-v-vnext-is-going-to-support-nested-virtualization/ ) but that doesn't help you today.

Matthew:
Thanks for stating where the problem was. Hopefully anyone else running Hyper-V Generation 2 Fedora VM's will be able to use the information here to understand and workaround secure boot issues.


Note You need to log in before you can comment on or make changes to this bug.