Bug 1014527

Summary: Unhelpful error messages in misconfigured system when booting in FIPS mode (.vmlinuz*.hmac does not exist)
Product: Red Hat Enterprise Linux 6 Reporter: Hubert Kario <hkario>
Component: dracutAssignee: Harald Hoyer <harald>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: low Docs Contact:
Priority: low    
Version: 6.5CC: arubin, dracut-maint-list, ebenes, gxing, hkario, jrieden, ksrot, liliu, mbanas, omoris, salmy, sgrubb, tmraz
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-13 13:06:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 993793, 1172231, 1270825    
Attachments:
Description Flags
full console log from failing boot
none
bootup screenshot
none
Run with UUID and quiet
none
Run with UUID and quiet
none
Run with UUID, rdinitdebug and quiet
none
Run with /dev path, rdinitdebug and quiet
none
Run with UUID, rdinitdebug and quiet, dracut-fips-aesni installed none

Description Hubert Kario 2013-10-02 09:42:05 UTC
Description of problem:
After enabling FIPS mode on some machines, dracut halts bootup because it can't find /boot/.vmlinuz-2.6.32-358.18.1.el6.x86_64.hmac file

Version-Release number of selected component (if applicable):
dracut-04-303.el6.noarch                     
dracut-fips-004-303.el6.noarch

How reproducible:
depends on hardware

Steps to Reproduce:
1. Enable FIPS mode
2. reboot machine
3.

Actual results:
dracut: Checking integrity of kernel 
dracut Warning: /boot/.vmlinuz-2.6.32-358.18.1.el6.x86_64.hmac does not exist 
dracut: FATAL: FIPS integrity test failed 
dracut: Refusing to continue 
dracut Warning: Signal caught! 
  
  
 %G %G  
  
dracut Warning: dracut: FATAL: FIPS integrity test failed 
dracut Warning: dracut: Refusing to continue 
Kernel panic - not syncing: Attempted to kill init! 
Pid: 1, comm: init Not tainted 2.6.32-358.18.1.el6.x86_64 #1 
Call Trace: 
 [<ffffffff8150da18>] ? panic+0xa7/0x16f 
 [<ffffffff81073be2>] ? do_exit+0x862/0x870 
 [<ffffffff81182c55>] ? fput+0x25/0x30 
 [<ffffffff81073c48>] ? do_group_exit+0x58/0xd0 
 [<ffffffff81073cd7>] ? sys_exit_group+0x17/0x20 
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b 
panic occurred, switching back to text console 
[-- MARK -- Tue Oct  1 17:55:00 2013] 

Expected results:
system booted to FIPS mode

Additional info:
Problem is machine dependent

Comment 7 Harald Hoyer 2013-10-07 13:38:04 UTC
If /boot is on a separate partition, it has to be specified on the kernel command line with "boot=<boot device>"

       boot=<boot device>
           specify the device, where /boot is located. e.g.

               boot=/dev/sda1
               boot=/dev/disk/by-path/pci-0000:00:1f.1-scsi-0:0:1:0-part1
               boot=UUID=<uuid>
               boot=LABEL=<label>

Comment 8 Hubert Kario 2013-10-07 14:24:26 UTC
yes, we had this problem too, it was specified

Comment 9 Harald Hoyer 2013-10-08 14:00:07 UTC
please boot with "rdinitdebug" on the kernel command line and remove "quiet rhgb" and show me the console messages.

Comment 10 Hubert Kario 2013-10-09 10:06:46 UTC
Created attachment 809835 [details]
full console log from failing boot

Looks like "quiet rhgb" are not present by default on the machine.

grub.conf:

default=0
timeout=5
serial --unit=0 --speed=115200
terminal --timeout=5 serial console
title Red Hat Enterprise Linux (2.6.32-358.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/vg_sunx845001-lv_root rd_NO_LUKS rd_LVM_LV=vg_sunx845001/lv_swap LANG=en_US.UTF-8 rd_LVM_LV=vg_sunx845001/lv_root rd_NO_MD console=ttyS0,115200 crashkernel=auto SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM fips=1 boot=/dev/sdb1 rdinitdebug
        initrd /initramfs-2.6.32-358.el6.x86_64.img



The panic with rdinitdebug:

dracut: Mounted root filesystem /dev/mapper/vg_sunx845001-lv_root 
 %Gdracut: Checking integrity of kernel 
dracut Warning: /boot/.vmlinuz-2.6.32-358.el6.x86_64.hmac does not exist 
dracut: FATAL: FIPS integrity test failed 
dracut: Refusing to continue 
dracut Warning: Signal caught! 
+ echo  
  
+ echo  
  
+ warn Signal 'caught!'  
+ check_quiet  
+ '[' -z yes ']'  
+ echo '<4>dracut Warning: Signal' 'caught!'  
+ '[' yes '!=' yes ']'  
+ source_all emergency  
+ local f  
+ '[' emergency ']'  
+ '[' -d /emergency ']'  
+ for f in '"/$1"/*.sh'  
+ '[' -e /emergency/00plymouth-emergency.sh ']'  
+ . /emergency/00plymouth-emergency.sh  
++ '[' -x /bin/plymouth ']'  
++ /bin/plymouth --hide-splash  
 %G %G+ echo  
  
+ echo  
  
+ warn Signal 'caught!'  
+ check_quiet  
+ '[' -z yes ']'  
+ echo '<4>dracut Warning: Signal' 'caught!'  
+ '[' yes '!=' yes ']'  
+ source_all emergency  
+ local f  
+ '[' emergency ']'  
+ '[' -d /emergency ']'  
+ for f in '"/$1"/*.sh'  
+ '[' -e /emergency/00plymouth-emergency.sh ']'  
+ . /emergency/00plymouth-emergency.sh  
++ '[' -x /bin/plymouth ']'  
++ /bin/plymouth --hide-splash  
+ for f in '"/$1dracut Warning: dracut: FATAL: FIPS integrity test failed 
"/*.sh' 
+ '[' -dracut Warning: dracut: Refusing to continue 
e /emergency/01-Kernel panic - not syncing: Attempted to kill init! 
Pid: 1, comm: init Not tainted 2.6.32-358.el6.x86_64 #1 
Call Trace: 
die.sh ']' 
+ .  [<ffffffff8150cfc8>] ? panic+0xa7/0x16f 
/emergency/01-di [<ffffffff81073ae2>] ? do_exit+0x862/0x870 
 [<ffffffff81073b48>] ? do_group_exit+0x58/0xd0 
e.sh 
++ warn dr [<ffffffff81073bd7>] ? sys_exit_group+0x17/0x20 
acut: FATAL: 'FI [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b 
panic occurred, switching back to text console 
[-- MARK -- Wed Oct  9 09:55:00 2013] 
[-- MARK -- Wed Oct  9 10:00:00 2013] 

full console.log output attached

Comment 18 Ondrej Moriš 2013-10-10 15:38:42 UTC
Let me point out that there is always a straightforward workaround for this bug, you can always use boot=<dev-file> instead of boot=<UUID>. With <dev-file> you will not hit the bug.

Comment 19 Hubert Kario 2013-10-10 16:08:18 UTC
(In reply to Ondrej Moriš from comment #18)
> Let me point out that there is always a straightforward workaround for this
> bug, you can always use boot=<dev-file> instead of boot=<UUID>. With
> <dev-file> you will not hit the bug.

The workaround works for most machines, but not all. The original bug description was using boot= specified as a path in /dev. Using UUID just makes the FIPS setup fail on more machines, not on a different set of machines.

Comment 21 Ondrej Moriš 2013-10-11 07:41:13 UTC
(In reply to Hubert Kario from comment #19)
> (In reply to Ondrej Moriš from comment #18)
> > Let me point out that there is always a straightforward workaround for this
> > bug, you can always use boot=<dev-file> instead of boot=<UUID>. With
> > <dev-file> you will not hit the bug.
> 
> The workaround works for most machines, but not all. The original bug
> description was using boot= specified as a path in /dev. Using UUID just
> makes the FIPS setup fail on more machines, not on a different set of
> machines.

Ah, I see now (BZ#1014527#c10). Please ignore my previous comment (c#19). 

Hubert, I've updated the test to use UUID for boot device when it is executed with USE_UUID parameter (see QA Whiteboard).

Comment 23 Harald Hoyer 2013-10-11 08:11:08 UTC

* ibm-x3650m4-01-vm-16.lab.eng.bos.redhat.com
  - boot on /dev/vda1
  - kernel parameter boot=UUID="62106b0d-ee5d-48c5-96b4-5406009bcd18"

* ibm-p730-06-lp3.rhts.eng.bos.redhat.com 	
  - boot on /dev/sda1
  - kernel parameter boot=UUID="62106b0d-ee5d-48c5-96b4-5406009bcd18"


How can these two machines have the boot partition with the same UUID?

Has anyone verified, that the UUID actually corresponds to the filesystem UUID of /boot?

Comment 24 Harald Hoyer 2013-10-11 08:13:47 UTC
A question for Steve Grubb: Can we drop the hmac check of the kernel? This check is so bogus, causes only problems to find the boot partition and proves _nothing_.

This could even be done very late in the boot process.

Comment 25 xingge 2013-10-11 10:10:02 UTC
I failed reboot the system with FIPS enabled too, I use RHEL6.5-Snapshot-3 server64 system. I don't know if the reason is the same with this bug, if not please let me know and I'll move this comment to a new bug report. 

Version-Release number of selected component (if applicable):
dracut-004-330.el6.noarch                     
dracut-fips-004-330.el6.noarch
dracut-kernel-004-330.el6.noarch

How reproducible:
always

Steps to Reproduce:
1. Enable FIPS mode
 1.1 Install dracut-fips and hamccalc 
 1.2 modify the /boot/grub/grub.conf to enable FIPS mode
2. reboot machine


Actual results:
Reboot failed.To check the details please see the attachment

Expected results:
system booted to FIPS mode

Additional info:
After I set prelink to NO and run "prelink -ua" reboot succeed but can't log in to the system "Authentication failed".

Comment 26 xingge 2013-10-11 10:13:18 UTC
Created attachment 810952 [details]
bootup screenshot

Comment 27 Hubert Kario 2013-10-11 10:28:39 UTC
(In reply to Harald Hoyer from comment #23)
> 
> * ibm-x3650m4-01-vm-16.lab.eng.bos.redhat.com
>   - boot on /dev/vda1
>   - kernel parameter boot=UUID="62106b0d-ee5d-48c5-96b4-5406009bcd18"
> 
> * ibm-p730-06-lp3.rhts.eng.bos.redhat.com 	
>   - boot on /dev/sda1
>   - kernel parameter boot=UUID="62106b0d-ee5d-48c5-96b4-5406009bcd18"
> 
> 
> How can these two machines have the boot partition with the same UUID?
> 
> Has anyone verified, that the UUID actually corresponds to the filesystem
> UUID of /boot?

It's done automatically using:
BOOT_DEV=`df /boot/ | tail -1 | awk '{print $1}'`
BOOT_DEV=$(blkid $BOOT_DEV | head -n 1 | grep -E --only-matching 'UUID="[^"]*"')

I don't see how this code can be failing (or providing wrong UUID) on rhel-6 but work just fine on RHEL-5 and RHEL-7.

Looks like Ondrej copied the same data twice, the log from ibm-x3650m4-01-vm-16.lab.eng.bos.redhat.com has:

    boot=UUID="f06760ef-f72b-45ae-9c5c-513a56aa7e9c"

Comment 28 Hubert Kario 2013-10-11 10:30:04 UTC
(In reply to xingge from comment #26)
> Created attachment 810952 [details]
> bootup screenshot

It's different issue.

In our case the boot is aborted before getting to initrd.

Comment 29 Harald Hoyer 2013-10-11 11:02:03 UTC
(In reply to xingge from comment #25)
> I failed reboot the system with FIPS enabled too, I use RHEL6.5-Snapshot-3
> server64 system. I don't know if the reason is the same with this bug, if
> not please let me know and I'll move this comment to a new bug report. 
> 
> Version-Release number of selected component (if applicable):
> dracut-004-330.el6.noarch                     
> dracut-fips-004-330.el6.noarch
> dracut-kernel-004-330.el6.noarch
> 
> How reproducible:
> always
> 
> Steps to Reproduce:
> 1. Enable FIPS mode
>  1.1 Install dracut-fips and hamccalc 
>  1.2 modify the /boot/grub/grub.conf to enable FIPS mode
> 2. reboot machine

did you recreate the initramfs?

> 
> 
> Actual results:
> Reboot failed.To check the details please see the attachment
> 
> Expected results:
> system booted to FIPS mode
> 
> Additional info:
> After I set prelink to NO and run "prelink -ua" reboot succeed but can't log
> in to the system "Authentication failed".

Comment 30 Harald Hoyer 2013-10-11 11:03:20 UTC
(In reply to xingge from comment #26)
> Created attachment 810952 [details]
> bootup screenshot

This is not a dracut bug!

Comment 31 Harald Hoyer 2013-10-11 11:04:03 UTC
(In reply to Hubert Kario from comment #28)
> (In reply to xingge from comment #26)
> > Created attachment 810952 [details]
> > bootup screenshot
> 
> It's different issue.
> 
> In our case the boot is aborted before getting to initrd.

before getting to initrd? Then it's not a dracut bug also

Comment 32 Hubert Kario 2013-10-11 11:12:00 UTC
(In reply to Harald Hoyer from comment #31)
> (In reply to Hubert Kario from comment #28)
> > (In reply to xingge from comment #26)
> > > Created attachment 810952 [details]
> > > bootup screenshot
> > 
> > It's different issue.
> > 
> > In our case the boot is aborted before getting to initrd.
> 
> before getting to initrd? Then it's not a dracut bug also

I mixed up the terms, I meant system initialization process, when deamons are started.

The stacktrace in Comment #0 clearly shows that the kernel panic is caused by dracut already running inside init process. Sorry if that caused any confusion.

Comment 33 Harald Hoyer 2013-10-11 11:29:37 UTC
(In reply to Hubert Kario from comment #27)
> It's done automatically using:
> BOOT_DEV=`df /boot/ | tail -1 | awk '{print $1}'`
> BOOT_DEV=$(blkid $BOOT_DEV | head -n 1 | grep -E --only-matching
> 'UUID="[^"]*"')

This would not work on my system! That catches the PARTUUID also and worse, it rewrites it to PARTUUID.

# BOOT_DEV=`df /boot/ | tail -1 | awk '{print $1}'`
# BOOT_DEV=$(blkid $BOOT_DEV | head -n 1 | grep -E --only-matching 'UUID="[^"]*"')
# echo $BOOT_DEV
UUID="B919-3D05" UUID="752799f4-f8f5-4435-9306-0d1550536049"

# blkid /dev/sda1
/dev/sda1: SEC_TYPE="msdos" UUID="B919-3D05" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="752799f4-f8f5-4435-9306-0d1550536049" 

...

# blkid -s UUID -o value /dev/sda1
B919-3D05

Comment 34 Harald Hoyer 2013-10-11 11:30:51 UTC
(In reply to Harald Hoyer from comment #33)
> (In reply to Hubert Kario from comment #27)
> > It's done automatically using:
> > BOOT_DEV=`df /boot/ | tail -1 | awk '{print $1}'`
> > BOOT_DEV=$(blkid $BOOT_DEV | head -n 1 | grep -E --only-matching
> > 'UUID="[^"]*"')
> 
> This would not work on my system! That catches the PARTUUID also and worse,
> it rewrites it to PARTUUID.

Correction:
This would not work on my system! That catches the PARTUUID also and worse,
it rewrites it to UUID.

Comment 35 Hubert Kario 2013-10-11 13:34:46 UTC
(In reply to Harald Hoyer from comment #33)
> (In reply to Hubert Kario from comment #27)
> > It's done automatically using:
> > BOOT_DEV=`df /boot/ | tail -1 | awk '{print $1}'`
> > BOOT_DEV=$(blkid $BOOT_DEV | head -n 1 | grep -E --only-matching
> > 'UUID="[^"]*"')
> 
> This would not work on my system! That catches the PARTUUID also and worse,
> it rewrites it to PARTUUID.
> 
> # BOOT_DEV=`df /boot/ | tail -1 | awk '{print $1}'`
> # BOOT_DEV=$(blkid $BOOT_DEV | head -n 1 | grep -E --only-matching
> 'UUID="[^"]*"')
> # echo $BOOT_DEV
> UUID="B919-3D05" UUID="752799f4-f8f5-4435-9306-0d1550536049"
> 
> # blkid /dev/sda1
> /dev/sda1: SEC_TYPE="msdos" UUID="B919-3D05" TYPE="vfat" PARTLABEL="EFI
> System Partition" PARTUUID="752799f4-f8f5-4435-9306-0d1550536049" 
> 
> ...
> 
> # blkid -s UUID -o value /dev/sda1
> B919-3D05

that would cause the kernel command line to be set to something like this:

boot=UUID="B919-3D05" UUID="752799f4-f8f5-4435-9306-0d1550536049"

The boot= is then correct and UUID= would be ignored so it would not be fatal. It also requires /boot to be on FAT, I don't think we use it in any default config. I'll fix the script to use -s -o syntax just in case. Thanks for pointing it out!

Anyway, both jobs on sun-x8450-01.rhts.eng.bos.redhat.com and the job on dell-pem805-01.rhts.eng.bos.redhat.com used ext4 as the /boot file system and /dev path name so it's certainly not the reason why they fail.

Comment 36 Steve Grubb 2013-10-11 13:50:18 UTC
Yes, the hmac files are required. It is weak, but requirement being fulfilled is merely to detect accidental misconfiguration rather than prevent tampering with the kernel.

Comment 37 Harald Hoyer 2013-10-11 14:21:56 UTC
(In reply to Steve Grubb from comment #36)
> Yes, the hmac files are required. It is weak, but requirement being
> fulfilled is merely to detect accidental misconfiguration rather than
> prevent tampering with the kernel.

Can we delay that check for rc.sysinit time? It would make life soooo much easier!

Comment 38 Harald Hoyer 2013-10-11 15:06:56 UTC
please provide the console output with "rhgb" removed and "quiet rdinitdebug" added to the kernel command line.

note: add "quiet" ... that lets dracut output directly to the console, instead of using /dev/kmsg, which does not seem to be flushed in this case.

Comment 39 Steve Grubb 2013-10-11 15:38:10 UTC
Just so there is a record of this. The check needs to be at the earliest possible time before using the kernel (much) and with FIPS-140 certified software. If we make big changes in how we do it, Atsec will have to explain how the new check works and that will raise questions about whether the old way was correct and cause even more scrutiny of the new way. That's not good for a simple re-validation.

We can make limitations in our Security Policy about what kind of file systems is permissible for /boot if that is a source of problems. Mkinitrd placed the following restrictions in the RHEL5 Security Policy:

-/boot must be on a separate partition (it usually is)
-/boot must *not* be on any of the following which
  are normally supported:
  -nfs
  -dmraid
  -mdraid (note we normally support level 1 only)
-/boot ofcourse also must not be on anything which we normally do
  not support for it, the following come to mind:
  -any kind of device mapper virtual device (ie lvm / dmcrypt)
  -multipath
  -gfs2

So, there is a precedent of restricting use on exotic file systems if it causes problems. Let me know if we need to change documentation.

Comment 40 Hubert Kario 2013-10-14 14:20:58 UTC
Created attachment 812038 [details]
Run with UUID and quiet

(In reply to Harald Hoyer from comment #38)
> please provide the console output with "rhgb" removed and "quiet
> rdinitdebug" added to the kernel command line.
> 
> note: add "quiet" ... that lets dracut output directly to the console,
> instead of using /dev/kmsg, which does not seem to be flushed in this case.

That didn't work out, there is no additional info:


The highlighted entry will be booted automatically in 5 seconds.          
      
The highlighted entry will be booted automatically in 4 seconds.          
      
The highlighted entry will be booted automatically in 3 seconds.          
      
The highlighted entry will be booted automatically in 2 seconds.          
      
The highlighted entry will be booted automatically in 1 seconds.          
  
  
%G %Gdracut: FATAL: FIPS integrity test failed 
dracut: Refusing to continue 
  
  
 %G %G  
  
Kernel panic - not syncing: Attempted to kill init! 
Pid: 1, comm: init Not tainted 2.6.32-358.el6.x86_64 #1 
Call Trace: 
 [<ffffffff8150cfc8>] ? panic+0xa7/0x16f 
 [<ffffffff81073ae2>] ? do_exit+0x862/0x870 
 [<ffffffff81182885>] ? fput+0x25/0x30 
 [<ffffffff81073b48>] ? do_group_exit+0x58/0xd0 
 [<ffffffff81073bd7>] ? sys_exit_group+0x17/0x20 
 [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

Comment 43 Hubert Kario 2013-10-14 16:15:11 UTC
Created attachment 812102 [details]
Run with UUID and quiet

(In reply to Harald Hoyer from comment #38)
> please provide the console output with "rhgb" removed and "quiet
> rdinitdebug" added to the kernel command line.
> 
> note: add "quiet" ... that lets dracut output directly to the console,
> instead of using /dev/kmsg, which does not seem to be flushed in this case.

run with uuid option and quiet, the dracut output is too big to quote here.

I'll schedule run with normal dev path and the sun machine in a moment

Comment 45 Harald Hoyer 2013-10-14 16:43:38 UTC
dracut: ++ '[' -e '/dev/disk/by-uuid/"28920d27-5c30-44c3-84a8-c3be44ead9d0"' ']' 

hmm, according to this log entry, you set:

boot=UUID="28920d27-5c30-44c3-84a8-c3be44ead9d0"

on the kernel command line.

Please remove the apostrophes and retry!

it should look like this: boot=UUID=28920d27-5c30-44c3-84a8-c3be44ead9d0

!!!!!!!!!

Comment 46 Hubert Kario 2013-10-14 17:03:45 UTC
(In reply to Harald Hoyer from comment #45)
> dracut: ++ '[' -e '/dev/disk/by-uuid/"28920d27-5c30-44c3-84a8-c3be44ead9d0"'
> ']' 
> 
> hmm, according to this log entry, you set:
> 
> boot=UUID="28920d27-5c30-44c3-84a8-c3be44ead9d0"
> 
> on the kernel command line.
> 
> Please remove the apostrophes and retry!
> 
> it should look like this: boot=UUID=28920d27-5c30-44c3-84a8-c3be44ead9d0
> 
> !!!!!!!!!

Not to say that it should be with apostrophes, but why does the exact same line (with apostrophes) work on RHEL-5 and RHEL-7?

Comment 47 Harald Hoyer 2013-10-14 18:12:52 UTC
(In reply to Hubert Kario from comment #46)
> Not to say that it should be with apostrophes, but why does the exact same
> line (with apostrophes) work on RHEL-5 and RHEL-7?

Well, in RHEL-5 there is no dracut and the old mkinitrd does not even understand boot=... kernel command line parameter.

Hmm, and if it works in RHEL-7, I would call it a bug.

Comment 48 Hubert Kario 2013-10-14 18:20:28 UTC
(In reply to Harald Hoyer from comment #47)
> (In reply to Hubert Kario from comment #46)
> > Not to say that it should be with apostrophes, but why does the exact same
> > line (with apostrophes) work on RHEL-5 and RHEL-7?
> 
> Well, in RHEL-5 there is no dracut and the old mkinitrd does not even
> understand boot=... kernel command line parameter.
> 
> Hmm, and if it works in RHEL-7, I would call it a bug.

ok

I scheduled a run on the sun machine, using /dev path, will see what dracut reports then

Comment 49 Suzanne Forsberg 2013-10-14 20:13:12 UTC
This may be totally unrelated, but check out https://bugzilla.redhat.com/show_bug.cgi?id=1016146

Is this the same bug?

Comment 50 Harald Hoyer 2013-10-15 07:04:02 UTC
(In reply to Suzanne Forsberg from comment #49)
> This may be totally unrelated, but check out
> https://bugzilla.redhat.com/show_bug.cgi?id=1016146
> 
> Is this the same bug?

Hmm, not really.

see https://bugzilla.redhat.com/show_bug.cgi?id=1016706

sha512sum -c <empty-file> always return with 0 (true)

But in this case, when the file could not even be found, dracut errors out earlier:

    if ! [ -e "/boot/.vmlinuz-${KERNEL}.hmac" ]; then
        warn "/boot/.vmlinuz-${KERNEL}.hmac does not exist"
        return 1
    fi

Comment 52 Hubert Kario 2013-10-16 11:39:27 UTC
Created attachment 812872 [details]
Run with UUID, rdinitdebug and quiet

The system crashes when boot= is specified with UUID without quotes too, same trace as before.

Comment 54 Hubert Kario 2013-10-16 14:38:30 UTC
Created attachment 812967 [details]
Run with /dev path, rdinitdebug and quiet

Run from the dell machine with /dev/ path for boot= kernel option and dracut debug messages.

Comment 56 Harald Hoyer 2013-10-17 14:25:55 UTC
(In reply to Hubert Kario from comment #54)
> Created attachment 812967 [details]
> Run with /dev path, rdinitdebug and quiet
> 
> Run from the dell machine with /dev/ path for boot= kernel option and dracut
> debug messages.

dracut: ++ mount -oro /dev/sdb1 /boot  
dracut: mount: unknown filesystem type 'LVM2_member'  

If /boot is in the same LVM as the root partition, you must not specify boot=...

user error

Comment 57 Harald Hoyer 2013-10-17 14:37:14 UTC
(In reply to Harald Hoyer from comment #56)
> (In reply to Hubert Kario from comment #54)
> > Created attachment 812967 [details]
> > Run with /dev path, rdinitdebug and quiet
> > 
> > Run from the dell machine with /dev/ path for boot= kernel option and dracut
> > debug messages.
> 
> dracut: ++ mount -oro /dev/sdb1 /boot  
> dracut: mount: unknown filesystem type 'LVM2_member'  
> 
> If /boot is in the same LVM as the root partition, you must not specify
> boot=...
> 
> user error

or boot=/dev/sdb1 points to the wrong partition here.

Comment 58 Hubert Kario 2013-10-17 15:19:22 UTC
(In reply to Harald Hoyer from comment #56)
> (In reply to Hubert Kario from comment #54)
> > Created attachment 812967 [details]
> > Run with /dev path, rdinitdebug and quiet
> > 
> > Run from the dell machine with /dev/ path for boot= kernel option and dracut
> > debug messages.
> 
> dracut: ++ mount -oro /dev/sdb1 /boot  
> dracut: mount: unknown filesystem type 'LVM2_member'  
> 
> If /boot is in the same LVM as the root partition, you must not specify
> boot=...
> 
> user error

not user error, for some reason kernel does reassign device names

Note that the system is installed on two disks.

What with the UUID run?

Comment 59 Harald Hoyer 2013-10-18 12:09:44 UTC
(In reply to Hubert Kario from comment #58)
> (In reply to Harald Hoyer from comment #56)
> > (In reply to Hubert Kario from comment #54)
> > > Created attachment 812967 [details]
> > > Run with /dev path, rdinitdebug and quiet
> > > 
> > > Run from the dell machine with /dev/ path for boot= kernel option and dracut
> > > debug messages.
> > 
> > dracut: ++ mount -oro /dev/sdb1 /boot  
> > dracut: mount: unknown filesystem type 'LVM2_member'  
> > 
> > If /boot is in the same LVM as the root partition, you must not specify
> > boot=...
> > 
> > user error
> 
> not user error, for some reason kernel does reassign device names

Yes, the kernel does that, and therefore we recommend the usage of UUID= or LABEL=.

> 
> Note that the system is installed on two disks.
> 
> What with the UUID run?

The UUID run is a total different error. The aesni-intel module could not be loaded on this machine. Apparently you installed dracut-fips-aesni on a non-intel CPU machine, which cannot work. There is a reason, why dracut-fips-aesni is in a separate package.

dracut: FATAL: Error inserting aesni_intel (/lib/modules/2.6.32-358.el6.x86_64/kernel/arch/x86/crypto/aesni-intel.ko): No such device

Comment 60 Hubert Kario 2013-10-18 12:57:34 UTC
(In reply to Harald Hoyer from comment #59)
> (In reply to Hubert Kario from comment #58)
> > (In reply to Harald Hoyer from comment #56)
> > > (In reply to Hubert Kario from comment #54)
> > > > Created attachment 812967 [details]
> > > > Run with /dev path, rdinitdebug and quiet
> > > > 
> > > > Run from the dell machine with /dev/ path for boot= kernel option and dracut
> > > > debug messages.
> > > 
> > > dracut: ++ mount -oro /dev/sdb1 /boot  
> > > dracut: mount: unknown filesystem type 'LVM2_member'  
> > > 
> > > If /boot is in the same LVM as the root partition, you must not specify
> > > boot=...
> > > 
> > > user error
> > 
> > not user error, for some reason kernel does reassign device names
> 
> Yes, the kernel does that, and therefore we recommend the usage of UUID= or
> LABEL=.
> 
> > 
> > Note that the system is installed on two disks.
> > 
> > What with the UUID run?
> 
> The UUID run is a total different error. The aesni-intel module could not be
> loaded on this machine. Apparently you installed dracut-fips-aesni on a
> non-intel CPU machine, which cannot work. There is a reason, why
> dracut-fips-aesni is in a separate package.
> 
> dracut: FATAL: Error inserting aesni_intel
> (/lib/modules/2.6.32-358.el6.x86_64/kernel/arch/x86/crypto/aesni-intel.ko):
> No such device

that was installed by errata tool, all runs that used the older dracut (303) won't have it installed

I've scheduled a run with a script that makes sure that the package is installed only on supported arches

Comment 61 Hubert Kario 2013-10-18 14:49:59 UTC
Created attachment 813804 [details]
Run with UUID, rdinitdebug and quiet, dracut-fips-aesni installed

It crashes with dracut-fips-aesni installed on machine with aes-ni when using boot=UUID=

Comment 63 Harald Hoyer 2013-10-21 07:38:54 UTC
(In reply to Hubert Kario from comment #61)
> Created attachment 813804 [details]
> Run with UUID, rdinitdebug and quiet, dracut-fips-aesni installed
> 
> It crashes with dracut-fips-aesni installed on machine with aes-ni when
> using boot=UUID=

Please control the boot parameters before you attach the logs here!!!

For this run, you specified an empty "boot=UUID="

Comment 64 Ann Marie Rubin 2013-10-21 14:51:52 UTC
If this does turn out to be "user error," let's make sure it is documented clearly for the user.

Comment 65 Hubert Kario 2013-10-21 15:44:25 UTC
(In reply to Ann Marie Rubin from comment #64)
> If this does turn out to be "user error," let's make sure it is documented
> clearly for the user.

I'm verifying now different configurations (with/without UUID, AES-NI, no AES-NI, etc.) and can't hit the problem again, either with new or old dracut so ATM it does look like user error (I still have few tests to run on the first box I encountered the problem on so please don't close the bug).

I'd say that not only it should be documented a bit better, but the error messages should be related to the mistake perpetrated by the user. From my point of view, the "/boot/.vmlinuz-2.6.32-358.18.1.el6.x86_64.hmac does not exist" is a "general failure" kind of error...

Comment 66 Hubert Kario 2013-10-23 13:27:47 UTC
Looks like it was indeed just configuration/scripting problem. Updated summary and keywords to reflect that.

Comment 67 Harald Hoyer 2013-10-23 13:53:32 UTC
(In reply to Hubert Kario from comment #66)
> Looks like it was indeed just configuration/scripting problem. Updated
> summary and keywords to reflect that.

CLOSED, NOTABUG ????

Comment 68 Harald Hoyer 2013-10-23 13:55:01 UTC
Sorry, didn't see the Summary change...

Summary: Enabling FIPS mode does cause kernel panic (.vmlinuz*.hmac does not exist) → Unhelpful error messages in misconfigured system when booting in FIPS mode (.vmlinuz*.hmac does not exist)

Comment 71 Harald Hoyer 2016-01-13 13:06:59 UTC
fixed since dracut >= 004-327

Comment 72 Harald Hoyer 2016-01-13 13:50:00 UTC
oops, closed wrong bug