Bug 1005242

Summary: cryptsetup breaks with fips=1
Product: Red Hat Enterprise Linux 7 Reporter: Matěj Cepl <mcepl>
Component: cryptsetupAssignee: Ondrej Kozina <okozina>
Status: CLOSED DEFERRED QA Contact: Release Test Team <release-test-team-automation>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: agk, harald, mcepl, okozina, prajnoha, tmraz, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-22 10:20:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
rdsosreport.txt
none
rdsosreport.txt log
none
/etc/crypttab
none
required log none

Description Matěj Cepl 2013-09-06 13:51:39 UTC
Created attachment 794754 [details]
rdsosreport.txt

Description of problem:
I have followed steps on https://access.redhat.com/site/solutions/137833 and started computer with fips=1 on the kernel command line.

Result was that boot died even before switchroot.

See attached logs (just BTW, I have observed, that the file /tmp/fipsdone was created and it had 0 bytes length).

Version-Release number of selected component (if applicable):
dracut-fips-032-23.el7.x86_64
openssl-fips-1.0.1e-19.el7.x86_64
cryptsetup-1.6.1-1.el7.x86_64
kernel-3.10.0-4.el7.x86_64

How reproducible:
happened twice from two attempts

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
systemctl status --failed gives 

systemd-cryptsetup -> '/org/freedesktop/systemd1/unit/systemd_2dcryptsetup_40swap_2eservice'
systemd-cryptsetup - Cryptography Setup for swap
   Loaded: loaded (/etc/crypttab)
   Active: failed (Result: exit-code) since Fri 2013-09-06 12:57:57 UTC; 1min 42s ago
     Docs: man:systemd-cryptsetup@.service(8)
           man:crypttab(5)
  Process: 639 ExecStart=/usr/lib/systemd/systemd-cryptsetup attach swap /dev/vg_wycliff/lv_swap none  (code=exited, status=1/FAILURE)
 Main PID: 639 (code=exited, status=1/FAILURE)

Sep 06 12:57:57 wycliff.ceplovi.cz systemd[1]: systemd-cryptsetup: main process exited, code=exited, status=1/FAILURE
Sep 06 12:57:57 wycliff.ceplovi.cz systemd[1]: Failed to start Cryptography Setup for swap.
Sep 06 12:57:57 wycliff.ceplovi.cz systemd[1]: Unit systemd-cryptsetup entered failed state.

Comment 1 Matěj Cepl 2013-09-06 14:09:42 UTC
Created attachment 794776 [details]
rdsosreport.txt log

I have managed to upgrade to

matej@wycliff: logs$ uname -a
Linux wycliff.ceplovi.cz 3.10.0-16.el7.x86_64 #1 SMP Tue Sep 3 16:58:22 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
matej@wycliff: logs$ 

so I am attaching the correct last version log

Comment 3 Ondrej Kozina 2013-09-10 11:06:17 UTC
Could you please paste content of /etc/crypttab? Going through the logs, it looks to me like an issue with systemd-cryptsetup generator or related service more than with cryptsetup itself.

Comment 4 Ondrej Kozina 2013-09-10 11:18:40 UTC
Also, do you experience same issue without fips being activated?

Comment 5 Matěj Cepl 2013-09-10 11:35:38 UTC
Created attachment 795962 [details]
/etc/crypttab

(In reply to Ondrej Kozina from comment #4)
> Also, do you experience same issue without fips being activated?

No, removing fips=1 from the kernel command line is the only way how to make my machine working.

Comment 6 Ondrej Kozina 2013-09-13 08:00:33 UTC
Harald, could you have a quick look at this one? Looks to me like there's something wrong with generated services in pre-switch-root stage.

Comment 7 Harald Hoyer 2013-09-13 11:21:19 UTC
Hmm...

> ExecStart=/usr/lib/systemd/systemd-cryptsetup attach swap /dev/vg_wycliff/lv_swap none  (code=exited, status=1/FAILURE)

Please do

- install strace
# yum install strace

- recreate your initramfs with the debug module
# dracut -f -a debug

- add "rd.debug" to the kernel command line

- reboot

- run strace on systemd-cryptsetup
# strace -f /usr/lib/systemd/systemd-cryptsetup attach swap /dev/vg_wycliff/lv_swap none 2>strace.txt

attach strace.txt and rdsosreport.txt

Comment 8 Alasdair Kergon 2013-09-13 11:43:53 UTC
N.B. If this is a non-test system, don't forget to make sure any trace you post doesn't contain information about your encryption key!

Comment 11 Harald Hoyer 2013-09-13 14:49:49 UTC
[pid   782] open("/lib64/.libcrypto.so.10.hmac", O_RDONLY) = -1 ENOENT (No such file or directory)

Comment 12 Harald Hoyer 2013-09-13 14:50:45 UTC
Matěj Cepl: can you attach the output of:

# dracut -f --debug

?

Comment 13 Matěj Cepl 2013-09-14 01:06:34 UTC
Created attachment 797552 [details]
required log

Comment 14 Harald Hoyer 2013-09-14 07:38:31 UTC
(In reply to Matěj Cepl from comment #13)
> Created attachment 797552 [details]
> required log

Do you have the file "/lib64/.libcrypto.so.10.hmac" in your real root?

What is the output of:

# lsinitrd | grep -E 'libcrypto.*.hmac'

Comment 15 Harald Hoyer 2013-09-14 07:43:34 UTC
When I ran on a F19 machine, I get:

# dracut --debug -f |& grep -E ': (cp|ln).*libcrypto.*.hmac'
dracut-install: cp '/usr/lib64/.libcrypto.so.1.0.1e.hmac' '/var/tmp/initramfs.knxfw3/usr/lib64/.libcrypto.so.1.0.1e.hmac'
dracut-install: ln -s '.libcrypto.so.1.0.1e.hmac' '/var/tmp/initramfs.knxfw3/lib64/.libcrypto.so.10.hmac'

Comment 16 Matěj Cepl 2013-09-14 15:52:03 UTC
(In reply to Harald Hoyer from comment #14)
> Do you have the file "/lib64/.libcrypto.so.10.hmac" in your real root?

No.

matej@wycliff: thunderbird-ensemble (master)$ ls /lib64/.libcrypto.so.10.hmac
ls: cannot access /lib64/.libcrypto.so.10.hmac: No such file or directory
matej@wycliff: thunderbird-ensemble (master)$ rpm -qf /lib64/.libcrypto.so.10.hmac 
error: file /lib64/.libcrypto.so.10.hmac: No such file or directory
matej@wycliff: thunderbird-ensemble (master)$ rpm -qf /lib64/libcrypto.so.10
openssl-libs-1.0.1e-19.el7.x86_64
matej@wycliff: thunderbird-ensemble (master)$ 

Adding openssl maintainer to CC of this bug.

> What is the output of:
> 
> # lsinitrd | grep -E 'libcrypto.*.hmac'

wycliff:~# lsinitrd |grep -E 'libcrypto.*.hmac'
wycliff:~#

Comment 17 Tomas Mraz 2013-09-16 12:26:49 UTC
Harald, the problem might be that the files got recently renamed - you need to use wildcard .lib{crypto,ssl}*.hmac  to copy them to initrd.

Comment 18 Harald Hoyer 2013-09-16 14:28:30 UTC
(In reply to Tomas Mraz from comment #17)
> Harald, the problem might be that the files got recently renamed - you need
> to use wildcard .lib{crypto,ssl}*.hmac  to copy them to initrd.

what? the strace clearly shows, that it wants /lib64/.libcrypto.so.10.hmac.

[pid   782] open("/lib64/.libcrypto.so.10.hmac", O_RDONLY) = -1 ENOENT (No such file or directory)

Comment 19 Tomas Mraz 2013-09-17 09:18:54 UTC
Ah, you're looking at F19 - yes, there is still the old file name.

But on F20 and RHEL-7 the files have longer suffix than just .hmac. The version-release from the rpm is included. And the files are in openssl-fips subpackage. Of course if you want to turn on the fips mode the -fips subpackages must be installed.

Comment 20 Harald Hoyer 2013-09-17 15:00:58 UTC
(In reply to Tomas Mraz from comment #19)
> Ah, you're looking at F19 - yes, there is still the old file name.

Hmm, Matěj Cepl should have used RHEL-7 to get the strace in comment 9

> 
> But on F20 and RHEL-7 the files have longer suffix than just .hmac. The
> version-release from the rpm is included. And the files are in openssl-fips
> subpackage. Of course if you want to turn on the fips mode the -fips
> subpackages must be installed.

Maybe Matěj Cepl does not have those installed

Comment 21 Matěj Cepl 2013-09-17 15:17:43 UTC
(In reply to Harald Hoyer from comment #20)
> Maybe Matěj Cepl does not have those installed

He has.

wycliff:~# rpm -qa openssl\*
openssl-static-1.0.1e-20.el7.x86_64
openssl-fips-1.0.1e-20.el7.x86_64
openssl-1.0.1e-20.el7.x86_64
openssl-debuginfo-1.0.1e-20.el7.x86_64
openssl-devel-1.0.1e-20.el7.x86_64
openssl-libs-1.0.1e-20.el7.i686
openssl-libs-1.0.1e-20.el7.x86_64

Comment 22 Harald Hoyer 2013-09-17 15:25:54 UTC
(In reply to Matěj Cepl from comment #21)
> (In reply to Harald Hoyer from comment #20)
> > Maybe Matěj Cepl does not have those installed
> 
> He has.
> 


What is the contents of this package?

> openssl-fips-1.0.1e-20.el7.x86_64

Comment 23 Matěj Cepl 2013-09-17 20:05:57 UTC
matej@wycliff: empathy (rhel-7 %)$ rpm -ql openssl-fips
/etc/prelink.conf.d
/etc/prelink.conf.d/openssl-fips.conf
/usr/lib64/.libcrypto.so.1.0.1e.1.0.1e-20.el7.hmac
/usr/lib64/.libcrypto.so.10.1.0.1e-20.el7.hmac
/usr/lib64/.libssl.so.1.0.1e.1.0.1e-20.el7.hmac
/usr/lib64/.libssl.so.10.1.0.1e-20.el7.hmac
matej@wycliff: empathy (rhel-7 %)$

Comment 24 Matěj Cepl 2013-09-17 20:07:30 UTC
(In reply to Matěj Cepl from comment #23)
> matej@wycliff: empathy (rhel-7 %)$ rpm -ql openssl-fips
> /etc/prelink.conf.d

BTW, this is unrealted but IMHO a packaging bug. This directory should be owned only by prelink package, shouldn't it?

matej@wycliff: empathy (rhel-7 %)$ rpm -qf /etc/prelink.conf.d/
prelink-0.5.0-2.el7.x86_64
openssl-fips-1.0.1e-20.el7.x86_64
openssh-server-fips-6.2p2-7.el7.x86_64
openssh-clients-fips-6.2p2-7.el7.x86_64
matej@wycliff: empathy (rhel-7 %)$

Comment 25 Tomas Mraz 2013-09-18 07:24:06 UTC
Nope, we cannot depend on prelink so we need to own it and it is allowed AFAIK by packaging guidelines.

Comment 26 Harald Hoyer 2013-09-20 06:03:17 UTC
(In reply to Matěj Cepl from comment #23)
> matej@wycliff: empathy (rhel-7 %)$ rpm -ql openssl-fips
> /etc/prelink.conf.d
> /etc/prelink.conf.d/openssl-fips.conf
> /usr/lib64/.libcrypto.so.1.0.1e.1.0.1e-20.el7.hmac
> /usr/lib64/.libcrypto.so.10.1.0.1e-20.el7.hmac
> /usr/lib64/.libssl.so.1.0.1e.1.0.1e-20.el7.hmac
> /usr/lib64/.libssl.so.10.1.0.1e-20.el7.hmac
> matej@wycliff: empathy (rhel-7 %)$

so, is openssl-fips missing a symlink? or was cryptsetup compiled with the wrong tools? Or does it have to be recompiled?

Comment 27 Tomas Mraz 2013-09-20 06:46:51 UTC
I believe we are mixing things here seriously - I think the strace is coming from older openssl where there was a problem with looking up the .hmac file however the 1.0.1e-20.el7 packages are fixed in this regard. I suppose the initrd was not regenerated appropriately.

Comment 28 Matěj Cepl 2013-09-20 15:35:47 UTC
(In reply to Tomas Mraz from comment #27)
> I believe we are mixing things here seriously - I think the strace is coming
> from older openssl where there was a problem with looking up the .hmac file
> however the 1.0.1e-20.el7 packages are fixed in this regard. I suppose the
> initrd was not regenerated appropriately.

I am sorry, but that isn't so ... this was generated on the latest EL-7 and I don't think I have any significant openssl upgrades meanwhile. And mkinitrd has been recreated with grub2-mkconfig. If there is anything wrong with using that, let me know, but I don't see any PEBKAC here.