Bug 1005242 - cryptsetup breaks with fips=1
cryptsetup breaks with fips=1
Status: CLOSED DEFERRED
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: cryptsetup (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Ondrej Kozina
Release Test Team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-06 09:51 EDT by Matěj Cepl
Modified: 2013-10-22 06:20 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-22 06:20:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rdsosreport.txt (101.14 KB, text/plain)
2013-09-06 09:51 EDT, Matěj Cepl
no flags Details
rdsosreport.txt log (96.76 KB, text/plain)
2013-09-06 10:09 EDT, Matěj Cepl
no flags Details
/etc/crypttab (69 bytes, text/plain)
2013-09-10 07:35 EDT, Matěj Cepl
no flags Details
required log (9.28 MB, text/plain)
2013-09-13 21:06 EDT, Matěj Cepl
no flags Details

  None (edit)
Description Matěj Cepl 2013-09-06 09:51:39 EDT
Created attachment 794754 [details]
rdsosreport.txt

Description of problem:
I have followed steps on https://access.redhat.com/site/solutions/137833 and started computer with fips=1 on the kernel command line.

Result was that boot died even before switchroot.

See attached logs (just BTW, I have observed, that the file /tmp/fipsdone was created and it had 0 bytes length).

Version-Release number of selected component (if applicable):
dracut-fips-032-23.el7.x86_64
openssl-fips-1.0.1e-19.el7.x86_64
cryptsetup-1.6.1-1.el7.x86_64
kernel-3.10.0-4.el7.x86_64

How reproducible:
happened twice from two attempts

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
systemctl status --failed gives 

systemd-cryptsetup@swap.service -> '/org/freedesktop/systemd1/unit/systemd_2dcryptsetup_40swap_2eservice'
systemd-cryptsetup@swap.service - Cryptography Setup for swap
   Loaded: loaded (/etc/crypttab)
   Active: failed (Result: exit-code) since Fri 2013-09-06 12:57:57 UTC; 1min 42s ago
     Docs: man:systemd-cryptsetup@.service(8)
           man:crypttab(5)
  Process: 639 ExecStart=/usr/lib/systemd/systemd-cryptsetup attach swap /dev/vg_wycliff/lv_swap none  (code=exited, status=1/FAILURE)
 Main PID: 639 (code=exited, status=1/FAILURE)

Sep 06 12:57:57 wycliff.ceplovi.cz systemd[1]: systemd-cryptsetup@swap.service: main process exited, code=exited, status=1/FAILURE
Sep 06 12:57:57 wycliff.ceplovi.cz systemd[1]: Failed to start Cryptography Setup for swap.
Sep 06 12:57:57 wycliff.ceplovi.cz systemd[1]: Unit systemd-cryptsetup@swap.service entered failed state.
Comment 1 Matěj Cepl 2013-09-06 10:09:42 EDT
Created attachment 794776 [details]
rdsosreport.txt log

I have managed to upgrade to

matej@wycliff: logs$ uname -a
Linux wycliff.ceplovi.cz 3.10.0-16.el7.x86_64 #1 SMP Tue Sep 3 16:58:22 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
matej@wycliff: logs$ 

so I am attaching the correct last version log
Comment 3 Ondrej Kozina 2013-09-10 07:06:17 EDT
Could you please paste content of /etc/crypttab? Going through the logs, it looks to me like an issue with systemd-cryptsetup generator or related service more than with cryptsetup itself.
Comment 4 Ondrej Kozina 2013-09-10 07:18:40 EDT
Also, do you experience same issue without fips being activated?
Comment 5 Matěj Cepl 2013-09-10 07:35:38 EDT
Created attachment 795962 [details]
/etc/crypttab

(In reply to Ondrej Kozina from comment #4)
> Also, do you experience same issue without fips being activated?

No, removing fips=1 from the kernel command line is the only way how to make my machine working.
Comment 6 Ondrej Kozina 2013-09-13 04:00:33 EDT
Harald, could you have a quick look at this one? Looks to me like there's something wrong with generated services in pre-switch-root stage.
Comment 7 Harald Hoyer 2013-09-13 07:21:19 EDT
Hmm...

> ExecStart=/usr/lib/systemd/systemd-cryptsetup attach swap /dev/vg_wycliff/lv_swap none  (code=exited, status=1/FAILURE)

Please do

- install strace
# yum install strace

- recreate your initramfs with the debug module
# dracut -f -a debug

- add "rd.debug" to the kernel command line

- reboot

- run strace on systemd-cryptsetup
# strace -f /usr/lib/systemd/systemd-cryptsetup attach swap /dev/vg_wycliff/lv_swap none 2>strace.txt

attach strace.txt and rdsosreport.txt
Comment 8 Alasdair Kergon 2013-09-13 07:43:53 EDT
N.B. If this is a non-test system, don't forget to make sure any trace you post doesn't contain information about your encryption key!
Comment 11 Harald Hoyer 2013-09-13 10:49:49 EDT
[pid   782] open("/lib64/.libcrypto.so.10.hmac", O_RDONLY) = -1 ENOENT (No such file or directory)
Comment 12 Harald Hoyer 2013-09-13 10:50:45 EDT
Matěj Cepl: can you attach the output of:

# dracut -f --debug

?
Comment 13 Matěj Cepl 2013-09-13 21:06:34 EDT
Created attachment 797552 [details]
required log
Comment 14 Harald Hoyer 2013-09-14 03:38:31 EDT
(In reply to Matěj Cepl from comment #13)
> Created attachment 797552 [details]
> required log

Do you have the file "/lib64/.libcrypto.so.10.hmac" in your real root?

What is the output of:

# lsinitrd | grep -E 'libcrypto.*.hmac'
Comment 15 Harald Hoyer 2013-09-14 03:43:34 EDT
When I ran on a F19 machine, I get:

# dracut --debug -f |& grep -E ': (cp|ln).*libcrypto.*.hmac'
dracut-install: cp '/usr/lib64/.libcrypto.so.1.0.1e.hmac' '/var/tmp/initramfs.knxfw3/usr/lib64/.libcrypto.so.1.0.1e.hmac'
dracut-install: ln -s '.libcrypto.so.1.0.1e.hmac' '/var/tmp/initramfs.knxfw3/lib64/.libcrypto.so.10.hmac'
Comment 16 Matěj Cepl 2013-09-14 11:52:03 EDT
(In reply to Harald Hoyer from comment #14)
> Do you have the file "/lib64/.libcrypto.so.10.hmac" in your real root?

No.

matej@wycliff: thunderbird-ensemble (master)$ ls /lib64/.libcrypto.so.10.hmac
ls: cannot access /lib64/.libcrypto.so.10.hmac: No such file or directory
matej@wycliff: thunderbird-ensemble (master)$ rpm -qf /lib64/.libcrypto.so.10.hmac 
error: file /lib64/.libcrypto.so.10.hmac: No such file or directory
matej@wycliff: thunderbird-ensemble (master)$ rpm -qf /lib64/libcrypto.so.10
openssl-libs-1.0.1e-19.el7.x86_64
matej@wycliff: thunderbird-ensemble (master)$ 

Adding openssl maintainer to CC of this bug.

> What is the output of:
> 
> # lsinitrd | grep -E 'libcrypto.*.hmac'

wycliff:~# lsinitrd |grep -E 'libcrypto.*.hmac'
wycliff:~#
Comment 17 Tomas Mraz 2013-09-16 08:26:49 EDT
Harald, the problem might be that the files got recently renamed - you need to use wildcard .lib{crypto,ssl}*.hmac  to copy them to initrd.
Comment 18 Harald Hoyer 2013-09-16 10:28:30 EDT
(In reply to Tomas Mraz from comment #17)
> Harald, the problem might be that the files got recently renamed - you need
> to use wildcard .lib{crypto,ssl}*.hmac  to copy them to initrd.

what? the strace clearly shows, that it wants /lib64/.libcrypto.so.10.hmac.

[pid   782] open("/lib64/.libcrypto.so.10.hmac", O_RDONLY) = -1 ENOENT (No such file or directory)
Comment 19 Tomas Mraz 2013-09-17 05:18:54 EDT
Ah, you're looking at F19 - yes, there is still the old file name.

But on F20 and RHEL-7 the files have longer suffix than just .hmac. The version-release from the rpm is included. And the files are in openssl-fips subpackage. Of course if you want to turn on the fips mode the -fips subpackages must be installed.
Comment 20 Harald Hoyer 2013-09-17 11:00:58 EDT
(In reply to Tomas Mraz from comment #19)
> Ah, you're looking at F19 - yes, there is still the old file name.

Hmm, Matěj Cepl should have used RHEL-7 to get the strace in comment 9

> 
> But on F20 and RHEL-7 the files have longer suffix than just .hmac. The
> version-release from the rpm is included. And the files are in openssl-fips
> subpackage. Of course if you want to turn on the fips mode the -fips
> subpackages must be installed.

Maybe Matěj Cepl does not have those installed
Comment 21 Matěj Cepl 2013-09-17 11:17:43 EDT
(In reply to Harald Hoyer from comment #20)
> Maybe Matěj Cepl does not have those installed

He has.

wycliff:~# rpm -qa openssl\*
openssl-static-1.0.1e-20.el7.x86_64
openssl-fips-1.0.1e-20.el7.x86_64
openssl-1.0.1e-20.el7.x86_64
openssl-debuginfo-1.0.1e-20.el7.x86_64
openssl-devel-1.0.1e-20.el7.x86_64
openssl-libs-1.0.1e-20.el7.i686
openssl-libs-1.0.1e-20.el7.x86_64
Comment 22 Harald Hoyer 2013-09-17 11:25:54 EDT
(In reply to Matěj Cepl from comment #21)
> (In reply to Harald Hoyer from comment #20)
> > Maybe Matěj Cepl does not have those installed
> 
> He has.
> 


What is the contents of this package?

> openssl-fips-1.0.1e-20.el7.x86_64
Comment 23 Matěj Cepl 2013-09-17 16:05:57 EDT
matej@wycliff: empathy (rhel-7 %)$ rpm -ql openssl-fips
/etc/prelink.conf.d
/etc/prelink.conf.d/openssl-fips.conf
/usr/lib64/.libcrypto.so.1.0.1e.1.0.1e-20.el7.hmac
/usr/lib64/.libcrypto.so.10.1.0.1e-20.el7.hmac
/usr/lib64/.libssl.so.1.0.1e.1.0.1e-20.el7.hmac
/usr/lib64/.libssl.so.10.1.0.1e-20.el7.hmac
matej@wycliff: empathy (rhel-7 %)$
Comment 24 Matěj Cepl 2013-09-17 16:07:30 EDT
(In reply to Matěj Cepl from comment #23)
> matej@wycliff: empathy (rhel-7 %)$ rpm -ql openssl-fips
> /etc/prelink.conf.d

BTW, this is unrealted but IMHO a packaging bug. This directory should be owned only by prelink package, shouldn't it?

matej@wycliff: empathy (rhel-7 %)$ rpm -qf /etc/prelink.conf.d/
prelink-0.5.0-2.el7.x86_64
openssl-fips-1.0.1e-20.el7.x86_64
openssh-server-fips-6.2p2-7.el7.x86_64
openssh-clients-fips-6.2p2-7.el7.x86_64
matej@wycliff: empathy (rhel-7 %)$
Comment 25 Tomas Mraz 2013-09-18 03:24:06 EDT
Nope, we cannot depend on prelink so we need to own it and it is allowed AFAIK by packaging guidelines.
Comment 26 Harald Hoyer 2013-09-20 02:03:17 EDT
(In reply to Matěj Cepl from comment #23)
> matej@wycliff: empathy (rhel-7 %)$ rpm -ql openssl-fips
> /etc/prelink.conf.d
> /etc/prelink.conf.d/openssl-fips.conf
> /usr/lib64/.libcrypto.so.1.0.1e.1.0.1e-20.el7.hmac
> /usr/lib64/.libcrypto.so.10.1.0.1e-20.el7.hmac
> /usr/lib64/.libssl.so.1.0.1e.1.0.1e-20.el7.hmac
> /usr/lib64/.libssl.so.10.1.0.1e-20.el7.hmac
> matej@wycliff: empathy (rhel-7 %)$

so, is openssl-fips missing a symlink? or was cryptsetup compiled with the wrong tools? Or does it have to be recompiled?
Comment 27 Tomas Mraz 2013-09-20 02:46:51 EDT
I believe we are mixing things here seriously - I think the strace is coming from older openssl where there was a problem with looking up the .hmac file however the 1.0.1e-20.el7 packages are fixed in this regard. I suppose the initrd was not regenerated appropriately.
Comment 28 Matěj Cepl 2013-09-20 11:35:47 EDT
(In reply to Tomas Mraz from comment #27)
> I believe we are mixing things here seriously - I think the strace is coming
> from older openssl where there was a problem with looking up the .hmac file
> however the 1.0.1e-20.el7 packages are fixed in this regard. I suppose the
> initrd was not regenerated appropriately.

I am sorry, but that isn't so ... this was generated on the latest EL-7 and I don't think I have any significant openssl upgrades meanwhile. And mkinitrd has been recreated with grub2-mkconfig. If there is anything wrong with using that, let me know, but I don't see any PEBKAC here.

Note You need to log in before you can comment on or make changes to this bug.