Bug 1015750

Summary: systemd-cryptsetup@swap.service fails to start
Product: Red Hat Enterprise Linux 7 Reporter: Matěj Cepl <mcepl>
Component: systemdAssignee: systemd-maint
Status: CLOSED DUPLICATE QA Contact: qe-baseos-daemons
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.0CC: harald, mcepl, rstrode, systemd-maint-list
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-25 20:58:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
output of journalctl SYSLOG_IDENTIFIER=dracut (as root)
none
/var/log/dracut
none
/boot/initramfs-3.10.0-65.el7.x86_64.img (after upgrade before dracut -f)
none
shot of the screen when the boot frozen
none
/boot/initramfs-3.10.0-65.el7.x86_64.img (after upgrade after dracut -f)
none
rdsosreport.txt
none
/boot/logs/strace.txt
none
output of ausearch -m AVC after the failed boot
none
audit2allow processed of ausearch output
none
journalctl -f when the installation happened none

Description Matěj Cepl 2013-10-05 06:20:30 UTC
Description of problem:
wycliff:~# systemctl restart systemd-cryptsetup
Please enter passphrase for disk vg_wycliff-lv_swap (swap) on swap! *********
Job for systemd-cryptsetup failed. See 'systemctl status systemd-cryptsetup' and 'journalctl -xn' for details.
wycliff:~# systemctl status systemd-cryptsetup
systemd-cryptsetup - Cryptography Setup for swap
   Loaded: loaded (/etc/crypttab)
   Active: failed (Result: exit-code) since Sat 2013-10-05 08:09:54 CEST; 1min 31s ago
     Docs: man:systemd-cryptsetup@.service(8)
           man:crypttab(5)
  Process: 618 ExecStop=/usr/lib/systemd/systemd-cryptsetup detach swap (code=killed, signal=TERM)
  Process: 2745 ExecStart=/usr/lib/systemd/systemd-cryptsetup attach swap /dev/vg_wycliff/lv_swap none  (code=exited, status=1/FAILURE)
 Main PID: 2745 (code=exited, status=1/FAILURE)

Oct 05 08:09:46 wycliff.ceplovi.cz systemd[1]: Starting Cryptography Setup for swap...
Oct 05 08:09:53 wycliff.ceplovi.cz systemd-cryptsetup[2745]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/vg_wycliff/lv_swap.
Oct 05 08:09:54 wycliff.ceplovi.cz systemd[1]: systemd-cryptsetup: main process exited, code=exited, status=1/FAILURE
Oct 05 08:09:54 wycliff.ceplovi.cz systemd[1]: Failed to start Cryptography Setup for swap.
Oct 05 08:09:54 wycliff.ceplovi.cz systemd[1]: Unit systemd-cryptsetup entered failed state.
wycliff:~# 


Version-Release number of selected component (if applicable):
systemd-207-2.el7.x86_64


How reproducible:
Happened now, but so far I haven't been able to start swap at all.

Steps to Reproduce:
1.see above
2.
3.

Actual results:
no swap

Expected results:
swap should be set on and working using encrypted storage

Additional info:

Comment 2 Harald Hoyer 2013-10-15 09:31:19 UTC
what is in /etc/crypttab?

Comment 3 Matěj Cepl 2013-10-15 10:08:55 UTC
(In reply to Harald Hoyer from comment #2)
> what is in /etc/crypttab?

You saw it couple of times already?

matej@wycliff: ~$ sudo cat /etc/crypttab 
swap /dev/vg_wycliff/lv_swap none
home /dev/vg_wycliff/lv_home none 
matej@wycliff: ~$

Comment 4 Matěj Cepl 2013-10-15 10:13:31 UTC
But meanwhile swap works again, so you probably are lucky.

Comment 5 Matěj Cepl 2014-01-07 15:58:09 UTC
OK, reopening. Currently the situation is that with kernel as upgraded it doesn't boot up without me prior running 'dracut -f' on the command line as a root after the upgrade.

Comment 6 Matěj Cepl 2014-01-07 16:20:57 UTC
Created attachment 846771 [details]
sosreport (just before upgrade)

Comment 8 Matěj Cepl 2014-01-08 08:26:42 UTC
Created attachment 847004 [details]
output of journalctl SYSLOG_IDENTIFIER=dracut (as root)

After upgrade

Comment 9 Matěj Cepl 2014-01-08 08:27:51 UTC
Created attachment 847005 [details]
/var/log/dracut

Comment 10 Matěj Cepl 2014-01-08 08:31:30 UTC
Created attachment 847007 [details]
/boot/initramfs-3.10.0-65.el7.x86_64.img (after upgrade before dracut -f)

wycliff:~# rpm -q kernel
kernel-3.10.0-60.el7.x86_64
kernel-3.10.0-64.el7.x86_64
kernel-3.10.0-65.el7.x86_64
wycliff:~#

Comment 11 Matěj Cepl 2014-01-10 21:56:04 UTC
Created attachment 848414 [details]
shot of the screen when the boot frozen

I have unfortunate development in the case. With kernel-3.10.0-65.el7.x86_64 and kernel-3.10.0-67.el7.x86_64 my trick with rebuilding initramfs doesn’t work anymore, and even after multiple rebuildings (I have checked everything thrice to be sure I haven’t screwed up anywhere). Unfortunately, I don’t know how to get journalctl when I have never got a prompt during the failed boot in the first place. Here is at least the shot of the screen in moment when it got frozen.

Comment 12 Matěj Cepl 2014-01-10 21:59:19 UTC
Created attachment 848416 [details]
/boot/initramfs-3.10.0-65.el7.x86_64.img (after upgrade after dracut -f)

Comment 13 Matěj Cepl 2014-01-10 22:00:34 UTC
Created attachment 848417 [details]
rdsosreport.txt

Comment 14 Matěj Cepl 2014-01-10 22:01:12 UTC
Created attachment 848418 [details]
/boot/logs/strace.txt

Comment 15 Matěj Cepl 2014-01-10 22:02:03 UTC
Created attachment 848419 [details]
output of ausearch -m AVC after the failed boot

Comment 16 Matěj Cepl 2014-01-10 22:02:41 UTC
Created attachment 848420 [details]
audit2allow processed of ausearch output

Comment 17 Ray Strode [halfline] 2014-01-13 21:01:10 UTC
are you saying that older versions of dracut showed this problem, but rebuilding the initrd with dracut -f on these older kernels fixes it?

If so, that's totally expected.  "yum update" won't rebuild old initrds, it will only rebuild the initrd of new kernels (when those kernels get installed)

Comment 18 Matěj Cepl 2014-01-13 21:48:05 UTC
(In reply to Ray Strode [halfline] from comment #17)
> are you saying that older versions of dracut showed this problem, but
> rebuilding the initrd with dracut -f on these older kernels fixes it?

“fixed it” up until and including kernel -60. More recent versions of kernel are hopeless.

> If so, that's totally expected.  "yum update" won't rebuild old initrds, it
> will only rebuild the initrd of new kernels (when those kernels get
> installed)

No, I haven’t meant rebuilding old initrds. I meant that even the kernel being upgraded was useless and I had to manually run dracut -f on it to get working initramfs?

Comment 19 Ray Strode [halfline] 2014-01-13 23:06:23 UTC
that's very strange, installing a new kernel causes dracut -f to get run "behind the scenes".  maybe some selinux issue?

Comment 20 Matěj Cepl 2014-01-14 13:09:32 UTC
(In reply to Ray Strode [halfline] from comment #19)
> that's very strange, installing a new kernel causes dracut -f to get run
> "behind the scenes".  maybe some selinux issue?

Not sure, how to reproduce it without further kernel update. I believe I have checked ausearch output, but not 100% certain. Let me know, when new kernel update happens.

Comment 21 Matěj Cepl 2014-01-17 11:20:35 UTC
(In reply to Ray Strode [halfline] from comment #19)
> that's very strange, installing a new kernel causes dracut -f to get run
> "behind the scenes".  maybe some selinux issue?

Actually, attachment 848419 [details] and attachment 848420 [details] is the SELinux stuff.

Comment 22 Matěj Cepl 2014-01-17 11:22:57 UTC
Created attachment 851510 [details]
journalctl -f when the installation happened

We have new kernel in the system, so I have managed to capture journalctl during it.

Comment 23 Matěj Cepl 2014-01-17 11:24:09 UTC
Also, ausearch -m AVC -ts <relevant-time> shows '<no matches>'

Comment 25 Matěj Cepl 2014-01-18 20:32:44 UTC
Upgraded to 
dracut-033-84.el7.x86_64
and
plymouth-0.8.9-0.3.20140113.el7.x86_64.rpm
and this happened

wycliff:~# dracut -f --kver=3.10.0-71.el7.x86_64
/usr/lib/dracut/modules.d/99base/dracut-lib.sh: line 1: !/bin/sh: No such file or directory
wycliff:~# rpm -q dracuát
package dracuát is not installed
wycliff:~# rpm -q dracut

wycliff:~#

Comment 26 Matěj Cepl 2014-02-25 17:54:33 UTC
Actually with

kernel-3.10.0-89.el7.x86_64
dracut-033-124.el7.x86_64
systemd-208-4.el7.x86_64
plymouth-0.8.9-0.2014.01.13.1.el7.x86_64

this seems to be working. Without any rebuilding mkinitrd I get the system booting again. Thanks!

Comment 27 Ray Strode [halfline] 2014-02-25 20:58:49 UTC
presumably a dupe of bug 1015564 then.

*** This bug has been marked as a duplicate of bug 1015564 ***