Bug 1688283

Summary: Clevis fails to unlock encrypted partition with iot (ostree) installation
Product: [Fedora] Fedora Reporter: Paul Whalen <pwhalen>
Component: clevisAssignee: Sergio Correia <scorreia>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: fmartine, npmccallum, pbrobinson, pk.eskse, scorreia
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
If this bug requires documentation, please select an appropriate Doc Type value.http://iot.stg.fedoraproject.org/
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-31 00:03:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1269538    
Attachments:
Description Flags
tpm2-luks1-ks.cfg
none
tpm2-luks2-ks.cfg none

Description Paul Whalen 2019-03-13 12:58:13 UTC
Description of problem:
Attempting to use clevis/tpm2 with Fedora IoT edition (ostree) to unlock an encrypted partition at boot. The installation is completed without error but requires user input to unlock the root partition.

Version-Release number of selected component (if applicable):
clevis-11-5.fc30.x86_64
clevis-luks-11-5.fc30.x86_64
clevis-systemd-11-5.fc30.x86_64
clevis-dracut-11-5.fc30.x86_64

tpm2-tools-3.1.3-4.fc30.x86_64
tpm2-tss-2.2.0-1.fc30.x86_64



How reproducible:
Always when using an ostree based installation.

Steps to Reproduce:
1. Encrypted install using the latest IoT installation media and tpm2 hardware
2. In the kickstart (or manually enter the following commands after installation) use:


    autopart --encrypted --type=lvm --luks-version=luks1 --passphrase=fedora

    %post --erroronfail
    # Find the LUKS volume
    UUID=$(lsblk | grep luks | sed 's/^.*luks-//' | cut -d ' ' -f1)
    DEV=$(blkid --uuid $UUID)
    # Bind the LUKS volume to a TPM2 for automatic unlocking on boot
    clevis luks bind -f -k- -d $DEV tpm2 '{}' <<< fedora

Additional info:

Works when using a standard Fedora 30 installation with the same package versions and kickstart snippets.

Clevis is included in the initramfs for IoT:

lsinitrd initramfs-5.0.0-0.rc8.git0.1.fc30.x86_64.img | grep clevis
clevis
-rw-r--r--   1 root     root            1 Feb 14 14:03 etc/cmdline.d/99clevis.conf
-rwxr-xr-x   1 root     root         1563 Aug 13  2018 usr/bin/clevis
-rwxr-xr-x   1 root     root         1614 Aug 13  2018 usr/bin/clevis-decrypt
-rwxr-xr-x   1 root     root        33304 Jan 31 09:46 usr/bin/clevis-decrypt-sss
-rwxr-xr-x   1 root     root         2721 Aug 13  2018 usr/bin/clevis-decrypt-tang
-rwxr-xr-x   1 root     root         3794 Aug 13  2018 usr/bin/clevis-decrypt-tpm2
-rwxr-xr-x   1 root     root           49 Jan 31 09:46 usr/lib/dracut/hooks/initqueue/online/60-clevis-hook.sh
-rwxr-xr-x   1 root     root           49 Jan 31 09:46 usr/lib/dracut/hooks/initqueue/settled/60-clevis-hook.sh
-rwxr-xr-x   1 root     root         2873 Aug 13  2018 usr/libexec/clevis-luks-askpass

Comment 1 Paul Whalen 2019-03-13 13:10:42 UTC
After booting the system to check:

[root@fitlet2 ~]# luksmeta show -d /dev/sda3
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
2 inactive empty
3 inactive empty
4 inactive empty
5 inactive empty
6 inactive empty
7 inactive empty

Password is correct:

[root@fitlet2 ~]# cryptsetup luksOpen --test-passphrase --key-slot 0 /dev/sda3 && echo correct
Enter passphrase for /dev/sda3:
correct

Remove the password and try again:

[root@fitlet2 ~]# clevis luks unbind -d /dev/sda3 -s 1
The unbind operation will wipe a slot. This operation is unrecoverable.
Do you wish to erase LUKS slot 1 on /dev/sda3? [ynYN] y
Enter any remaining passphrase:

[root@fitlet2 ~]# luksmeta show -d /dev/sda3
0   active empty
1 inactive empty
2 inactive empty
3 inactive empty
4 inactive empty
5 inactive empty
6 inactive empty
7 inactive empty

Try to use the clevis commands again:

[root@fitlet2 ~]# UUID=$(lsblk | grep luks | sed 's/^.*luks-//' | cut -d ' ' -f1)
[root@fitlet2 ~]# DEV=$(blkid --uuid $UUID)
[root@fitlet2 ~]# echo $DEV
/dev/sda3
[root@fitlet2 ~]# clevis luks bind -f -k- -d $DEV tpm2 '{}' <<< fedora
[root@fitlet2 ~]# luksmeta show -d /dev/sda3
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
2 inactive empty
3 inactive empty
4 inactive empty
5 inactive empty
6 inactive empty
7 inactive empty

Reboot and it again fails to unlock. The same commands work on a standard F29/30 installation.

Comment 2 Javier Martinez Canillas 2019-03-13 14:47:39 UTC
(In reply to Paul Whalen from comment #0)

[snip]

> 
> Works when using a standard Fedora 30 installation with the same package
> versions and kickstart snippets.
> 
> Clevis is included in the initramfs for IoT:
> 
> lsinitrd initramfs-5.0.0-0.rc8.git0.1.fc30.x86_64.img | grep clevis
> clevis

Are the tpm2-tools and libraries also included in the initramfs? i.e:

lsinitrd /boot/initramfs-$(uname -r).img | grep -E 'clevis|tpm2|tss2'

Comment 3 Paul Whalen 2019-03-13 18:48:54 UTC
lsinitrd initramfs-5.0.0-300.fc30.x86_64.img | grep -E 'clevis|tpm2|tss2'

clevis
-rw-r--r--   1 root     root            1 Feb 14 14:03 etc/cmdline.d/99clevis.conf
-rwxr-xr-x   1 root     root         1563 Aug 13  2018 usr/bin/clevis
-rwxr-xr-x   1 root     root         1614 Aug 13  2018 usr/bin/clevis-decrypt
-rwxr-xr-x   1 root     root        33304 Jan 31 09:46 usr/bin/clevis-decrypt-sss
-rwxr-xr-x   1 root     root         2721 Aug 13  2018 usr/bin/clevis-decrypt-tang
-rwxr-xr-x   1 root     root         3794 Aug 13  2018 usr/bin/clevis-decrypt-tpm2
-rwxr-xr-x   1 root     root        95408 Feb  3 04:10 usr/bin/tpm2_createprimary
-rwxr-xr-x   1 root     root        92712 Feb  3 04:10 usr/bin/tpm2_load
-rwxr-xr-x   1 root     root        87968 Feb  3 04:10 usr/bin/tpm2_pcrlist
-rwxr-xr-x   1 root     root       108424 Feb  3 04:10 usr/bin/tpm2_unseal
-rwxr-xr-x   1 root     root       303880 Feb 14 14:03 usr/lib64/libtss2-mu.so.0.0.0
lrwxrwxrwx   1 root     root           19 Feb 14 14:03 usr/lib64/libtss2-mu.so.0 -> libtss2-mu.so.0.0.0
-rwxr-xr-x   1 root     root       403216 Feb 14 14:03 usr/lib64/libtss2-sys.so.0.0.0
lrwxrwxrwx   1 root     root           20 Feb 14 14:03 usr/lib64/libtss2-sys.so.0 -> libtss2-sys.so.0.0.0
-rwxr-xr-x   1 root     root        36416 Feb 14 14:03 usr/lib64/libtss2-tcti-device.so.0.0.0
lrwxrwxrwx   1 root     root           28 Feb 14 14:03 usr/lib64/libtss2-tcti-device.so.0 -> libtss2-tcti-device.so.0.0.0
-rwxr-xr-x   1 root     root           49 Jan 31 09:46 usr/lib/dracut/hooks/initqueue/online/60-clevis-hook.sh
-rwxr-xr-x   1 root     root           49 Jan 31 09:46 usr/lib/dracut/hooks/initqueue/settled/60-clevis-hook.sh
-rwxr-xr-x   1 root     root         2873 Aug 13  2018 usr/libexec/clevis-luks-askpass

Comment 4 Javier Martinez Canillas 2019-03-14 09:25:18 UTC
With Fedora-IoT-ostree-x86_64-30-20190215.0.iso I get:

# echo foobar | clevis encrypt tpm2 '{}' | clevis decrypt
WARNING:tcti:src/tss2-tcti/tcti-device.c:254:tcti_device_receive() Got EOF instead of response. 
ERROR: 
Create Object Failed ! ErrorCode: 0xa0008

ERROR: Unable to run tpm2_create
Creating TPM2 object for jwk failed!

# tpm2_rc_decode 0xa0008
error layer
  hex: 0xa0000
  identifier: TSS2_TCTI_RC_LAYER
  description: Error from the TCTI
base error code
  identifier: TSS2_BASE_RC_NO_CONNECTION
  description: Fails to connect to next lower layer

# uname -r
5.0.0-0.rc6.git1.1.fc30.x86_64

Which is reported to be a regression in the kernel:

https://github.com/tpm2-software/tpm2-tools/issues/1356

Comment 5 Javier Martinez Canillas 2019-03-14 09:32:36 UTC
Following is the thread in the linux-integrity mailing list where the problem and possible solutions are being discussed:

https://www.spinics.net/lists/linux-integrity/msg06386.html

Comment 6 Javier Martinez Canillas 2019-03-22 09:37:40 UTC
Upstream patch that solves this bug:

https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1959715.html

Comment 7 Fedora Update System 2019-03-26 16:14:54 UTC
kernel-tools-5.0.4-300.fc30 kernel-headers-5.0.4-300.fc30 kernel-5.0.4-300.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-4e7590b99c

Comment 8 Fedora Update System 2019-03-26 16:16:32 UTC
kernel-tools-5.0.4-200.fc29 kernel-headers-5.0.4-200.fc29 kernel-5.0.4-200.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0ba1e6642f

Comment 9 Fedora Update System 2019-03-27 00:45:26 UTC
kernel-5.0.4-300.fc30, kernel-headers-5.0.4-300.fc30, kernel-tools-5.0.4-300.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-4e7590b99c

Comment 10 Fedora Update System 2019-03-27 04:34:42 UTC
kernel-5.0.4-200.fc29, kernel-headers-5.0.4-200.fc29, kernel-tools-5.0.4-200.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0ba1e6642f

Comment 11 Fedora Update System 2019-03-29 02:59:06 UTC
kernel-5.0.4-200.fc29, kernel-headers-5.0.4-200.fc29, kernel-tools-5.0.4-200.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 12 Fedora Update System 2019-03-31 00:03:59 UTC
kernel-5.0.4-300.fc30, kernel-headers-5.0.4-300.fc30, kernel-tools-5.0.4-300.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 13 Paul Whalen 2019-04-02 14:26:47 UTC
Using compose Fedora-IoT-30-20190327.0

[root@fitlet2 ~]# echo foo | clevis encrypt tpm2 '{}' | clevis decrypt
foo
[root@fitlet2 ~]# uname -r
5.0.0-300.fc30.x86_64

But the automatic decryption still fails on boot. If I add 'rd.break=initqueue' to the kernel args to get a shell prior to decryption, then 'exit' the system will decrypt the volume and boot to log in. It appears to be working to some degree but perhaps a timing issue?

This also happens with a F29 compose which includes kernel-4.20.16-200.fc29.

Comment 14 Javier Martinez Canillas 2019-04-16 15:09:19 UTC
Created attachment 1555593 [details]
tpm2-luks1-ks.cfg

Comment 15 Javier Martinez Canillas 2019-04-16 15:11:01 UTC
Created attachment 1555594 [details]
tpm2-luks2-ks.cfg

Comment 16 Javier Martinez Canillas 2019-04-16 15:29:50 UTC
I tried to reproduce this issue on the same hardware than Paul used and couldn't reproduce the exact issue but had a different one that could be related.

Using the kickstart tpm2-luks1-ks.cfg file attached in Comment 14, the system was installed correctly but the LUKS volume was not unlocked automatically by clevis. Looking at the LUKSv1 and LUKSMeta headers, I noticed that a LUKSMeta slot was added for the bound pin but no a LUKS slot:

$ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active"
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e

$ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED
Key Slot 0: ENABLED

But if I bind a pin to a LUKS volume after the installation, it did work correctly:

$ sudo luksmeta wipe -d /dev/sda3 -s 1

$ sudo clevis luks bind -d /dev/sda3 tpm2 '{}'

$ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active"
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e

$ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED
Key Slot 0: ENABLED
Key Slot 1: ENABLED

And also the LUKS volume was automatically unlocked on boot.

It also worked correctly if I used LUKSv2 instead of LUKSv1 like in the tpm2-luks2-ks.cfg kickstart file attached in Comment 15.

So the problem seems to be only when storing the clevis pin on a LUKSv1 volume and when the clevis luks bind command is executed in the kickstart file.

Comment 17 pk.eskse 2019-07-16 18:03:20 UTC
I've been having a similar issue on Fedora Silverblue, as this seems to still be open I thought I'd post it here rather than open a new bug report.

It seems to be that the required initramfs modules are not loaded by the layer provided by what I assume would be the "clevis-dracut" package (if that's not the correct terminology let me know as I only installed Silverblue yesterday to play around with).

Enabling client side generation of the initramfs via "rpm-ostree initramfs --enable" resolves this.

Comment 19 Sergio Correia 2020-01-17 11:52:14 UTC
(In reply to Javier Martinez Canillas from comment #16)
> I tried to reproduce this issue on the same hardware than Paul used and
> couldn't reproduce the exact issue but had a different one that could be
> related.
> 
> Using the kickstart tpm2-luks1-ks.cfg file attached in Comment 14, the
> system was installed correctly but the LUKS volume was not unlocked
> automatically by clevis. Looking at the LUKSv1 and LUKSMeta headers, I
> noticed that a LUKSMeta slot was added for the bound pin but no a LUKS slot:
> 
> $ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active"
> 0   active empty
> 1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
> 
> $ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED
> Key Slot 0: ENABLED
> 
> But if I bind a pin to a LUKS volume after the installation, it did work
> correctly:
> 
> $ sudo luksmeta wipe -d /dev/sda3 -s 1
> 
> $ sudo clevis luks bind -d /dev/sda3 tpm2 '{}'
> 
> $ sudo luksmeta show -d /dev/sda3 | grep "^[0-9][ ]*active"
> 0   active empty
> 1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
> 
> $ sudo cryptsetup luksDump /dev/sda3 | grep ENABLED
> Key Slot 0: ENABLED
> Key Slot 1: ENABLED
> 
> And also the LUKS volume was automatically unlocked on boot.
> 
> It also worked correctly if I used LUKSv2 instead of LUKSv1 like in the
> tpm2-luks2-ks.cfg kickstart file attached in Comment 15.
> 
> So the problem seems to be only when storing the clevis pin on a LUKSv1
> volume and when the clevis luks bind command is executed in the kickstart
> file.

Javier: Can you still reproduce this issue?

Comment 20 Javier Martinez Canillas 2020-01-17 11:55:28 UTC
(In reply to Sergio Correia from comment #19)
> (In reply to Javier Martinez Canillas from comment #16)

[snip]

> > 
> > So the problem seems to be only when storing the clevis pin on a LUKSv1
> > volume and when the clevis luks bind command is executed in the kickstart
> > file.
> 
> Javier: Can you still reproduce this issue?

I don't have access to the machine where I was able to reproduce the issue.

But you could ask Paul Whalen or Peter Robinson if they are still facing issues with clevis and TPM2 devices in the Fedora IoT spin.