This is also reproducible in Fedora Server 31.

With the same configuration, I get a broken device when I set the _netdev property in crypttab. And if I remove it, I get the password prompt as expected.

(In reply to nicolasoliver03 from comment #0)

[snip]

> 
> I have added the entry in /etc/crypttab 
> 
> [root@fedora-iot-2 ~]# cat /etc/crypttab 
> encryptedhome /dev/mapper/fedora--iot-home none _netdev
> 
> And the entry in /etc/fstab
> 
> [root@fedora-iot-2 ~]# cat /etc/fstab 
> 
> #
> # /etc/fstab
> # Created by anaconda on Wed Sep 25 13:06:52 2019
> #
> # Accessible filesystems, by reference, are maintained under '/dev/disk/'.
> # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
> #
> # After editing this file, run 'systemctl daemon-reload' to update systemd
> # units generated from this file.
> #
> /dev/mapper/fedora--iot-root /                       ext4    defaults       
> 1 1
> UUID=32e96389-4eb8-4a8d-8f39-383d1fe3b13b /boot                   ext4   
> defaults        1 2
> UUID=E058-A35D          /boot/efi               vfat   
> umask=0077,shortname=winnt 0 2
> /dev/mapper/fedora--iot-swap none                    swap    defaults       
> 0 0
> /dev/mapper/encryptedhome /var/lib/docker ext4 defaults  0 2
> 

My understanding is that this is expected.

Because clevis-luks-askpass is executed after the network-online.target, which means that the /dev/mapper/fedora--iot-home LUKS volume will only be unlocked after the network has been brought up. But the network is started after the local file systems have been mounted (in local-fs.target) but local-fs.target is waiting for /dev/mapper/encryptedhome to be mounted and so the network is never brought up.

The crypttab(5) man page explains this situation much better than me:

"Hint: if this device is used for a mount point that is specified in fstab(5), the _netdev option should also be used for the mount point. Otherwise, a dependency loop might be created where the mount point will be pulled in by local-fs.target, while the service to configure the network is usually only started after the local file system has been mounted."

So if you want to use the _netdev option in your /etc/crypttab entry, then you need to do something like the following in /etc/fstab:

/dev/mapper/encryptedhome /var/lib/docker ext4 defaults,_netdev  0 2

> 
> If I remove the _netdev parameter in cryptab, I get the password prompt on
> late boot, input the correct password, and my system boots with the correct
> mapper and mount point.
> 

Yes, that's also expected because in that case systemd-ask-password doesn't have to wait for the network to be brought up to ask for a password, so you can enter the password for /dev/mapper/fedora--iot-home to be unlocked, and after that /dev/mapper/encryptedhome is present so the partition can be mounted and the local-fs.target complete.

So this is not really a bug but I agree that at least the clevis documentation should be improved to mention this. I also got confused by this and had to dig in the different man pages and clevis source to understand what was happening.

Now I've two questions about this behaviour in clevis though:

1- Why the clevis systemd-ask-password responder has to be executed after the network-online.target? (which makes the /etc/crypttab entries to require a _netdev option).

2- Could the clevis-luks-askpass.path watcher be more flexible to not require this for some cases?

I think (1) is just because tang was the only pin supported and that of course required the network. But if you are only using the clevis tpm2 pin then something like After=dev-tpm0.device should be enough.

I don't know enough about systemd unit files to figure out how to accomplish (2) but that's something that we should discuss with clevis upstream.

Thank you Javier!

I updated my /etc/fstab with _netdev, and my Fedora Server 31 automatically unlocked and mounted the disk :)
Same happened in my Fedora IoT 31. So it was a configuration issue on my side.

I actually had to guess how all this after reading clevis, and crypttab documentation, and I missed the _netdev in fstab.
The script I used to reproduce this is below, in case that somebody see this comment in the future:


pvcreate /dev/sda
vgcreate lvmvolume /dev/sda
lvcreate -y -L 5G -n root lvmvolume
lvcreate -y -L 512M -n swap lvmvolume
lvcreate -y -L 2G -n encryptedhome lvmvolume
openssl rand -hex 8 > key
cryptsetup -q luksFormat /dev/lvmvolume/encryptedhome key
clevis luks bind -f -k key -d /dev/lvmvolume/encryptedhome tpm2 '{}'
clevis luks unlock -d /dev/lvmvolume/encryptedhome -n c1
mkfs.ext4 /dev/mapper/c1
sleep 1
cryptsetup luksClose c1

echo "encryptedhome /dev/mapper/fedora--iot-home none _netdev" >> /etc/crypttab
echo "/dev/mapper/encryptedhome /var/home ext4 defaults,_netdev  0 2" >> /etc/fstab


Apparently, _netdev causes to wait for the network-online.target. Does this means that my disk will not get unlocked if I have no network interface enabled? Will test that!

I disabled wifi, gsm, and unplugged the ethernet cable. Apparently this works as long as I get a loopback address configured. 
This is good for the TPM case, but could that cause problems in the TANG scenario?

Hello Javier and  Nicolas,

I have the same issue but I still can not resolve it. Right now I have encrypted my 2 partitions, like /dev/sda2 and /dev/sda4. I created these 2 partitions and format them as LUKS and created 2 different CRYPT mapper, 
I encrypted like this:
for sda2:
cryptsetup  luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 Enc
mkfs.xfs /dev/mapper/Enc 

for sda4:
cryptsetup  luksFormat /dev/sda4 
cryptsetup luksOpen /dev/sda4 system 
mkfs.xfs /dev/mapper/system 

then bindto tpm:

clevis luks bind -f -k- -d /dev/sda2 tpm2 '{"pcr_ids":"7,11"}'
clevis luks bind -f -k- -d /dev/sda4 tpm2 '{"pcr_ids":"7,11"}'

After that I follow: https://bugzilla.redhat.com/show_bug.cgi?id=1784084 setup:
# cat > /etc/systemd/system/clevis-luks-askpass.path << EOF
[Unit]
Description=Clevis systemd-ask-password Watcher
DefaultDependencies=no
Conflicts=shutdown.target
Before=cryptsetup.target

[Path]
PathChanged=/run/systemd/ask-password

[Install]
WantedBy=cryptsetup.target
EOF
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
# cat > /etc/systemd/system/clevis-luks-askpass.service << EOF
[Unit]
Description=Clevis LUKS systemd-ask-password Responder
DefaultDependencies=no
Conflicts=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/libexec/clevis-luks-askpass -l
EOF

Inorder to not prompt password input I follow: https://discussion.fedoraproject.org/t/automatic-decrypt-with-tpm2-on-silverblue/8424
sudo systemctl edit systemd-ask-password-plymouth.service

[Service]
ExecStartPre=/bin/sleep 120
and

sudoedit /etc/dracut.conf.d/systemd-ask-password-plymouth.conf

install_items+=" /etc/systemd/system/systemd-ask-password-plymouth.service.d/override.conf "

then : dracut -fv --regenerate-all 
reboot, I found it automatically boot up system and unlock the encrypted partions but sometimes it does not. I am not sure what is going on my system. But it is look like what you said.
these 2 partitions are not my root partitions, under production requirement we can not connect with network, if using dracut -fv --regenerate-all to rebuild initramfs then system can not bring up without network. I have to pass rd.neednet=0 from command line in /etc/default/grub and rebuild initramfs. 

Question:  
1. Nicolas, how come you can bring up without network disabled in such situation? how do you configure your loopback ip? your rd.neednet=1?
2. Do you know what my issue is? I am using tpm2, in your comment you said: After=dev-tpm0.device should be enough, how to do that?

Thank you! Please help and appreciate.

Hello!

I have this setup automated, and for me it is consistently working. 

The first step is to enable the clevis-luks-askpass.path service. The service is installed by default, at least in Fedora IoT 31.

Then, I create my encrypted partitions with the following commands:

# Encrypted Partition: /var/opt
lvcreate -y -l 20%VG -n opt system
printf "PASSWORD" | cryptsetup -q luksFormat /dev/system/opt -d -
printf "PASSWORD" | clevis luks bind -f -k- -d /dev/system/opt tpm2 '{}'
printf "PASSWORD" | cryptsetup luksOpen /dev/system/opt c1 -d -
mkfs.ext4 /dev/mapper/c1
sleep 1
cryptsetup luksClose c1
echo "luks-system-opt /dev/system/opt none _netdev" >> /etc/crypttab
echo "/dev/mapper/luks-system-opt /var/opt ext4 defaults,_netdev  0 2" >> /etc/fstab

In my case, I am using LVM for my partitions, but should work with standard partitions as well. 

In the boot process, I see the Clevis Luks Askpass service sometimes failing as well, but it immediately retries the unlock process and succeed. 
The only mayor difference I see from my setup and yours is the usage of PCR 7 and 11 as the integrity policy.
In PCR 7, I can inspect the measurements from the TPM Event Log, which can be seen below. So, if something measured there changes, you will get a locked partition, my guess is the DBX (which gets periodically updated)
I have never seen any software using the PCR 11. Is that something custom? My guess is that those measurements will not be included  in the event log.

$ sudo tsseventextend -sim -if /sys/kernel/security/tpm0/binary_bios_measurements -v | grep 'PCR index 7' --context=2

eventextend: line 3
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 80000001 EV_EFI_VARIABLE_DRIVER_CONFIG
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 4
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 80000001 EV_EFI_VARIABLE_DRIVER_CONFIG
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 5
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 80000001 EV_EFI_VARIABLE_DRIVER_CONFIG
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 6
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 80000001 EV_EFI_VARIABLE_DRIVER_CONFIG
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 7
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 80000001 EV_EFI_VARIABLE_DRIVER_CONFIG
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 8
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 00000004 EV_SEPARATOR
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 21
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 800000e0 EV_EFI_VARIABLE_AUTHORITY
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 24
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 800000e0 EV_EFI_VARIABLE_AUTHORITY
TSS_EVENT2_Line_Trace: digest count 2
--

eventextend: line 81
TSS_EVENT2_Line_Trace: PCR index 7
TSS_EVENT_EventType_Trace: 800000e0 EV_EFI_VARIABLE_AUTHORITY
TSS_EVENT2_Line_Trace: digest count 2

Thank you Nicolas for prompt reply. 

Yes It looks like our setting are very similar. From boot console I always have boot issue like 2-3 out of 10 when Clevis Luks askpass service failed to run and then hang. Actually I am the newbie of this TPM stuff and I will test your way without specify to bind PCR 7 and 11. 

Right now based on my setup I add following 2 line in clevis-luks-askpass.service Unit section, then it boots and automatically unlock consistently, but I can not tell the reason. Why I add this 2 lines because every time it boots successfully, I see network online service started before Clevis Luks askpass service, I am afraid there is race condition? So I add 2 line to tell Clevis Luks askpass service to run after network online service. 

Wants=metwork-online.target
After=network-online.target

I will also try your encryption way and let you know my status. Thank you again.

Regards

In my case, I have those dependencies declared by default with the installed service. The network-online.target is reached when the loopback interface is ready, you don't actually need external network access.

[root@first ~]# rpm -qa clevis*
clevis-12-1.fc31.x86_64
clevis-luks-12-1.fc31.x86_64
clevis-systemd-12-1.fc31.x86_64
clevis-dracut-12-1.fc31.x86_64

[root@first ~]# systemctl cat clevis-luks-askpass.service
# /usr/lib/systemd/system/clevis-luks-askpass.service
[Unit]
Description=Clevis LUKS systemd-ask-password Responder
Requires=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/libexec/clevis-luks-askpass -l

[root@first ~]# systemctl cat clevis-luks-askpass.path 
# /usr/lib/systemd/system/clevis-luks-askpass.path
[Unit]
Description=Clevis systemd-ask-password Watcher
Before=remote-fs-pre.target
Wants=remote-fs-pre.target

[Path]
PathChanged=/run/systemd/ask-password

[Install]
WantedBy=remote-fs.target

I noticed that there is a typo in your "Wants=metwork-online.target" (should be network)

In the case that there is a service that needs to wait until your encrypted partitions are ready, you need to add a configuration for that.
Otherwise, your service may start (and fail) before your partition is mounted.
In my case, I have a partition for the Docker stuff mounted in /var/lib/docker.
To make the Docker service wait until that partition is unlocked and ready, I added this override configuration to the docker service

mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/override.conf << EOF
[Unit]
After=var-lib-docker.mount
EOF

So then, docker.service start after the var-lib-docker.mount service.
var-lib-docker.mount is auto generated, and I actually don't know if it is clevis or the OS that generated them.
You can check your generated mount service name with 

$ systemctl list-units | grep '\.mount '

Thank you so much Nicola. For point pout my typo helpful information you posted. You are very helpful and you are my life saver. :) Appreciated!

Right now I determine to encrypt SWAP or root (/) partition, I am wondering swap and / partition are all belongs to early boot. I am not sure if it will work in such a way. Have you ever tried encrypt Swap & root ( / )?

For the swap encryption, it is very simple:

I have my swap partition created as a LVM volume:

$ logvol swap --vgname system --percent=5 --name=swap

And then, the encryption configuration is done with the following line

$ echo "system-swap /dev/system/swap /dev/urandom swap,cipher=aes-xts-plain64,size=256" >> /etc/crypttab

For the root partition encryption, I got it working on Fedora Server, but Fedora IoT have some problems.
You can check the details in this bug https://bugzilla.redhat.com/show_bug.cgi?id=1688283

Hope it helps!

I did encrypt my standard partition /dev/sda3 which is a swap when partition in installation.

swapoff -a ;
echo "YES" | cryptsetup open --type plain /dev/sda3 cswap
mkswap /dev/mapper/cswap ;

echo ""cswap /dev/sda3 /dev/urandom swap,cipher=aes-xts-plain64" >> /etc/crypttab

Then it works! Before I always failed. Probably since this race condition has been resolved. 

in case same issue happen, I took your suggestion to configure  to wait until my encrypted partitions are ready, everything works consistently so far! Thanks a lot!

Question: ever try encrypt swap via TPM?


Next I will try root ( / ) partition and let you know after.

Regards

Hi Nicolas,

We are not deciding to encrypt root partition so far. But since I have added swap partition to be encrypted like:

echo "cswap /dev/sda3 /dev/urandom swap,cipher=aes-xts-plain64,size=256" > /etc/crypttab ;

echo "/dev/mapper/cswap swap swap defaults        0 0" >> /etc/fstab

It sometime boot still hang but very less. like 1 out of 50 hang. When it hang I could see my following 3 encrypted device in turn keep flashing..

A start job is running for dev-mapper-scwap.device   
A start job is running for dev-mapper-system.device  
A start job is running for dev-mapper-Enc.device

It looks like still has race condition in there.

Do you have any idea?

Regards,

FEDORA-2020-b35219394f has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-b35219394f

FEDORA-2020-d42f4e90f9 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-d42f4e90f9

FEDORA-2020-d42f4e90f9 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-d42f4e90f9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-d42f4e90f9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-b35219394f has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-b35219394f`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-b35219394f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2020-d42f4e90f9 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2020-b35219394f has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.