Created attachment 913791 [details] journctl -xb output Description of problem: When I updated to cryptsetup-1.6.5-1.fc21 my systems would not successfully boot. Downgrading to cryptsetup-1.6.4-3.fc21.x86_64 got things working again. How reproducible: It happens every boot. It gets through early boot with encrypted root, but fails near the root pivot. I have /home as a separate luks partition.
Proposing as an f21 alpha blocker because: Expected installed system boot behavior A working mechanism to create a user account must be clearly presented during installation and/or first boot of the installed system. A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility. A system installed without a graphical package set must boot to a state where it is possible to log in through at least one of the default virtual consoles. Encrypted partitions In all of the above cases, if any system partitions were encrypted as part of the installation, the boot process must prompt for the passphrase(s) and correctly unlock the partition(s) when provided with the correct passphrase(s). User intervention
There are two LUKS devvice, only one is failing (I think it is one with ESSIV). Please can you paste lsblk and blkid from working system? I this standard install or some older upgraded system?
Do you run with selinux enforcing? Can you try to disable selinux and try again? I was able to reproduce this problem (only for devices using ESSIV IV) and seems that for some reason socket write for kernel crypto API returns ENOACCESS. (in new wrapper in 1.6.5) With selinux disabled it works for me...
This is what I get: type=SYSCALL msg=audit(1404241784.204:359): arch=c000003e syscall=46 success=yes exit=16 a0=a a1=7fff34e833f0 a2=0 a3=140 items=0 ppid=1 pid=1083 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="systemd-cryptse" exe="/usr/lib/systemd/systemd-cryptsetup" subj=system_u:system_r:lvm_t:s0 key=(null) type=AVC msg=audit(1404241784.204:359): avc: denied { write } for pid=1083 comm="systemd-cryptse" scontext=system_u:system_r:lvm_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=socket permissive=1 cryptsetup works ok (it uses the same library) IMHO this means, that systemd-cryptsetup cannot write to socket which is used to communicate with kernel crypto api (in cryptsetup 1.6.5 this used for ESSIV calculation). Also it probably means, that it *cannot* map any Truecrypt based device... seems nobody is testing it! (See man cryptab - tcrypt parameters)
Created attachment 913918 [details] blkid output
Created attachment 913920 [details] lsblk output
I saw this with both an i686 machine and an x86_64 machine. One is running selinux in permissive mode (because of other issues) and one is running in enforcing mode.
The machines have been upgraded for a while, they aren't recent rawhide installs.
What I see from logs that it fails with your /home only (which is aes-cbc-essiv:sha256 encryption) and that you have enforcing selinux: /dev/md14: UUID="ab1ed7c7-d153-4d85-aff7-b852dfc937f6" TYPE="crypto_LUKS" -> /dev/mapper/luks-ab1ed7c7-d153-4d85-aff7-b852dfc937f6: LABEL="home" UUID="fbf3394c-bd40-479a-b59e-7a34b5b8b82b" TYPE="ext4" Jun 30 16:53:25 cerberus.csd.uwm.edu kernel: audit: type=1404 audit(1404165202.214:2): enforcing=1 old_enforcing=0 auid=4294967295 ses=4294967295 ... Jun 30 16:53:32 cerberus.csd.uwm.edu systemd-cryptsetup[932]: Set cipher aes, mode cbc-essiv:sha256, key size 256 bits for device /dev/disk/by-uuid/ab1ed7c7-d153-4d85-aff7-b852dfc937f6. Jun 30 16:53:32 cerberus.csd.uwm.edu systemd-cryptsetup[932]: Failed to activate: Input/output error Jun 30 16:53:32 cerberus.csd.uwm.edu systemd[1]: systemd-cryptsetup@luks\x2dab1ed7c7\x2dd153\x2d4d85\x2daff7\x2db852dfc937f6.service: main process exited, code=exited, status=1/FAILURE I think this is exactly the problem I reproduced. If it fails with in permissive mode, please can you post correct log?
The two machines involved are in different locations. The one I am physically at is setup in enforcing mode, the one at the other location is in permissive mode. I am in the process of doing an update now. (Today's rawhide plus getting back to the problematic version of cryptsetup). When that finishes I'm going to try a permissive mode boot and if that fails, boot with selinux disabled. (The latter wil trigger a relabel for the following boot which will make the machine unavailable for a while, so I want to do that test at a good time.)
ok, thx. Let me know the results - it is possible that there are two separate issues.
Some good news for me. The reboot with enforcing=0 worked. So it looks like there is just one problem and I don't need to test disabled and then wait for a relabel. I double checked the system at home and it was set to boot in enforcing mode when I did the test there. I have been doing a mix of changing the configuration and manually changing to permissive mode after boot (to work around a sound issue) and fogot what the current status is.
Ok, so we nee to modify selinux policy here to allow systemd-cryptsetup to communicate with kernel socket, see comment#4.
The problem is we see unlabeled_t socket. Milan, did you get it too? Does the socket come from systemd?
It is kernel crypto API interface, this is how it is opened (but for me write fails, not open). fd = socket(AF_ALG, SOCK_SEQPACKET, 0); This is unix socket handled by kernel directly. (You can strace e.g. "cryptsetup benchmark -c aes-xts" to see how libcryptsetup communicates with it.)
I'm confused why this was assigned to the kernel. Bruno clearly states that if he downgrades cryptsetup things work fine in comment #1. So what changed in cryptsetup and/or SELinux that broke this?
I believe the issue is that AF_ALG does not correctly setup the sock struct's SELinux state due to sock_init_data() missing a call to security_sock_graft() when assigning a sock struct to a socket. If I don't get a patch out today I will do so early next week.
Paul, that's fine but my question from my previous comment still stands. What changed to cause an issue here?
Based on the reporter's comment that downgrading cryptsetup correct the problem, I'm guessing that in the new version of cryptsetup they use a different approach or a different interface. I'm not familiar with the cryptsetup code so I can't say for certain. Regardless, there is a kernel bug that needs to be addressed. A kernel bug that appears to date back to at least 2010 when the AF_ALG code was merged.
Cryptsetup (libcryptsetup which is used in systemd-cryptsetup) uses kernel crypto socket interface (for block ciphers) to process Truecrypt header. In version 1.6.5 I added the same logic for LUKS devices. This allows to use cryptsetup for LUKS header management under normal user account. But in reality the only place where we need to use it is ESSIV initialization vector, where we need to encrypt IV before use. (That's why aes-xts-plain64 works). Old way (map temporary dmcrypt device) is just falback if this fails. Unfortunately it allows open the socket but not use it (so it fails with EIO even if running as root). The conclusion is: Truecrypt mapping from systemd-cryptsetup is broken for months the same way but apparently nobody is using it. The LUKS one is new functionality and fails now because it is new feature in version 1.6.5. In any way, the unlabeled socket is a bug and must be fixed otherwise tools cannot use kernel userspace crypto interface properly in enforcing mode.
Thanks Milan. Your explanation clarifies things nicely.
Actually, correcting myself: it should fail even for aes-xts-plain64 (just not in IV but during keyslot decryption). And during boot it is not failing for system disks (later it does). How it is possible that in initramdisk selinux policy does not stop it then?
During boot if the cryptsetup operation happens before the SELinux policy is loaded then I would expect the operation to succeed regardless of the kernel bug.
Is there a simple cryptsetup test that I can do to trigger/reproduce this bug without creating a LUKS partition? I can create a dummy test program if needed, but it would be nice to verify a fix using cryptsetup.
(In reply to Paul Moore from comment #17) > I believe the issue is that AF_ALG does not correctly setup the sock > struct's SELinux state due to sock_init_data() missing a call to > security_sock_graft() when assigning a sock struct to a socket. I'm no longer 100% certain this is the problem, let me look into this a bit more.
Cryptsetup itself works (process is labeled differently, not sure). If you able to run it in the same context as systemd-cryptsetup, then "cryptsetup benchmark -c aes-xts" should be enough to reproduce it. But I am able to reproduce it this way: 1) prepare image # dd if=/dev/zero of=/mnt/tst.img bs=1M count=16 # "echo "password" | cryptsetup luksFormat -q /mnt/tst.img # echo "tst /mnt/tst.img" >> /etc/crypttab 2) enable enforcing mode and and reload systemd crypto mappings # setenforce 1 # systemctl daemon-reload # systemctl restart cryptsetup.target ... it should ask for password through systemd magic and then fail, in journalctl is systemd-cryptsetup[1576]: Failed to activate: Input/output error 3) switch to permissive mode # setenforce 0 # systemctl restart cryptsetup.target And now it should work.
(In reply to Milan Broz from comment #26) > But I am able to reproduce it this way ... Great, thanks.
Or even simpler, in permissive mode you should get these socket avc denials too (except others, it will not work in enforcing). runcon -u system_u -t init_t -r system_r /usr/sbin/cryptsetup benchmark -c aes-xts
Any new info what's wrong with the kernel socket label?
Nothing yet, I'll update the BZ when I have something to report.
Discussed at the 2014-07-09 Fedora 21 alpha blocker review meeting. Accepted as a blocker for Fedora 21 alpha due to violation of the following alpha release criteria [1]: A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility. A system installed without a graphical package set must boot to a state where it is possible to log in through at least one of the default virtual consoles. In all of the above cases, if any system partitions were encrypted as part of the installation, the boot process must prompt for the passphrase(s) and correctly unlock the partition(s) when provided with the correct passphrase(s). [1] https://fedoraproject.org/wiki/Fedora_21_Alpha_Release_Criteria#Expected_installed_system_boot_behavior
(In reply to Milan Broz from comment #28) > Or even simpler, in permissive mode you should get these socket avc denials > too (except others, it will not work in enforcing). > > runcon -u system_u -t init_t -r system_r /usr/sbin/cryptsetup benchmark -c > aes-xts Good news, using the above reproducer the socket created via the socket() syscall appears to be labeled correctly, and looking closely at the code I see no reason why it shouldn't be labeled correctly. Next step is to look at the accept() syscall on AF_ALG sockets and the kcrypto API to make sure we are labeled things correctly there. Looking at the code, I'm pretty sure what we have is wrong, I just need to understand the correct/desired behavior.
(In reply to Paul Moore from comment #32) > Next step is to look at the accept() syscall on AF_ALG sockets and the > kcrypto API to make sure we are labeled things correctly there. Looking at > the code, I'm pretty sure what we have is wrong, I just need to understand > the correct/desired behavior. We need to change the sock_graft() hook such that in the case of AF_ALG the sock security struct inherits the label from the parent socket. I'll have a patch soonish to test.
Created attachment 916950 [details] 09-XXX-selinux-sock_graft_default_fix.patch
Koji scratch build using the patch from this BZ: * http://koji.fedoraproject.org/koji/taskinfo?taskID=7122536
The patch/build above appears to solve the problem on my test system, can someone else give it a quick test to make sure everything is working properly? If everything looks good I'll push this upstream and send it to the stable folks as well.
Yes, test build fixes the problem on my test machine and systemd activates device properly in enforcing mode. So for the kernel patch... Tested-by: Milan Broz <gmazyland> Thanks!
(BTW do you plan to update also older Fedora kernel? I would like to build the cryptsetup update there.)
updated to kernel-3.16.0-0.rc4.git2.1, I still can't boot without selinux=0 on encrypted systems. Is the patch public now?
The patch is currently undergoing some discussion and tweaks, it has not been merged upstream or to any Fedora kernel that I am aware of at this time. Patience :)
I just tested 3.16.0-0.rc4.git2.1.bz1115120.1.fc21.x86_64 and successfully booted in enforcing mode.
Great, thanks Bruno. Hopefully we'll have something upstream soon.
I've included the patch in the SELinux stable-3.16 branch and sent a pull request to the LSM maintainer. As soon as the patch hits Linus' tree I'll update the BZ. * http://git.infradead.org/users/pcmoore/selinux/shortlog/refs/heads/stable-3.16
Since I had to test another patch anyway, I tested this one on top of Linus' 3.16-rc6 plus another patch for a scheduler bug and all three correctly mounted their encrypted devices and booted successfully.
Do you know if the LSM maintainer is planning to submit this for 3.16 or are they planning on waiting for the 3.17 merge window? If they are going to wait for the merge window, we may want to carry this as a Fedora patch to 3.16 for the alpha release. (Even with a 3 week slip we aren't going to be using a 3.17 kernel for alpha.)
It was pushed upstream to James as a fix for 3.16 so I would expect it to be in the next 3.16-rc (it is not present in 3.16-rc6). Regardless, I see no reason why Fedora couldn't carry this patch until it lands in an upstream kernel.
Patch added. Will be in tomorrow's rawhide. Thanks Paul!
I found this got merged into linux next about a week ago. So it looks like it is slotted for 3.17. And that makes some sense, as from the kernel perspective this seems like an enhancement and it is late in the cycle for that. Thinks for getting this into Fedora's 3.16 kernels.
Even though this is closed now, I'd like to state for the record that I tested 3.16.0-0.rc6.git2.2.fc22.i686+PAE (from rawhide nodebug) and things look good. (Booting in enforcing mode with encrypted file systems works.)
Created attachment 921748 [details] Kernel panis screenshot Is this just coincidence or does this patch beaks my BT mouse? My Kernel panics as soon as I try to move my BT mouse :/
Seems related. Paul?
I'll take a look ...
Okay, my best guess/hunch at this point is the patch is running afoul of commit 6230c9b4f8957c8938ee4cf2d03166d3c2dc89de which added some LSM awareness to the BT stack. I need to think about the best way to resolve this, in the meantime, the best solution is probably to revert the selinux_sock_graft() patch from this BZ. Sorry guys.
Thank makes some sense. None of the machines I used for testing have bluetooth, so I wouldn't see the issue.
Same here, I don't use BT on a regular basis and didn't think to check. I really *should* have remembered this, but unfortunately I didn't. In my defense, I can claim that the BT patch is three years old :) Anyway, thanks to vondruch for testing and identifying this.
(In reply to Paul Moore from comment #55) > Anyway, thanks to vondruch for testing and identifying this. Heh, I'm not that happy to find this bug, since I cannot use my mouse now ;)
Reverting breaks LUKS partitions though. Instead, can we tweak the patch to only do the new selinux_sock_graft stuff on the AF_ALG address family? Or does BT also use that?
We might be able to hack something up quick and dirty if it is really needed, but it wouldn't be anything I'm going to accept upstream for SELinux. The correct fix is probably to do for AF_ALG what we did for AF_BLUETOOTH, but unfortunately, I'm probably not going to get a chance to work on this today.
Note that in order to release F21 alpha we need some sort of fix for this, even if it isn't the final fix.
Do we need to ship the latest version of cryptsetup which exercises this bug? Why not simply hold the version of cryptsetup for the time being and once we fix the kernel bug we can update cryptsetup.
I would think that downgrading cryptsetup for f21 would be an acceptable option if push comes to shove. We'd need to hear from the cryptsetup maintainer to see if there are any significant ramifications to that.
As I said above, it is not only LUKS. The whole cryptsetup Truecrypt support (in systemd) is broken and there is no other fix for this other than properly label the socket. If you know the situation around Truecrypt know (development has been stopped) it is kind of very bad timing to not fix this for release.
I don't think there is a release requirement for truecrypt to work. I expect that the problem will be fixed before F21's final release. For the alpha release we only have a few weeks to get things ready and we really need testable images now.
I would say that hacking the AF_ALG in kernel where the problem really lies is more appropriate solution... Fedora already forces users to use selinux and now forces developers to downgrade packages... Hmmm. Whatever, the problem is known, if you do not have resource to fix it in time, just downgrade the package and sweep it under the carpet for now.
The problem in the kernel will get fixed, it is just a matter of time, and if F21 needs to have something working *NOW* then I think downgrading cryptsetup (assuming no major functional regression) is the best option.
The problem is what to do in the short term. We seem to have three bad choices. Bluetooth doesn't work. cyrptsetup doesn't work for anyone in enforcing mode or cryptsetup doesn't work for truecrypt users in enforcing mode. The latter seems to be the best choice right now. The alpha change deadline is in two weeks. The alpha release is in 4, but it needs to be completely frozen about a week before that.
*** Bug 1123993 has been marked as a duplicate of this bug. ***
Created attachment 922040 [details] Properly label AF_ALG socket This should be (I hope) proper fix. Paul, please check it if it is what you meant. I'll send it upstream if it looks correct. The testing kernel build with patch is here http://koji.fedoraproject.org/koji/taskinfo?taskID=7204964 (I just replaced the old patch with new one.) Can anyone test it and verify problems with Bluetooth are gone? Thanks.
I tested that 3.16.0-0.rc7.git0.1.bz1115120.2.fc22.x86_64 still solves the booting with encryoted file systems issue. I can't test bluetooth though.
Nice, such a comfort! I can use my mouse again ;) In other words, my system boots and it does not panic anymore when using my BT mouse. So thanks Milan, the patch seems to fix the issue.
Thanks guys. Patch sent to linux kernel crypto list.
Upstream patch posting: * http://marc.info/?l=selinux&m=140665964115604&w=2
Thanks Milan for the patch and to Bruno and Vít for the testing. I do apologize that I haven't been able to pay as much attention to this lately, but unfortunately there have been other, much nastier bugs which have required my attention. I'll review the patch tomorrow and give my Acked-by for SELinux if all looks well.
I tested 3.16.0-0.rc7.git0.1.bz1115120.2.fc22.i686+PAE on two more systems overnight. Things looked good, but again I had no way to test bluetooth on these machines.
Patch is applied only on AF_ALG (crypto API) module, no changes in any common code, so I am pretty sure it should not cause any bluetooth problems again.
FYI: the patch looks good to me so I added my ACK, CC'd Herbert and asked for this to get pushed this week (it is up to Herbert) so we can make the 3.16 release. Thanks again for everyone's help developing and testing these patches.
Josh, I've got no problems with Milan's latest patch if you want to apply it to the current Fedora kernels.
Applied. Thanks all!
Should be fixed with the rc7-git3 build I just kicked off. Thanks again everyone.
*** Bug 1125389 has been marked as a duplicate of this bug. ***
Upstream: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=5196626dae54fdb5bee89fdd1232b2e03cc60956
I've tested the cryptsetup functionality (but not if the bluetooth regression is still present) using 3.16.0-0.rc7.git3.1 on three machines and not had a problem.