Created attachment 1953921 [details] screenshot console Description of problem: The systemd.volatile=overlay mode is useful for testing purposes. However, Fedora 38 hangs during the startup if you boot the kernel with the systemd.volatile=overlay option. It works, if SELinux is disabled. Version-Release number of selected component (if applicable): selinux-policy-38.8-2.fc38.noarch systemd-253-6.fc38.x86_64 How reproducible: Always on F38. Steps to Reproduce: 1. Install F38 beta. 2. Boot F38. 3. Optional: Install the latest updates (dnf update). 4. Add the "overlay" driver to the initrd: # echo 'add_drivers+=" overlay "' > /etc/dracut.conf.d/01-overlay.conf # dracut -vf --regenerate-all 5. Reboot. 6. Append "systemd.volatile=overlay" to the kernel command line in GRUB, and boot. Actual results: System hangs during boot. The last line shown on the console is: Starting systemd-hostnamed.service - Hostname Service. Expected results: System should boot and overlay mode should work as in previous Fedora versions. Additional info: - It works on Fedora 37 and previous versions. - If you disable SELinux in /etc/sysconfig/selinux, the overlay mode works.
Problem still exists with latest updates: selinux-policy-38.10-1.fc38.noarch systemd-253.2-1.fc38.x86_64
Marc, This seems to be a less used scenario, can you give us a use case example? Do you happen to know what has changed since the fix for bz#2128246 was confirmed working?
> This seems to be a less used scenario, can you give us a use case example? My use case for having a temporary overlay filesystem on root is that I use this feature to test things when I don't want to mess up my system. I can install software or do other things and, after the next reboot, the changes are gone. Another use case I know from a friend: They have some Linux hosts that random people use. It's a kind of public host to access the internet. They boot the machines with the systemd.volatile=overlay option. Once they reboot, the system is back in its previous clean state. > Do you happen to know what has changed since the fix for bz#2128246 was confirmed working? I don't know what has changed. On a laptop with an up-to-date Fedora 37 (this was the version for which I filed bz#2128246), the feature works. On an up-to-date Fedora 38 (fresh install, no upgrade), the system hangs during the boot. Meanwhile, I tried the following workaround (and it works): 1) Download the latest selinux-policy and selinux-policy-targeted packages from Fedora 37: # wget https://ftp.halifax.rwth-aachen.de/fedora/linux/updates/37/Everything/x86_64/Packages/s/selinux-policy-37.19-1.fc37.noarch.rpm # wget https://ftp.halifax.rwth-aachen.de/fedora/linux/updates/37/Everything/x86_64/Packages/s/selinux-policy-targeted-37.19-1.fc37.noarch.rpm 2) Remove smartmontools (it has some selinux dependencies): # dnf remove smartmontools 3) Install the F37 packages on F38: # dnf install selinux-policy*.rpm 4) Reboot and add "systemd.volatile=overlay" to the kernel command line. Result: System boots as expected and the root file system is an overlayfs. From a quick check, the system seems to work as expected. So the problem is in the diff of the F37 and F38 packages. Last time, debugging was easier because the system booted and only systemd-resolved didn't work, so I could provide logs. This time, the system hangs during the boot. Therefore, I'm not sure how I can get any logs. Because of the tmpfs overlay, the log entries that were written during the boot when it hangs are gone as soon as I reboot without systemd.volatile=overlay.
I suppose it is in a state when debug-shell works: boot the system with the volatile options and systemd.unit=debug-shell.service then ctrl-alt-f9 I will try to reproduce and find the details on my own.
Zdenek, I am also affected by this regression. I've tested if with the latest versions of those packages and can confirm that the bug still persists: selinux-policy-38.17-1.fc38 systemd-253.5-1.fc38
The problem still exists on Fedora 38. I retried it today with: selinux-policy-38.27-1.fc38.noarch systemd-253.9-1.fc38.x86_64 Additionally, I tried this on the latest Fedora 39 nightly (2023-09-16) and the system also hangs during boot: selinux-policy-38.28-1.fc39.noarch systemd-254.2-6.fc39.x86_64 Zdenek, what can I check in the debug shell to bring this ticket forward?
Marc, I am sorry for the delay, it went out of my radar for some time. I'd say find out which service is blocked or why the boot process does not continue. Look at processes/services list, journal, audit log if auditd is running. # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts boot You can also try if the following module helps and is sufficient: # cat local_kernel_relabel.cil (allow kernel_t tmp_t (dir (relabelfrom relabelto))) # semodule -i local_kernel_relabel.cil and boot.
An additional observation: I tried some old F38 packages from Koji and, acctually, there were packages in F38 where the overlay mode worked. The last working package was selinux-policy-38.1-1.fc38.noarch.rpm (https://koji.fedoraproject.org/koji/buildinfo?buildID=2091670). In selinux-policy-38.2-1.fc38.noarch.rpm (https://koji.fedoraproject.org/koji/buildinfo?buildID=2097652) and later, the system hangs during boot.
(In reply to Zdenek Pytela from comment #7) > You can also try if the following module helps and is sufficient: > > # cat local_kernel_relabel.cil > (allow kernel_t tmp_t (dir (relabelfrom relabelto))) > # semodule -i local_kernel_relabel.cil > > and boot. Unfortunately, this doesn't help.
Created attachment 1989718 [details] ausearch output in permissive mode I don't know if this helps to debug the problem, but I switched to permissive mode, booted into volatile=overlay mode (system boots), and ran # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts boot I attached the output of the command.
(In reply to Marc Muehlfeld from comment #8) > An additional observation: > > I tried some old F38 packages from Koji and, acctually, there were packages > in F38 where the overlay mode worked. > > The last working package was selinux-policy-38.1-1.fc38.noarch.rpm > (https://koji.fedoraproject.org/koji/buildinfo?buildID=2091670). > > In selinux-policy-38.2-1.fc38.noarch.rpm > (https://koji.fedoraproject.org/koji/buildinfo?buildID=2097652) and later, > the system hangs during boot. In this policy version major changes were introduced with higher impact than expected. (In reply to Marc Muehlfeld from comment #10) > Created attachment 1989718 [details] > ausearch output in permissive mode > > I don't know if this helps to debug the problem, but I switched to > permissive mode, booted into volatile=overlay mode (system boots), and ran > # ausearch -i -m avc,user_avc,selinux_err,user_selinux_err -ts boot > > I attached the output of the command. Yes, this helps, thank you. audit2allow shortens the AVCs to: allow kernel_t tmp_t:dir { relabelfrom relabelto }; allow kernel_t tmpfs_t:chr_file { create link rename unlink }; allow kernel_t var_log_t:chr_file unlink; allow kernel_t xserver_log_t:chr_file unlink; where the chr_file class confuses me, unfortunately even with full auditing enabled there are no additional entries. Can you try to boot in enforcing with the local module enhanced like this? (allow kernel_t tmp_t (dir (relabelfrom relabelto))) (allow kernel_t tmpfs_t (chr_file (create link rename unlink))) (allow kernel_t var_log_t (chr_file (unlink))) (allow kernel_t xserver_log_t (chr_file (unlink))) (allow kernel_t xdm_var_lib_t (chr_file (unlink))) It can help you at least until I get to understanding the problem.
Thanks Zdenek, your workaround works (tested on F38 and F39 Beta). 1) Create a file "workaround.cil" with the following content: (allow kernel_t tmp_t (dir (relabelfrom relabelto))) (allow kernel_t tmpfs_t (chr_file (create link rename unlink))) (allow kernel_t var_log_t (chr_file (unlink))) (allow kernel_t xserver_log_t (chr_file (unlink))) (allow kernel_t xdm_var_lib_t (chr_file (unlink))) 2) Run: # semodule -i workaround.cil 3) Reboot into systemd.volatile=overlay mode.
Quick update: With the workaround from #c12, the system is still not fully functional in volatile=overlay mode with SELinux in enforcing. For example, installing packages is not possible: # dnf install libusb-compat-0.1 ... Running transaction Preparing : 1/1 Installing : libusb-compat-0.1-0.1.8-5.fc39.x86_64 1/1 error: lsetfilecon: (33 /usr/lib/.build-id, system_u:object_r:lib_t:s0) Permission denied error: Plugin selinux: hook fsm_file_prepare failed Error unpacking rpm package libusb-compat-0.1-0.1.8-5.fc39.x86_64 Verifying : libusb-compat-0.1-0.1.8-5.fc39.x86_64 1/1 Failed: libusb-compat-0.1-0.1.8-5.fc39.x86_64 Error: Transaction failed
(In reply to Marc Muehlfeld from comment #13) > Quick update: With the workaround from #c12, the system is still not fully > functional in volatile=overlay mode with SELinux in enforcing. For example, > installing packages is not possible: > > # dnf install libusb-compat-0.1 > ... > Running transaction > Preparing : > 1/1 > > Installing : libusb-compat-0.1-0.1.8-5.fc39.x86_64 > 1/1 > > error: lsetfilecon: (33 /usr/lib/.build-id, system_u:object_r:lib_t:s0) > Permission denied > error: Plugin selinux: hook fsm_file_prepare failed > > Error unpacking rpm package libusb-compat-0.1-0.1.8-5.fc39.x86_64 > Verifying : libusb-compat-0.1-0.1.8-5.fc39.x86_64 > 1/1 > > > Failed: > libusb-compat-0.1-0.1.8-5.fc39.x86_64 > > Error: Transaction failed Are you sure this is caused by applying the workaround?
I guess this is not caused by the workaround, but the workaround is not sufficient to have a fully working system as before the selinux-policy changes. After applying the workaround: * If I boot the system normal, I can install packages * If I boot in volatile=overlay mode, installing packages fail
OK. Are there any AVC denials?
The denials depend on the packages. Some random examples: * dhcp-server: type=ADD_GROUP msg=audit(1696849908.210:222): pid=1071 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-group id=177 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" ID="dhcpd" type=GRP_MGMT msg=audit(1696849908.211:223): pid=1071 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-shadow-group id=177 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" ID="dhcpd" type=ADD_USER msg=audit(1696849908.242:224): pid=1078 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 msg='op=add-user acct="dhcpd" exe="/usr/sbin/useradd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" type=AVC msg=audit(1696849908.270:225): avc: denied { relabelfrom } for pid=1065 comm="dnf" name="dhcp" dev="tmpfs" ino=379 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:dhcp_etc_t:s0 tclass=dir permissive=0 * usbguard: type=AVC msg=audit(1696849972.419:226): avc: denied { relabelfrom } for pid=1094 comm="dnf" name=".build-id" dev="tmpfs" ino=393 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1696849972.421:227): avc: denied { relabelfrom } for pid=1094 comm="dnf" name="6c" dev="tmpfs" ino=394 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:lib_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1696849972.431:228): avc: denied { relabelfrom } for pid=1094 comm="dnf" name="contrib" dev="tmpfs" ino=402 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:object_r:usr_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1696849972.433:229): avc: denied { relabelfrom } for pid=1094 comm="dnf" name="usbguard" dev="tmpfs" ino=403 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:object_r:etc_t:s0 tclass=dir permissive=0 * postfix: type=AVC msg=audit(1696850145.553:230): avc: denied { relabelfrom } for pid=1120 comm="dnf" name="make-dummy-cert;6523e0e1" dev="tmpfs" ino=412 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:object_r:bin_t:s0 tclass=file permissive=0 type=ADD_GROUP msg=audit(1696850145.570:231): pid=1127 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-group id=90 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" ID="postdrop" type=GRP_MGMT msg=audit(1696850145.572:232): pid=1127 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-shadow-group id=90 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" ID="postdrop" type=ADD_GROUP msg=audit(1696850145.596:233): pid=1131 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-group id=89 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" ID="postfix" type=GRP_MGMT msg=audit(1696850145.597:234): pid=1131 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-shadow-group id=89 exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" ID="postfix" type=ADD_GROUP msg=audit(1696850145.650:235): pid=1135 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 msg='op=add-group acct="mail" exe="/usr/sbin/groupadd" hostname=? addr=? terminal=? res=failed'UID="root" AUID="root" type=ADD_USER msg=audit(1696850145.659:236): pid=1136 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 msg='op=add-user acct="postfix" exe="/usr/sbin/useradd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" type=USER_MGMT msg=audit(1696850145.659:237): pid=1136 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 msg='op=add-user-to-group grp="mail" acct="postfix" exe="/usr/sbin/useradd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" type=USER_MGMT msg=audit(1696850145.660:238): pid=1136 uid=0 auid=0 ses=2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 msg='op=add-to-shadow-group grp="mail" acct="postfix" exe="/usr/sbin/useradd" hostname=? addr=? terminal=? res=success'UID="root" AUID="root" type=AVC msg=audit(1696850145.707:239): avc: denied { relabelfrom } for pid=1120 comm="dnf" name="smtp.postfix;6523e0e1" dev="tmpfs" ino=432 scontext=system_u:system_r:kernel_t:s0 tcontext=unconfined_u:object_r:etc_t:s0 tclass=file permissive=0
I believe this should all be fixed with https://github.com/fedora-selinux/selinux-policy/commit/2f172413e104c7ff5ad9799c2030464e29422d53, though that change isn't any build yet... Could you please test with this Packit build from the pull request? https://dashboard.packit.dev/results/copr-builds/1057573
(In reply to Ondrej Mosnáček from comment #18) > I believe this should all be fixed with > https://github.com/fedora-selinux/selinux-policy/commit/ > 2f172413e104c7ff5ad9799c2030464e29422d53, though that change isn't any build > yet... > > Could you please test with this Packit build from the pull request? > https://dashboard.packit.dev/results/copr-builds/1057573 This looks very promising. I tried it on F38, and the system boots with volatile=overlay and package installation also works.
Marc, The PR has been merged in rawhide some time ago, can you confirm if the fix was sufficient?
Zdenek, I use the selinux-policy and selinux-policy-targeted RPMs from https://download.copr.fedorainfracloud.org/results/packit/fedora-selinux-selinux-policy-1893/fedora-39-x86_64/06499891-selinux-policy/ since two months and everything works as expected and I haven't encountered any side effects. Good work! If you want, I can test it again with the latest version if you add a job to Koji.
Marc, it seems most of the related changes are actually in F38, not only in rawhide as I thought, so closing this bz. Thank you for your cooperation, issues like this one are not really tested.