User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0 Build Identifier: Yesterday, I updated my PC from fedora 30 to 31. The update turned out to be problematic, and now I could only boot into the 'emergency shell'. Fedora installation disk layout is like this: All partitions are LUKS encrypted, root (/) is (directly) on LVM on a SSD. /home and /opt are stratis managed (HDDs partition SSD cached). Investigations showed the following: boot into 'emergency shell' is because /home could not be mounted. However, I have problems to recover stratis both from the 'emergency shell' and from the fedora 31 live cd. In 'emergency shell' stratisd warns that there is no DBus. Hence the stratis cli is on no use. I have not found a way to successfully start DBus in the 'emergency shell'. From the live cd, I could install stratisd and stratis-cli. But it is not possible to start stratisd with systemd (some IO Error, permission denied). In chroot from live cd to my installed root (/) it is not possible to start stratisd because systemd does not allow this in chroot. So, has somebody an idea what to do to recover the system? Reproducible: Always Steps to Reproduce: n/a, as this is a question on recover from desaster
Created attachment 1631433 [details] sosreport after boot/emergency shell
Created attachment 1631434 [details] sosreport after network (dhclient) and restart stratisd in emergency shell
Observations: * ls /dev/mapper control fedora-00 fedora-root fedora-thinpool fedora-thinpool_tdata fedora-thinpool_tmeta luks-8cfdfb51-cdfa-401a-9815-d3be9a527942 luks-stratis-hdd-vg luks-stratis-ssd-vg * mount -a mount: /home: UUID=17155... not found mount: /opt: UUID=fb19a... not found * after boot stratisd is up, trying to restart is leads to IO Error: ● stratisd.service - A daemon that manages a pool of block devices to create flexible file systems Loaded: loaded (/usr/lib/systemd/system/stratisd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2019-11-01 11:19:56 CET; 4min 23s ago Docs: man:stratisd(8) Process: 2585 ExecStart=/usr/libexec/stratisd --debug (code=exited, status=1/FAILURE) Main PID: 2585 (code=exited, status=1/FAILURE) CPU: 4ms Nov 01 11:19:56 blacksnapper systemd[1]: Started A daemon that manages a pool of block devices to create flexible file systems. Nov 01 11:19:56 blacksnapper stratisd[2585]: DEBUG libstratis::stratis::buff_log: BuffLogger: pass_through: true hold time: none Nov 01 11:19:56 blacksnapper stratisd[2585]: INFO stratisd: Using StratEngine Nov 01 11:19:56 blacksnapper stratisd[2585]: IO error: Permission denied (os error 13) Nov 01 11:19:56 blacksnapper systemd[1]: stratisd.service: Main process exited, code=exited, status=1/FAILURE Nov 01 11:19:56 blacksnapper systemd[1]: stratisd.service: Failed with result 'exit-code'.
I've commented out the /opt and /home stuff in /etc/fstab. Hence I'm able to use the system again (but as have no access to my real /home, this is _not_ a real solution). After login, I restarted DBus (as it reported an error) and than I restarted stratisd. As before there is a IO Error and I found the following in /var/log/message: Oct 30 20:01:38 blacksnapper audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:syst em_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res= success' Oct 30 20:01:38 blacksnapper kernel: audit: type=1131 audit(1572462097.999:117): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Oct 30 20:01:38 blacksnapper systemd[1]: Started A daemon that manages a pool of block devices to create flexible file systems. Oct 30 20:01:38 blacksnapper audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Oct 30 20:01:38 blacksnapper kernel: audit: type=1130 audit(1572462098.000:118): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Oct 30 20:01:38 blacksnapper stratisd[23157]: DEBUG libstratis::stratis::buff_log: BuffLogger: pass_through: true hold time: none Oct 30 20:01:38 blacksnapper stratisd[23157]: INFO stratisd: Using StratEngine Oct 30 20:01:38 blacksnapper audit[23157]: AVC avc: denied { read } for pid=23157 comm="stratisd" name="dm-6" dev="devtmpfs" ino=19373 scontext=system_u:system_r:stratisd_t:s0 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0 Oct 30 20:01:38 blacksnapper stratisd[23157]: IO error: Permission denied (os error 13) Oct 30 20:01:38 blacksnapper systemd[1]: stratisd.service: Main process exited, code=exited, status=1/FAILURE Oct 30 20:01:38 blacksnapper systemd[1]: stratisd.service: Failed with result 'exit-code'. Oct 30 20:01:38 blacksnapper audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Oct 30 20:01:38 blacksnapper kernel: audit: type=1400 audit(1572462098.005:119): avc: denied { read } for pid=23157 comm="stratisd" name="dm-6" dev="devtmpfs" ino=19373 scontext=system_u:system_r:stratisd_t:s0 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0 Oct 30 20:01:38 blacksnapper kernel: audit: type=1131 audit(1572462098.005:120): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' So it looks like a AVC/SELinux (?) problem: stratisd is not allowed to read /dev/dm-6 on restart. Is this a fedora bug?
Opened a SELinux Policy bug: https://bugzilla.redhat.com/show_bug.cgi?id=1767773
Probably partly upstream as well as packaging, taking.
Probably related: https://bugzilla.redhat.com/show_bug.cgi?id=1755396.
From comment 3 there is an "os error 13" reported "Nov 01 11:19:56 blacksnapper stratisd[2585]: IO error: Permission denied (os error 13)" We've seen a similar error in issues 853. https://github.com/stratis-storage/stratisd/issues/853 I believe there was a fix to the lock file location for stratis recently because of a different boot issue.
Mysteriously, the issue has been handle with https://bugzilla.redhat.com/show_bug.cgi?id=1755396 (which is closed) but is still there on my system.
Do you still see a problem with the selinux-policy-3.14.4-40 package installed? Reference: BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1755396 Comment 6 "Issue is fixed in: # rpm -q selinux-policy selinux-policy-3.14.4-40.fc31.noarch # sesearch -A -s stratisd_t -t fixed_disk_device_t -c blk_file allow stratisd_t fixed_disk_device_t:blk_file getattr; You can install it via: # sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2019-aec8f7ab50"
Nope. Not fixed in selinux-policy-3.14.4-43.fc31.noarch. See https://bugzilla.redhat.com/show_bug.cgi?id=1755396#c21 for details.
This issue has been solved with https://bugzilla.redhat.com/show_bug.cgi?id=1794645 *** This bug has been marked as a duplicate of bug 1794645 ***