Bug 1767743 - recover from stratis failure (update to fedora 31)
Summary: recover from stratis failure (update to fedora 31)
Keywords:
Status: CLOSED DUPLICATE of bug 1794645
Alias: None
Product: Fedora
Classification: Fedora
Component: stratisd
Version: 31
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Dennis Keefe
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On: 1794645
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-01 09:52 UTC by aannoaanno
Modified: 2020-02-14 15:12 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-02-14 15:12:42 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
sosreport after boot/emergency shell (11.56 MB, application/x-xz)
2019-11-01 10:44 UTC, aannoaanno
no flags Details
sosreport after network (dhclient) and restart stratisd in emergency shell (11.59 MB, application/x-xz)
2019-11-01 10:46 UTC, aannoaanno
no flags Details

Description aannoaanno 2019-11-01 09:52:29 UTC
User-Agent:       Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0
Build Identifier: 

Yesterday, I updated my PC from fedora 30 to 31. The update turned out to be problematic, and now I could only boot into the 'emergency shell'.

Fedora installation disk layout is like this: All partitions are LUKS encrypted, root (/) is (directly) on LVM on a SSD. /home and /opt are stratis managed (HDDs partition SSD cached).

Investigations showed the following: boot into 'emergency shell' is because /home could not be mounted. However, I have problems to recover stratis both from the 'emergency shell' and from the fedora 31 live cd.

In 'emergency shell' stratisd warns that there is no DBus. Hence the stratis cli is on no use. I have not found a way to successfully start DBus in the 'emergency shell'.

From the live cd, I could install stratisd and stratis-cli. But it is not possible to start stratisd with systemd (some IO Error, permission denied).

In chroot from live cd to my installed root (/) it is not possible to start stratisd because systemd does not allow this in chroot.

So, has somebody an idea what to do to recover the system?

Reproducible: Always

Steps to Reproduce:
n/a, as this is a question on recover from desaster

Comment 1 aannoaanno 2019-11-01 10:44:43 UTC
Created attachment 1631433 [details]
sosreport after boot/emergency shell

Comment 2 aannoaanno 2019-11-01 10:46:55 UTC
Created attachment 1631434 [details]
sosreport after network (dhclient) and restart stratisd in emergency shell

Comment 3 aannoaanno 2019-11-01 10:56:40 UTC
Observations:

* ls /dev/mapper
control
fedora-00
fedora-root
fedora-thinpool
fedora-thinpool_tdata
fedora-thinpool_tmeta
luks-8cfdfb51-cdfa-401a-9815-d3be9a527942
luks-stratis-hdd-vg
luks-stratis-ssd-vg

* mount -a 
mount: /home: UUID=17155... not found
mount: /opt: UUID=fb19a... not found

* after boot stratisd is up, trying to restart is leads to IO Error:
● stratisd.service - A daemon that manages a pool of block devices to create flexible file systems
   Loaded: loaded (/usr/lib/systemd/system/stratisd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-11-01 11:19:56 CET; 4min 23s ago
     Docs: man:stratisd(8)
  Process: 2585 ExecStart=/usr/libexec/stratisd --debug (code=exited, status=1/FAILURE)
 Main PID: 2585 (code=exited, status=1/FAILURE)
      CPU: 4ms

Nov 01 11:19:56 blacksnapper systemd[1]: Started A daemon that manages a pool of block devices to create flexible file systems.
Nov 01 11:19:56 blacksnapper stratisd[2585]: DEBUG libstratis::stratis::buff_log: BuffLogger: pass_through: true hold time: none
Nov 01 11:19:56 blacksnapper stratisd[2585]:  INFO stratisd: Using StratEngine
Nov 01 11:19:56 blacksnapper stratisd[2585]: IO error: Permission denied (os error 13)
Nov 01 11:19:56 blacksnapper systemd[1]: stratisd.service: Main process exited, code=exited, status=1/FAILURE
Nov 01 11:19:56 blacksnapper systemd[1]: stratisd.service: Failed with result 'exit-code'.

Comment 4 aannoaanno 2019-11-01 11:35:54 UTC
I've commented out the /opt and /home stuff in /etc/fstab. Hence I'm able to use the system again (but as have no access to my real /home, this is _not_ a real solution).

After login, I restarted DBus (as it reported an error) and than I restarted stratisd. As before there is a IO Error and I found the following in /var/log/message:

Oct 30 20:01:38 blacksnapper audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:syst
em_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=
success'
Oct 30 20:01:38 blacksnapper kernel: audit: type=1131 audit(1572462097.999:117): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 30 20:01:38 blacksnapper systemd[1]: Started A daemon that manages a pool of block devices to create flexible file systems.
Oct 30 20:01:38 blacksnapper audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 30 20:01:38 blacksnapper kernel: audit: type=1130 audit(1572462098.000:118): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 30 20:01:38 blacksnapper stratisd[23157]: DEBUG libstratis::stratis::buff_log: BuffLogger: pass_through: true hold time: none
Oct 30 20:01:38 blacksnapper stratisd[23157]: INFO stratisd: Using StratEngine
Oct 30 20:01:38 blacksnapper audit[23157]: AVC avc:  denied  { read } for  pid=23157 comm="stratisd" name="dm-6" dev="devtmpfs" ino=19373 scontext=system_u:system_r:stratisd_t:s0 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
Oct 30 20:01:38 blacksnapper stratisd[23157]: IO error: Permission denied (os error 13)
Oct 30 20:01:38 blacksnapper systemd[1]: stratisd.service: Main process exited, code=exited, status=1/FAILURE
Oct 30 20:01:38 blacksnapper systemd[1]: stratisd.service: Failed with result 'exit-code'.
Oct 30 20:01:38 blacksnapper audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Oct 30 20:01:38 blacksnapper kernel: audit: type=1400 audit(1572462098.005:119): avc:  denied  { read } for  pid=23157 comm="stratisd" name="dm-6" dev="devtmpfs" ino=19373 scontext=system_u:system_r:stratisd_t:s0 tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file permissive=0
Oct 30 20:01:38 blacksnapper kernel: audit: type=1131 audit(1572462098.005:120): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=stratisd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'

So it looks like a AVC/SELinux (?) problem: stratisd is not allowed to read /dev/dm-6 on restart. Is this a fedora bug?

Comment 5 aannoaanno 2019-11-01 11:52:27 UTC
Opened a SELinux Policy bug: https://bugzilla.redhat.com/show_bug.cgi?id=1767773

Comment 6 mulhern 2019-11-01 12:37:01 UTC
Probably partly upstream as well as packaging, taking.

Comment 7 mulhern 2019-11-01 18:30:55 UTC
Probably related: https://bugzilla.redhat.com/show_bug.cgi?id=1755396.

Comment 8 Dennis Keefe 2019-11-02 04:00:33 UTC
From comment 3 there is an "os error 13" reported
"Nov 01 11:19:56 blacksnapper stratisd[2585]: IO error: Permission denied (os error 13)"
We've seen a similar error in issues 853.
https://github.com/stratis-storage/stratisd/issues/853

I believe there was a fix to the lock file location for stratis recently because of a different boot issue.

Comment 9 aannoaanno 2019-12-19 17:26:03 UTC
Mysteriously, the issue has been handle with https://bugzilla.redhat.com/show_bug.cgi?id=1755396 (which is closed) but is still there on my system.

Comment 10 Dennis Keefe 2019-12-19 18:47:51 UTC
Do you still see a problem with the selinux-policy-3.14.4-40 package installed?

Reference: 
BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1755396
Comment 6
"Issue is fixed in:
# rpm -q selinux-policy
selinux-policy-3.14.4-40.fc31.noarch

# sesearch -A -s  stratisd_t -t fixed_disk_device_t -c blk_file 
allow stratisd_t fixed_disk_device_t:blk_file getattr;

You can install it via:
# sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2019-aec8f7ab50"

Comment 11 aannoaanno 2019-12-24 09:39:40 UTC
Nope. Not fixed in selinux-policy-3.14.4-43.fc31.noarch. See https://bugzilla.redhat.com/show_bug.cgi?id=1755396#c21 for details.

Comment 12 aannoaanno 2020-02-14 15:12:42 UTC
This issue has been solved with https://bugzilla.redhat.com/show_bug.cgi?id=1794645

*** This bug has been marked as a duplicate of bug 1794645 ***


Note You need to log in before you can comment on or make changes to this bug.