Description of problem: Silverblue 31 can't fstrim some partitions of SSD hard drive Version-Release number of selected component (if applicable): [junjie@xps ~]$ fstrim --version fstrim from util-linux 2.34 [junjie@xps ~]$ rpm-ostree status State: idle AutomaticUpdates: disabled Deployments: ● ostree://fedora:fedora/31/x86_64/silverblue Version: 31.20200210.0 (2020-02-10T00:40:47Z) BaseCommit: 5d20cf79b4efde70a873d815015554f169f58085576aabcf4fe34c1ea54edf4e GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4 LocalPackages: google-chrome-stable-80.0.3987.87-1.x86_64 ostree://fedora:fedora/31/x86_64/silverblue Version: 31.20200209.0 (2020-02-09T00:44:42Z) BaseCommit: 1d84699a832851471d71804bb2933fe2241d3e341e0d34ca360543e10d8d2fe3 GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4 LocalPackages: google-chrome-stable-80.0.3987.87-1.x86_64 How reproducible: [junjie@xps ~]$ sudo fstrim -av /boot/efi: 569.5 MiB (597159936 bytes) trimmed on /dev/sda1 /boot: 0 B (0 bytes) trimmed on /dev/sda2 [junjie@xps ~]$ sudo fstrim / fstrim: /: the discard operation is not supported [junjie@xps ~]$ sudo fstrim /home fstrim: /home: the discard operation is not supported [junjie@xps ~]$ sudo fstrim /var/home fstrim: /var/home: the discard operation is not supported Steps to Reproduce: 1.sudo fstrim / 2.sudo fstrim /home 3.sudo fstrim -av Actual results: fstrim is only available in /boot and /boot/efi Expected results: fstrim works on all partitions, not just /boot and /boot/efi Additional info: I installed Silverblue 31 using the default installation option without modifying the file system. [junjie@xps ~]$ df -Th Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 3.7G 0 3.7G 0% /dev tmpfs tmpfs 3.7G 65M 3.7G 2% /dev/shm tmpfs tmpfs 3.7G 1.9M 3.7G 1% /run /dev/mapper/fedora-root ext4 69G 21G 44G 33% /sysroot tmpfs tmpfs 3.7G 100K 3.7G 1% /tmp /dev/sda2 ext4 976M 141M 769M 16% /boot /dev/sda1 vfat 599M 30M 570M 5% /boot/efi /dev/mapper/fedora-home ext4 40G 33G 4.2G 89% /var/home tmpfs tmpfs 756M 46M 711M 6% /run/user/1000 [junjie@xps ~]$ mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3834560k,nr_inodes=958640,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) /dev/mapper/fedora-root on /sysroot type ext4 (rw,relatime,seclabel) /dev/mapper/fedora-root on / type ext4 (rw,relatime,seclabel) /dev/mapper/fedora-root on /usr type ext4 (ro,relatime,seclabel) selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=7584) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) /dev/mapper/fedora-root on /var type ext4 (rw,relatime,seclabel) /dev/sda2 on /boot type ext4 (rw,relatime,seclabel) /dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro) /dev/mapper/fedora-home on /var/home type ext4 (rw,relatime,seclabel) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=773424k,mode=700,uid=1000,gid=1000) gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000) /dev/fuse on /run/user/1000/doc type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
Are you using disk encryption? I came across a similar problem on my machines. It seems that somehow LUKS (dm-crypt) in Silverblue is not properly configured to pass discard commands. If I select disk encryption in Anaconda when installing Silverblue, fstrim does not work on the LUKS partition. Without disk encryption, TRIM works just fine. I have tested Silverblue 31 and Silverblue 32, the problem exists on both of them (only when using disk encryption). The problem is not present in regular Fedora Workstation 31 and 32. Output of lsblk on my Fedora Silverblue 31 installation signalling that even though the storage device supports TRIM, it is blocked by LUKS (Silverblue 32 gives the same output): [david@localhost ~]$ lsblk -D NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO sda 0 4K 1G 0 ├─sda1 0 4K 1G 0 ├─sda2 0 4K 1G 0 └─sda3 0 4K 1G 0 └─luks-31a68890-6d8b-42fc-a80f-2262577c1178 0 0B 0B 0 ├─fedora-root 0 0B 0B 0 ├─fedora-swap 0 0B 0B 0 └─fedora-home 0 0B 0B 0 sr0 0 0B 0B 0
I found a way to solve my problem with fstrim not working on LUKS on Silverblue. Comments in an old bug report #901888 provide some useful information. The problem seems to be caused by an omission of /etc/crypttab file in the initramfs image. This file is present in initramfs of Fedora Workstation (can be listed by the lsinitrd command), so I tried to include it in the initramfs of my Silverblue too: [david@localhost ~]$ rpm-ostree initramfs --enable --arg=-I --arg=/etc/crypttab After a reboot, fstrim works! [david@localhost ~]$ lsblk -D NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO sda 0 4K 1G 0 ├─sda1 0 4K 1G 0 ├─sda2 0 4K 1G 0 └─sda3 0 4K 1G 0 └─luks-31a68890-6d8b-42fc-a80f-2262577c1178 0 4K 1G 0 ├─fedora-root 0 4K 1G 0 ├─fedora-swap 0 4K 1G 0 └─fedora-home 0 4K 1G 0 sr0 0 0B 0B 0
(In reply to David Sebek from comment #1) > Are you using disk encryption? I came across a similar problem on my > machines. It seems that somehow LUKS (dm-crypt) in Silverblue is not > properly configured to pass discard commands. If I select disk encryption in > Anaconda when installing Silverblue, fstrim does not work on the LUKS > partition. Without disk encryption, TRIM works just fine. I have tested > Silverblue 31 and Silverblue 32, the problem exists on both of them (only > when using disk encryption). The problem is not present in regular Fedora > Workstation 31 and 32. > > Output of lsblk on my Fedora Silverblue 31 installation signalling that even > though the storage device supports TRIM, it is blocked by LUKS (Silverblue > 32 gives the same output): > [david@localhost ~]$ lsblk -D > NAME DISC-ALN DISC-GRAN DISC-MAX > DISC-ZERO > sda 0 4K 1G > 0 > ├─sda1 0 4K 1G > 0 > ├─sda2 0 4K 1G > 0 > └─sda3 0 4K 1G > 0 > └─luks-31a68890-6d8b-42fc-a80f-2262577c1178 > 0 0B 0B > 0 > ├─fedora-root 0 0B 0B > 0 > ├─fedora-swap 0 0B 0B > 0 > └─fedora-home 0 0B 0B > 0 > sr0 0 0B 0B > 0 (In reply to David Sebek from comment #2) > I found a way to solve my problem with fstrim not working on LUKS on > Silverblue. Comments in an old bug report #901888 provide some useful > information. The problem seems to be caused by an omission of /etc/crypttab > file in the initramfs image. This file is present in initramfs of Fedora > Workstation (can be listed by the lsinitrd command), so I tried to include > it in the initramfs of my Silverblue too: > [david@localhost ~]$ rpm-ostree initramfs --enable --arg=-I > --arg=/etc/crypttab > > After a reboot, fstrim works! > [david@localhost ~]$ lsblk -D > NAME DISC-ALN DISC-GRAN DISC-MAX > DISC-ZERO > sda 0 4K 1G > 0 > ├─sda1 0 4K 1G > 0 > ├─sda2 0 4K 1G > 0 > └─sda3 0 4K 1G > 0 > └─luks-31a68890-6d8b-42fc-a80f-2262577c1178 > 0 4K 1G > 0 > ├─fedora-root 0 4K 1G > 0 > ├─fedora-swap 0 4K 1G > 0 > └─fedora-home 0 4K 1G > 0 > sr0 0 0B 0B > 0 Yes, I used luks to encrypt my hard drive. Thank you very much for the solution! :)
You can just add `rd.luks.options=discard` to the kargs. That way rpm-ostree doesn't have to rebuild the initramfs on each new deployment.
In my case, I also solved it by replicating the content of /etc/crypttab to the kargs (instead of rebuilding the initramfs): rpm-ostree kargs --append rd.luks.options=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx=discard Is there any reason why it is not done automatically during the installation, or is this a bug?
This isn't really an rpm-ostree bug. The LUKS mount needs to have the discard option enabled, either via kargs, or by editing crypttab and rebuilding the initramfs (I suggest the former). Re-assigning to anaconda.
This message is a reminder that Fedora 32 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '32'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 32 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
At a glance, it seems that the root cause is the same as bug 1890085 - a config file is missing in initramfs. The reason why it is missing is that Anaconda does not regenerate initramfs for rpm-ostree payloads, and that is because they are signed. This problem can be solved the same way, if it is still present.
This problem can still be reproduced in fedora 34.
(In reply to Jonathan Lebon from comment #6) > This isn't really an rpm-ostree bug. The LUKS mount needs to have the > discard option enabled, either via kargs, or by editing crypttab and > rebuilding the initramfs (I suggest the former). > > Re-assigning to anaconda. I just installed Silverblue 35. I noticed "discard" was the last item on /etc/crypttab, however nothing in the boot options /boot/loader/entries/ostree-1-fedora.conf, /boot/loader/entries/ostree-2-fedora.confetc Seeing as it's installed now, what is the way to fix this outside of Anaconda?
(In reply to David Sebek from comment #5) > In my case, I also solved it by replicating the content of /etc/crypttab to > the kargs (instead of rebuilding the initramfs): > rpm-ostree kargs --append > rd.luks.options=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx=discard > > Is there any reason why it is not done automatically during the > installation, or is this a bug? My bad. I missed this post.
This message is a reminder that Fedora Linux 34 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 34 on 2022-06-07. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '34'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 34 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
This message is a reminder that Fedora Linux 36 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '36'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 36 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Hello, Just a quick note to report that this issue is still persisting in Fedora 40. Regards, Vincent.
This message is a reminder that Fedora Linux 38 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 38 on 2024-05-21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '38'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 38 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
This bur is still valid, for Fedora 40. @QA please bump to F40. @ Upstream bug: https://github.com/fedora-silverblue/issue-tracker/issues/31 Additional information on the topic: This tracks the work to enable encryption/integrity by default in Fedora Silverblue (an related rpm-ostree variants) https://github.com/fedora-silverblue/issue-tracker/issues/447
*** Bug 2349783 has been marked as a duplicate of this bug. ***
This issue is pretty nasty time bomb in all the ostree based systems. Every Fedora user who has - LUKS partition - SSD drive - ostree based system won't have trimming working on LUKS partition. Which at the end will result in unresponsive system with kernel errors somewhere in the future. This should be documented and resolved ASAP. If possible even on running systems. Raising priority of this issue.
Hi Vojto, seems that this is about how the LUKS partition is created. The workaround for me was: sudo cryptsetup --allow-discards --persistent refresh /dev/mapper/luks* Could this be enabled by default from blivet?
Yes, we can set the allow discard flag from blivet during installation (we are already setting the "discard" option in crypttab by default so this won't be a change in default behaviour), but I really think that this should be fixed somewhere else and all distributions should use the "discard" option in crypttab if possible. But the change in blivet would be relatively simple, so if this is the best or only way to fix this for Silverblue I am not against. Feel free to reassign this bug to python-blivet.
Thanks a lot for your info Vojto. Currently, I'm trying to find out if we can't update systemd-cryptsetup-generator to cover also existing installations. So I'll leave it on `anaconda' right now but yeah it will get reassigned.
rewording the description to be clearer, and bumping version as 40 will go EOL relatively soon (but we should fix on all branches if possible). For blocker purposes: remember we do have release-blocking ostree-based editions, IoT and CoreOS.
this may not affect CoreOS, I guess, as it doesn't deploy via anaconda/blivet. But the anaconda-based IoT installer is release-blocking.
This one strongly relates to migrating away from /etc/fstab to kargs for the root mount, see https://github.com/containers/bootc/issues/971 > but I really think that this should be fixed somewhere else and all distributions should use the "discard" option in crypttab if possible. Why not switch to having this configured in the LUKS metadata everywhere?
> this may not affect CoreOS, Yes, I was curious so I looked at Ignition's code here: https://github.com/coreos/ignition/blob/50a4776e9823355b2055c467fa20e67f7f4d165b/internal/exec/stages/disks/luks.go#L526-L533 And indeed it's persistently writing it into the LUKS metadata.
Hi Colin, > This one strongly relates to migrating away from /etc/fstab to kargs for the root mount, see https://github.com/containers/bootc/issues/971 I don't think it is good idea to migrate everything to kargs. The issue is that kargs is limited and we are trying to rather get out of this because there are situations were we hitting the limit already. I'm rather convinced to do the change in blivet during the LUKS creation. One more question, are bootc systems also affected. Is this affecting Image Mode on CentOS Stream, RHEL.
> Why not switch to having this configured in the LUKS metadata everywhere? For that reason, I'm asking systemd team if they are willing to change systemd generator.
I've read through the links and I see your point. We are talking only about root not whole fstab. So, if I understand it correctly, this could be fixed by bug 2332319, am I correct?
I've discussed this issue today with systemd developer and it looks they should be able to enable this when unlocking the LUKS device, however, they are not aware of hardware during that time because this runs before udev. So, the question is if we can enable this feature just everywhere nevertheless the HW. I need to reach out LUKS developer to find that out.
LUKS2 devel here: I do not think it's a good idea to change _every_ device persistent flags settings upon activation after boot (that's what would change in systemd-cryptsetup-generator mean, right?). The discards flag have implications wrt to encryption of data at rest. For freshly initialized LUKS2 devices during installation it's ok. And Fedora communicated the change publicly. But changing all existing devices _without_ user's awareness is not a good idea. Whatever you do, please make sure it does not affect storage devices settings for people who do _not_ want such change. It's worse that by making it in systemd-cryptsetup-generator, the change would be pretty much hidden to most. Just my 2 cents.
Thanks a lot for your input. I'm switching this bug to the blivet then as we can store this information to LUKS2 metadata without adding stuff on kargs which is more fragile and problematic in general.
> But changing all existing devices _without_ user's awareness is not a good idea. I'd say it more strongly, it would need to be classified as a CVE to change existing systems unilaterally, especially those not created via Anaconda! So yes, must only do this the first time the device is initialized in Anaconda - which is what Ignition is doing (Ignition always runs at most once). > One more question, are bootc systems also affected. Is this affecting Image Mode on CentOS Stream, RHEL. Almost certainly yes.
upstream PR: https://github.com/storaged-project/blivet/pull/1347
Discussed during the 2025-03-10 blocker review meeting [1]: !agreed 1801539 - Rejected as a Final Blocker - Silverblue is not a blocking deliverable, so we can't consider it here. IoT should be affected by this in theory, but in practice it seems unlikely that full disk encryption is used at scale there, by our estimate. Also, many SD cards (used in IoT) might not even support trim. Additionally, this problem has been around for 5 years already, further weakening the blocker proposal. We will document this as a CommonBug instead. [1] https://meetbot.fedoraproject.org/blocker-review_matrix_fedoraproject-org/2025-03-10/f42-blocker-review.2025-03-10-16.01.log.html
FEDORA-2025-d9c862cb8d (libblockdev-3.3.0-2.fc42 and python-blivet-3.12.0-2.fc42) has been submitted as an update to Fedora 42. https://bodhi.fedoraproject.org/updates/FEDORA-2025-d9c862cb8d
FEDORA-2025-d9c862cb8d has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-d9c862cb8d` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-d9c862cb8d See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2025-d9c862cb8d (libblockdev-3.3.0-2.fc42 and python-blivet-3.12.0-2.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.