Description of problem: When the rootfs is reprovisioned, such as when LUKS disk encryption is configured, the default karg "rootflags=prjquota" is dropped. This mount option is required to enable container storage quotas (see https://github.com/coreos/coreos-assembler/pull/303/commits/6103effbd006bb6109467830d6a3e42dd847668d and BZ 1658386) Version-Release number of selected component (if applicable): 4.7.1 (47.83.202103041352-0) How reproducible: 100% Steps to Reproduce: 1. Deploy RHCOS with Tang encryption (https://docs.openshift.com/container-platform/4.7/installing/install_config/installing-customizing.html#installation-special-config-encrypt-disk-tang_installing-customizing) Actual results: rootfs is mounted with 'noquota' option # findmnt /var TARGET SOURCE FSTYPE OPTIONS /var /dev/mapper/root[/ostree/deploy/rhcos/var] xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota Expected results: root is mounted with 'prjquota' option # findmnt /var TARGET SOURCE FSTYPE OPTIONS /var /dev/mapper/root[/ostree/deploy/rhcos/var] xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota Additional info: Benjamin pointed me at https://github.com/coreos/fedora-coreos-config/blob/rhcos-4.7/overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-rootflags.sh Simply reproducer for testing on a single node: { "ignition": { "config": { "merge": [ { "source": "http://myserver/rhcos/test.json" } ] }, "version": "3.2.0" }, "storage": { "filesystems": [ { "device": "/dev/mapper/root", "format": "xfs", "label": "root", "wipeFilesystem": true } ], "luks": [ { "clevis": { "tang": [ { "thumbprint": "my_thumbprint", "url": "http://mytangserver" } ] }, "device": "/dev/disk/by-partlabel/root", "name": "root", "options": [ "--cipher", "aes-cbc-essiv:sha256" ], "wipeVolume": true } ] } } test.json is just a simple ignition config that provides a password hash and ssh key so I can log in to the test system
Assigning to jlebon for initial evaluation.
At the "Ignition distro integration" level, the assumption is that as soon as you reprovision the root filesystem, you're choosing your own adventure. So e.g. you should be able to add whatever mount options you'd like. There's at least two problems here though: 1. So far `prjquota` has mostly been an implementation detail we haven't exposed to users. This would require documenting it everywhere root reprovisioning is documented. 2. Changing mount options right now for the rootfs is painful. We don't support the `mountOptions` flag (https://github.com/coreos/fedora-coreos-config/issues/805) nor Ignition kargs (but soon: https://github.com/coreos/ignition/issues/1168). So right now, this would require using `rpm-ostree kargs` in a systemd service as documented in https://docs.fedoraproject.org/en-US/fedora-coreos/kernel-args/. It's tempting to try to get https://github.com/coreos/fedora-coreos-config/issues/805 into 4.7, because at a technical level it's pretty trivial to do. That would leave (1), i.e. updating all the documented MachineConfigs and RCC snippets to include a mount_options/mountOptions. Though specifically for prjquota, it doesn't seem like there's any harm in just *always* turning it on at the OS level if the rootfs is XFS, regardless of whether it was reprovisioned or not. AFAICT there isn't really any overhead or performance issues associated with this (I mean... this *has* been the default for a long time). So then it remains an implementation detail and avoids documentation churn. Or another way to frame this is that it simplifies the rootflags messaging to just: "by default, we use prjquota if the rootfs is XFS". And when we implement https://github.com/coreos/fedora-coreos-config/issues/805 (which is generally useful), anyone who for whatever reason *doesn't* want prjquota can just do `mountOptions: []` (short-term, they can fallback to using `rpm-ostree kargs` to modify the `rootflags` karg). I'll take a look at this.
PR to retain prjquota on XFS: https://github.com/coreos/fedora-coreos-config/pull/903 PR to support mountOptions: https://github.com/coreos/fedora-coreos-config/pull/904 (though not planning to backport that one since it's not strictly needed).
This fix landed in RHCOS 48.83.202103221318-0
Booted RHCOS 48.83.202103221318-0 with luks enabled and prjquota option is set [core@localhost ~]$ rpm-ostree status State: idle Deployments: * ostree://328a44d7c259ca1e3ed31ae020f09d922f460be998657a92f684f6760443077b Version: 48.83.202103221318-0 (2021-03-22T13:22:02Z) [core@localhost ~]$ findmnt /var | less TARGET SOURCE FSTYPE OPTIONS /var /dev/mapper/root[/ostree/deploy/rhcos/var] xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota [core@localhost ~]$ sudo cryptsetup luksDump /dev/disk/by-partlabel/root LUKS header information Version: 2 Epoch: 6 Metadata area: 16384 [bytes] Keyslots area: 16744448 [bytes] UUID: d8404c5d-8db6-418e-bf57-5f9b3e43534d Label: (no label) Subsystem: (no subsystem) Flags: (no flags) Data segments: 0: crypt offset: 16777216 [bytes] length: (whole device) cipher: aes-cbc-essiv:sha256 sector: 512 [bytes] Keyslots: 1: luks2 Key: 256 bits Priority: normal Cipher: aes-cbc-essiv:sha256 Cipher key: 256 bits PBKDF: argon2i Time cost: 4 Memory: 629526 Threads: 2 Salt: e0 8d 9a f2 99 f0 43 d2 46 95 37 a4 2e fa e6 9f 16 b4 33 83 05 f7 3c 29 42 d2 d2 b1 89 d3 9e dd AF stripes: 4000 AF hash: sha256 Area offset:163840 [bytes] Area length:131072 [bytes] Digest ID: 0 Tokens: 0: clevis Keyslot: 1 Digests: 0: pbkdf2 Hash: sha256 Iterations: 191625 Salt: 76 6a de 2e 82 74 12 3c f9 95 a2 cd 1a bf 4b 4d 65 63 77 8a 4f 88 b0 27 26 53 3e 21 92 bb 93 d9 Digest: ae 5e 0b 24 88 06 be ee 6c 2e 84 0d ba e9 08 e2 83 ac d3 01 92 4e c1 06 47 00 a1 ad dd 8d be 52 [core@localhost ~]$ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk |-vda1 252:1 0 1M 0 part |-vda2 252:2 0 127M 0 part |-vda3 252:3 0 384M 0 part /boot `-vda4 252:4 0 19.5G 0 part `-root 253:0 0 19.5G 0 crypt /sysroot [core@localhost ~]$ sudo clevis luks list -d /dev/disk/by-partlabel/root 1: sss '{"t":1,"pins":{"tang":[{"url":"http://192.168.1.176"}]}}'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438