+++ This bug was initially created as a clone of Bug #1934174 +++ Description of problem: rootfs is created too small ``` # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 127M 0 part ├─sda3 8:3 0 384M 0 part /boot ├─sda4 8:4 0 3G 0 part │ └─root 253:0 0 3G 0 crypt /sysroot └─sda5 8:5 0 65M 0 part ``` when using the documented procedure for NBDE https://github.com/openshift/openshift-docs/blob/enterprise-4.7/modules/installation-special-config-encrypt-disk-tang.adoc without NBDE, rootfs is spaning the full disk size: ``` $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 127M 0 part ├─sda3 8:3 0 384M 0 part /boot ├─sda4 8:4 0 29.4G 0 part /sysroot └─sda5 8:5 0 65M 0 part ``` Version-Release number of selected component (if applicable): 4.8-nightly How reproducible: everytime Steps to Reproduce: 1. follow official steps to enable nbde Actual results: mcd crash because not enough disk space Expected results: installation to succeed Additional info: --- Additional comment from Micah Abbott on 2021-03-12 13:54:18 UTC --- See also: https://github.com/openshift/os/pull/514 and https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1234 The fix for this landed in RHCOS 48.83.202103111918-0 --- Additional comment from OpenShift Automated Release Tooling on 2021-03-12 14:26:50 UTC --- Elliott changed bug status from MODIFIED to ON_QA. --- Additional comment from Michael Nguyen on 2021-03-15 16:03:31 UTC --- Verified on RHCOS 48.83.202103122318-0 [core@localhost ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 20G 0 disk |-vda1 252:1 0 1M 0 part |-vda2 252:2 0 127M 0 part |-vda3 252:3 0 384M 0 part /boot `-vda4 252:4 0 19.5G 0 part `-root 253:0 0 19.5G 0 crypt /sysroot Also need to verify this after the boot image bump. --- Additional comment from Jonathan Lebon on 2021-03-17 21:29:33 UTC --- There's a bug in RHEL's lsblk (which we've hit before; see https://github.com/coreos/coreos-installer/pull/453) breaking the new code in ignition-ostree-growfs.sh. Working on a patch to work around it. But sadly, we'll need a new bootimage bump for this. Re-moving to ASSIGNED. --- Additional comment from Micah Abbott on 2021-03-22 15:45:25 UTC --- The fix for this landed in RHCOS 48.83.202103221318-0 --- Additional comment from OpenShift Automated Release Tooling on 2021-03-22 15:58:13 UTC --- Elliott changed bug status from MODIFIED to ON_QA.
Verified on 47.83.202103251640-0. Overrode the boot image with the ami from the boot image bump https://github.com/openshift/installer/pull/4791 on 4.7.0-0.nightly-2021-03-26-105314. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2021-03-26-105314 True False 10m Cluster version is 4.7.0-0.nightly-2021-03-26-105314 $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-130-216.us-west-2.compute.internal Ready master 36m v1.20.0+bafe72f ip-10-0-140-126.us-west-2.compute.internal Ready worker 23m v1.20.0+bafe72f ip-10-0-160-193.us-west-2.compute.internal Ready worker 24m v1.20.0+bafe72f ip-10-0-169-235.us-west-2.compute.internal Ready master 36m v1.20.0+bafe72f ip-10-0-194-63.us-west-2.compute.internal Ready master 34m v1.20.0+bafe72f ip-10-0-212-179.us-west-2.compute.internal Ready worker 23m v1.20.0+bafe72f $ oc debug node/ip-10-0-130-216.us-west-2.compute.internal Starting pod/ip-10-0-130-216us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# rpm-ostree status State: idle Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ad297b22e7b96e04e45aefcc57f571361c87bdc3110e692bb239f2dfbe64050 CustomOrigin: Managed by machine-config-operator Version: 47.83.202103251640-0 (2021-03-25T16:44:03Z) ostree://3fdd1488024f054e39b1be508781d535d1ac7ed423bb3b4b656c2f345934220d Version: 47.83.202103251640-0 (2021-03-25T16:44:03Z) sh-4.4# cryptsetup luksDump /dev/disk/by-partlabel/root LUKS header information Version: 2 Epoch: 6 Metadata area: 16384 [bytes] Keyslots area: 16744448 [bytes] UUID: 40fb8592-a819-412e-8dc6-25c58c915edf Label: (no label) Subsystem: (no subsystem) Flags: (no flags) Data segments: 0: crypt offset: 16777216 [bytes] length: (whole device) cipher: aes-cbc-essiv:sha256 sector: 512 [bytes] Keyslots: 1: luks2 Key: 256 bits Priority: normal Cipher: aes-cbc-essiv:sha256 Cipher key: 256 bits PBKDF: argon2i Time cost: 5 Memory: 1048576 Threads: 4 Salt: 50 79 32 2f 29 ec 5a 33 8a 05 17 47 80 89 bf 2d 63 54 ce e7 dd 99 70 23 bf b6 28 74 22 65 68 6c AF stripes: 4000 AF hash: sha256 Area offset:163840 [bytes] Area length:131072 [bytes] Digest ID: 0 Tokens: 0: clevis Keyslot: 1 Digests: 0: pbkdf2 Hash: sha256 Iterations: 214520 Salt: 23 0f c2 81 42 2c ca 5b 82 0a 3e 9b e7 af 61 5d ec af 0d c4 12 65 4d e4 94 5c 8d 92 07 2d 54 29 Digest: e3 54 29 61 44 be 29 39 68 da 62 01 da e5 0c 8f c7 17 32 59 02 56 f2 ab 32 d6 fb f0 a9 0e 13 31 sh-4.4# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 120G 0 disk |-nvme0n1p1 259:1 0 1M 0 part |-nvme0n1p2 259:2 0 127M 0 part |-nvme0n1p3 259:3 0 384M 0 part /boot `-nvme0n1p4 259:4 0 119.5G 0 part `-root 253:0 0 119.5G 0 crypt /sysroot sh-4.4# findmnt /var | more TARGET SOURCE FSTYPE OPTIONS /var /dev/mapper/root[/ostree/deploy/rhcos/var] xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota sh-4.4# clevis luks list -d /dev/disk/by-partlabel/root 1: sss '{"t":1,"pins":{"tang":[{"url":"http://34.217.25.205"}]}}' sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ oc debug node/ip-10-0-140-126.us-west-2.compute.internal Starting pod/ip-10-0-140-126us-west-2computeinternal-debug ... To use host binaries, run `chroot /host` If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 120G 0 disk |-nvme0n1p1 259:1 0 1M 0 part |-nvme0n1p2 259:2 0 127M 0 part |-nvme0n1p3 259:3 0 384M 0 part /boot `-nvme0n1p4 259:4 0 119.5G 0 part `-root 253:0 0 119.5G 0 crypt /sysroot sh-4.4# rpm-ostree status State: idle Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ad297b22e7b96e04e45aefcc57f571361c87bdc3110e692bb239f2dfbe64050 CustomOrigin: Managed by machine-config-operator Version: 47.83.202103251640-0 (2021-03-25T16:44:03Z) ostree://3fdd1488024f054e39b1be508781d535d1ac7ed423bb3b4b656c2f345934220d Version: 47.83.202103251640-0 (2021-03-25T16:44:03Z) sh-4.4# findmnt /var | more TARGET SOURCE FSTYPE OPTIONS /var /dev/mapper/root[/ostree/deploy/rhcos/var] xfs rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota sh-4.4# cryptsetup luksDump /dev/disk/by-partlabel/root LUKS header information Version: 2 Epoch: 6 Metadata area: 16384 [bytes] Keyslots area: 16744448 [bytes] UUID: 32f7868b-af6c-45d2-8d80-b041fed469d2 Label: (no label) Subsystem: (no subsystem) Flags: (no flags) Data segments: 0: crypt offset: 16777216 [bytes] length: (whole device) cipher: aes-cbc-essiv:sha256 sector: 512 [bytes] Keyslots: 1: luks2 Key: 256 bits Priority: normal Cipher: aes-cbc-essiv:sha256 Cipher key: 256 bits PBKDF: argon2i Time cost: 4 Memory: 852086 Threads: 2 Salt: 01 2a 15 b9 81 ea 5b 1a e2 41 26 05 2b 81 74 64 19 51 61 81 4e 46 55 28 b2 70 5d 45 51 72 6e 54 AF stripes: 4000 AF hash: sha256 Area offset:163840 [bytes] Area length:131072 [bytes] Digest ID: 0 Tokens: 0: clevis Keyslot: 1 Digests: 0: pbkdf2 Hash: sha256 Iterations: 217366 Salt: 1a 51 2d 2d 06 42 96 2a de ef 7a 79 2f d9 57 38 7d 50 8a 33 9f 65 f6 ba f4 83 01 57 73 a3 b4 d9 Digest: 49 0c 7c 64 de 41 2e 38 6c 2a b9 24 22 5e 5f 03 9b 31 5d fd d8 4c 58 60 4c 34 04 5f e0 84 14 34 sh-4.4# clevis luks list /dev/disk/by-partlabel/root Did not specify a device! Usage: clevis luks list -d DEV [-s SLT] Lists pins bound to a LUKSv1 or LUKSv2 device: -d DEV The LUKS device to list bound pins -s SLOT The slot number to list sh-4.4# clevis luks list -d /dev/disk/by-partlabel/root 1: sss '{"t":1,"pins":{"tang":[{"url":"http://34.217.25.205"}]}}' sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ oc -n openshift-machine-api get machinesets/mnguyen47bootimage-cmmk5-worker-us-west-2a -o yaml | grep ami- id: ami-0617611237b58ac93
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.6 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:1075