Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1934174

Summary: rootfs too small when enabling NBDE
Product: OpenShift Container Platform Reporter: Yuval Kashtan <ykashtan>
Component: RHCOSAssignee: Jonathan Lebon <jlebon>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.7CC: bbreard, imcleod, jcall, jlebon, jligon, miabbott, nstielau, omichael
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1934863 1941760 (view as bug list) Environment:
Last Closed: 2021-07-27 22:49:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1934557, 1934863, 1941760    

Description Yuval Kashtan 2021-03-02 16:22:18 UTC 
Description of problem:
rootfs is created too small
```
# lsblk
NAME     MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda        8:0    0   30G  0 disk  
├─sda1     8:1    0    1M  0 part  
├─sda2     8:2    0  127M  0 part  
├─sda3     8:3    0  384M  0 part  /boot
├─sda4     8:4    0    3G  0 part  
│ └─root 253:0    0    3G  0 crypt /sysroot
└─sda5     8:5    0   65M  0 part
```

when using the documented procedure for NBDE 
https://github.com/openshift/openshift-docs/blob/enterprise-4.7/modules/installation-special-config-encrypt-disk-tang.adoc

without NBDE, rootfs is spaning the full disk size:
```
$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   30G  0 disk 
├─sda1   8:1    0    1M  0 part 
├─sda2   8:2    0  127M  0 part 
├─sda3   8:3    0  384M  0 part /boot
├─sda4   8:4    0 29.4G  0 part /sysroot
└─sda5   8:5    0   65M  0 part 
```

Version-Release number of selected component (if applicable):
4.8-nightly

How reproducible:
everytime

Steps to Reproduce:
1. follow official steps to enable nbde

Actual results:
mcd crash because not enough disk space

Expected results:
installation to succeed 

Additional info:
Comment 1 Micah Abbott 2021-03-12 13:54:18 UTC 
See also:  https://github.com/openshift/os/pull/514 and https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1234

The fix for this landed in RHCOS 48.83.202103111918-0
Comment 3 Michael Nguyen 2021-03-15 16:03:31 UTC 
Verified on RHCOS  48.83.202103122318-0

[core@localhost ~]$ lsblk
NAME     MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda      252:0    0   20G  0 disk  
|-vda1   252:1    0    1M  0 part  
|-vda2   252:2    0  127M  0 part  
|-vda3   252:3    0  384M  0 part  /boot
`-vda4   252:4    0 19.5G  0 part  
  `-root 253:0    0 19.5G  0 crypt /sysroot

Also need to verify this after the boot image bump.
Comment 4 Jonathan Lebon 2021-03-17 21:29:33 UTC 
There's a bug in RHEL's lsblk (which we've hit before; see https://github.com/coreos/coreos-installer/pull/453) breaking the new code in ignition-ostree-growfs.sh. Working on a patch to work around it. But sadly, we'll need a new bootimage bump for this. Re-moving to ASSIGNED.
Comment 5 Micah Abbott 2021-03-22 15:45:25 UTC 
The fix for this landed in RHCOS 48.83.202103221318-0
Comment 7 Michael Nguyen 2021-03-23 14:38:30 UTC 
nvme disk
----------------
[core@cosa-devsh ~]$ rpm-ostree status
State: idle
Deployments:
● ostree://328a44d7c259ca1e3ed31ae020f09d922f460be998657a92f684f6760443077b
                   Version: 48.83.202103221318-0 (2021-03-22T13:22:02Z)
[core@cosa-devsh ~]$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0          11:0    1 1024M  0 rom   
nvme0n1     259:0    0   16G  0 disk  
|-nvme0n1p1 259:1    0    1M  0 part  
|-nvme0n1p2 259:2    0  127M  0 part  
|-nvme0n1p3 259:3    0  384M  0 part  /boot
`-nvme0n1p4 259:4    0 15.5G  0 part  
  `-root    253:0    0 15.5G  0 crypt /sysroot
[core@cosa-devsh ~]$ sudo cryptsetup luksDump  /dev/disk/by-partlabel/root
LUKS header information
Version:       	2
Epoch:         	6
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	68c753b1-8f2f-427d-afc4-f3b9d0db550b
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)
	cipher: aes-cbc-essiv:sha256
	sector: 512 [bytes]

Keyslots:
  1: luks2
	Key:        256 bits
	Priority:   normal
	Cipher:     aes-cbc-essiv:sha256
	Cipher key: 256 bits
	PBKDF:      argon2i
	Time cost:  4
	Memory:     1048576
	Threads:    4
	Salt:       d8 78 43 e1 1a 40 b1 40 a3 91 b1 fb 25 66 2a 05 
	            b1 dc 4f f8 20 5a 7b 83 bc 99 5c 10 e5 d5 92 1d 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:163840 [bytes]
	Area length:131072 [bytes]
	Digest ID:  0
Tokens:
  0: clevis
	Keyslot:  1
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 203212
	Salt:       8c b8 d3 99 97 1f f5 8b 6b b8 a7 d8 ba b4 57 5e 
	            57 59 3c 6e 30 87 bc 6e 30 62 ba 44 90 95 d3 83 
	Digest:     6e 07 4f bb cb 1f 8f d0 d6 46 6b 35 d3 d6 ef 08 
	            18 70 29 68 2b 88 3d 9e f8 4e 47 c0 18 fb b2 22 
[core@cosa-devsh ~]$ clevis luks list -d /dev/disk/by-partlabel/root
Device /dev/disk/by-partlabel/root does not exist or access denied.
Device /dev/disk/by-partlabel/root does not exist or access denied.
Device /dev/disk/by-partlabel/root does not exist or access denied.
/dev/disk/by-partlabel/root is not a supported LUKS device!
No used slots detected for device /dev/disk/by-partlabel/root!
[core@cosa-devsh ~]$ sudo clevis luks list -d /dev/disk/by-partlabel/root
1: sss '{"t":1,"pins":{"tang":[{"url":"http://192.168.1.176"}]}}'
[core@cosa-devsh ~]$ findmnt /var
TARGET SOURCE                                     FSTYPE OPTIONS
/var   /dev/mapper/root[/ostree/deploy/rhcos/var] xfs    rw,relatime,seclabel,at
[core@cosa-devsh ~]$ findmnt /var | less
TARGET SOURCE                                     FSTYPE OPTIONS
/var   /dev/mapper/root[/ostree/deploy/rhcos/var] xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
[core@cosa-devsh ~]$ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0          11:0    1 1024M  0 rom   
nvme0n1     259:0    0   16G  0 disk  
|-nvme0n1p1 259:1    0    1M  0 part  
|-nvme0n1p2 259:2    0  127M  0 part  
|-nvme0n1p3 259:3    0  384M  0 part  /boot
`-nvme0n1p4 259:4    0 15.5G  0 part  
  `-root    253:0    0 15.5G  0 crypt /sysroot




regular disk
-------------------
[core@localhost ~]$ rpm-ostree status
State: idle
Deployments:
● ostree://328a44d7c259ca1e3ed31ae020f09d922f460be998657a92f684f6760443077b
                   Version: 48.83.202103221318-0 (2021-03-22T13:22:02Z)
[core@localhost ~]$ lsblk
NAME     MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda      252:0    0   20G  0 disk  
├─vda1   252:1    0    1M  0 part  
├─vda2   252:2    0  127M  0 part  
├─vda3   252:3    0  384M  0 part  /boot
└─vda4   252:4    0 19.5G  0 part  
  └─root 253:0    0 19.5G  0 crypt /sysroot
[core@localhost ~]$ sudo cryptsetup luksDump /dev/disk/by-partlabel/root
LUKS header information
Version:       	2
Epoch:         	6
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	d8404c5d-8db6-418e-bf57-5f9b3e43534d
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)
	cipher: aes-cbc-essiv:sha256
	sector: 512 [bytes]

Keyslots:
  1: luks2
	Key:        256 bits
	Priority:   normal
	Cipher:     aes-cbc-essiv:sha256
	Cipher key: 256 bits
	PBKDF:      argon2i
	Time cost:  4
	Memory:     629526
	Threads:    2
	Salt:       e0 8d 9a f2 99 f0 43 d2 46 95 37 a4 2e fa e6 9f 
	            16 b4 33 83 05 f7 3c 29 42 d2 d2 b1 89 d3 9e dd 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:163840 [bytes]
	Area length:131072 [bytes]
	Digest ID:  0
Tokens:
  0: clevis
	Keyslot:  1
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 191625
	Salt:       76 6a de 2e 82 74 12 3c f9 95 a2 cd 1a bf 4b 4d 
	            65 63 77 8a 4f 88 b0 27 26 53 3e 21 92 bb 93 d9 
	Digest:     ae 5e 0b 24 88 06 be ee 6c 2e 84 0d ba e9 08 e2 
	            83 ac d3 01 92 4e c1 06 47 00 a1 ad dd 8d be 52 
[core@localhost ~]$ sudo clevis luks list -d /dev/disk/by-partlabel
Device /dev/disk/by-partlabel is not compatible.
Device /dev/disk/by-partlabel is not compatible.
Device /dev/disk/by-partlabel is not compatible.
/dev/disk/by-partlabel is not a supported LUKS device!
No used slots detected for device /dev/disk/by-partlabel!
[core@localhost ~]$ sudo clevis luks list -d /dev/disk/by-partlabel/root
1: sss '{"t":1,"pins":{"tang":[{"url":"http://192.168.1.176"}]}}'
[core@localhost ~]$ findmnt /var | less
[core@localhost ~]$ findmnt /var
TARGET SOURCE                                     FSTYPE OPTIONS
/var   /dev/mapper/root[/ostree/deploy/rhcos/var] xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
Comment 11 errata-xmlrpc 2021-07-27 22:49:01 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438