Bug 1934557

The bump updated the AMI but the luks issue we needed the bump for still exists and is currently being investigated.

Per https://bugzilla.redhat.com/show_bug.cgi?id=1934174#c4, we will need another boot image bump to gain additional fixes for the LUKS problem.

Setting back to ASSIGNED

We'll need to fix https://bugzilla.redhat.com/show_bug.cgi?id=1940704 in 4.7 so let's make sure it's in 4.8 too. Will clone RHBZ.

Verified on 4.8.0-0.nightly-2021-03-25-063034.  Boot image is updated and the fixes are working.


$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-03-25-063034   True        False         7m30s   Cluster version is 4.8.0-0.nightly-2021-03-25-063034

$ oc -n openshift-machine-api get machineset
NAME                                         DESIRED   CURRENT   READY   AVAILABLE   AGE
mnguyen48bootimage-z926b-worker-us-west-2a   1         1         1       1           40m
mnguyen48bootimage-z926b-worker-us-west-2b   1         1         1       1           40m
mnguyen48bootimage-z926b-worker-us-west-2c   1         1         1       1           40m
mnguyen48bootimage-z926b-worker-us-west-2d   0         0                             40m
$ oc -n openshift-machine-api get machineset/mnguyen48bootimage-z926b-worker-us-west-2a -o yaml | grep ami
                f:ami: {}
          ami:
            id: ami-0c6da162537298ad6

$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-134-208.us-west-2.compute.internal   Ready    worker   21m   v1.20.0+39c0afe
ip-10-0-143-75.us-west-2.compute.internal    Ready    master   31m   v1.20.0+39c0afe
ip-10-0-164-207.us-west-2.compute.internal   Ready    worker   21m   v1.20.0+39c0afe
ip-10-0-184-86.us-west-2.compute.internal    Ready    master   32m   v1.20.0+39c0afe
ip-10-0-205-239.us-west-2.compute.internal   Ready    master   32m   v1.20.0+39c0afe
ip-10-0-214-4.us-west-2.compute.internal     Ready    worker   21m   v1.20.0+39c0afe

$ oc debug node/ip-10-0-143-75.us-west-2.compute.internal
Starting pod/ip-10-0-143-75us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
nvme0n1     259:0    0   120G  0 disk  
|-nvme0n1p1 259:1    0     1M  0 part  
|-nvme0n1p2 259:2    0   127M  0 part  
|-nvme0n1p3 259:3    0   384M  0 part  /boot
`-nvme0n1p4 259:4    0 119.5G  0 part  
  `-root    253:0    0 119.5G  0 crypt /sysroot
sh-4.4# clevis luks list -d /dev/disk/by-partlabel/root
1: sss '{"t":1,"pins":{"tang":[{"url":"http://18.237.82.232"}]}}'
sh-4.4# cryptsetup luksDump /dev/disk/by-partlabel/root
LUKS header information
Version:       	2
Epoch:         	6
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	32631113-dd5e-4a69-8897-18f2fcb77199
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)
	cipher: aes-cbc-essiv:sha256
	sector: 512 [bytes]

Keyslots:
  1: luks2
	Key:        256 bits
	Priority:   normal
	Cipher:     aes-cbc-essiv:sha256
	Cipher key: 256 bits
	PBKDF:      argon2i
	Time cost:  5
	Memory:     1048576
	Threads:    4
	Salt:       91 16 15 e3 47 64 e0 81 61 e6 ca 56 f9 9f 5e c6 
	            5b 76 83 a4 c1 25 0f c5 eb fb 82 ba f8 a2 5c b1 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:163840 [bytes]
	Area length:131072 [bytes]
	Digest ID:  0
Tokens:
  0: clevis
	Keyslot:  1
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 217366
	Salt:       80 f3 5b 7d 00 0e 21 9e 11 8e aa 7c cf ca 95 d8 
	            34 d1 c4 8c c1 36 6b a0 ae 99 6f 60 f2 34 d8 aa 
	Digest:     6e d5 24 0b 0c 5f 9a ae 21 76 10 af c7 b7 ca a2 
	            56 a0 7c f3 a4 84 2a 05 7a d5 35 c5 84 05 25 25 
sh-4.4# findmnt /var | more
TARGET SOURCE                                     FSTYPE OPTIONS
/var   /dev/mapper/root[/ostree/deploy/rhcos/var] xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,log
bsize=32k,prjquota
sh-4.4# rpm-ostree status
State: idle
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f0c628ec5d669a574ad114c89f4af9e669e7da89e7a2705c95fe83e98eaf570
              CustomOrigin: Managed by machine-config-operator
                   Version: 48.83.202103221318-0 (2021-03-22T13:22:02Z)

  ostree://328a44d7c259ca1e3ed31ae020f09d922f460be998657a92f684f6760443077b
                   Version: 48.83.202103221318-0 (2021-03-22T13:22:02Z)
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

$ oc debug node/ip-10-0-134-208.us-west-2.compute.internal
Starting pod/ip-10-0-134-208us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
nvme0n1     259:0    0   120G  0 disk  
|-nvme0n1p1 259:1    0     1M  0 part  
|-nvme0n1p2 259:2    0   127M  0 part  
|-nvme0n1p3 259:3    0   384M  0 part  /boot
`-nvme0n1p4 259:4    0 119.5G  0 part  
  `-root    253:0    0 119.5G  0 crypt /sysroot
sh-4.4# clevis luks list -d /dev/disk/by-partlabel/root
1: sss '{"t":1,"pins":{"tang":[{"url":"http://18.237.82.232"}]}}'
sh-4.4# cryptsetup luksDump /dev/disk/by-partlabel/root
LUKS header information
Version:       	2
Epoch:         	6
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	e83863d3-0e80-4bfd-a58f-4d26dace2d9c
Label:         	(no label)
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)
	cipher: aes-cbc-essiv:sha256
	sector: 512 [bytes]

Keyslots:
  1: luks2
	Key:        256 bits
	Priority:   normal
	Cipher:     aes-cbc-essiv:sha256
	Cipher key: 256 bits
	PBKDF:      argon2i
	Time cost:  4
	Memory:     849082
	Threads:    2
	Salt:       8d f9 a9 2b 0e 2b 48 37 06 b8 06 bf 6d 9f 28 0b 
	            88 6c 9b b2 93 91 0d 72 b9 3a 71 71 fa 13 ec 7c 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:163840 [bytes]
	Area length:131072 [bytes]
	Digest ID:  0
Tokens:
  0: clevis
	Keyslot:  1
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 217366
	Salt:       75 b6 39 b5 63 5b 17 87 ac 4a 19 e3 5c 47 e7 95 
	            81 be 07 d0 df 0d d2 0e 67 fe cc 4f 62 73 52 4f 
	Digest:     73 75 23 80 e4 e4 99 82 4a 1f 4e cf 64 68 c0 b6 
	            ba a7 3b b9 de fe e7 67 39 af 2d 54 3d 84 cb c0 
sh-4.4# findmnt /var | more
TARGET SOURCE                                     FSTYPE OPTIONS
/var   /dev/mapper/root[/ostree/deploy/rhcos/var] xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438