Bug 1953475 - worker pool went degraded due to no rpm-ostree on rhel worker during applying new mc
Summary: worker pool went degraded due to no rpm-ostree on rhel worker during applying...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.z
Assignee: Sinny Kumari
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On: 1952368
Blocks: 1953493
TreeView+ depends on / blocked
 
Reported: 2021-04-26 08:08 UTC by OpenShift BugZilla Robot
Modified: 2021-06-15 09:27 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-15 09:27:08 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2544 0 None open [release-4.7] Bug 1953475: daemon: return nil for unsupported operation on an OS 2021-05-20 16:55:19 UTC
Red Hat Product Errata RHSA-2021:2286 0 None None None 2021-06-15 09:27:55 UTC

Comment 5 Michael Nguyen 2021-06-02 17:06:52 UTC
Verified on 4.7.0-0.nightly-2021-06-01-051359

$ oc get nodes
NAME                                        STATUS   ROLES    AGE    VERSION
ip-10-0-54-214.us-east-2.compute.internal   Ready    worker   140m   v1.20.0+a0b09eb
ip-10-0-56-151.us-east-2.compute.internal   Ready    master   151m   v1.20.0+a0b09eb
ip-10-0-57-232.us-east-2.compute.internal   Ready    master   150m   v1.20.0+a0b09eb
ip-10-0-59-23.us-east-2.compute.internal    Ready    worker   95m    v1.20.0+a0b09eb
ip-10-0-71-237.us-east-2.compute.internal   Ready    master   151m   v1.20.0+a0b09eb
ip-10-0-77-130.us-east-2.compute.internal   Ready    worker   139m   v1.20.0+a0b09eb
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-06-01-051359   True        False         131m    Cluster version is 4.7.0-0.nightly-2021-06-01-051359
$ cat << EOF > trifecta.yaml
>apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: worker-extensions-usbguard
spec:
  config:
    ignition:
      version: 3.2.0
  extensions:
    - usbguard
  kernelType: realtime
  kernelArguments:
  - 'z=10'
> EOF
$ oc create -f trifecta.yaml 
machineconfig.machineconfiguration.openshift.io/worker-extensions-usbguard created

$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             151m
00-worker                                          3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             151m
01-master-container-runtime                        3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             151m
01-master-kubelet                                  3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             151m
01-worker-container-runtime                        3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             151m
01-worker-kubelet                                  3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             150m
99-master-generated-registries                     3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             150m
99-master-ssh                                                                                 3.2.0             155m
99-worker-generated-registries                     3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             150m
99-worker-ssh                                                                                 3.2.0             155m
rendered-master-196f36ec6654bc8d3062140cdf89bb51   3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             150m
rendered-worker-cd34e86dcbf505c2bba1316b55b2e3b0   3c1fc49624d0a9edbbd4ac20223afbdbd4b5ccf4   3.2.0             150m
worker-extensions-usbguard                                                                    3.2.0             3s

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-196f36ec6654bc8d3062140cdf89bb51   True      False      False      3              3                   3                     0                      153m
worker   rendered-worker-cd34e86dcbf505c2bba1316b55b2e3b0   False     True       False      3              0                   0                     0                      153m

$ oc get nodes
NAME                                        STATUS                     ROLES    AGE    VERSION
ip-10-0-54-214.us-east-2.compute.internal   Ready,SchedulingDisabled   worker   144m   v1.20.0+a0b09eb
ip-10-0-56-151.us-east-2.compute.internal   Ready                      master   155m   v1.20.0+a0b09eb
ip-10-0-57-232.us-east-2.compute.internal   Ready                      master   154m   v1.20.0+a0b09eb
ip-10-0-59-23.us-east-2.compute.internal    Ready                      worker   98m    v1.20.0+a0b09eb
ip-10-0-71-237.us-east-2.compute.internal   Ready                      master   154m   v1.20.0+a0b09eb
ip-10-0-77-130.us-east-2.compute.internal   Ready                      worker   143m   v1.20.0+a0b09eb

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-196f36ec6654bc8d3062140cdf89bb51   True      False      False      3              3                   3                     0                      170m
worker   rendered-worker-de58a78a7aa3c8bffdbb9321b35366da   True      False      False      3              3                   3                     0                      170m

$ oc debug node/ip-10-0-54-214.us-east-2.compute.internal
Starting pod/ip-10-0-54-214us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# rpm -qa | grep kernel
kernel-rt-core-4.18.0-240.22.1.rt7.77.el8_3.x86_64
kernel-rt-kvm-4.18.0-240.22.1.rt7.77.el8_3.x86_64
kernel-rt-modules-4.18.0-240.22.1.rt7.77.el8_3.x86_64
kernel-rt-modules-extra-4.18.0-240.22.1.rt7.77.el8_3.x86_64
sh-4.4# uname -a
Linux ip-10-0-54-214 4.18.0-240.22.1.rt7.77.el8_3.x86_64 #1 SMP PREEMPT_RT Fri Mar 26 18:44:48 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
sh-4.4# cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-5c2248babeb9f76b3bc4b66df095be360f33546a5d8ed15d1fb77c415e2c348e/vmlinuz-4.18.0-240.22.1.rt7.77.el8_3.x86_64 random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.1/rhcos/5c2248babeb9f76b3bc4b66df095be360f33546a5d8ed15d1fb77c415e2c348e/0 ignition.platform.id=aws root=UUID=aa912f04-f50d-4703-86c1-06d97217dfd7 rw rootflags=prjquota z=10
sh-4.4# exit
exit
sh-4.2# exit    
exit

Removing debug pod ...

$ oc debug node/ip-10-0-59-23.us-east-2.compute.internal
Starting pod/ip-10-0-59-23us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.2# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 (Maipo)
sh-4.2# rpm -qa | grep kernel
kernel-tools-3.10.0-1127.el7.x86_64
kernel-tools-libs-3.10.0-1127.el7.x86_64
kernel-3.10.0-1160.25.1.el7.x86_64
kernel-3.10.0-1127.el7.x86_64
sh-4.2# uname -a
Linux ip-10-0-59-23.us-east-2.compute.internal 3.10.0-1160.25.1.el7.x86_64 #1 SMP Tue Apr 13 18:55:45 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
sh-4.2# cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.10.0-1160.25.1.el7.x86_64 root=UUID=5a000634-a1fc-467d-8ef4-5fcf5dbc6033 ro console=ttyS0,115200n8 console=tty0 net.ifnames=0 rd.blacklist=nouveau nvme_core.io_timeout=4294967295 crashkernel=auto LANG=en_US.UTF-8
sh-4.2# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Comment 6 Siddharth Sharma 2021-06-04 18:38:56 UTC
This bug will be shipped as part of next z-stream release 4.7.15 on June 14th, as 4.7.14 was dropped due to a regression https://bugzilla.redhat.com/show_bug.cgi?id=1967614

Comment 10 errata-xmlrpc 2021-06-15 09:27:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.16 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2286


Note You need to log in before you can comment on or make changes to this bug.