Bug 1872313 - realtime jobs 100% blocked: appears to be failure to ignite workers
Summary: realtime jobs 100% blocked: appears to be failure to ignite workers
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.6
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.0
Assignee: Yu Qi Zhang
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-25 13:33 UTC by David Eads
Modified: 2020-10-27 16:32 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:32:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2024 0 None closed Bug 1872313: controller: improve merging of MCs during rendering 2020-10-26 01:47:09 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:32:44 UTC

Comment 4 Micah Abbott 2020-09-08 17:17:40 UTC
Verified with 4.6.0-fc.4

```
$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-fc.4   True        False         32m     Cluster version is 4.6.0-fc.4

$ oc get nodes
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-0-134-86.ec2.internal    Ready    master   52m   v1.19.0-rc.2+514f31a                           
ip-10-0-148-100.ec2.internal   Ready    worker   41m   v1.19.0-rc.2+514f31a
ip-10-0-150-160.ec2.internal   Ready    master   53m   v1.19.0-rc.2+514f31a
ip-10-0-180-51.ec2.internal    Ready    worker   41m   v1.19.0-rc.2+514f31a
ip-10-0-218-54.ec2.internal    Ready    worker   41m   v1.19.0-rc.2+514f31a
ip-10-0-223-205.ec2.internal   Ready    master   53m   v1.19.0-rc.2+514f31a

$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
00-worker                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-master-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-master-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-worker-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-worker-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
99-master-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
99-master-ssh                                                                                 3.1.0             57m
99-worker-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
99-worker-ssh                                                                                 3.1.0             57m
rendered-master-b0985b6c15f0f96c125e32b7a5b1a081   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
rendered-worker-27ca4280a07e7e99a33a3347cfff1f6b   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m

$ cat bz1872313.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: "worker"
  name: zz-test
spec:
  kernelArguments:
    - 'loglevel=6'


$ oc apply -f bz1872313.yaml 
machineconfig.machineconfiguration.openshift.io/zz-test created

$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
00-worker                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-master-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-master-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-worker-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-worker-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
99-master-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
99-master-ssh                                                                                 3.1.0             103m
99-worker-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
99-worker-ssh                                                                                 3.1.0             103m
rendered-master-b0985b6c15f0f96c125e32b7a5b1a081   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
rendered-worker-27ca4280a07e7e99a33a3347cfff1f6b   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
rendered-worker-8daaa44d8998b182fd082a2f3dac2feb   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             38m
zz-test                                                                                                         38m

$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-fc.4   True        False         85m     Cluster version is 4.6.0-fc.4

$ oc get nodes
NAME                           STATUS   ROLES    AGE    VERSION
ip-10-0-134-86.ec2.internal    Ready    master   105m   v1.19.0-rc.2+514f31a
ip-10-0-148-100.ec2.internal   Ready    worker   94m    v1.19.0-rc.2+514f31a
ip-10-0-150-160.ec2.internal   Ready    master   107m   v1.19.0-rc.2+514f31a
ip-10-0-180-51.ec2.internal    Ready    worker   94m    v1.19.0-rc.2+514f31a
ip-10-0-218-54.ec2.internal    Ready    worker   94m    v1.19.0-rc.2+514f31a
ip-10-0-223-205.ec2.internal   Ready    master   106m   v1.19.0-rc.2+514f31a

$ oc debug node/ip-10-0-148-100.ec2.internal -- chroot /host cat /proc/cmdline
Starting pod/ip-10-0-148-100ec2internal-debug ...
To use host binaries, run `chroot /host`
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-21c3e2ed2266506a60374b77103a4666f1eef52e07ee045463988c7274b034d3/vmlinuz-4.18.0-211.el8.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.1/rhcos/21c3e2ed2266506a60374b77103a4666f1eef52e07ee045463988c7274b034d3/0 ignition.platform.id=aws loglevel=6

Removing debug pod ...
```

Comment 6 errata-xmlrpc 2020-10-27 16:32:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.