Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1872313

Summary: realtime jobs 100% blocked: appears to be failure to ignite workers
Product: OpenShift Container Platform Reporter: David Eads <deads>
Component: Machine Config OperatorAssignee: Yu Qi Zhang <jerzhang>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.6CC: jerzhang, miabbott
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:32:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Eads 2020-08-25 13:33:24 UTC 
the overall job is here https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#release-openshift-ocp-installer-e2e-gcp-rt-4.6

In https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-rt-4.6/1298201056227889152 for example.

Ingress doesn't go available, but it's because workers aren't present.  The machine API indicates that the machines were created (see thread https://coreos.slack.com/archives/CBZHF4DHC/p1598361928052500?thread_ts=1598361007.051600&cid=CBZHF4DHC ) and mgugino found a failure in the MCS: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-rt-4.6/1298201056227889152/artifacts/e2e-gcp/pods/openshift-machine-config-operator_machine-config-server-7h9cg_machine-config-server.log
Comment 4 Micah Abbott 2020-09-08 17:17:40 UTC 
Verified with 4.6.0-fc.4

```
$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-fc.4   True        False         32m     Cluster version is 4.6.0-fc.4

$ oc get nodes
NAME                           STATUS   ROLES    AGE   VERSION
ip-10-0-134-86.ec2.internal    Ready    master   52m   v1.19.0-rc.2+514f31a                           
ip-10-0-148-100.ec2.internal   Ready    worker   41m   v1.19.0-rc.2+514f31a
ip-10-0-150-160.ec2.internal   Ready    master   53m   v1.19.0-rc.2+514f31a
ip-10-0-180-51.ec2.internal    Ready    worker   41m   v1.19.0-rc.2+514f31a
ip-10-0-218-54.ec2.internal    Ready    worker   41m   v1.19.0-rc.2+514f31a
ip-10-0-223-205.ec2.internal   Ready    master   53m   v1.19.0-rc.2+514f31a

$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
00-worker                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-master-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-master-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-worker-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
01-worker-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
99-master-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
99-master-ssh                                                                                 3.1.0             57m
99-worker-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
99-worker-ssh                                                                                 3.1.0             57m
rendered-master-b0985b6c15f0f96c125e32b7a5b1a081   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m
rendered-worker-27ca4280a07e7e99a33a3347cfff1f6b   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             51m

$ cat bz1872313.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: "worker"
  name: zz-test
spec:
  kernelArguments:
    - 'loglevel=6'


$ oc apply -f bz1872313.yaml 
machineconfig.machineconfiguration.openshift.io/zz-test created

$ oc get mc
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
00-worker                                          130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-master-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-master-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-worker-container-runtime                        130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
01-worker-kubelet                                  130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
99-master-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
99-master-ssh                                                                                 3.1.0             103m
99-worker-generated-registries                     130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
99-worker-ssh                                                                                 3.1.0             103m
rendered-master-b0985b6c15f0f96c125e32b7a5b1a081   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
rendered-worker-27ca4280a07e7e99a33a3347cfff1f6b   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             97m
rendered-worker-8daaa44d8998b182fd082a2f3dac2feb   130947243313dcfa8a4f0ef487f458f923df1128   3.1.0             38m
zz-test                                                                                                         38m

$ oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-fc.4   True        False         85m     Cluster version is 4.6.0-fc.4

$ oc get nodes
NAME                           STATUS   ROLES    AGE    VERSION
ip-10-0-134-86.ec2.internal    Ready    master   105m   v1.19.0-rc.2+514f31a
ip-10-0-148-100.ec2.internal   Ready    worker   94m    v1.19.0-rc.2+514f31a
ip-10-0-150-160.ec2.internal   Ready    master   107m   v1.19.0-rc.2+514f31a
ip-10-0-180-51.ec2.internal    Ready    worker   94m    v1.19.0-rc.2+514f31a
ip-10-0-218-54.ec2.internal    Ready    worker   94m    v1.19.0-rc.2+514f31a
ip-10-0-223-205.ec2.internal   Ready    master   106m   v1.19.0-rc.2+514f31a

$ oc debug node/ip-10-0-148-100.ec2.internal -- chroot /host cat /proc/cmdline
Starting pod/ip-10-0-148-100ec2internal-debug ...
To use host binaries, run `chroot /host`
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-21c3e2ed2266506a60374b77103a4666f1eef52e07ee045463988c7274b034d3/vmlinuz-4.18.0-211.el8.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.1/rhcos/21c3e2ed2266506a60374b77103a4666f1eef52e07ee045463988c7274b034d3/0 ignition.platform.id=aws loglevel=6

Removing debug pod ...
```
Comment 6 errata-xmlrpc 2020-10-27 16:32:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196