Bug 1809921
| Summary: | [UPI on self-signed OSP16] MCO is degraded due to nodes are reporting machineconfig not found | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | weiwei jiang <wjiang> | ||||
| Component: | Installer | Assignee: | Martin André <m.andre> | ||||
| Installer sub component: | OpenShift on OpenStack | QA Contact: | David Sanz <dsanzmor> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | high | ||||||
| Priority: | high | CC: | amurdaca, aos-bugs, bart, jerzhang, jokerman, m.andre, mitr, pprinett, rphillips | ||||
| Version: | 4.4 | Keywords: | TestBlocker | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.5.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1813822 (view as bug list) | Environment: | |||||
| Last Closed: | 2020-07-13 17:17:56 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1813822 | ||||||
| Attachments: |
|
||||||
Created attachment 1667629 [details]
mcs-machine-config-content.json on master-0
Add mcs-machine-config-content.json as attachments.
$ oc get machineconfig
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
00-master c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
00-worker c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
01-master-container-runtime c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
01-master-kubelet c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
01-worker-container-runtime c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
01-worker-kubelet c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
99-master-6d03836a-379b-4d9f-925a-3bb684ce2c01-registries c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
99-master-ssh 2.2.0 39h
99-worker-cb9ef1b5-6e5d-4b6f-b7c5-b8bf2348bb24-registries c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
99-worker-ssh 2.2.0 39h
rendered-master-2ff3968f86f4dadbbf2118dde83da8ad c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
rendered-master-b0bfe7cf925981b79f6641233b38a1d0 c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 20h
rendered-master-e938388901494b038eb7dfe835a2f4b4 c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
rendered-worker-4dc1d4af4078f2921cc7a57e2b96f71f c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
rendered-worker-c2c8c5f9a6b4e405b0694fedc294abe6 c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 39h
rendered-worker-f7c06bfb75d1f8cb0eb8b3d3c3541333 c0e0c323241932282de12bc9fc5fe5ef5696092d 2.2.0 20h
The diff between the very first MCO-generated rendered-master MC and the bootstrap generated rendered-master MC is this: ``` < source: data:, --- > source: data:,-----BEGIN%20CERTIFICATE-----%0AMIIDpTCCAo2gAwIBAgIUUkVbKASTDq8lts1k6FBiXYJpZ6gwDQYJKoZIhvcNAQEL%0ABQAwYjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdo%0AMRAwDgYDVQQKDAdSZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2%0AOC4yNC4yMB4XDTIwMDIyMzE2MjgzOFoXDTIxMDIyMjE2MjgzOFowYjELMAkGA1UE%0ABhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdS%0AZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2OC4yNC4yMIIBIjAN%0ABgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo8hDyhKXBpOcVfzP5ZnGi6T8W9P9%0AHyol%2FojaT958JoO5V7luoIgshPVA8feD9xMaT3QOfVDPgkAE4rMvpdohbCwx1aYY%0A8OeumzkEQQEezYmlY8XRNBXB%2Bh0%2F%2BxO54GVXbtkekCDJ3Z5yeQyA6dB2mcf%2FNSF0%0Ai3ox%2BBKVQryrpHvjSbkrb0dKQzAdPe59BuTyy7uphwhUB9K4EPuotUTDY19uPyMC%0A7Hg0%2Fd%2FiEKgHcahbep%2FPTD05%2FQxa9mPx3lqXr9YO5e1VSQFhG3cL5P%2BiDqtpksP7%0AQ62dBsbe16a6bTPH9OCJ3hl8lABGbfgEcNlbaEIA1e0TDIspgYz%2Fp75%2FGQIDAQAB%0Ao1MwUTAdBgNVHQ4EFgQUROo72sMpfR09Yfs34dPtgInoAogwHwYDVR0jBBgwFoAU%0AROo72sMpfR09Yfs34dPtgInoAogwDwYDVR0TAQH%2FBAUwAwEB%2FzANBgkqhkiG9w0B%0AAQsFAAOCAQEAWYclYSNIGeSmk6nehhpwzi1vEpxvlxF%2F%2FTLUo7N82y1zVXrH1zDS%0AKsEOBJDem%2BJotw%2FO4jEM2i84t3n3UyjFjfakKS7L0FjI8PXVGyRxmtT9AovT7%2ByZ%0AiZR7BhbC3IWrXnZ8c3y%2BOfxs75Ofb%2BQxF%2Bz4TZMiqeRmkrysn6NTi%2B11Z90q3qE8%0A6KdhWGc2TGORBijRcJyKQEUbQ2b5SeEPlWV1yZbQWhJfASMbAAfi3ZsYwEUqi%2F8W%0AECzUQcqMAJLPaHM1NtVkCVcpAkKVOIu21dCC5QHQjkfVAz06%2FCtmCS0hjadbUlcp%0AY%2BxtFK2KtggdOWriZiNH%2FuUvVQglh1kLcg%3D%3D%0A-----END%20CERTIFICATE-----%0A 309c314 < source: data:text/plain,unqualified-search-registries%20%3D%20%5B%22registry.access.redhat.com%22%2C%20%22docker.io%22%5D%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22quay.io%2Fopenshift-release-dev%2Focp-v4.0-art-dev%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22registry.svc.ci.openshift.org%2Focp%2Frelease%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A --- > source: data:text/plain,unqualified-search-registries%20%3D%20%5B%22registry.access.redhat.com%22%2C%20%22docker.io%22%5D%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22quay.io%2Fopenshift-release-dev%2Focp-v4.0-art-dev%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22quay.io%2Fopenshift%2Forigin-must-gather%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Fopenshift%2Forigin-must-gather%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22registry.redhat.io%2Frhel7%2Fsupport-tools%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Frhel7%2Fsupport-tools%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22registry.svc.ci.openshift.org%2Focp%2Frelease%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A ``` It looks like something is weird between the registries.conf content, not sure what's going on there, I'll move to Node for Miloslav/Urvashi to investigate. What's basically happening is bootstrap generates an MC with a given content of registries.conf but when the MCO kicks in, you can see it's adding something which makes it differ Is this fully reproducible? CC'ing Miloslav as well who worked on this Before: > unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] > > [[registry]] > prefix = "" > location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev" > mirror-by-digest-only = true > > [[registry.mirror]] > location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release" > > [[registry]] > prefix = "" > location = "registry.svc.ci.openshift.org/ocp/release" > mirror-by-digest-only = true > > [[registry.mirror]] > location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release" After: > unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] > > [[registry]] > prefix = "" > location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev" > mirror-by-digest-only = true > > [[registry.mirror]] > location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release" > > [[registry]] > prefix = "" > location = "quay.io/openshift/origin-must-gather" > mirror-by-digest-only = true > > [[registry.mirror]] > location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift/origin-must-gather" > > [[registry]] > prefix = "" > location = "registry.redhat.io/rhel7/support-tools" > mirror-by-digest-only = true > > [[registry.mirror]] > location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/rhel7/support-tools" > > [[registry]] > prefix = "" > location = "registry.svc.ci.openshift.org/ocp/release" > mirror-by-digest-only = true > > [[registry.mirror]] > location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release" Is this really just a difference between bootstrap and first run? It looks far more likely like an effect of the debugging attempts from comment#4. *** Bug 1810980 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |
Description of problem: When trying to setup a restricted networking UPI on OSP16, got the following error: # openshift-install' wait-for install-complete --dir '/home/jenkins/workspace/Launch Environment Flexy/workdir/install-dir' --log-level debug evel=fatal msg="failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Failed to resync 4.4.0-0.nightly-2020-03-03-011650 because: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: configuration status for pool master is empty: pool is degraded because nodes fail with \"3 nodes are reporting degraded status on sync\": \"Node wjuos163037-lc6gf-master-0 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-1 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-2 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\"\", retrying" Version-Release number of selected component (if applicable): 4.4.0-0.nightly-2020-03-03-011650: How reproducible: Always Steps to Reproduce: 1. create restricted UPI on OSP16 cluster with install-config.yaml like following, we use dns posioning to construct a restricted networking --- apiVersion: v1 controlPlane: architecture: amd64 hyperthreading: Enabled name: master replicas: 3 compute: - architecture: amd64 hyperthreading: Enabled name: worker replicas: 0 metadata: name: wjuos163037 platform: openstack: cloud: openstack computeFlavor: m4.xlarge externalNetwork: nova lbFloatingIP: 10.8.100.242 region: regionOne trunkSupport: '0' octaviaSupport: '0' externalDNS: - 10.0.77.163 pullSecret: HIDDEN additionalTrustBundle: | -----BEGIN CERTIFICATE----- MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x EDAOBgNVBAgMB0JlaWppbmcxEDAOBgNVBAcMB0JlaWppbmcxDDAKBgNVBAoMA09D UDEPMA0GA1UECwwGT0NQLVFFMRcwFQYDVQQDDA5PQ1AtUUUtUk9PVC1DQTAeFw0x OTA4MTgwNjA4MzRaFw0yOTA4MTUwNjA4MzRaMF4xCzAJBgNVBAYTAkNOMRAwDgYD VQQIDAdCZWlqaW5nMQwwCgYDVQQKDANPQ1AxFTATBgNVBAsMDEluc3RhbGxlci1R RTEYMBYGA1UEAwwPSW5zdGFsbGVyLVFFLUNBMIICIjANBgkqhkiG9w0BAQEFAAOC Ag8AMIICCgKCAgEAwt0MujtrS6uPOx9pV71W5o0Nk9a6Fe4bSojyyOJw1SmDihaC AvxrWK3NHaqYV8cqQWLB1ZXtw8LF74BK98/b94PvauqgTn3Kg+Vcqnq3JlpyrgKN n5g4ORYScQXlyN/Kzn98cv07qHn1MhwZt8W8lYI9m6z2un0VyPkr8UgSmvDo0cx0 zwjB5Q7zCvXcoc1IQFa3JkYH4Z6Ccz9FNYnDRtoqu8K3SiWid50WEXcpycMLCSwb SVSDAsUR5wwA4aTgW7s32Fdd4fAtNcnfZ2AnLTwyJBZoPeoa5npvmpCr8khLyDdW Y9rWDfaKXhB++Ou27FDE6NLWQK/FPMVNPIr+P3xPbHIDlwzWq0eSK8SMsiOZrI9N dzMNGtcxv3sfxMYqKhnl3HrZbXbM1ouD9lsv5zGCAIdrnmZoMRI9NTjBatOevZXQ ojby2XQzNDX1ouQK4gSTi9q3aa1e8WQfiLbaNPxAU9FlLqS7J16nFsTsWQ6Qt6iN yEFaw3pYWeZk6sacGQECvmfrbaHxlI63rQUI3mRxs8mZqb3zJapcbNtUlimEAsqE 1oj/Tv3oVQKei2MpQHctenJqOZGC0Q/iWeRALD9E656MqbIt5dudEnx56Nq8av4r sad+OquDKFB/EnQ69VViYs9s6Ck426bqX5dx6T0Y0Tgk0WcnR5aPO+YrEtUCAwEA AaNmMGQwHQYDVR0OBBYEFJhUiRBfCjzfjHxoPLwYEwz5jHuuMB8GA1UdIwQYMBaA FG5nokgYqmIIwaW7blM6wHVIQwBIMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0P AQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBhcO+rA1blMP7SKt3/qzqsX5di BRxqOWqlmpKDgmC9rJts76t/PEodI2XNUVnKtybQD7Fh768b4fo0WO/evWUxs2LM 4d7jQp5KTqEPhv6oKlrTp9fzw3BGwdnzZSPk6L8ahZvyr0i7Hls9oe5Pvhy5F87e qWt/SuDMCztYR3gs78IxBYMv4BPEuCeLsvLlPFW4vl+4lpGjOGcS8GbwwZIwq5X4 LIdkk00NAMQ6Nmztoc+k/EVnj7O/bj66FY4WZFYUgnKUMlJ33UZy+Uao2GKUAM8j znFOl8fHgLYlcHsRYyLWeMGmOk0ukN06AvygnWh0UVBQCRrmTPNsShK+PlRyHmFW Zw4TDuPOqEwLx1VcmlEbLbpgc4f4GUWKGegaLHUltfwTwlb/6m1J4HomiYrBhdLJ LDReBo7dNYr7mpGPfZIMRdmywz6w10F1zTKe2F1KHb7mR7tyORaZ7NcAtmQmuxDF T8sUTrIop4GaQMZnNTPImtPGt23zsNTXUY93IeISJ6eUDKlnDgzYJDQ3pnKWbWHz wdWcyjh0Ojh/snItIm6/h1+CQ/FRlnt3+LRP9GxvWHbn1+sS51Kb979m/R0W7Djt y4p+AwCHpLwi9sU17Lg1JafgJVFB9Tu2wz/DIocfzdpP+7MUrqTkeDmN0p+Ia1Y9 bTSegOgySxp2uzPJqg== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIFuDCCA6CgAwIBAgIJAJk39xzKHHf9MA0GCSqGSIb3DQEBCwUAMGkxCzAJBgNV BAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMRAwDgYDVQQHDAdCZWlqaW5nMQwwCgYD VQQKDANPQ1AxDzANBgNVBAsMBk9DUC1RRTEXMBUGA1UEAwwOT0NQLVFFLVJPT1Qt Q0EwHhcNMTkwODE4MDYwNzU4WhcNMzkwODEzMDYwNzU4WjBpMQswCQYDVQQGEwJD TjEQMA4GA1UECAwHQmVpamluZzEQMA4GA1UEBwwHQmVpamluZzEMMAoGA1UECgwD T0NQMQ8wDQYDVQQLDAZPQ1AtUUUxFzAVBgNVBAMMDk9DUC1RRS1ST09ULUNBMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA338oV6IIvllZpr/lWOjMMVZq 4Smm0qA6BOe3ezZlr2LU5TLsgZeY+Oa1YtwXSAka8rRnuuqNa6gZEHGdL1SHTynB rEyq05KErChLabRVYb9aotQHt1+G1GG2Mi11QZ4Pdgsfmrs8NC05703C5V4kEL+q NXG88O3J54ySsKp+aD4xvOtZ0uXcVdjAo347/CJEm/2HF9C/uIR8ktJ43ZQPq55c tgsJjjY/UBSmOOhDsTfRzv9DVrcWuZYW0ZztG7gfC3d2i2l7dLhaAr76kzZ68aH2 402ghE1Xh9zDlmWugfqOyT/v6RsE7gL/Dkkuk27Eau3jyRdWVIJroqK2Sd/yJcrQ DiG1wAzwb7JVlPi5lkQBrWXti+qgm415+Xfcc9KRZP3hv3tbGVuKmNxONpGjbrMw GKV2EMWGnpdKepQ0STWb9SC916iNXO9ffCsPlqgEoV1ONiNfvU9G3cCcRcc1yjtF 8zbMcqmtsvl+AC1RfmM4n8TesSx56vk/obNsUljtU1/FGQIKRlamey4r/dKDR8kJ oyDibv7dUGm5pX5/L7bahRb7LoVg0MbV9bGlqL+hpCbjIO1rouMyy3qu3z+NMGh7 nzVYULulOjdbVw5u14O4VeonavWByyCFUMK4JKqfUOPNjjS7OEXue1HoCy9LBjIv qfPUdeulyX0OtbZ8EhECAwEAAaNjMGEwHQYDVR0OBBYEFG5nokgYqmIIwaW7blM6 wHVIQwBIMB8GA1UdIwQYMBaAFG5nokgYqmIIwaW7blM6wHVIQwBIMA8GA1UdEwEB /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQAcnBrb Cde2jE+iumzlN3TNm6nOvMnomIrMupBInuWI0GvA9rGjv8SC8ZAjfx/fZOY28uLx ACiZqKWQT0YARjKCgOSe0RxTG+vpNH6E8FpTEiVIq/N+rgdHCZUJiWoY7BA1FNNq 3UTlqV6RM+RqsVIptu8lk7fVDehng+zQzYYs4ZV6bSLjBQG3yBUBN1lYnFWe3pnS WmLuw22Riuunc5MVdH97modji1UDzQHDbYy0FXt8gLM8DRPIrOe039XO1lO+eWWM /NI7sZBU6bSotDh3aTLnHIyJdJ0dnh+/wMIK6h5au/7BMV1oK4JsSmpNCmzP+s3O cpNINYhkBRqFViA72D/Vim/meP2Q4J/dKsT2JbprY7X/XIYd1+aS48QAyusat2Gn KJ1JQNOoYHGijz8bYHm5JVytMIKU5LJ/Rp9SgK3d0ByqmJR76alzyRdUKa3Pmsw3 Beq8GQSAdjlyIB6C1FpG7XD4ySz1EjGEcOXiGiEi8l9wjDgLtA20U9ALaMcEdODY K8zhyirrdXdV8XHBAE7QBkzcuQAVc9iyTNoqCfJBtvl2HYpH2XoRhxP0rX9NtAYE Gc+Yc4Tgf2HAERrwj0B6AfWQaDfcjAJtQ0xorONJJpEZpItV8Cl5dSeOtX7howTB BvBHcmyVbaW7PGNBmIM1FBKwi/fBJoawSJlslA== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDpTCCAo2gAwIBAgIUUkVbKASTDq8lts1k6FBiXYJpZ6gwDQYJKoZIhvcNAQEL BQAwYjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdo MRAwDgYDVQQKDAdSZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2 OC4yNC4yMB4XDTIwMDIyMzE2MjgzOFoXDTIxMDIyMjE2MjgzOFowYjELMAkGA1UE BhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdS ZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2OC4yNC4yMIIBIjAN BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo8hDyhKXBpOcVfzP5ZnGi6T8W9P9 Hyol/ojaT958JoO5V7luoIgshPVA8feD9xMaT3QOfVDPgkAE4rMvpdohbCwx1aYY 8OeumzkEQQEezYmlY8XRNBXB+h0/+xO54GVXbtkekCDJ3Z5yeQyA6dB2mcf/NSF0 i3ox+BKVQryrpHvjSbkrb0dKQzAdPe59BuTyy7uphwhUB9K4EPuotUTDY19uPyMC 7Hg0/d/iEKgHcahbep/PTD05/Qxa9mPx3lqXr9YO5e1VSQFhG3cL5P+iDqtpksP7 Q62dBsbe16a6bTPH9OCJ3hl8lABGbfgEcNlbaEIA1e0TDIspgYz/p75/GQIDAQAB o1MwUTAdBgNVHQ4EFgQUROo72sMpfR09Yfs34dPtgInoAogwHwYDVR0jBBgwFoAU ROo72sMpfR09Yfs34dPtgInoAogwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B AQsFAAOCAQEAWYclYSNIGeSmk6nehhpwzi1vEpxvlxF//TLUo7N82y1zVXrH1zDS KsEOBJDem+Jotw/O4jEM2i84t3n3UyjFjfakKS7L0FjI8PXVGyRxmtT9AovT7+yZ iZR7BhbC3IWrXnZ8c3y+Ofxs75Ofb+QxF+z4TZMiqeRmkrysn6NTi+11Z90q3qE8 6KdhWGc2TGORBijRcJyKQEUbQ2b5SeEPlWV1yZbQWhJfASMbAAfi3ZsYwEUqi/8W ECzUQcqMAJLPaHM1NtVkCVcpAkKVOIu21dCC5QHQjkfVAz06/CtmCS0hjadbUlcp Y+xtFK2KtggdOWriZiNH/uUvVQglh1kLcg== -----END CERTIFICATE----- networking: clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 serviceNetwork: - 172.30.0.0/16 networkType: OpenShiftSDN machineNetwork: - cidr: 192.168.0.0/18 publish: External imageContentSources: - mirrors: - upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release source: quay.io/openshift-release-dev/ocp-v4.0-art-dev - mirrors: - upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release source: registry.svc.ci.openshift.org/ocp/release baseDomain: qe.devcluster.openshift.com sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa37P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D openshift-qe 2. Check the install output 3. Actual results: # openshift-install' wait-for install-complete --dir '/home/jenkins/workspace/Launch Environment Flexy/workdir/install-dir' --log-level debug ... evel=fatal msg="failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Failed to resync 4.4.0-0.nightly-2020-03-03-011650 because: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: configuration status for pool master is empty: pool is degraded because nodes fail with \"3 nodes are reporting degraded status on sync\": \"Node wjuos163037-lc6gf-master-0 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-1 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-2 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\"\", retrying" Expected results: MCO should not be degraded Additional info: our dns posioning configuration on externalDNS server: [root@v4-disconnected-registry ~]# cat /etc/dnsmasq.d/disconnected-dns.conf server=/gcr.io/192.168.199.99 server=/docker.io/192.168.199.99 server=/quay.io/192.168.199.99 #rhos-d.infra.prod.upshift.rdu2.redhat.com have to be in whitelist #server=/redhat.com/192.168.199.99 server=/redhat.io/192.168.199.99 server=/openshift.org/192.168.199.99 #apps url would ended with openshift.com, so do not block openshift.com server=/api.openshift.com/192.168.199.99 server=/grafana.com/192.168.199.99 server=/googleapis.com/192.168.199.99