Bug 1809921 - [UPI on self-signed OSP16] MCO is degraded due to nodes are reporting machineconfig not found
Summary: [UPI on self-signed OSP16] MCO is degraded due to nodes are reporting machine...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Martin André
QA Contact: David Sanz
URL:
Whiteboard:
: 1810980 (view as bug list)
Depends On:
Blocks: 1813822
TreeView+ depends on / blocked
 
Reported: 2020-03-04 08:13 UTC by weiwei jiang
Modified: 2020-07-13 17:18 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1813822 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:17:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
mcs-machine-config-content.json on master-0 (154.66 KB, text/plain)
2020-03-05 03:23 UTC, weiwei jiang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 3290 0 None closed Bug 1809921: Openstack UPI: Document use with Self-signed cert 2020-09-09 22:36:43 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:18:17 UTC

Description weiwei jiang 2020-03-04 08:13:20 UTC
Description of problem:
When trying to setup a restricted networking UPI on OSP16, got the following error:
# openshift-install' wait-for install-complete --dir '/home/jenkins/workspace/Launch Environment Flexy/workdir/install-dir' --log-level debug
evel=fatal msg="failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Failed to resync 4.4.0-0.nightly-2020-03-03-011650 because: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: configuration status for pool master is empty: pool is degraded because nodes fail with \"3 nodes are reporting degraded status on sync\": \"Node wjuos163037-lc6gf-master-0 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-1 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-2 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\"\", retrying"

Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-03-03-011650:

How reproducible:
Always

Steps to Reproduce:
1. create restricted UPI on OSP16 cluster with install-config.yaml like following, we use dns posioning to construct a restricted networking
---
apiVersion: v1
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  replicas: 3
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  replicas: 0
metadata:
  name: wjuos163037
platform:
  openstack:
    cloud: openstack
    computeFlavor: m4.xlarge
    externalNetwork: nova
    lbFloatingIP: 10.8.100.242
    region: regionOne
    trunkSupport: '0'
    octaviaSupport: '0'
    externalDNS:
    - 10.0.77.163
pullSecret: HIDDEN
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIFqTCCA5GgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwaTELMAkGA1UEBhMCQ04x
  EDAOBgNVBAgMB0JlaWppbmcxEDAOBgNVBAcMB0JlaWppbmcxDDAKBgNVBAoMA09D
  UDEPMA0GA1UECwwGT0NQLVFFMRcwFQYDVQQDDA5PQ1AtUUUtUk9PVC1DQTAeFw0x
  OTA4MTgwNjA4MzRaFw0yOTA4MTUwNjA4MzRaMF4xCzAJBgNVBAYTAkNOMRAwDgYD
  VQQIDAdCZWlqaW5nMQwwCgYDVQQKDANPQ1AxFTATBgNVBAsMDEluc3RhbGxlci1R
  RTEYMBYGA1UEAwwPSW5zdGFsbGVyLVFFLUNBMIICIjANBgkqhkiG9w0BAQEFAAOC
  Ag8AMIICCgKCAgEAwt0MujtrS6uPOx9pV71W5o0Nk9a6Fe4bSojyyOJw1SmDihaC
  AvxrWK3NHaqYV8cqQWLB1ZXtw8LF74BK98/b94PvauqgTn3Kg+Vcqnq3JlpyrgKN
  n5g4ORYScQXlyN/Kzn98cv07qHn1MhwZt8W8lYI9m6z2un0VyPkr8UgSmvDo0cx0
  zwjB5Q7zCvXcoc1IQFa3JkYH4Z6Ccz9FNYnDRtoqu8K3SiWid50WEXcpycMLCSwb
  SVSDAsUR5wwA4aTgW7s32Fdd4fAtNcnfZ2AnLTwyJBZoPeoa5npvmpCr8khLyDdW
  Y9rWDfaKXhB++Ou27FDE6NLWQK/FPMVNPIr+P3xPbHIDlwzWq0eSK8SMsiOZrI9N
  dzMNGtcxv3sfxMYqKhnl3HrZbXbM1ouD9lsv5zGCAIdrnmZoMRI9NTjBatOevZXQ
  ojby2XQzNDX1ouQK4gSTi9q3aa1e8WQfiLbaNPxAU9FlLqS7J16nFsTsWQ6Qt6iN
  yEFaw3pYWeZk6sacGQECvmfrbaHxlI63rQUI3mRxs8mZqb3zJapcbNtUlimEAsqE
  1oj/Tv3oVQKei2MpQHctenJqOZGC0Q/iWeRALD9E656MqbIt5dudEnx56Nq8av4r
  sad+OquDKFB/EnQ69VViYs9s6Ck426bqX5dx6T0Y0Tgk0WcnR5aPO+YrEtUCAwEA
  AaNmMGQwHQYDVR0OBBYEFJhUiRBfCjzfjHxoPLwYEwz5jHuuMB8GA1UdIwQYMBaA
  FG5nokgYqmIIwaW7blM6wHVIQwBIMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0P
  AQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBhcO+rA1blMP7SKt3/qzqsX5di
  BRxqOWqlmpKDgmC9rJts76t/PEodI2XNUVnKtybQD7Fh768b4fo0WO/evWUxs2LM
  4d7jQp5KTqEPhv6oKlrTp9fzw3BGwdnzZSPk6L8ahZvyr0i7Hls9oe5Pvhy5F87e
  qWt/SuDMCztYR3gs78IxBYMv4BPEuCeLsvLlPFW4vl+4lpGjOGcS8GbwwZIwq5X4
  LIdkk00NAMQ6Nmztoc+k/EVnj7O/bj66FY4WZFYUgnKUMlJ33UZy+Uao2GKUAM8j
  znFOl8fHgLYlcHsRYyLWeMGmOk0ukN06AvygnWh0UVBQCRrmTPNsShK+PlRyHmFW
  Zw4TDuPOqEwLx1VcmlEbLbpgc4f4GUWKGegaLHUltfwTwlb/6m1J4HomiYrBhdLJ
  LDReBo7dNYr7mpGPfZIMRdmywz6w10F1zTKe2F1KHb7mR7tyORaZ7NcAtmQmuxDF
  T8sUTrIop4GaQMZnNTPImtPGt23zsNTXUY93IeISJ6eUDKlnDgzYJDQ3pnKWbWHz
  wdWcyjh0Ojh/snItIm6/h1+CQ/FRlnt3+LRP9GxvWHbn1+sS51Kb979m/R0W7Djt
  y4p+AwCHpLwi9sU17Lg1JafgJVFB9Tu2wz/DIocfzdpP+7MUrqTkeDmN0p+Ia1Y9
  bTSegOgySxp2uzPJqg==
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  MIIFuDCCA6CgAwIBAgIJAJk39xzKHHf9MA0GCSqGSIb3DQEBCwUAMGkxCzAJBgNV
  BAYTAkNOMRAwDgYDVQQIDAdCZWlqaW5nMRAwDgYDVQQHDAdCZWlqaW5nMQwwCgYD
  VQQKDANPQ1AxDzANBgNVBAsMBk9DUC1RRTEXMBUGA1UEAwwOT0NQLVFFLVJPT1Qt
  Q0EwHhcNMTkwODE4MDYwNzU4WhcNMzkwODEzMDYwNzU4WjBpMQswCQYDVQQGEwJD
  TjEQMA4GA1UECAwHQmVpamluZzEQMA4GA1UEBwwHQmVpamluZzEMMAoGA1UECgwD
  T0NQMQ8wDQYDVQQLDAZPQ1AtUUUxFzAVBgNVBAMMDk9DUC1RRS1ST09ULUNBMIIC
  IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA338oV6IIvllZpr/lWOjMMVZq
  4Smm0qA6BOe3ezZlr2LU5TLsgZeY+Oa1YtwXSAka8rRnuuqNa6gZEHGdL1SHTynB
  rEyq05KErChLabRVYb9aotQHt1+G1GG2Mi11QZ4Pdgsfmrs8NC05703C5V4kEL+q
  NXG88O3J54ySsKp+aD4xvOtZ0uXcVdjAo347/CJEm/2HF9C/uIR8ktJ43ZQPq55c
  tgsJjjY/UBSmOOhDsTfRzv9DVrcWuZYW0ZztG7gfC3d2i2l7dLhaAr76kzZ68aH2
  402ghE1Xh9zDlmWugfqOyT/v6RsE7gL/Dkkuk27Eau3jyRdWVIJroqK2Sd/yJcrQ
  DiG1wAzwb7JVlPi5lkQBrWXti+qgm415+Xfcc9KRZP3hv3tbGVuKmNxONpGjbrMw
  GKV2EMWGnpdKepQ0STWb9SC916iNXO9ffCsPlqgEoV1ONiNfvU9G3cCcRcc1yjtF
  8zbMcqmtsvl+AC1RfmM4n8TesSx56vk/obNsUljtU1/FGQIKRlamey4r/dKDR8kJ
  oyDibv7dUGm5pX5/L7bahRb7LoVg0MbV9bGlqL+hpCbjIO1rouMyy3qu3z+NMGh7
  nzVYULulOjdbVw5u14O4VeonavWByyCFUMK4JKqfUOPNjjS7OEXue1HoCy9LBjIv
  qfPUdeulyX0OtbZ8EhECAwEAAaNjMGEwHQYDVR0OBBYEFG5nokgYqmIIwaW7blM6
  wHVIQwBIMB8GA1UdIwQYMBaAFG5nokgYqmIIwaW7blM6wHVIQwBIMA8GA1UdEwEB
  /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQAcnBrb
  Cde2jE+iumzlN3TNm6nOvMnomIrMupBInuWI0GvA9rGjv8SC8ZAjfx/fZOY28uLx
  ACiZqKWQT0YARjKCgOSe0RxTG+vpNH6E8FpTEiVIq/N+rgdHCZUJiWoY7BA1FNNq
  3UTlqV6RM+RqsVIptu8lk7fVDehng+zQzYYs4ZV6bSLjBQG3yBUBN1lYnFWe3pnS
  WmLuw22Riuunc5MVdH97modji1UDzQHDbYy0FXt8gLM8DRPIrOe039XO1lO+eWWM
  /NI7sZBU6bSotDh3aTLnHIyJdJ0dnh+/wMIK6h5au/7BMV1oK4JsSmpNCmzP+s3O
  cpNINYhkBRqFViA72D/Vim/meP2Q4J/dKsT2JbprY7X/XIYd1+aS48QAyusat2Gn
  KJ1JQNOoYHGijz8bYHm5JVytMIKU5LJ/Rp9SgK3d0ByqmJR76alzyRdUKa3Pmsw3
  Beq8GQSAdjlyIB6C1FpG7XD4ySz1EjGEcOXiGiEi8l9wjDgLtA20U9ALaMcEdODY
  K8zhyirrdXdV8XHBAE7QBkzcuQAVc9iyTNoqCfJBtvl2HYpH2XoRhxP0rX9NtAYE
  Gc+Yc4Tgf2HAERrwj0B6AfWQaDfcjAJtQ0xorONJJpEZpItV8Cl5dSeOtX7howTB
  BvBHcmyVbaW7PGNBmIM1FBKwi/fBJoawSJlslA==
  -----END CERTIFICATE-----

  -----BEGIN CERTIFICATE-----
  MIIDpTCCAo2gAwIBAgIUUkVbKASTDq8lts1k6FBiXYJpZ6gwDQYJKoZIhvcNAQEL
  BQAwYjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdo
  MRAwDgYDVQQKDAdSZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2
  OC4yNC4yMB4XDTIwMDIyMzE2MjgzOFoXDTIxMDIyMjE2MjgzOFowYjELMAkGA1UE
  BhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdS
  ZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2OC4yNC4yMIIBIjAN
  BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo8hDyhKXBpOcVfzP5ZnGi6T8W9P9
  Hyol/ojaT958JoO5V7luoIgshPVA8feD9xMaT3QOfVDPgkAE4rMvpdohbCwx1aYY
  8OeumzkEQQEezYmlY8XRNBXB+h0/+xO54GVXbtkekCDJ3Z5yeQyA6dB2mcf/NSF0
  i3ox+BKVQryrpHvjSbkrb0dKQzAdPe59BuTyy7uphwhUB9K4EPuotUTDY19uPyMC
  7Hg0/d/iEKgHcahbep/PTD05/Qxa9mPx3lqXr9YO5e1VSQFhG3cL5P+iDqtpksP7
  Q62dBsbe16a6bTPH9OCJ3hl8lABGbfgEcNlbaEIA1e0TDIspgYz/p75/GQIDAQAB
  o1MwUTAdBgNVHQ4EFgQUROo72sMpfR09Yfs34dPtgInoAogwHwYDVR0jBBgwFoAU
  ROo72sMpfR09Yfs34dPtgInoAogwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0B
  AQsFAAOCAQEAWYclYSNIGeSmk6nehhpwzi1vEpxvlxF//TLUo7N82y1zVXrH1zDS
  KsEOBJDem+Jotw/O4jEM2i84t3n3UyjFjfakKS7L0FjI8PXVGyRxmtT9AovT7+yZ
  iZR7BhbC3IWrXnZ8c3y+Ofxs75Ofb+QxF+z4TZMiqeRmkrysn6NTi+11Z90q3qE8
  6KdhWGc2TGORBijRcJyKQEUbQ2b5SeEPlWV1yZbQWhJfASMbAAfi3ZsYwEUqi/8W
  ECzUQcqMAJLPaHM1NtVkCVcpAkKVOIu21dCC5QHQjkfVAz06/CtmCS0hjadbUlcp
  Y+xtFK2KtggdOWriZiNH/uUvVQglh1kLcg==
  -----END CERTIFICATE-----
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  serviceNetwork:
  - 172.30.0.0/16
  networkType: OpenShiftSDN
  machineNetwork:
  - cidr: 192.168.0.0/18
publish: External
imageContentSources:
- mirrors:
  - upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release
  source: registry.svc.ci.openshift.org/ocp/release
baseDomain: qe.devcluster.openshift.com
sshKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCWkwurd8TNAi+D7ffvyDdhGBSQtJx3/Yedlwvvha0q772vLlOAGlKCw4dajKy6qty1/GGQDgTJ17h3C9TEArI8ZqILnyydeY56DL+ELN3dtGBVof/N2qtW0+SmEnd1Mi7Qy5Tx4e/GVmB3NgX9szwNOVXhebzgBsXc9x+RtCVLPLC8J+qqSdTUZ0UfJsh2ptlQLGHmmTpF//QlJ1tngvAFeCOxJUhrLAa37P9MtFsiNk31EfKyBk3eIdZljTERmqFaoJCohsFFEdO7tVgU6p5NwniAyBGZVjZBzjELoI1aZ+/g9yReIScxl1R6PWqEzcU6lGo2hInnb6nuZFGb+90D
  openshift-qe
2. Check the install output
3.

Actual results:
# openshift-install' wait-for install-complete --dir '/home/jenkins/workspace/Launch Environment Flexy/workdir/install-dir' --log-level debug
...
evel=fatal msg="failed to initialize the cluster: Cluster operator machine-config is reporting a failure: Failed to resync 4.4.0-0.nightly-2020-03-03-011650 because: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: configuration status for pool master is empty: pool is degraded because nodes fail with \"3 nodes are reporting degraded status on sync\": \"Node wjuos163037-lc6gf-master-0 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-1 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\", Node wjuos163037-lc6gf-master-2 is reporting: \\\"machineconfig.machineconfiguration.openshift.io \\\\\\\"rendered-master-ffeeb07b30522b801c8e9cf20f73c023\\\\\\\" not found\\\"\", retrying"

Expected results:
MCO should not be degraded

Additional info:

our dns posioning configuration on externalDNS server:
[root@v4-disconnected-registry ~]# cat /etc/dnsmasq.d/disconnected-dns.conf 
server=/gcr.io/192.168.199.99
server=/docker.io/192.168.199.99
server=/quay.io/192.168.199.99
#rhos-d.infra.prod.upshift.rdu2.redhat.com have to be in whitelist
#server=/redhat.com/192.168.199.99
server=/redhat.io/192.168.199.99
server=/openshift.org/192.168.199.99
#apps url would ended with openshift.com, so do not block openshift.com
server=/api.openshift.com/192.168.199.99
server=/grafana.com/192.168.199.99
server=/googleapis.com/192.168.199.99

Comment 8 weiwei jiang 2020-03-05 03:23:44 UTC
Created attachment 1667629 [details]
mcs-machine-config-content.json on master-0

Add mcs-machine-config-content.json as attachments.

$ oc get machineconfig
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                                   c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
00-worker                                                   c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
01-master-container-runtime                                 c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
01-master-kubelet                                           c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
01-worker-container-runtime                                 c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
01-worker-kubelet                                           c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
99-master-6d03836a-379b-4d9f-925a-3bb684ce2c01-registries   c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
99-master-ssh                                                                                          2.2.0             39h
99-worker-cb9ef1b5-6e5d-4b6f-b7c5-b8bf2348bb24-registries   c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
99-worker-ssh                                                                                          2.2.0             39h
rendered-master-2ff3968f86f4dadbbf2118dde83da8ad            c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
rendered-master-b0bfe7cf925981b79f6641233b38a1d0            c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             20h
rendered-master-e938388901494b038eb7dfe835a2f4b4            c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
rendered-worker-4dc1d4af4078f2921cc7a57e2b96f71f            c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
rendered-worker-c2c8c5f9a6b4e405b0694fedc294abe6            c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             39h
rendered-worker-f7c06bfb75d1f8cb0eb8b3d3c3541333            c0e0c323241932282de12bc9fc5fe5ef5696092d   2.2.0             20h

Comment 11 Antonio Murdaca 2020-03-05 11:14:58 UTC
The diff between the very first MCO-generated rendered-master MC and the bootstrap generated rendered-master MC is this:

```
<           source: data:,
---
>           source: data:,-----BEGIN%20CERTIFICATE-----%0AMIIDpTCCAo2gAwIBAgIUUkVbKASTDq8lts1k6FBiXYJpZ6gwDQYJKoZIhvcNAQEL%0ABQAwYjELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdo%0AMRAwDgYDVQQKDAdSZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2%0AOC4yNC4yMB4XDTIwMDIyMzE2MjgzOFoXDTIxMDIyMjE2MjgzOFowYjELMAkGA1UE%0ABhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdS%0AZWQgSEF0MQswCQYDVQQLDAJRRTEVMBMGA1UEAwwMMTkyLjE2OC4yNC4yMIIBIjAN%0ABgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAo8hDyhKXBpOcVfzP5ZnGi6T8W9P9%0AHyol%2FojaT958JoO5V7luoIgshPVA8feD9xMaT3QOfVDPgkAE4rMvpdohbCwx1aYY%0A8OeumzkEQQEezYmlY8XRNBXB%2Bh0%2F%2BxO54GVXbtkekCDJ3Z5yeQyA6dB2mcf%2FNSF0%0Ai3ox%2BBKVQryrpHvjSbkrb0dKQzAdPe59BuTyy7uphwhUB9K4EPuotUTDY19uPyMC%0A7Hg0%2Fd%2FiEKgHcahbep%2FPTD05%2FQxa9mPx3lqXr9YO5e1VSQFhG3cL5P%2BiDqtpksP7%0AQ62dBsbe16a6bTPH9OCJ3hl8lABGbfgEcNlbaEIA1e0TDIspgYz%2Fp75%2FGQIDAQAB%0Ao1MwUTAdBgNVHQ4EFgQUROo72sMpfR09Yfs34dPtgInoAogwHwYDVR0jBBgwFoAU%0AROo72sMpfR09Yfs34dPtgInoAogwDwYDVR0TAQH%2FBAUwAwEB%2FzANBgkqhkiG9w0B%0AAQsFAAOCAQEAWYclYSNIGeSmk6nehhpwzi1vEpxvlxF%2F%2FTLUo7N82y1zVXrH1zDS%0AKsEOBJDem%2BJotw%2FO4jEM2i84t3n3UyjFjfakKS7L0FjI8PXVGyRxmtT9AovT7%2ByZ%0AiZR7BhbC3IWrXnZ8c3y%2BOfxs75Ofb%2BQxF%2Bz4TZMiqeRmkrysn6NTi%2B11Z90q3qE8%0A6KdhWGc2TGORBijRcJyKQEUbQ2b5SeEPlWV1yZbQWhJfASMbAAfi3ZsYwEUqi%2F8W%0AECzUQcqMAJLPaHM1NtVkCVcpAkKVOIu21dCC5QHQjkfVAz06%2FCtmCS0hjadbUlcp%0AY%2BxtFK2KtggdOWriZiNH%2FuUvVQglh1kLcg%3D%3D%0A-----END%20CERTIFICATE-----%0A
309c314
<           source: data:text/plain,unqualified-search-registries%20%3D%20%5B%22registry.access.redhat.com%22%2C%20%22docker.io%22%5D%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22quay.io%2Fopenshift-release-dev%2Focp-v4.0-art-dev%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22registry.svc.ci.openshift.org%2Focp%2Frelease%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A
---
>           source: data:text/plain,unqualified-search-registries%20%3D%20%5B%22registry.access.redhat.com%22%2C%20%22docker.io%22%5D%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22quay.io%2Fopenshift-release-dev%2Focp-v4.0-art-dev%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22quay.io%2Fopenshift%2Forigin-must-gather%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Fopenshift%2Forigin-must-gather%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22registry.redhat.io%2Frhel7%2Fsupport-tools%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Frhel7%2Fsupport-tools%22%0A%0A%5B%5Bregistry%5D%5D%0A%20%20prefix%20%3D%20%22%22%0A%20%20location%20%3D%20%22registry.svc.ci.openshift.org%2Focp%2Frelease%22%0A%20%20mirror-by-digest-only%20%3D%20true%0A%0A%20%20%5B%5Bregistry.mirror%5D%5D%0A%20%20%20%20location%20%3D%20%22upshift.mirror-registry.qe.devcluster.openshift.com%3A5000%2Focp%2Frelease%22%0A
```

It looks like something is weird between the registries.conf content, not sure what's going on there, I'll move to Node for Miloslav/Urvashi to investigate.

What's basically happening is bootstrap generates an MC with a given content of registries.conf but when the MCO kicks in, you can see it's adding something which makes it differ

Is this fully reproducible?


CC'ing Miloslav as well who worked on this

Comment 12 Miloslav Trmač 2020-03-05 20:21:06 UTC
Before:
> unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
> 
> [[registry]]
>   prefix = ""
>   location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
>   mirror-by-digest-only = true
> 
>   [[registry.mirror]]
>     location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release"
> 
> [[registry]]
>   prefix = ""
>   location = "registry.svc.ci.openshift.org/ocp/release"
>   mirror-by-digest-only = true
> 
>   [[registry.mirror]]
>     location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release"

After:
> unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
> 
> [[registry]]
>   prefix = ""
>   location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
>   mirror-by-digest-only = true
> 
>   [[registry.mirror]]
>     location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release"
> 
> [[registry]]
>   prefix = ""
>   location = "quay.io/openshift/origin-must-gather"
>   mirror-by-digest-only = true
> 
>   [[registry.mirror]]
>     location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/openshift/origin-must-gather"
> 
> [[registry]]
>   prefix = ""
>   location = "registry.redhat.io/rhel7/support-tools"
>   mirror-by-digest-only = true
> 
>   [[registry.mirror]]
>     location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/rhel7/support-tools"
> 
> [[registry]]
>   prefix = ""
>   location = "registry.svc.ci.openshift.org/ocp/release"
>   mirror-by-digest-only = true
> 
>   [[registry.mirror]]
>     location = "upshift.mirror-registry.qe.devcluster.openshift.com:5000/ocp/release"

Is this really just a difference between bootstrap and first run? It looks far more likely like an effect of the debugging attempts from comment#4.

Comment 30 Martin André 2020-03-12 15:32:47 UTC
*** Bug 1810980 has been marked as a duplicate of this bug. ***

Comment 36 errata-xmlrpc 2020-07-13 17:17:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.