Hide Forgot
Description of problem: When applying new mirrorReigstryRef to agentServiceConfig (with corresponding configMap containing disconnected registry CA), the assisted service is still failing to pull images from the registry stating: ""x509: certificate signed by unknown authority"" Version-Release number of selected component (if applicable): Latest assisted service operator bundle / service image from date of this bz. How reproducible: 100% Steps to Reproduce: 1. Deploy assisted service operator in a disconnected ipv6 env with a disconnected image registry 2. Create configmap with registries.conf and disconnected registry ca, ex: apiVersion: v1 data: ca-bundle.crt: | -----BEGIN CERTIFICATE----- MIIEnzCCA4egAwIBAgIUReZSesNwvO8QzPhaDhEo2W54PwUwDQYJKoZIhvcNAQEL BQAwgYgxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWln aDEVMBMGA1UECgwMVGVzdCBDb21wYW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYD VQQDDChzZWFsdXNhMTIubW9iaXVzLmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMB4X DTIxMDQyODAxNDE1MloXDTIyMDQyODAxNDE1MlowgYgxCzAJBgNVBAYTAlVTMQsw CQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEVMBMGA1UECgwMVGVzdCBDb21w YW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYDVQQDDChzZWFsdXNhMTIubW9iaXVz LmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAzQyl4WJElWE3izEVIvUHGpcandkJclEGMgvvP9YaTMkp8E8MJC7p /PxnNFZozyaeXggepsEEwnCpgC3I0xXsdcneqMmoacWg9w2Vk6O0gprbeRC4uz49 2WYGimsVmg+tfVj8AOBDQTpxc+N2TTDeTxftXdoljUnzKEw4SS87h2iO//jG4cpW QKBtEMVlnaGHWGk7R4MNfOTiCehhQdankCNxcKxvV42I3VS2+mFS0qpVLOQfN5bK 9LOTa+pclUSL56IaTLzzAM8iM6OBbieno8RuG2TwTpWmXXzsGgJ/DsUGi6FmlE+X y1cXjWdaDrudE2zzHviyQ+vf6gFFFNL/YwIDAQABo4H+MIH7MA8GA1UdEwQIMAYB Af8CAQAwgecGA1UdEQSB3zCB3IItcmVnaXN0cnkub2NwLWVkZ2UtY2x1c3Rlci0w LnFlLmxhYi5yZWRoYXQuY29tgglzZWFsdXNhMTKHEBABDbkAAAAAAAAAAAAAAAGH EP6AAAAAAAAAUFQA//7Nok+HEP6AAAAAAAAA/FQA//6/W2GHEP6AAAAAAAAADsR6 //7Skb6HEP0ub0Rd2AAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//57hnKHEP6AAAAA AAAAehCwqjYaVTiHEBABDbkAAAAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//6LAcgw DQYJKoZIhvcNAQELBQADggEBAJgwEuObMzjjh1KqEKHuoSgGJl7byxK76W1py1Yh NJctqkMZtHEUzhcipT80RhhpKYREnEXlX2r7VDe8aSVa4XnJ9h3CbNT8RrXmW40T wdo49QG/h0n17I1oilNaz44pReGKg8F/p6DSr1++ElCh+Cj8354EsS+QCTJ8Ciqq Ycyrev1TRu1Bh9PqWP9WbnFyOk1UG3AV/Ad97nMvOODuGFFCY2V6g/pUOJDRoJCh 8lmxh7gBcGTxZazWNMbw6Mbu6lBSYmpsUhh64uNhNvUC3b0kDepT6NB5/mPmy4jj aiWA55l1J0Ho5/kyPv1YK+9aPBzd8yBHY09nxxBoRbd0Lq8= -----END CERTIFICATE----- registries.conf: | unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] [[registry]] prefix = "" location = "quay.io/openshift-release-dev/ocp-release" mirror-by-digest-only = true kind: ConfigMap metadata: labels: app: assisted-service name: mirror-registry-ca namespace: assisted-installer 3. Check assisted service pod assisted service container logs oc logs -f assisted-service-979c79679-cxgbg -c assisted-service Actual results: Repeated failed attempts to pull release image, but x509 auth failing: error msg="failed to add OCP version" func="github.com/openshift/assisted-service/internal/controller/controllers.(*ClusterDeploymentsReconciler).createNewCluster" file="/go/src/github.com/openshift/origin/internal/controller/controllers/clusterdeployments_controller.go:546" error="command oc adm release info -o template --template '{{.metadata.version}}' --insecure=false registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.8.0-fc.1-x86_64 exited with non-zero exit code 1: \nerror: unable to connect to image repository registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.8.0-fc.1-x86_64: Get \"https://registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/v2/\": x509: certificate signed by unknown authority\n" Expected results: Able to pull release image without errir Additional info: If I use the same configmap, but replace mirror_ca.pem with tls-ca-bundle.pem, the assisted service pod comes up correctly (This was used in the original work around). I'm not sure what the difference is between the two names. Here is a diff of the relevant working and non-working assisted-service deployment objects: - configMap: - configMap: defaultMode: 420 defaultMode: 420 items: items: - key: ca-bundle.crt - key: ca-bundle.crt path: tls-ca-bundle.pem | path: mirror_ca.pem name: mirror-registry-ca name: mirror-registry-ca optional: true optional: true name: mirror-registry-ca name: mirror-registry-ca - mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-b | - mountPath: /etc/pki/ca-trust/extracted/pem/mirror_c name: mirror-registry-ca name: mirror-registry-ca subPath: tls-ca-bundle.pem | subPath: mirror_ca.pem Another note is that I also tried excluding the registries.conf info and nothing changed.
Related to merged pr https://github.com/openshift/assisted-service/pull/1563/files
I validated that I can set a disconnected registry ca using this commit [1]. I'm able to deploy assisted service using the mirror registry change included in the above commit [2]. No x509 errors in logs with this change. This bz is related to [3]. [1] https://github.com/openshift/assisted-service/commit/d42b8c859f82eb4938142dd0ca32d7357df435af [2] https://github.com/openshift/assisted-service/blob/master/docs/operator.md#mirror-registry-configuration [3] https://bugzilla.redhat.com/show_bug.cgi?id=1943558