Bug 1956937 - add configuration for disconnected registry ca still giving "x509: certificate signed by unknown authority"
Summary: add configuration for disconnected registry ca still giving "x509: certificat...
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.8
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: ---
Assignee: Antoni Segura Puimedon
QA Contact: Chad Crum
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-04 17:20 UTC by Chad Crum
Modified: 2021-05-10 14:14 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Chad Crum 2021-05-04 17:20:17 UTC
Description of problem:
When applying new mirrorReigstryRef to agentServiceConfig (with corresponding configMap containing disconnected registry CA), the assisted service is still failing to pull images from the registry stating: ""x509: certificate signed by unknown authority""

Version-Release number of selected component (if applicable):
Latest assisted service operator bundle / service image from date of this bz.

How reproducible:
100%

Steps to Reproduce:
1. Deploy assisted service operator in a disconnected ipv6 env with a disconnected image registry
2. Create configmap with registries.conf and disconnected registry ca, ex:
apiVersion: v1
data:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    MIIEnzCCA4egAwIBAgIUReZSesNwvO8QzPhaDhEo2W54PwUwDQYJKoZIhvcNAQEL
    BQAwgYgxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWln
    aDEVMBMGA1UECgwMVGVzdCBDb21wYW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYD
    VQQDDChzZWFsdXNhMTIubW9iaXVzLmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMB4X
    DTIxMDQyODAxNDE1MloXDTIyMDQyODAxNDE1MlowgYgxCzAJBgNVBAYTAlVTMQsw
    CQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEVMBMGA1UECgwMVGVzdCBDb21w
    YW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYDVQQDDChzZWFsdXNhMTIubW9iaXVz
    LmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
    MIIBCgKCAQEAzQyl4WJElWE3izEVIvUHGpcandkJclEGMgvvP9YaTMkp8E8MJC7p
    /PxnNFZozyaeXggepsEEwnCpgC3I0xXsdcneqMmoacWg9w2Vk6O0gprbeRC4uz49
    2WYGimsVmg+tfVj8AOBDQTpxc+N2TTDeTxftXdoljUnzKEw4SS87h2iO//jG4cpW
    QKBtEMVlnaGHWGk7R4MNfOTiCehhQdankCNxcKxvV42I3VS2+mFS0qpVLOQfN5bK
    9LOTa+pclUSL56IaTLzzAM8iM6OBbieno8RuG2TwTpWmXXzsGgJ/DsUGi6FmlE+X
    y1cXjWdaDrudE2zzHviyQ+vf6gFFFNL/YwIDAQABo4H+MIH7MA8GA1UdEwQIMAYB
    Af8CAQAwgecGA1UdEQSB3zCB3IItcmVnaXN0cnkub2NwLWVkZ2UtY2x1c3Rlci0w
    LnFlLmxhYi5yZWRoYXQuY29tgglzZWFsdXNhMTKHEBABDbkAAAAAAAAAAAAAAAGH
    EP6AAAAAAAAAUFQA//7Nok+HEP6AAAAAAAAA/FQA//6/W2GHEP6AAAAAAAAADsR6
    //7Skb6HEP0ub0Rd2AAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//57hnKHEP6AAAAA
    AAAAehCwqjYaVTiHEBABDbkAAAAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//6LAcgw
    DQYJKoZIhvcNAQELBQADggEBAJgwEuObMzjjh1KqEKHuoSgGJl7byxK76W1py1Yh
    NJctqkMZtHEUzhcipT80RhhpKYREnEXlX2r7VDe8aSVa4XnJ9h3CbNT8RrXmW40T
    wdo49QG/h0n17I1oilNaz44pReGKg8F/p6DSr1++ElCh+Cj8354EsS+QCTJ8Ciqq
    Ycyrev1TRu1Bh9PqWP9WbnFyOk1UG3AV/Ad97nMvOODuGFFCY2V6g/pUOJDRoJCh
    8lmxh7gBcGTxZazWNMbw6Mbu6lBSYmpsUhh64uNhNvUC3b0kDepT6NB5/mPmy4jj
    aiWA55l1J0Ho5/kyPv1YK+9aPBzd8yBHY09nxxBoRbd0Lq8=
    -----END CERTIFICATE-----
  registries.conf: |
    unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

    [[registry]]
      prefix = ""
      location = "quay.io/openshift-release-dev/ocp-release"
      mirror-by-digest-only = true

kind: ConfigMap
metadata:
  labels:
    app: assisted-service
  name: mirror-registry-ca
  namespace: assisted-installer

3. Check assisted service pod assisted service container logs
oc logs -f assisted-service-979c79679-cxgbg  -c assisted-service

Actual results:
Repeated failed attempts to pull release image, but x509 auth failing:
error msg="failed to add OCP version" func="github.com/openshift/assisted-service/internal/controller/controllers.(*ClusterDeploymentsReconciler).createNewCluster" file="/go/src/github.com/openshift/origin/internal/controller/controllers/clusterdeployments_controller.go:546" error="command oc adm release info -o template --template '{{.metadata.version}}' --insecure=false registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.8.0-fc.1-x86_64 exited with non-zero exit code 1: \nerror: unable to connect to image repository registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.8.0-fc.1-x86_64: Get \"https://registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/v2/\": x509: certificate signed by unknown authority\n"

Expected results:
Able to pull release image without errir

Additional info:
If I use the same configmap, but replace mirror_ca.pem with tls-ca-bundle.pem, the assisted service pod comes up correctly (This was used in the original work around). I'm not sure what the difference is between the two names. 

Here is a diff of the relevant working and non-working assisted-service deployment objects:

      - configMap:						      - configMap:
          defaultMode: 420					          defaultMode: 420
          items:						          items:
          - key: ca-bundle.crt					          - key: ca-bundle.crt
            path: tls-ca-bundle.pem			      |	            path: mirror_ca.pem
          name: mirror-registry-ca				          name: mirror-registry-ca
          optional: true					          optional: true
        name: mirror-registry-ca				        name: mirror-registry-ca

        - mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-b |	        - mountPath: /etc/pki/ca-trust/extracted/pem/mirror_c
          name: mirror-registry-ca				          name: mirror-registry-ca
          subPath: tls-ca-bundle.pem			      |	          subPath: mirror_ca.pem


Another note is that I also tried excluding the registries.conf info and nothing changed.

Comment 1 Chad Crum 2021-05-04 17:34:48 UTC
Related to merged pr https://github.com/openshift/assisted-service/pull/1563/files

Comment 3 Chad Crum 2021-05-10 14:14:56 UTC
I validated that I can set a disconnected registry ca using this commit [1].

I'm able to deploy assisted service using the mirror registry change included in the above commit [2].


No x509 errors in logs with this change.


This bz is related to [3].


[1] https://github.com/openshift/assisted-service/commit/d42b8c859f82eb4938142dd0ca32d7357df435af

[2] https://github.com/openshift/assisted-service/blob/master/docs/operator.md#mirror-registry-configuration

[3] https://bugzilla.redhat.com/show_bug.cgi?id=1943558


Note You need to log in before you can comment on or make changes to this bug.