Bug 1956937 - add configuration for disconnected registry ca still giving "x509: certificate signed by unknown authority"
Summary: add configuration for disconnected registry ca still giving "x509: certificat...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.8
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: ---
Assignee: yevgeny shnaidman
QA Contact: Chad Crum
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-04 17:20 UTC by Chad Crum
Modified: 2023-09-15 01:33 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-28 08:45:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Chad Crum 2021-05-04 17:20:17 UTC
Description of problem:
When applying new mirrorReigstryRef to agentServiceConfig (with corresponding configMap containing disconnected registry CA), the assisted service is still failing to pull images from the registry stating: ""x509: certificate signed by unknown authority""

Version-Release number of selected component (if applicable):
Latest assisted service operator bundle / service image from date of this bz.

How reproducible:
100%

Steps to Reproduce:
1. Deploy assisted service operator in a disconnected ipv6 env with a disconnected image registry
2. Create configmap with registries.conf and disconnected registry ca, ex:
apiVersion: v1
data:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    MIIEnzCCA4egAwIBAgIUReZSesNwvO8QzPhaDhEo2W54PwUwDQYJKoZIhvcNAQEL
    BQAwgYgxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWln
    aDEVMBMGA1UECgwMVGVzdCBDb21wYW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYD
    VQQDDChzZWFsdXNhMTIubW9iaXVzLmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMB4X
    DTIxMDQyODAxNDE1MloXDTIyMDQyODAxNDE1MlowgYgxCzAJBgNVBAYTAlVTMQsw
    CQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEVMBMGA1UECgwMVGVzdCBDb21w
    YW55MRAwDgYDVQQLDAdUZXN0aW5nMTEwLwYDVQQDDChzZWFsdXNhMTIubW9iaXVz
    LmxhYi5lbmcucmR1Mi5yZWRoYXQuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
    MIIBCgKCAQEAzQyl4WJElWE3izEVIvUHGpcandkJclEGMgvvP9YaTMkp8E8MJC7p
    /PxnNFZozyaeXggepsEEwnCpgC3I0xXsdcneqMmoacWg9w2Vk6O0gprbeRC4uz49
    2WYGimsVmg+tfVj8AOBDQTpxc+N2TTDeTxftXdoljUnzKEw4SS87h2iO//jG4cpW
    QKBtEMVlnaGHWGk7R4MNfOTiCehhQdankCNxcKxvV42I3VS2+mFS0qpVLOQfN5bK
    9LOTa+pclUSL56IaTLzzAM8iM6OBbieno8RuG2TwTpWmXXzsGgJ/DsUGi6FmlE+X
    y1cXjWdaDrudE2zzHviyQ+vf6gFFFNL/YwIDAQABo4H+MIH7MA8GA1UdEwQIMAYB
    Af8CAQAwgecGA1UdEQSB3zCB3IItcmVnaXN0cnkub2NwLWVkZ2UtY2x1c3Rlci0w
    LnFlLmxhYi5yZWRoYXQuY29tgglzZWFsdXNhMTKHEBABDbkAAAAAAAAAAAAAAAGH
    EP6AAAAAAAAAUFQA//7Nok+HEP6AAAAAAAAA/FQA//6/W2GHEP6AAAAAAAAADsR6
    //7Skb6HEP0ub0Rd2AAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//57hnKHEP6AAAAA
    AAAAehCwqjYaVTiHEBABDbkAAAAAAAAAAAAAAAGHEP6AAAAAAAAAUFQA//6LAcgw
    DQYJKoZIhvcNAQELBQADggEBAJgwEuObMzjjh1KqEKHuoSgGJl7byxK76W1py1Yh
    NJctqkMZtHEUzhcipT80RhhpKYREnEXlX2r7VDe8aSVa4XnJ9h3CbNT8RrXmW40T
    wdo49QG/h0n17I1oilNaz44pReGKg8F/p6DSr1++ElCh+Cj8354EsS+QCTJ8Ciqq
    Ycyrev1TRu1Bh9PqWP9WbnFyOk1UG3AV/Ad97nMvOODuGFFCY2V6g/pUOJDRoJCh
    8lmxh7gBcGTxZazWNMbw6Mbu6lBSYmpsUhh64uNhNvUC3b0kDepT6NB5/mPmy4jj
    aiWA55l1J0Ho5/kyPv1YK+9aPBzd8yBHY09nxxBoRbd0Lq8=
    -----END CERTIFICATE-----
  registries.conf: |
    unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

    [[registry]]
      prefix = ""
      location = "quay.io/openshift-release-dev/ocp-release"
      mirror-by-digest-only = true

kind: ConfigMap
metadata:
  labels:
    app: assisted-service
  name: mirror-registry-ca
  namespace: assisted-installer

3. Check assisted service pod assisted service container logs
oc logs -f assisted-service-979c79679-cxgbg  -c assisted-service

Actual results:
Repeated failed attempts to pull release image, but x509 auth failing:
error msg="failed to add OCP version" func="github.com/openshift/assisted-service/internal/controller/controllers.(*ClusterDeploymentsReconciler).createNewCluster" file="/go/src/github.com/openshift/origin/internal/controller/controllers/clusterdeployments_controller.go:546" error="command oc adm release info -o template --template '{{.metadata.version}}' --insecure=false registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.8.0-fc.1-x86_64 exited with non-zero exit code 1: \nerror: unable to connect to image repository registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.8.0-fc.1-x86_64: Get \"https://registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/v2/\": x509: certificate signed by unknown authority\n"

Expected results:
Able to pull release image without errir

Additional info:
If I use the same configmap, but replace mirror_ca.pem with tls-ca-bundle.pem, the assisted service pod comes up correctly (This was used in the original work around). I'm not sure what the difference is between the two names. 

Here is a diff of the relevant working and non-working assisted-service deployment objects:

      - configMap:						      - configMap:
          defaultMode: 420					          defaultMode: 420
          items:						          items:
          - key: ca-bundle.crt					          - key: ca-bundle.crt
            path: tls-ca-bundle.pem			      |	            path: mirror_ca.pem
          name: mirror-registry-ca				          name: mirror-registry-ca
          optional: true					          optional: true
        name: mirror-registry-ca				        name: mirror-registry-ca

        - mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-b |	        - mountPath: /etc/pki/ca-trust/extracted/pem/mirror_c
          name: mirror-registry-ca				          name: mirror-registry-ca
          subPath: tls-ca-bundle.pem			      |	          subPath: mirror_ca.pem


Another note is that I also tried excluding the registries.conf info and nothing changed.

Comment 1 Chad Crum 2021-05-04 17:34:48 UTC
Related to merged pr https://github.com/openshift/assisted-service/pull/1563/files

Comment 3 Chad Crum 2021-05-10 14:14:56 UTC
I validated that I can set a disconnected registry ca using this commit [1].

I'm able to deploy assisted service using the mirror registry change included in the above commit [2].


No x509 errors in logs with this change.


This bz is related to [3].


[1] https://github.com/openshift/assisted-service/commit/d42b8c859f82eb4938142dd0ca32d7357df435af

[2] https://github.com/openshift/assisted-service/blob/master/docs/operator.md#mirror-registry-configuration

[3] https://bugzilla.redhat.com/show_bug.cgi?id=1943558

Comment 6 Red Hat Bugzilla 2023-09-15 01:33:52 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days


Note You need to log in before you can comment on or make changes to this bug.