Bug 1966156 - Issue with Internal Registry CA on the service pod
Summary: Issue with Internal Registry CA on the service pod
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.8.0
Assignee: Fred Rolland
QA Contact: Chad Crum
URL:
Whiteboard: AI-Team-Hive
Depends On:
Blocks: mint
TreeView+ depends on / blocked
 
Reported: 2021-05-31 14:30 UTC by Juan Manuel Parrilla Madrid
Modified: 2023-09-15 01:08 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:10:38 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:11:14 UTC

Description Juan Manuel Parrilla Madrid 2021-05-31 14:30:01 UTC
Context:

- ACM Downstream Snapshot deployed: 2.3.0-DOWNSTREAM-2021-05-27-19-55-15
- ACM Operator Bundle: v2.3.0-111
- IPv6/Disconnected


Issue Description:

Once deployed the ACM Downstream version (Disconnected/IPv6), we deploy the Operand and the typical CRs, ClusterDeployment, AgentClusterInstall and Infraenv. When the Assisted Service pod tries to download the ClusterImageSet version declared in the proper CR, gives an error related with the Internal Registry CA:

===============================================
time="2021-05-31T13:07:30Z" level=error msg="failed to add OCP version" func="github.com/openshift/assisted-service/internal/controller/controllers.(*ClusterDeploymentsReconciler).createNewCluster" file="/remote-source/app/internal/controller/controllers/clusterdeployments_controller.go:757" agent_cluster_install=test-cluster-virtual-aci agent_cluster_install_namespace=open-cluster-management cluster_deployment=test-cluster-virtual cluster_deployment_namespace=open-cluster-management error="command oc adm release info -o template --template '{{.metadata.version}}' --insecure=false bm-cluster-1-hyper.e2e.bos.redhat.com:5000/ocp4:4.8.0-0.nightly-2021-05-13-134354 exited with non-zero exit code 1: \nerror: unable to read image bm-cluster-1-hyper.e2e.bos.redhat.com:5000/ocp4:4.8.0-0.nightly-2021-05-13-134354: Get \"https://bm-cluster-1-hyper.e2e.bos.redhat.com:5000/v2/\": x509: certificate signed by unknown authority\n" go-id=884 request_id=e34e5afb-e908-4965-a700-4208d9ab519b
===============================================

I've validated:
- The Certificate is the same in the Internal Registry
- The ConfigMap (pod description):
 - mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
   name: mirror-registry-ca
   subPath: tls-ca-bundle.pem
- The master node: /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
- The Pod: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

The problem is the path on the pod side, looks like it's not in the right one. Right now it's in /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and this does not like to the oc adm release info command


Jira Bug: MGMT-6741

Comment 1 Fred Rolland 2021-06-02 06:23:49 UTC
Trying to run update-ca-trust before the service

https://code.engineering.redhat.com/gerrit/c/assisted-installer-projects/+/244512

Comment 2 Fred Rolland 2021-06-03 09:42:38 UTC
Downstream PR: https://code.engineering.redhat.com/gerrit/c/assisted-installer-projects/+/244553

No changes needed Upstream

Comment 6 Fred Rolland 2021-06-17 09:53:24 UTC
Juan, is this one fixed? Can you move to verified?

Comment 7 Chad Crum 2021-06-19 14:16:36 UTC
I have validated this in:
ACM DS Build: 2.3.0-DOWNSTREAM-2021-06-17-01-26-58
OCP Hub: 4.8.0-fc.7

Steps:
- Deployed disconnected ipv6 with D/S ACM build, using mirror-registry-ca to map registries.conf + self signed CA to assisted pod
- Tried to run the oc adm release command directly from the assisted-service pod:

[root@sealusa10 ~]# oc rsh assisted-service-554499cbf6-nzj28 
Defaulted container "assisted-service" out of: assisted-service, postgres
sh-4.4$ oc adm release info -o template --template '{{.metadata.version}}' --insecure=false registry.ocp-edge-cluster-assisted-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.6.16-x86_64
error: unable to read image registry.ocp-edge-cluster-assisted-0.qe.lab.redhat.com:5000/openshift-release-dev/ocp-release:4.6.16-x86_64: Head "https://registry.ocp-edge-cluster-assisted-0.qe.lab.redhat.com:5000/v2/openshift-release-dev/ocp-release/manifests/4.6.16-x86_64": no basic auth credentials
sh-4.4$ 

- No issue with x509 in above step - no basic auth creds is expected in this case

- Next deployed all CRs to deploy a SNO cluster and did not see any errors in the assisted service pod logs related to x509. SNO cluster creation completed successfully.



Hi @jparrill - I'm not reproducing this with the latest build. Can you also confirm you no longer see it and then we can flip to VERIFIED.

Comment 8 Chad Crum 2021-06-21 12:56:10 UTC
Verified with Juan that he no longer sees this either.

Comment 10 errata-xmlrpc 2021-07-27 23:10:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Comment 11 Red Hat Bugzilla 2023-09-15 01:08:47 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days


Note You need to log in before you can comment on or make changes to this bug.