Bug 2100527

Summary:	Submariner operator uses image tag names rather than image digest
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	Felix Dewaleyne <fdewaley>
Component:	Submariner	Assignee:	Stephen Kitt <skitt>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Noam Manos <nmanos>
Severity:	medium	Docs Contact:	Christopher Dawson <cdawson>
Priority:	high
Version:	rhacm-2.4	CC:	fdewaley, gekis, jkho, maafried, marpears, mbabushk, mlele, njean, nyechiel, sgaddam, skitt
Target Milestone:	---	Flags:	nmanos: qe_test_coverage? nyechiel: rhacm-2.7+
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-02-09 05:37:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Felix Dewaleyne 2022-06-23 15:44:17 UTC

**What happened**:
When deploying the Submariner operator using RHACM to a cluster set in a disconnected environment which uses imagecontentsourcepolicies, the images referenced by the pods created by the operator use a tag name rather than image digest values. This is not currently supported by Openshift 4.10


**What you expected to happen**:
we need a solution that is supported with 4.10 and under regarding imagecontentsourcepolicies

**How to reproduce it (as minimally and precisely as possible)**:
consultant deployed RHACM 2.4 following the disconnected installation process and then submariner from ACM but had to make changes (see more info) to be able to finish the deployment

**Anything else we need to know?**:
the information regarding imagecontentsourcepolicies comes from https://access.redhat.com/solutions/4817401
As a work-around, the submariner config was altered on the hub cluster to specify the images directly in the registry in the spec.imagePullSpecs section. This was in-turn applied to the managed clusters who were then able to pull the images.

Pods affected :
submariner-lighthouse-agent
submariner-lighthouse-coredns
submariner-gateway
submariner-routeagent

Work-around :
spec: 
  imagePullSpecs:
    lighthouseAgentImagePullSpec: myregistry.local.co.uk/rhacm2-tech-preview/lighthouse-agent-rhel8@sha256[...value...]
    lighthouseCoreDNSImagePullSpec: myregistry.local.co.uk/rhacm2-tech-preview/lighthouse-coredns-rhel8@sha256[...value...]
    submarinerImagePullSpec: myregistry.local.co.uk/rhacm2-tech-preview/submariner-gateway-rhel8@sha256[...value...]
    submarinerRouteAgentImagePullSpec: myregistry.local.co.uk/rhacm2-tech-preview/submariner-route-agent-rhel8@sha256[...value...]

**Environment**:
- Submariner version (use `subctl version`):
subctl version: v0.12.1
- Kubernetes version (use `kubectl version`):
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v0.23.0", GitCommit:"3e24949fea37244367d50a1f3a226ec20d51eef1", GitTreeState:"clean", BuildDate:"2022-04-01T12:50:14Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5+9ce5071", GitCommit:"3c28e7a79b58e78b4c1dc1ab7e5f6c6c2d3aedd3", GitTreeState:"clean", BuildDate:"2022-04-04T17:59:32Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (0.23) and server (1.23) exceeds the supported minor version skew of +/-1
- Others:
[n145503@ploocpadmin01 ~]$ oc version
Client Version: 4.10.0-202204090935.p0.g3e24949.assembly.stream-3e24949
Server Version: 4.10.11
Kubernetes Version: v1.23.5+9ce5071

They cannot move to 2.5 at this stage of the deployment. If changes cannot be made to the deployment, we will need documentation updates to cover this problem.

Comment 2 Stephen Kitt 2022-06-27 10:19:40 UTC

Was Submariner deployed using the ACM UI directly, or were any changes made to account for the disconnected environment?

Comment 4 Stephen Kitt 2022-06-30 12:48:17 UTC

Thanks, I think I have all the information I need here.

Comment 8 Nir Yechiel 2022-09-05 08:44:51 UTC

Workaround:

The ImageContentSourcePolicy has a limitation of working only with digest(sha): https://access.redhat.com/solutions/4817401. In addition, Submariner is currently using tagging references instead of digest.

By default, the pull sources are the official Red Hat registry, so in order to redirect them to the other images, two steps need to be performed:

1. Create a registry override by using the MachineSet manifest. The override should point to the internal registry of the cluster or some other internal registry that is used. In addition, the "mirror-by-digest-only = false" option should be set as mentioned in the article.

2. In case no central internal registry is used, import the images and tag them into the cluster internal registry.


During deployment, the submariner-operator will try to pull the images from the official Red Hat registry but because of the override, it will point to the alternative location.

Comment 9 Stephen Kitt 2022-09-21 09:20:23 UTC

The plan to fix this is to specify related images in the CSV; see https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/contributors/design-proposals/related-images.md

Comment 12 Stephen Kitt 2023-01-26 08:41:09 UTC

This is fixed in ACM 2.7.

Comment 13 Maxim Babushkin 2023-01-26 11:43:41 UTC

Deployment of the operator by using digest reference instead of tag has been verified.
Now, instead of MachineSet with mirror configuration, a ImageContentSourcePolicy could be used to reference another registry.