Bug 1882689

Summary: c adm catalog mirror --filter-by-os="linux/amd64" doesn't mirror all necessary images in "Using Operator Lifecycle Manager on restricted networks" guide
Product: OpenShift Container Platform Reporter: Andreas Karis <akaris>
Component: ocAssignee: Sally <somalley>
Status: CLOSED DUPLICATE QA Contact: zhou ying <yinzhou>
Severity: high Docs Contact:
Priority: high    
Version: 4.5CC: agabriel, ak.karis, aos-bugs, bjarolim, dmoessne, jokerman, krizza, maszulik, mfojtik, nhale, openshift-bugs-escalate, sagopina
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-05 16:29:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Karis 2020-09-25 10:33:56 UTC
Description of problem:
I already created a docs bug for this: https://bugzilla.redhat.com/show_bug.cgi?id=1879877

However, this might also be a problem rather in the product itself:

The way our consultant sees this, the oc tool doesn't do the job that it is supposed to do.  If a user follows the documentation, (s)he doesn't get the expected result.

The documentation makes sense, as a customer normally has a cluster with one architecture. They made it work by syncing all the images, but that's not what they want as unnecessary s390 images for instance just use storage on their mirror registry. Hence the oc tool itself needs a fix; the fix to documentation might just be a workaround.




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Document URL: 
https://docs.openshift.com/container-platform/4.5/operators/olm-restricted-networks.html#olm-updating-operator-catalog-image_olm-restricted-networks

Section Number and Name: 

Describe the issue: 

Suggestions for improvement: 

Additional information: 

One of our consultants ran into this while helping a customer set up their OpenShift environment. Recently they upgraded from OCP 4.4.3 to OCP 4.5.7 (going through 4.4.14). The customer runs a disconnected environment, and there is a mirror registry which has originally been setup as per [1]. 

As a preparation, they mirrored the new release into the mirror registry, and also created a new OLM catalog version and mirrored the content to the mirror registry as per [2]. For the oc adm catalog mirror command, they specified --filter-by-os="linux/amd64", so that no unnecessary images will be mirrored to the registry. The mirroring procedure went through without any troubles, but after activating the new OLM catalog in the upgraded cluster, OLM was not able to pull the operator images. They noticed the problem for the Elasticsearch operator and the Logging operator, it failed with an Image Pull Error. 

After digging some more, they found the following BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1800674

The last comment [3] pointed them toward using the following filter: "--filter-by-os=.*" The bug has been closed recently, stating that this filter should be used.

After running oc adm catalog mirror using this filter, it synced more images into the registry, amongst them also the missing images for ES and Logging.

The consultant thinks oc adm catalog mirror is not behaving correctly, he expects it to sync all the necessary images for a amd64 cluster when specifying linux/amd64 as a filter. Also, the documentation [2] tells the user to use this filter and doesn't mention the .* filter at all, so IMHO if users adhere to the documentation, they will end up in the situation he outlined here.

[1] https://docs.openshift.com/container-platform/4.2/installing/install_config/installing-restricted-networks-preparations.html#installation-creating-mirror-registry_installing-restricted-networks-preparations
https://docs.openshift.com/container-platform/4.5/operators/olm-restricted-networks.html
[2] https://docs.openshift.com/container-platform/4.5/operators/olm-restricted-networks.html#olm-updating-operator-catalog-image_olm-restricted-networks
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1800674#c42

Comment 1 Maciej Szulik 2020-09-28 09:59:33 UTC
Moving to OLM, since they own catalog sub-command.

Comment 7 Alexander Greene 2020-10-21 19:20:42 UTC
@akaris,

Thank you for submitting this bug.

I was unable to reproduce the bug that you described - I created a curated catalog that mirrored the following images correctly:
- registry.redhat.io/openshift4/ose-logging-elasticsearch6
- registry.redhat.io/openshift4/ose-elasticsearch-proxy
- registry.redhat.io/openshift4/ose-elasticsearch-operator
- registry.redhat.io/openshift4/ose-logging-elasticsearch5
- registry.redhat.io/openshift4/ose-logging-kibana6
- registry.redhat.io/openshift4/ose-oauth-proxy
```

Given that I was unable to reproduce the issue, could you provide:
- The exact steps taken to build the Mirror Catalog along with the logs.
- The ImageContentSourcePolicy and the mapping.txt generated from the oc mirror command.
- Access to an environment in which the issue is present (this is not mandatory but would speed things along).

Thanks!

Comment 12 Maciej Szulik 2020-10-23 10:18:13 UTC
Sally, is this possibly linked with ICSP?

Comment 15 Sally 2020-11-13 19:41:45 UTC
Opened this: https://github.com/openshift/oc/pull/642  

With this PR, the manifestList digest will always be mirrored whenever a user sets:
   --filter-by-os (wildcard OR a single os/arch) 
OR
   --keep-manifest-list=true

Comment 17 Sally 2020-12-05 00:34:39 UTC
Slight change to that PR referenced above, but it's still under review/discussion.  I think the best we can do is to ensure when `--keep-manifest-list=true` is passed, you get all manifests in the list as this is the only way to preserve the digest - again, this PR is still under review but will report back here.