Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1929775

Summary: Unable to create catalog from previous OCP versions
Product: OpenShift Container Platform Reporter: Victor Hernando <vhernand>
Component: OLMAssignee: Kevin Rizza <krizza>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: unspecified CC: mvalledi
Version: 4.5   
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-17 21:40:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Victor Hernando 2021-02-17 16:00:04 UTC
Description of problem:
While building an operator catalog image, using older 4.5 images or the latest 4.5, we see the same operator versions, so the customer is not being able to deploy different clusters with components on the same version on different dates.
That might be a problem for many customers running disconnected environments, so it may be required under business policies to run all the clusters on the same version.

Version-Release number of selected component (if applicable):
4.5

How reproducible:


Steps to Reproduce:
# First, build 3 different catalogs, the first is a 4.5 image built on 2020 December, the 2nd is the 4.5 latest, and the 3rd the 4.6 latest.
1. oc adm catalog build --appregistry-org=redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry@sha256:b3e72f89e9097f350404a1a5f2eb8d1723274b2a3d599b22eeea8e0fd98d82e4 --to=rhel7-docker.openshift.local:5000/test/redhat-operators:v1.0 --filter-by-os="amd64" --insecure=true --loglevel=10 --v=4 
2. oc adm catalog build --appregistry-org=redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.5 --to=rhel7-docker.openshift.local:5000/test/redhat-operators:v2.0 --filter-by-os="amd64" --insecure=true --loglevel=10 --v=4 
3. oc adm catalog build --appregistry-org=redhat-operators --from=registry.redhat.io/openshift4/ose-operator-registry:v4.6 --to=rhel7-docker.openshift.local:5000/test/redhat-operators:v3.0 --filter-by-os="amd64" --insecure=true --loglevel=10 --v=4  

# Checking images BUILD INFO, so we can verify we are using different builds, so different images. 
4. docker inspect registry.redhat.io/openshift4/ose-operator-registry:v4.5|grep BUILD                                                   
                "BUILD_RELEASE=202101300210.p0",
                "BUILD_VERSION=v4.5.0",
                "BUILD_RELEASE=202101300210.p0",
                "BUILD_VERSION=v4.5.0",
5. docker inspect registry.redhat.io/openshift4/ose-operator-registry@sha256:b3e72f89e9097f350404a1a5f2eb8d1723274b2a3d599b22eeea8e0fd98d82e4|grep BUILD
                "BUILD_RELEASE=202012050338.p0",
                "BUILD_VERSION=v4.5.0",
                "BUILD_RELEASE=202012050338.p0",
                "BUILD_VERSION=v4.5.0",
6. docker inspect registry.redhat.io/openshift4/ose-operator-registry:v4.6|grep BUILD
                "BUILD_RELEASE=202101300140.p0",
                "BUILD_VERSION=v4.6.0",
                "BUILD_RELEASE=202101300140.p0",
                "BUILD_VERSION=v4.6.0",

# RUN containers from each of those 3 images we have been built before and check the versions available for some package, i.e. servicemeshoperator
7. # podman run -p 50051:50051 -it rhel7-docker.openshift.local:5000/test/redhat-operators:v1.0
WARN[0000] unable to set termination log path            error="open /dev/termination-log: permission denied"
INFO[0000] serving registry                              database=/bundles.db port=50051

# ./grpcurl -plaintext -d '{"name":"servicemeshoperator"}' localhost:50051 api.Registry/GetPackage
{
  "name": "servicemeshoperator",
  "channels": [
    {
      "name": "1.0",
      "csvName": "servicemeshoperator.v2.0.2"
    },
    {
      "name": "stable",
      "csvName": "servicemeshoperator.v2.0.2"
    }
  ],
  "defaultChannelName": "stable"
}

8. # podman run -p 50051:50051 -it rhel7-docker.openshift.local:5000/test/redhat-operators:v2.0
WARN[0000] unable to set termination log path            error="open /dev/termination-log: permission denied"
INFO[0000] serving registry                              database=/bundles.db port=50051

# ./grpcurl -plaintext -d '{"name":"servicemeshoperator"}' localhost:50051 api.Registry/GetPackage
{
  "name": "servicemeshoperator",
  "channels": [
    {
      "name": "1.0",
      "csvName": "servicemeshoperator.v2.0.2"
    },
    {
      "name": "stable",
      "csvName": "servicemeshoperator.v2.0.2"
    }
  ],
  "defaultChannelName": "stable"
}
9. # podman run -p 50051:50051 -it rhel7-docker.openshift.local:5000/test/redhat-operators:v3.0
WARN[0000] unable to set termination log path            error="open /dev/termination-log: permission denied"
INFO[0000] serving registry                              database=/bundles.db port=50051

# ./grpcurl -plaintext -d '{"name":"servicemeshoperator"}' localhost:50051 api.Registry/GetPackage
{
  "name": "servicemeshoperator",
  "channels": [
    {
      "name": "1.0",
      "csvName": "servicemeshoperator.v2.0.2"
    },
    {
      "name": "stable",
      "csvName": "servicemeshoperator.v2.0.2"
    }
  ],
  "defaultChannelName": "stable"
}

Actual results:

Same version of packages on 3 different operator catalog images, each built from a different image.

Expected results:
Having different versions on each different build, so we can deploy a new cluster with previous versions of each component and not forcing the customer to use always the latest.

Additional info:

Comment 2 Kevin Rizza 2021-02-17 21:40:11 UTC
Hi Victor,

I think there's some fundamental misunderstandings here about how this command works and what it is for:

To start, the `--from` flag is not a pointer to the actual catalog, it is the version of the base image used to serve the content. Modifying that flag does not in any way define what catalog content you get or where it is pulled from.

In OCP 4.5 and earlier, there is never a distinction between previous and current catalog content. We did not store or save that in any way between minor openshift versions, and in fact the catalog content from 4.5 and earlier was all from the exact same source and identical. So there is no way to do what the customer is asking for because the content store was never built to contain that history.

That being said, starting in openshift 4.6 we now actually produce a catalog image tag per minor openshift version. That means that now there is potentially a difference between minor version published content, although that is still up to each individual operator developer's release cadence and support statements. It also means that the command you are running `oc adm catalog build` is no longer needed or supported in OCP 4.6+. For more details on that new process, please check out our 4.6 docs: https://docs.openshift.com/container-platform/4.6/operators/admin/olm-restricted-networks.html#olm-understanding-operator-catalog-images_olm-restricted-networks

Also keep in mind that even with this new catalog publishing mechanism, the catalogs are still published outside the scope of an actual OCP release, so there is no relationship between individual OCP builds or z-streams with a particular set of catalog content at a point in time. So that tag will always be the latest version of the catalog for minor openshift versions. If a customer wants to save the catalog at a specific point in time, they would need to retag that image and store it locally, as there aren't explicit guarantees for specific shas to be stored forever.

All that being said, this is expected behavior and not a bug. As a result, I'm closing this ticket as NOTABUG. If you have further questions or concerns, please feel free to reach out to the OLM maintainers on the coreos slack at #forum-operator-fw or on our mailing list aos-odin