Bug 1841178

Summary:	opm does not respect whiteout files when unpacking images with docker and podman
Product:	OpenShift Container Platform	Reporter:	lgallett
Component:	OLM	Assignee:	Nick Hale <nhale>
OLM sub component:	OLM	QA Contact:	Jian Zhang <jiazha>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	jiazha, krizza, nhale, tbuskey, vdinh
Version:	4.5
Target Milestone:	---
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: opm treats whiteout files the same as normal files during unpacking for podman and docker tooling options. Consequence: Whiteout files show up in unpacked content when using podman or docker tooling options. Fix: Use docker/podman run and copy to properly apply whiteout files during unpacking. Result: Whiteout files are no longer present after unpacking with podman and docker tooling options.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 16:01:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1845588

Description lgallett 2020-05-28 15:02:32 UTC

Description of problem:

When opm uses docker save / podman save to unpack the bundle image, whiteout files are still present in the extracted filesystem. This cause the bundle validate command to fail.

Version-Release number of selected component (if applicable):


How reproducible:

always


Steps to Reproduce:

opm alpha bundle validate --tag registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1

Actual results:

EOF error

Expected results:

passes validation

Additional info:

Comment 1 Vu Dinh 2020-05-28 17:11:05 UTC

There is work underway to address the issue with the unpacking process that will clean up the whiteout files before it's reaching the validation. From validation perspective, it will simply validate what has been given and will error out if there are unwanted existing in the provided directory.

Comment 2 Nick Hale 2020-05-29 13:42:27 UTC

We may need to run the image with a "do nothing" entrypoint and copy its filesystem with `docker cp` https://docs.docker.com/engine/reference/commandline/cp/

Comment 3 Nick Hale 2020-06-02 13:07:52 UTC

I'll take this bug. Marking for upcoming sprint and moving to assigned.

Comment 13 Jian Zhang 2020-07-07 09:08:18 UTC

Hi Tom,

> I need current binaries from https://github.com/operator-framework/operator-registry/releases
> ./opm-1.12.5 alpha bundle validate --image-builder podman  --tag registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1

If no available opm binary, we can set the status to `POST` instead of the `MODIFIED`. Otherwise, the `MODIFIED` will be changed to `ON_QA` by the Bugzilla script automatically.

For the olm binary, we can also build it by ourself, as follows:
1, Build the latest opm from the master branch.
[root@preserve-olm-env operator-registry]# git branch
  1.12.6
* master

[root@preserve-olm-env operator-registry]# make build
...
GOFLAGS="-mod=vendor" go build -ldflags "-X 'github.com/operator-framework/operator-registry/cmd/opm/version.gitCommit=3ca7ddd' -X 'github.com/operator-framework/operator-registry/cmd/opm/version.opmVersion=1.12.7' -X 'github.com/operator-framework/operator-registry/cmd/opm/version.buildDate=2020-07-07T08:48:26Z'"  -o bin/opm ./cmd/opm

[root@preserve-olm-env operator-registry]# ./bin/opm version
Version: version.Version{OpmVersion:"1.12.7", GitCommit:"3ca7ddd", BuildDate:"2020-07-07T08:48:26Z", GoOs:"linux", GoArch:"amd64"}

As we can see the GitCommit:"3ca7ddd" contains the fixed PR: https://github.com/operator-framework/operator-registry/pull/351 by checking https://github.com/operator-framework/operator-registry/commits/master

2, Run the `bundle validate`. It failed. As follows:

[root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle validate --tag registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
INFO[0000] Create a temp directory at /tmp/bundle-776951841  container-tool=docker
DEBU[0000] Pulling and unpacking container image         container-tool=docker
INFO[0000] running docker create                         container-tool=docker
DEBU[0000] [docker create registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1 ]  container-tool=docker
INFO[0000] running docker cp                             container-tool=docker
DEBU[0000] [docker cp Unable to find image 'registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1' locally
1.0.0-1: Pulling from rh-osbs/amq7-amq-streams-rhel7-operator-metadata
ec602282be3e: Already exists
Digest: sha256:8a4194473c8963f4419a0546b324995025147993bf2b583bab01207b601b85f5
Status: Downloaded newer image for registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
a30b6e8ff549e576f8080bc16e8906b5fc9afc891376e8bf94f647ba0d7a564f:/ /tmp/bundle-776951841]  container-tool=docker
ERRO[0000] Error: No such container:path: Unable to find image 'registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1' locally
1.0.0-1: Pulling from rh-osbs/amq7-amq-streams-rhel7-operator-metadata
ec602282be3e: Already exists
Digest: sha256:8a4194473c8963f4419a0546b324995025147993bf2b583bab01207b601b85f5
Status: Downloaded newer image for registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
a30b6e8ff549e576f8080bc16e8906b5fc9afc891376e8bf94f647ba0d7a564f:/  container-tool=docker
Error: error copying container directory Error: No such container:path: Unable to find image 'registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1' locally
1.0.0-1: Pulling from rh-osbs/amq7-amq-streams-rhel7-operator-metadata
ec602282be3e: Already exists
Digest: sha256:8a4194473c8963f4419a0546b324995025147993bf2b583bab01207b601b85f5
Status: Downloaded newer image for registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
a30b6e8ff549e576f8080bc16e8906b5fc9afc891376e8bf94f647ba0d7a564f:/
: exit status 1
Usage:
  opm alpha bundle validate [flags]

Examples:
$ opm alpha bundle validate --tag quay.io/test/test-operator:latest --image-builder docker

Flags:
  -h, --help                   help for validate
  -b, --image-builder string   Tool used to pull and unpack bundle images. One of: [none, docker, podman] (default "docker")
  -t, --tag string             The path of a registry to pull from, image name and its tag that present the bundle image (e.g. quay.io/test/test-operator:latest)

3, But, it works well after pull the image to the local first. @Nick Is it as expected? Thanks!
[root@preserve-olm-env operator-registry]# docker pull registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
1.0.0-1: Pulling from rh-osbs/amq7-amq-streams-rhel7-operator-metadata
ec602282be3e: Already exists 
Digest: sha256:8a4194473c8963f4419a0546b324995025147993bf2b583bab01207b601b85f5
Status: Downloaded newer image for registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1

[root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle validate --tag registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
INFO[0000] Create a temp directory at /tmp/bundle-118897420  container-tool=docker
DEBU[0000] Pulling and unpacking container image         container-tool=docker
INFO[0000] running docker create                         container-tool=docker
DEBU[0000] [docker create registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1 ]  container-tool=docker
INFO[0000] running docker cp                             container-tool=docker
DEBU[0000] [docker cp 33241cb9ed2c17e4f31121e317dcbc749ea15c9a6421325801e26457b9b467be:/ /tmp/bundle-118897420]  container-tool=docker
INFO[0000] running docker rm                             container-tool=docker
DEBU[0000] [docker rm 33241cb9ed2c17e4f31121e317dcbc749ea15c9a6421325801e26457b9b467be]  container-tool=docker
INFO[0000] Unpacked image layers, validating bundle image format & contents  container-tool=docker
DEBU[0000] Found manifests directory                     container-tool=docker
DEBU[0000] Found metadata directory                      container-tool=docker
DEBU[0000] Getting mediaType info from manifests directory  container-tool=docker
INFO[0000] Found annotations file                        container-tool=docker
INFO[0000] Could not find optional dependencies file     container-tool=docker
DEBU[0000] Validating bundle contents                    container-tool=docker
DEBU[0000] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafka.crd.yaml"  container-tool=docker
DEBU[0000] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkaconnect.crd.yaml"  container-tool=docker
DEBU[0000] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkaconnects2i.crd.yaml"  container-tool=docker
DEBU[0000] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkamirrormaker.crd.yaml"  container-tool=docker
DEBU[0000] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkatopic.crd.yaml"  container-tool=docker
DEBU[0000] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkauser.crd.yaml"  container-tool=docker
DEBU[0000] Validating "operators.coreos.com/v1alpha1, Kind=ClusterServiceVersion" from file "amq-streams.v1.0.0.clusterserviceversion.yaml"  container-tool=docker
INFO[0000] All validation tests have been completed successfully  container-tool=docker

Comment 15 Nick Hale 2020-07-16 03:15:20 UTC

I was able to reproduce the issue and filed a separate BZ and associated patch to cover fix it: https://bugzilla.redhat.com/show_bug.cgi?id=1857502 PTAL. After that PR merges we should be good to test this again at a newer commit.

Comment 18 Jian Zhang 2020-07-22 02:38:04 UTC

1, check the olm version:
[root@preserve-olm-env operator-registry]# ./bin/opm version
Version: version.Version{OpmVersion:"v1.12.5-78-g24965bb", GitCommit:"24965bb", BuildDate:"2020-07-22T02:25:53Z", GoOs:"linux", GoArch:"amd64"}

2, Run the validation: `bundle validate`. It works well, LGTM, verify it.

[root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle validate --tag registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1
INFO[0000] Create a temp directory at /tmp/bundle-053862621  container-tool=docker
DEBU[0000] Pulling and unpacking container image         container-tool=docker
INFO[0000] running /usr/bin/docker pull registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1  container-tool=docker
INFO[0001] running docker create                         container-tool=docker
DEBU[0001] [docker create registry-proxy.engineering.redhat.com/rh-osbs/amq7-amq-streams-rhel7-operator-metadata:1.0.0-1 ]  container-tool=docker
INFO[0001] running docker cp                             container-tool=docker
DEBU[0001] [docker cp bfd701d2fc59aceeda85b99fa3802875d9d05a09e207b3cd1823ba77378eddc0:/. /tmp/bundle-053862621]  container-tool=docker
INFO[0001] running docker rm                             container-tool=docker
DEBU[0001] [docker rm bfd701d2fc59aceeda85b99fa3802875d9d05a09e207b3cd1823ba77378eddc0]  container-tool=docker
INFO[0001] Unpacked image layers, validating bundle image format & contents  container-tool=docker
DEBU[0001] Found manifests directory                     container-tool=docker
DEBU[0001] Found metadata directory                      container-tool=docker
DEBU[0001] Getting mediaType info from manifests directory  container-tool=docker
INFO[0001] Found annotations file                        container-tool=docker
INFO[0001] Could not find optional dependencies file     container-tool=docker
DEBU[0001] Validating bundle contents                    container-tool=docker
DEBU[0001] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafka.crd.yaml"  container-tool=docker
DEBU[0001] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkaconnect.crd.yaml"  container-tool=docker
DEBU[0001] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkaconnects2i.crd.yaml"  container-tool=docker
DEBU[0001] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkamirrormaker.crd.yaml"  container-tool=docker
DEBU[0001] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkatopic.crd.yaml"  container-tool=docker
DEBU[0001] Validating "apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition" from file "amq-streams-kafkauser.crd.yaml"  container-tool=docker
DEBU[0001] Validating "operators.coreos.com/v1alpha1, Kind=ClusterServiceVersion" from file "amq-streams.v1.0.0.clusterserviceversion.yaml"  container-tool=docker
INFO[0001] All validation tests have been completed successfully  container-tool=docker

Comment 20 errata-xmlrpc 2020-10-27 16:01:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196