Description of problem: First of all this issue was detected while deploying Elasticsearch instance for Service Mesh, but will probably happen for cluster-logging as well as the operator is the same. While deploying Service Mesh using {"tracing":{"jaeger":{"template": "production-elasticsearch"}, Elasticsearch pods fail to start with ImagePullBackOff error, which in turn makes the pods jaeger-collector and jaeger-query to stay in CrashLoopBackOff. Checked the image on Elasticsearch and tried manually pulled the imaged and we get FATA[0004] Error parsing image name "docker://registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a": Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference #132.72221102.1596114759.1e32f5b3\n</BODY></HTML>\n" Version-Release number of selected component (if applicable): OCP v4.4.z on bare metal and AWS How reproducible: Every time Steps to Reproduce: 1. Deploy the Red Hat Elasticsearch operator 2. Follow the rest of the procedure to deploy Red Hat OCP Service Mesh 3. On the smcp instance file configure - {"tracing":{"jaeger":{"template": "production-elasticsearch"} 4. When the opertors start deployments pods elasticsearch, jaeger-collector and jaeger-query will fail to start. 5. Change the CSV for the elasticsearch-operator to use image registry.redhat.io/openshift4/ose-logging-elasticsearch5, delete the elasticsearch instance deployment in the Service Mesh project and the pods will then start. Actual results: Cri-o fails to pull image. Expected results: Image should have either the tags or correct the image digest Additional info: I haven't tried deploying ClusterLogging, but I will assume it will be the same as the Elasticsearch operator used is the same Tried pulling the image on my PC: podman --log-level=debug pull registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a INFO[0000] podman filtering at log level debug DEBU[0000] Called pull.PersistentPreRunE(podman --log-level=debug pull registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a) DEBU[0000] Found deprecated file /home/andcosta/.config/containers/containers.conf, please remove. Use /home/andcosta/.config/containers/containers.conf to override defaults. DEBU[0000] Reading configuration file "/home/andcosta/.config/containers/libpod.conf" DEBU[0000] Ignoring libpod.conf EventsLogger setting "/home/andcosta/.config/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files. DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] containers-default-0.14.6 [] private enabled [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] [] [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false private k8s-file -1 slirp4netns false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {true cgroupfs [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/libexec/crio/conmon /usr/local/lib/podman/conmon /usr/local/libexec/crio/conmon /usr/bin/conmon /usr/sbin/conmon /usr/lib/crio/bin/conmon] ctrl-p,ctrl-q true /run/user/1000/libpod/tmp/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /pause k8s.gcr.io/pause:3.1 /usr/libexec/podman/catatonit shm false 2048 /usr/bin/crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing false [/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc] [crun runc] [crun] [kata kata-runtime kata-qemu kata-fc] {false false false false false false} /etc/containers/policy.json false 3 /home/andcosta/.local/share/containers/storage/libpod 10 /run/user/1000/libpod/tmp /home/andcosta/.local/share/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}} DEBU[0000] Using conmon: "/usr/libexec/crio/conmon" DEBU[0000] Initializing boltdb state at /home/andcosta/.local/share/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /home/andcosta/.local/share/containers/storage DEBU[0000] Using run root /run/user/1000 DEBU[0000] Using static dir /home/andcosta/.local/share/containers/storage/libpod DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp DEBU[0000] Using volume path /home/andcosta/.local/share/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false DEBU[0000] Initializing event backend file DEBU[0000] using runtime "/usr/bin/crun" WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] using runtime "/usr/bin/runc" DEBU[0000] using runtime "/usr/bin/crun" INFO[0000] Setting parallel job count to 25 DEBU[0000] parsed reference into "[overlay@/home/andcosta/.local/share/containers/storage+/run/user/1000:overlay.mount_program=/usr/bin/fuse-overlayfs,overlay.mount_program=/usr/bin/fuse-overlayfs]registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a" Trying to pull registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a... DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Trying to access "registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a" DEBU[0000] Returning credentials from /home/andcosta/.docker/config.json DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration DEBU[0000] Using "default-docker" configuration DEBU[0000] No signature storage configuration found for registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io DEBU[0000] GET https://registry.redhat.io/v2/ DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=1979710%7Candre-13&scope=repository%3Aopenshift4%2Fose-logging-elasticsearch5%3Apull&service=docker-registry DEBU[0001] GET https://registry.redhat.io/v2/openshift4/ose-logging-elasticsearch5/manifests/sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a DEBU[0003] Content-Type from manifest GET is "text/html" DEBU[0003] Accessing "registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a" failed: Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference #132.72221102.1596116349.1e6e6df9\n</BODY></HTML>\n" DEBU[0003] Error pulling image ref //registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error initializing source docker://registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference #132.72221102.1596116349.1e6e6df9\n</BODY></HTML>\n" error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference #132.72221102.1596116349.1e6e6df9\n</BODY></HTML>\n" Error: unable to pull registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error initializing source docker://registry.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a: Error reading manifest sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a in registry.redhat.io/openshift4/ose-logging-elasticsearch5: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<HTML><HEAD><TITLE>Error</TITLE></HEAD><BODY>\nAn error occurred while processing your request.<p>\nReference #132.72221102.1596116349.1e6e6df9\n</BODY></HTML>\n"
Moving this to the release team for resolution
Moving BZ to urgent as this blocks the installation of ElasticSearch and Logging.
The shasum is pointing at an image logging-elasticsearch5-container-v4.4.0-202007171809.p0 that indeed we have never shipped. This is because we removed it from the 4.4.14 release last week, but still shipped the elasticsearch operator that refers to it. This will be remedied at least with this week's 4.4.15 release, but we're looking into doing it even before that.
$ podman pull registry.stage.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47ce cdd2c4065f7f5a1a71d2cf6dde53746c59a26a Trying to pull registry.stage.redhat.io/openshift4/ose-logging-elasticsearch5@sha256:3e77ecfd109fe44166669d47cecdd2c4065f7f5a1a71d2cf6dde53746c59a26a... Getting image source signatures Copying blob 1d62f0c6869b done Copying blob bc56ba89b5a1 done Copying blob 4575d7d12fc9 done Copying blob 34971b2d1eb9 done Copying blob 4fbc3bafa3d4 done Copying config 078412d0a0 done Writing manifest to image destination Storing signatures 078412d0a0a42b886ef8da40622963b132df350260a2b0bd754037d49956b4bc
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.4.14 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3288