Bug 1716697

Summary:

Build prints invalid image pull spec, containing both ":latest" tag and "@sha256" digest

Product:

OpenShift Container Platform

Reporter:

Miciah Dashiel Butler Masters <mmasters>

Component:

Build

Assignee:

Nalin Dahyabhai <nalin>

Status:

CLOSED ERRATA

QA Contact:

wewang <wewang>

Severity:

medium

Docs Contact:

Priority:

unspecified

Version:

4.1.0

CC:

adam.kaplan, aos-bugs, bparees, obulatov, pweil, wewang, wzheng

Target Milestone:

---

Target Release:

4.2.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

good-first-issue

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Cause: When pushing the just-built image to a registry, the builder would log a "Successfully pushed" message which indicated where it had pushed the image. The location it specified would include an initial "//". Consequence: Applications which parsed the log output and then attempted to pull an image using the location extracted from this part of the build log would encounter an error while attempting to parse the location. Fix: The builder now logs the location in a different format which is more in line with expectations. Result: Applications which parse the log output and then attempt to pull an image using the location extracted from this part of the build log should no longer encounter errors when attempting to parse the image's location.

Story Points:

---

Clone Of:

Environment:

Last Closed:

2019-10-16 06:29:26 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
buildconfig.yaml used in Step 1	none

Description Miciah Dashiel Butler Masters 2019-06-04 00:38:05 UTC

Created attachment 1576794 [details]
buildconfig.yaml used in Step 1

Description of problem:

A successful build prints an invalid image pull spec.

Trying to use the image pull spec in a pod spec will result in an error: "Failed to apply default image tag [...] couldn't parse image reference [...] invalid reference format" because the pull spec has a leading "//".

Trying to use the image pull spec with the leading "//" removed will result in an error: "Failed to pull image [...] Invalid image name [...] unknown transport" because the pull spec has both a tag (":latest") and a digest ("@sha256:").

Trying to use the image pull spec with both the leading "//" and the ":latest" string removed succeeds.


Version-Release number of selected component (if applicable):

Client Version: version.Info{Major:"4", Minor:"1+", GitVersion:"v4.1.0-201904211700+86afce9-dirty", GitCommit:"86afce9", GitTreeState:"", BuildDate:"2019-04-21T23:20:35Z", GoVersion:"", Compiler:"", Platform:""}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+2f6cda3", GitCommit:"2f6cda3", GitTreeState:"clean", BuildDate:"2019-05-31T00:18:28Z", GoVersion:"go1.11.9", Compiler:"gc", Platform:"linux/amd64"}


How reproducible:

100%.



Steps to Reproduce:

1. Create a build config:

       oc process -f hack/buildconfig.yaml -p GIT_URL=https://github.com/Miciah/cluster-ingress-operator.git -p GIT_BRANCH=master

   (buildconfig.yaml is attached.)

2. Run a build:

       oc -n openshift-ingress-operator start-build ingress-operator --follow --wait

3. Check the build's output.

4. Try to use the image pull spec from the build output:

       oc -n openshift-ingress-operator patch deploy/ingress-operator --type=strategic --patch='{"spec":{"template":{"spec":{"containers":[{"name":"ingress-operator","image":"//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9"}]}}}}'

    or

       oc -n openshift-ingress-operator patch deploy/ingress-operator --type=strategic --patch='{"spec":{"template":{"spec":{"containers":[{"name":"ingress-operator","image":"image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9"}]}}}}'

5. Check events:

       oc -n openshift-ingress-operator get events


Actual results:

The build prints an invalid image pull spec:

    Successfully pushed //image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9
    Push successful

Using this image pull spec causes the following failure:

    0s    Warning   InspectFailed   pod/ingress-operator-58b89c4586-hv5xp   Failed to apply default image tag "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9": couldn't parse image reference "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9": invalid reference format

or

    0s    Warning   Failed   pod/ingress-operator-697d8bc864-c5h8b   Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9": rpc error: code = Unknown desc = Invalid image name "image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9", unknown transport "image-registry.openshift-image-registry.svc"



Expected results:

The build should print a usable image pull spec.

Comment 1 Adam Kaplan 2019-07-02 14:23:45 UTC

This was discussed in https://github.com/openshift/builder/pull/75 - we ended up keeping the tag name.

However, since this creates an unusable pull spec we can go back to stripping the tag name.

Comment 2 Oleg Bulatov 2019-07-03 13:38:40 UTC

The error isn't about the tag name, it's about //

Comment 3 Ben Parees 2019-07-30 16:30:04 UTC

i ran a build w/ today's OCP and here's the build output (build log, i assume that's where you were looking?):

Successfully pushed image-registry.openshift-image-registry.svc:5000/p3/nodejs-ex:latest@sha256:e381bd969409842a4efbc2ca6387f19b6925e697256df7ec4bdd0ac862f455a2


so i do not see the // prefix, but the ":latest" is still problematic.

It seems like the correct spec should be:

Successfully pushed image-registry.openshift-image-registry.svc:5000/p3/nodejs-ex@sha256:e381bd969409842a4efbc2ca6387f19b6925e697256df7ec4bdd0ac862f455a2


Assigning to Nalin since he last touched this space.

Comment 6 wewang 2019-09-11 07:38:15 UTC

Verified  it, output is not include  "Successfully pushed //image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9"
version:
4.2.0-0.nightly-2019-09-10-181551

[wewang@Desktop Downloads]$ oc get event -n openshift-ingress-operator
LAST SEEN   TYPE      REASON              OBJECT                                   MESSAGE
69m         Normal    Scheduled           pod/ingress-operator-2-build             Successfully assigned openshift-ingress-operator/ingress-operator-2-build to ip-10-0-145-253.us-west-2.compute.internal
69m         Normal    Pulled              pod/ingress-operator-2-build             Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6fdc4d0db2e1292c1c498aaec3a7256ae7cb485c7c8c585f1cd42d3b47ac2f26" already present on machine
69m         Normal    Created             pod/ingress-operator-2-build             Created container git-clone
69m         Normal    Started             pod/ingress-operator-2-build             Started container git-clone
69m         Normal    Pulled              pod/ingress-operator-2-build             Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6fdc4d0db2e1292c1c498aaec3a7256ae7cb485c7c8c585f1cd42d3b47ac2f26" already present on machine
69m         Normal    Created             pod/ingress-operator-2-build             Created container manage-dockerfile
69m         Normal    Started             pod/ingress-operator-2-build             Started container manage-dockerfile
69m         Normal    Pulled              pod/ingress-operator-2-build             Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6fdc4d0db2e1292c1c498aaec3a7256ae7cb485c7c8c585f1cd42d3b47ac2f26" already present on machine
69m         Normal    Created             pod/ingress-operator-2-build             Created container docker-build
69m         Normal    Started             pod/ingress-operator-2-build             Started container docker-build
69m         Normal    BuildStarted        build/ingress-operator-2                 Build openshift-ingress-operator/ingress-operator-2 is now running
66m         Normal    BuildCompleted      build/ingress-operator-2                 Build openshift-ingress-operator/ingress-operator-2 completed successfully
59m         Normal    Scheduled           pod/ingress-operator-744b7955c-95srk     Successfully assigned openshift-ingress-operator/ingress-operator-744b7955c-95srk to ip-10-0-144-28.us-west-2.compute.internal
58m         Normal    Pulling             pod/ingress-operator-744b7955c-95srk     Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fa90696e5d88977845caafbfe1ae970cac9d51b526ea83349678c2a921688838"
58m         Normal    Pulled              pod/ingress-operator-744b7955c-95srk     Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fa90696e5d88977845caafbfe1ae970cac9d51b526ea83349678c2a921688838"
58m         Normal    Created             pod/ingress-operator-744b7955c-95srk     Created container ingress-operator
58m         Normal    Started             pod/ingress-operator-744b7955c-95srk     Started container ingress-operator
56m         Normal    Killing             pod/ingress-operator-744b7955c-95srk     Stopping container ingress-operator
55m         Normal    Scheduled           pod/ingress-operator-744b7955c-h5clt     Successfully assigned openshift-ingress-operator/ingress-operator-744b7955c-h5clt to ip-10-0-140-220.us-west-2.compute.internal
55m         Normal    Pulling             pod/ingress-operator-744b7955c-h5clt     Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fa90696e5d88977845caafbfe1ae970cac9d51b526ea83349678c2a921688838"
55m         Normal    Pulled              pod/ingress-operator-744b7955c-h5clt     Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fa90696e5d88977845caafbfe1ae970cac9d51b526ea83349678c2a921688838"
55m         Normal    Created             pod/ingress-operator-744b7955c-h5clt     Created container ingress-operator
55m         Normal    Started             pod/ingress-operator-744b7955c-h5clt     Started container ingress-operator
32s         Normal    Killing             pod/ingress-operator-744b7955c-h5clt     Stopping container ingress-operator
61m         Normal    Killing             pod/ingress-operator-744b7955c-j4sbm     Stopping container ingress-operator
61m         Normal    SuccessfulDelete    replicaset/ingress-operator-744b7955c    Deleted pod: ingress-operator-744b7955c-j4sbm
59m         Normal    SuccessfulCreate    replicaset/ingress-operator-744b7955c    Created pod: ingress-operator-744b7955c-95srk
56m         Normal    SuccessfulDelete    replicaset/ingress-operator-744b7955c    Deleted pod: ingress-operator-744b7955c-95srk
55m         Normal    SuccessfulCreate    replicaset/ingress-operator-744b7955c    Created pod: ingress-operator-744b7955c-h5clt
32s         Normal    SuccessfulDelete    replicaset/ingress-operator-744b7955c    Deleted pod: ingress-operator-744b7955c-h5clt
56m         Normal    Scheduled           pod/ingress-operator-7789b7d9cd-t2dz5    Successfully assigned openshift-ingress-operator/ingress-operator-7789b7d9cd-t2dz5 to ip-10-0-161-183.us-west-2.compute.internal
56m         Warning   InspectFailed       pod/ingress-operator-7789b7d9cd-t2dz5    Failed to apply default image tag "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:c8832095e889d75a39f1c48ec87dfeb563b572648e54209e864418168e87faff": couldn't parse image reference "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:c8832095e889d75a39f1c48ec87dfeb563b572648e54209e864418168e87faff": invalid reference format
56m         Warning   Failed              pod/ingress-operator-7789b7d9cd-t2dz5    Error: InvalidImageName
56m         Normal    SuccessfulCreate    replicaset/ingress-operator-7789b7d9cd   Created pod: ingress-operator-7789b7d9cd-t2dz5
55m         Normal    SuccessfulDelete    replicaset/ingress-operator-7789b7d9cd   Deleted pod: ingress-operator-7789b7d9cd-t2dz5
30s         Normal    Scheduled           pod/ingress-operator-c95d5d7cb-f6f64     Successfully assigned openshift-ingress-operator/ingress-operator-c95d5d7cb-f6f64 to ip-10-0-140-220.us-west-2.compute.internal
0s          Warning   InspectFailed       pod/ingress-operator-c95d5d7cb-f6f64     Failed to apply default image tag "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9": couldn't parse image reference "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9": invalid reference format
0s          Warning   Failed              pod/ingress-operator-c95d5d7cb-f6f64     Error: InvalidImageName
61m         Normal    Scheduled           pod/ingress-operator-c95d5d7cb-n5t8w     Successfully assigned openshift-ingress-operator/ingress-operator-c95d5d7cb-n5t8w to ip-10-0-140-220.us-west-2.compute.internal
59m         Warning   InspectFailed       pod/ingress-operator-c95d5d7cb-n5t8w     Failed to apply default image tag "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9": couldn't parse image reference "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-operator/ingress-operator:latest@sha256:ab0aa7ca1ad8fa9e8e41cb6e81fb8e795b1e043f42e8c15b24661b862eca3dc9": invalid reference format
59m         Warning   Failed              pod/ingress-operator-c95d5d7cb-n5t8w     Error: InvalidImageName
61m         Normal    SuccessfulCreate    replicaset/ingress-operator-c95d5d7cb    Created pod: ingress-operator-c95d5d7cb-n5t8w
59m         Normal    SuccessfulDelete    replicaset/ingress-operator-c95d5d7cb    Deleted pod: ingress-operator-c95d5d7cb-n5t8w
30s         Normal    SuccessfulCreate    replicaset/ingress-operator-c95d5d7cb    Created pod: ingress-operator-c95d5d7cb-f6f64
32s         Normal    ScalingReplicaSet   deployment/ingress-operator              Scaled down replica set ingress-operator-744b7955c to 0
30s         Normal    ScalingReplicaSet   deployment/ingress-operator              Scaled up replica set ingress-operator-c95d5d7cb to 1
59m         Normal    ScalingReplicaSet   deployment/ingress-operator              Scaled down replica set ingress-operator-c95d5d7cb to 0
55m         Normal    ScalingReplicaSet   deployment/ingress-operator              Scaled up replica set ingress-operator-744b7955c to 1
56m         Normal    ScalingReplicaSet   deployment/ingress-operator              Scaled up replica set ingress-operator-7789b7d9cd to 1
55m         Normal    ScalingReplicaSet   deployment/ingress-operator              Scaled down replica set ingress-operator-7789b7d9cd to 0

Comment 7 Nalin Dahyabhai 2019-09-11 14:37:07 UTC

(In reply to wewang from comment #6)
> 56m         Warning   InspectFailed      
> pod/ingress-operator-7789b7d9cd-t2dz5    Failed to apply default image tag
> "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-
> operator/ingress-operator:latest@sha256:
> c8832095e889d75a39f1c48ec87dfeb563b572648e54209e864418168e87faff": couldn't
> parse image reference
> "//image-registry.openshift-image-registry.svc:5000/openshift-ingress-
> operator/ingress-operator:latest@sha256:
> c8832095e889d75a39f1c48ec87dfeb563b572648e54209e864418168e87faff": invalid
> reference format

Do we have the log from the build that produced this image?  The "//" isn't supposed to be in the "Successfully pushed" log line, and it shouldn't include a tag, either.

Comment 9 errata-xmlrpc 2019-10-16 06:29:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922

Comment 10 Red Hat Bugzilla 2023-09-14 05:29:41 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days