Bug 1778888
| Summary: | CI: the following tags from the release could not be imported to stable-initial after five minutes | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Antonio Murdaca <amurdaca> |
| Component: | ImageStreams | Assignee: | Ricardo Maraschini <rmarasch> |
| Status: | CLOSED DEFERRED | QA Contact: | XiuJuan Wang <xiuwang> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.11.0 | CC: | adam.kaplan, agreene, aos-bugs, bbennett, bparees, cewong, erich, htariq, jcallen, jesusr, jokerman, lxia, obulatov, qiwan, shurley, skuznets, vlaad, wking, wzheng |
| Target Milestone: | --- | ||
| Target Release: | 4.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-05-18 10:56:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Antonio Murdaca
2019-12-02 18:01:46 UTC
Failing for vSphere as well: https://prow.svc.ci.openshift.org/?job=*vsphere*4.3* Moving to Image since this seems to be related to imagestream import, and not the registry. Slack thread with relevant conversation: https://coreos.slack.com/archives/CBN38N3MW/p1575286491375900 @Steve do you know how we can obtain the imagestream tag YAML so we can see what errors (if any) are being reported when this occurs? 33 of our e2e tests in the past 24 hours hit this (~2% of our failures for those jobs) [1]. That's not a huge flake rate, but it's not negligible either. Dunno if that warrants a severity bump? [1]: https://search.svc.ci.openshift.org/chart?search=the%20following%20tags%20from%20the%20release%20could%20not%20be%20imported *** Bug 1785421 has been marked as a duplicate of this bug. *** Still seeing this issue across a number of different release installer and upgrade jobs: https://search.svc.ci.openshift.org/?search=the+following+tags+from+the+release+could+not+be+imported+to+stable+after+five+minutes&maxAge=24h&context=3&type=all https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14300 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/14298 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-okd-installer-e2e-aws-4.3/92 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade-4.2/343 Steve, can you help us get YAMLs for the problematic image stream tags? The imagestream json is collected in artifacts. here's a recent recurrence: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-okd-installer-e2e-aws-4.3/109 and here's the imagestream json: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-okd-installer-e2e-aws-4.3/109/artifacts/build-resources/imagestreams.json spec: { "name": "aws-machine-controllers", "annotations": { "io.openshift.build.commit.id": "6814fc34f30a4c98d5edeb0a4ef73bbd03fa959b", "io.openshift.build.commit.ref": "master", "io.openshift.build.source-location": "https://github.com/openshift/cluster-api-provider-aws" }, "from": { "kind": "DockerImage", "name": "registry.svc.ci.openshift.org/origin/4.3-2019-10-31-164446@sha256:2b0c1ebc7e91a02d1f6d9a4345b978710162cca51b7d971a72086bbe6c302f93" }, "generation": 7, "importPolicy": {}, "referencePolicy": { "type": "Local" } }, status (w/ error): { "tag": "aws-machine-controllers", "items": [ { "created": "2020-01-31T19:21:37Z", "dockerImageReference": "docker-registry.default.svc:5000/ci-op-t9hf5z2f/stable@sha256:dc4ea87b51a509ae09665ff167d063d3e5f4fc3e577bf8faafad809b7fecf5c2", "image": "sha256:dc4ea87b51a509ae09665ff167d063d3e5f4fc3e577bf8faafad809b7fecf5c2", "generation": 1 } ], "conditions": [ { "type": "ImportSuccess", "status": "False", "lastTransitionTime": "2020-01-31T19:23:54Z", "reason": "NotFound", "message": "dockerimage.image.openshift.io \"registry.svc.ci.openshift.org/origin/4.3-2019-10-31-164446@sha256:2b0c1ebc7e91a02d1f6d9a4345b978710162cca51b7d971a72086bbe6c302f93\" not found", "generation": 6 } ] }, Which is a bit strange since it appears we did import it successfully at least once (and it's a SHA import, so it shouldn't change)? The image stream origin/4.3-2019-10-31-164446 doesn't exist on registry.svc.ci.openshift.org, so I'd assume it has already been removed. https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade-4.2/343 the following tags from the release could not be imported to stable after five minutes: aws-machine-controllers, ... https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade-4.2/343/artifacts/build-resources/imagestreams.json { "metadata": { "name": "stable", "namespace": "ci-op-vgcmz7jf", "selfLink": "/apis/image.openshift.io/v1/namespaces/ci-op-vgcmz7jf/imagestreams/stable", "uid": "d5f6b106-3688-11ea-a7f4-42010a8e0004", "resourceVersion": "421944318", "generation": 3, "creationTimestamp": "2020-01-14T04:46:32Z" }, "spec": { ... { "name": "aws-machine-controllers", "annotations": { "io.openshift.build.commit.id": "fcb0301bc83c38c9f904c61a63608996c5c7bbd9", "io.openshift.build.commit.ref": "release-4.2", "io.openshift.build.source-location": "https://github.com/openshift/cluster-api-provider-aws" }, "from": { "kind": "DockerImage", "name": "registry.svc.ci.openshift.org/ocp/4.2-2020-01-11-040651@sha256:c343de64c79c370f9a8355886d327fb15b0b4beec355358844f3d4f80ece081f" }, "generation": 3, "importPolicy": {}, "referencePolicy": { "type": "Local" } }, ... }, "status": { ... { "tag": "aws-machine-controllers", "items": [ { "created": "2020-01-14T04:46:35Z", "dockerImageReference": "docker-registry.default.svc:5000/ci-op-vgcmz7jf/stable@sha256:c343de64c79c370f9a8355886d327fb15b0b4beec355358844f3d4f80ece081f", "image": "sha256:c343de64c79c370f9a8355886d327fb15b0b4beec355358844f3d4f80ece081f", "generation": 1 } ] }, ... That's interesting, everything looks fine expect generation in status. It should be 3. *** Bug 1794836 has been marked as a duplicate of this bug. *** *** Bug 1820336 has been marked as a duplicate of this bug. *** *** Bug 1763588 has been marked as a duplicate of this bug. *** Seems like we already have a JIRA card to address this: https://issues.redhat.com/browse/DEVEXP-569. |