+++ This bug was initially created as a clone of Bug #1901982 +++ test: [sig-builds][Feature:Builds] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service is failing frequently in CI, see search results: https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-builds%5C%5D%5C%5BFeature%3ABuilds%5C%5D+build+can+reference+a+cluster+service++with+a+build+being+created+from+new-build+should+be+able+to+run+a+build+that+references+a+cluster+service Failed job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.7/1331837576172015616 From the logs, it appears this job is depending on a docker image, and that, given the new docker pull rate limiting, the test is failing due to having hit that rate limit. ``` error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://busybox" as "busybox": Error initializing source docker://busybox:latest: Error reading manifest latest in docker.io/library/busybox: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit ``` --- Additional comment from Joel Speed on 2020-11-26 14:43:39 UTC --- Also seeing a significant number of failures for "[sig-builds][Feature:Builds] prune builds based on settings in the buildconfig should prune completed builds based on the successfulBuildsHistoryLimit setting" reporting docker hub rate limiting issues as well https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere/1331455308672798720 --- Additional comment from Joel Speed on 2020-11-26 14:50:03 UTC --- Also seeing a significant number of failures for "[sig-auth][Feature:SecurityContextConstraints] TestPodDefaultCapabilities" reporting docker hub rate limiting issues as well https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere/1331447713761333248
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#periodic-ci-openshift-release-master-ocp-4.6-e2e-vsphere-upi https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.6-e2e-vsphere-upi/1332236668144455680 See 'oc new-app -h' for examples. W1127 08:56:40.309674 28517 dockerimagelookup.go:237] container image registry lookup failed: docker.io/openshift/hello-openshift:latest: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit error: local file access failed with: stat docker.io/openshift/hello-openshift: no such file or directory error: unable to locate any images in image streams, templates loaded in accessible projects, template files, local docker images with name "docker.io/openshift/hello-openshift" Argument 'docker.io/openshift/hello-openshift' was classified as an image, image~source, or loaded template reference. The 'oc new-app' command will match arguments to the following types: 1. Images tagged into image streams in the current project or the 'openshift' project - if you don't specify a tag, we'll add ':latest' 2. Images in the Docker Hub, on remote registries, or on the local Docker engine 3. Templates in the current project or the 'openshift' project 4. Git repository URLs or local paths that point to Git repositories --allow-missing-images can be used to point to an image that does not exist yet. See 'oc new-app -h' for examples. [AfterEach] github.com/openshift/origin/test/extended/builds/service.go:33 Nov 27 08:56:40.315: INFO: Dumping pod state for namespace e2e-test-build-service-dzdw5 Nov 27 08:56:40.315: INFO: Running 'oc --namespace=e2e-test-build-service-dzdw5 --kubeconfig=/tmp/tmp.sfUoFPwTvK get pods -o yaml' Nov 27 08:56:40.520: INFO: apiVersion: v1 items: [] kind: List metadata: resourceVersion: "" selfLink: "" Nov 27 08:56:40.520: INFO: Dumping configMap state for namespace e2e-test-build-service-dzdw5 Nov 27 08:56:40.520: INFO: Running 'oc --namespace=e2e-test-build-service-dzdw5 --kubeconfig=/tmp/tmp.sfUoFPwTvK get configmaps -o yaml' Nov 27 08:56:40.748: INFO: apiVersion: v1 items: [] kind: List metadata: resourceVersion: "" selfLink: "" [AfterEach] [sig-builds][Feature:Builds] build can reference a cluster service github.com/openshift/origin/test/extended/util/client.go:133 STEP: Collecting events from namespace "e2e-test-build-service-dzdw5". STEP: Found 0 events. Nov 27 08:56:40.817: INFO: POD NODE PHASE GRACE CONDITIONS Nov 27 08:56:40.817: INFO: Nov 27 08:56:40.892: INFO: skipping dumping cluster info - cluster too large Nov 27 08:56:40.944: INFO: Deleted {user.openshift.io/v1, Resource=users e2e-test-build-service-dzdw5-user}, err: <nil> Nov 27 08:56:40.996: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-build-service-dzdw5}, err: <nil> Nov 27 08:56:41.026: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens wEbY3OXyTg6w6KTKpgXoxwAAAAAAAAAA}, err: <nil> [AfterEach] [sig-builds][Feature:Builds] build can reference a cluster service github.com/openshift/origin/test/extended/util/client.go:134 Nov 27 08:56:41.026: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-test-build-service-dzdw5" for this suite. Nov 27 08:56:41.056: INFO: Running AfterSuite actions on all nodes Nov 27 08:56:41.056: INFO: Running AfterSuite actions on node 1 fail [github.com/openshift/origin/test/extended/builds/service.go:45]: Unexpected error: <*util.ExitError | 0xc001888b70>: { Cmd: "oc --namespace=e2e-test-build-service-dzdw5 --kubeconfig=/tmp/configfile699515232 new-app docker.io/openshift/hello-openshift", StdErr: "W1127 08:56:40.309674 28517 dockerimagelookup.go:237] container image registry lookup failed: docker.io/openshift/hello-openshift:latest: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\nerror: local file access failed with: stat docker.io/openshift/hello-openshift: no such file or director\n...\nhift' project\n - if you don't specify a tag, we'll add ':latest'\n 2. Images in the Docker Hub, on remote registries, or on the local Docker engine\n 3. Templates in the current project or the 'openshift' project\n 4. Git repository URLs or local paths that point to Git repositories\n\n--allow-missing-images can be used to point to an image that does not exist yet.\n\nSee 'oc new-app -h' for examples.", ExitError: { ProcessState: { pid: 28517, status: 256, rusage: { Utime: {Sec: 0, Usec: 162483}, Stime: {Sec: 0, Usec: 34616}, Maxrss: 139648, Ixrss: 0, Idrss: 0, Isrss: 0, Minflt: 8285, Majflt: 0, Nswap: 0, Inblock: 0, Oublock: 0, Msgsnd: 0, Msgrcv: 0, Nsignals: 0, Nvcsw: 415, Nivcsw: 233, }, }, Stderr: nil, }, } exit status 1 occurred
*** This bug has been marked as a duplicate of bug 1902029 ***
Still fail in https://prow.ci.openshift.org/job-history/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.6 https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.6/1419378227696111616 fail [github.com/openshift/origin/test/extended/builds/service.go:64]: Unexpected error: <*util.ExitError | 0xc00210a5a0>: { Cmd: "oc --namespace=e2e-test-build-service-zgvhz --kubeconfig=/tmp/configfile259238148 new-build -D - --to test:latest", StdErr: "W0725 20:39:27.968654 75325 dockerimagelookup.go:237] container image registry lookup failed: image-registry.openshift-image-registry.svc:5000/openshift/tools:latest: unknown: unable to pull manifest from registry.build01.ci.openshift.org/ci-op-f3ylwit7/release-scratch@sha256:59cbf5ebc4cf4880388d7be0ff1e300689ef07f4697e10a5028ac8f1581d3251: manifest unknown: manifest unknown\nerror: unable to locate \n...\nged into image streams in the current project or the 'openshift' project\n - if you don't specify a tag, we'll add ':latest'\n 2. Images in the Docker Hub, on remote registries, or on the local Docker engine\n 3. Git repository URLs or local paths that point to Git repositories\n\n--allow-missing-images can be used to force the use of an image that was not matched\n\nSee 'oc new-build -h' for examples.", ExitError: { ProcessState: { pid: 75325, status: 256, rusage: { Utime: {Sec: 0, Usec: 153968}, Stime: {Sec: 0, Usec: 28510}, Maxrss: 134020, Ixrss: 0, Idrss: 0, Isrss: 0, Minflt: 4779, Majflt: 0, Nswap: 0, Inblock: 0, Oublock: 0, Msgsnd: 0, Msgrcv: 0, Nsignals: 0, Nvcsw: 850, Nivcsw: 2, }, }, Stderr: nil, }, } exit status 1 occurred
that error is a little truncated A more complete one from the job run XiuJuan specified: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://image-registry.openshift-image-registry.svc:5000/openshift/tools:latest" as "image-registry.openshift-image-registry.svc:5000/openshift/tools:latest": Error initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift/tools:latest: Error reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift/tools: unknown: unable to pull manifest from registry.build01.ci.openshift.org/ci-op-f3ylwit7/release-scratch@sha256:59cbf5ebc4cf4880388d7be0ff1e300689ef07f4697e10a5028ac8f1581d3251: manifest unknown: manifest unknown as we are talking about an e2e-aws-mirror jobs and that manifest issue vs. the original docker.io throttling but this item noted, I feel Scott Dodson that this BZ was reopened in error yes, the same e2e is failing, but for an entirely different cause I am reclosing this as dupe. @XiuJuan - please open a new bz. As to where to open it against, given we are trying to pull an imagestream image from the image registry, start with the image registry or image components (both owned by Oleg and his team). Then Oleg and team can triage and see if things land with the image registry and operating in a mirrored environment. Or if there is something amiss with that the mirrors are set up, and the bz can then be routed accordingly. *** This bug has been marked as a duplicate of bug 1902029 ***
Thanks, Gabe I will check more and open a bug on image|image registry component.