Bug 1902221 - [sig-builds][Feature:Builds] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service
Summary: [sig-builds][Feature:Builds] build can reference a cluster service with a bu...
Keywords:
Status: CLOSED DUPLICATE of bug 1902029
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 4.6.z
Assignee: rvanderp
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On: 1901982
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-27 12:20 UTC by Jiří Mencák
Modified: 2021-07-27 03:35 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1901982
Environment:
[sig-builds][Feature:Builds] build can reference a cluster service with a build being created from new-build should be able to run a build that references a cluster service [sig-builds][Feature:Builds] prune builds based on settings in the buildconfig should prune completed builds based on the successfulBuildsHistoryLimit setting [sig-auth][Feature:SecurityContextConstraints] TestPodDefaultCapabilities [sig-builds][Feature:Builds] Optimized image builds should succeed
Last Closed: 2021-07-26 12:38:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Jiří Mencák 2020-11-27 12:20:39 UTC
+++ This bug was initially created as a clone of Bug #1901982 +++

test:
[sig-builds][Feature:Builds] build can reference a cluster service  with a build being created from new-build should be able to run a build that references a cluster service 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-builds%5C%5D%5C%5BFeature%3ABuilds%5C%5D+build+can+reference+a+cluster+service++with+a+build+being+created+from+new-build+should+be+able+to+run+a+build+that+references+a+cluster+service


Failed job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.7/1331837576172015616

From the logs, it appears this job is depending on a docker image, and that, given the new docker pull rate limiting, the test is failing due to having hit that rate limit.

```
error: build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://busybox" as "busybox": Error initializing source docker://busybox:latest: Error reading manifest latest in docker.io/library/busybox: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
```

--- Additional comment from Joel Speed on 2020-11-26 14:43:39 UTC ---

Also seeing a significant number of failures for "[sig-builds][Feature:Builds] prune builds based on settings in the buildconfig should prune completed builds based on the successfulBuildsHistoryLimit setting" reporting docker hub rate limiting issues as well

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere/1331455308672798720

--- Additional comment from Joel Speed on 2020-11-26 14:50:03 UTC ---

Also seeing a significant number of failures for "[sig-auth][Feature:SecurityContextConstraints] TestPodDefaultCapabilities" reporting docker hub rate limiting issues as well

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.7-e2e-vsphere/1331447713761333248

Comment 1 Jiří Mencák 2020-11-27 12:22:50 UTC
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.6-informing#periodic-ci-openshift-release-master-ocp-4.6-e2e-vsphere-upi
https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ocp-4.6-e2e-vsphere-upi/1332236668144455680

See 'oc new-app -h' for examples.
W1127 08:56:40.309674   28517 dockerimagelookup.go:237] container image registry lookup failed: docker.io/openshift/hello-openshift:latest: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
error:  local file access failed with: stat docker.io/openshift/hello-openshift: no such file or directory
error: unable to locate any images in image streams, templates loaded in accessible projects, template files, local docker images with name "docker.io/openshift/hello-openshift"

Argument 'docker.io/openshift/hello-openshift' was classified as an image, image~source, or loaded template reference.

The 'oc new-app' command will match arguments to the following types:

  1. Images tagged into image streams in the current project or the 'openshift' project
     - if you don't specify a tag, we'll add ':latest'
  2. Images in the Docker Hub, on remote registries, or on the local Docker engine
  3. Templates in the current project or the 'openshift' project
  4. Git repository URLs or local paths that point to Git repositories

--allow-missing-images can be used to point to an image that does not exist yet.

See 'oc new-app -h' for examples.
[AfterEach] 
  github.com/openshift/origin/test/extended/builds/service.go:33
Nov 27 08:56:40.315: INFO: Dumping pod state for namespace e2e-test-build-service-dzdw5
Nov 27 08:56:40.315: INFO: Running 'oc --namespace=e2e-test-build-service-dzdw5 --kubeconfig=/tmp/tmp.sfUoFPwTvK get pods -o yaml'
Nov 27 08:56:40.520: INFO: apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Nov 27 08:56:40.520: INFO: Dumping configMap state for namespace e2e-test-build-service-dzdw5
Nov 27 08:56:40.520: INFO: Running 'oc --namespace=e2e-test-build-service-dzdw5 --kubeconfig=/tmp/tmp.sfUoFPwTvK get configmaps -o yaml'
Nov 27 08:56:40.748: INFO: apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
[AfterEach] [sig-builds][Feature:Builds] build can reference a cluster service
  github.com/openshift/origin/test/extended/util/client.go:133
STEP: Collecting events from namespace "e2e-test-build-service-dzdw5".
STEP: Found 0 events.
Nov 27 08:56:40.817: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
Nov 27 08:56:40.817: INFO: 
Nov 27 08:56:40.892: INFO: skipping dumping cluster info - cluster too large
Nov 27 08:56:40.944: INFO: Deleted {user.openshift.io/v1, Resource=users  e2e-test-build-service-dzdw5-user}, err: <nil>
Nov 27 08:56:40.996: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients  e2e-client-e2e-test-build-service-dzdw5}, err: <nil>
Nov 27 08:56:41.026: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens  wEbY3OXyTg6w6KTKpgXoxwAAAAAAAAAA}, err: <nil>
[AfterEach] [sig-builds][Feature:Builds] build can reference a cluster service
  github.com/openshift/origin/test/extended/util/client.go:134
Nov 27 08:56:41.026: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready
STEP: Destroying namespace "e2e-test-build-service-dzdw5" for this suite.
Nov 27 08:56:41.056: INFO: Running AfterSuite actions on all nodes
Nov 27 08:56:41.056: INFO: Running AfterSuite actions on node 1
fail [github.com/openshift/origin/test/extended/builds/service.go:45]: Unexpected error:
    <*util.ExitError | 0xc001888b70>: {
        Cmd: "oc --namespace=e2e-test-build-service-dzdw5 --kubeconfig=/tmp/configfile699515232 new-app docker.io/openshift/hello-openshift",
        StdErr: "W1127 08:56:40.309674   28517 dockerimagelookup.go:237] container image registry lookup failed: docker.io/openshift/hello-openshift:latest: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\nerror:  local file access failed with: stat docker.io/openshift/hello-openshift: no such file or director\n...\nhift' project\n     - if you don't specify a tag, we'll add ':latest'\n  2. Images in the Docker Hub, on remote registries, or on the local Docker engine\n  3. Templates in the current project or the 'openshift' project\n  4. Git repository URLs or local paths that point to Git repositories\n\n--allow-missing-images can be used to point to an image that does not exist yet.\n\nSee 'oc new-app -h' for examples.",
        ExitError: {
            ProcessState: {
                pid: 28517,
                status: 256,
                rusage: {
                    Utime: {Sec: 0, Usec: 162483},
                    Stime: {Sec: 0, Usec: 34616},
                    Maxrss: 139648,
                    Ixrss: 0,
                    Idrss: 0,
                    Isrss: 0,
                    Minflt: 8285,
                    Majflt: 0,
                    Nswap: 0,
                    Inblock: 0,
                    Oublock: 0,
                    Msgsnd: 0,
                    Msgrcv: 0,
                    Nsignals: 0,
                    Nvcsw: 415,
                    Nivcsw: 233,
                },
            },
            Stderr: nil,
        },
    }
    exit status 1
occurred

Comment 2 Gabe Montero 2020-11-30 21:08:18 UTC

*** This bug has been marked as a duplicate of bug 1902029 ***

Comment 4 XiuJuan Wang 2021-07-26 12:23:13 UTC
Still fail in https://prow.ci.openshift.org/job-history/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.6

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-mirrors-4.6/1419378227696111616

fail [github.com/openshift/origin/test/extended/builds/service.go:64]: Unexpected error:
    <*util.ExitError | 0xc00210a5a0>: {
        Cmd: "oc --namespace=e2e-test-build-service-zgvhz --kubeconfig=/tmp/configfile259238148 new-build -D - --to test:latest",
        StdErr: "W0725 20:39:27.968654   75325 dockerimagelookup.go:237] container image registry lookup failed: image-registry.openshift-image-registry.svc:5000/openshift/tools:latest: unknown: unable to pull manifest from registry.build01.ci.openshift.org/ci-op-f3ylwit7/release-scratch@sha256:59cbf5ebc4cf4880388d7be0ff1e300689ef07f4697e10a5028ac8f1581d3251: manifest unknown: manifest unknown\nerror: unable to locate \n...\nged into image streams in the current project or the 'openshift' project\n     - if you don't specify a tag, we'll add ':latest'\n  2. Images in the Docker Hub, on remote registries, or on the local Docker engine\n  3. Git repository URLs or local paths that point to Git repositories\n\n--allow-missing-images can be used to force the use of an image that was not matched\n\nSee 'oc new-build -h' for examples.",
        ExitError: {
            ProcessState: {
                pid: 75325,
                status: 256,
                rusage: {
                    Utime: {Sec: 0, Usec: 153968},
                    Stime: {Sec: 0, Usec: 28510},
                    Maxrss: 134020,
                    Ixrss: 0,
                    Idrss: 0,
                    Isrss: 0,
                    Minflt: 4779,
                    Majflt: 0,
                    Nswap: 0,
                    Inblock: 0,
                    Oublock: 0,
                    Msgsnd: 0,
                    Msgrcv: 0,
                    Nsignals: 0,
                    Nvcsw: 850,
                    Nivcsw: 2,
                },
            },
            Stderr: nil,
        },
    }
    exit status 1
occurred

Comment 5 Gabe Montero 2021-07-26 12:38:48 UTC
that error is a little truncated

A more complete one from the job run XiuJuan specified:

build error: failed to pull image: After retrying 2 times, Pull image still failed due to error: while pulling "docker://image-registry.openshift-image-registry.svc:5000/openshift/tools:latest" as "image-registry.openshift-image-registry.svc:5000/openshift/tools:latest": Error initializing source docker://image-registry.openshift-image-registry.svc:5000/openshift/tools:latest: Error reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift/tools: unknown: unable to pull manifest from registry.build01.ci.openshift.org/ci-op-f3ylwit7/release-scratch@sha256:59cbf5ebc4cf4880388d7be0ff1e300689ef07f4697e10a5028ac8f1581d3251: manifest unknown: manifest unknown

as we are talking about an e2e-aws-mirror jobs and that manifest issue vs. the original docker.io throttling but this item noted, I feel Scott Dodson that this BZ was reopened in error


yes, the same e2e is failing, but for an entirely different cause

I am reclosing this as dupe.

@XiuJuan - please open a new bz.  As to where to open it against, given we are trying to pull an imagestream image from the image registry, start with the image registry or image components (both owned by Oleg and his team).  Then Oleg and team can triage and see if things land with the image registry and operating in a mirrored environment.  Or if there is something amiss with that the mirrors are set up, and the bz can then be routed accordingly.

*** This bug has been marked as a duplicate of bug 1902029 ***

Comment 6 XiuJuan Wang 2021-07-27 03:35:18 UTC
Thanks, Gabe
I will check more and open a bug on image|image registry component.


Note You need to log in before you can comment on or make changes to this bug.