Bug 2091110
| Summary: | [sig-node] should not encounter ErrImagePull in non-openshift namespace pods: denied: requested access to the resource is denied | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Dennis Periquet <dperique> |
| Component: | Test Framework | Assignee: | OpenShift Release Oversight <openshift-release-oversight> |
| Status: | CLOSED WONTFIX | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.11 | CC: | ehila, openshift-release-oversight |
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-04-30 18:04:53 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2117301 | ||
|
Description
Dennis Periquet
2022-05-27 15:07:31 UTC
relevant slack thread with more info: https://coreos.slack.com/archives/C01CQA76KMX/p1653664318699599 trying to determine where docker.io is being used ... this code has been there for a while to avoid docker.io: https://github.com/openshift/release/blob/8bbb676c5f8b158c2c8d69f3ea11bbefcc9fbb4f/ci-operator/step-registry/openshift/e2e/test/openshift-e2e-test-commands.sh#L45 # Override the upstream docker.io registry due to issues with rate limiting # https://bugzilla.redhat.com/show_bug.cgi?id=1895107 # sjenning: TODO: use of personal repo is temporary; should find long term location for these mirrored images export KUBE_TEST_REPO_LIST=${HOME}/repo_list.yaml cat <<EOF > ${KUBE_TEST_REPO_LIST} dockerLibraryRegistry: quay.io/sjenning dockerGluster: quay.io/sjenning EOF but is overridden by: https://github.com/openshift/origin/blob/28cbfd55f5bda6a814d98569906dbabf5f00b68c/cmd/openshift-tests/openshift-tests.go#L44-L46 if len(os.Getenv("KUBE_TEST_REPO_LIST")) > 0 { fmt.Fprintln(os.Stderr, "warning: KUBE_TEST_REPO_LIST may not be set when using openshift-tests and will be ignored") os.Setenv("KUBE_TEST_REPO_LIST", "") which has also been there for a while. Just noting that this test has been flaking for a long time which is why we haven't really noticed it. See the flake here (look for [sig-node] should not encounter ErrImagePull in openshift namespace pods): https://testgrid.k8s.io/redhat-openshift-ocp-release-4.11-informing#periodic-ci-openshift-multiarch-master-nightly-4.11-upgrade-from-nightly-4.10-ocp-e2e-aws-arm64 Two sets of those failures in the original description are failures created by a test: https://github.com/openshift/origin//blob/bba29efd04e32ca75a62d2a117c6e9ce333e6e77/vendor/k8s.io/kubernetes/test/e2e/common/node/runtime.go#L379 https://github.com/openshift/origin//blob/bba29efd04e32ca75a62d2a117c6e9ce333e6e77/vendor/k8s.io/kubernetes/test/e2e/common/node/runtime.go#L390 So, I will ensure we skip them in this PR: https://github.com/openshift/origin/pull/27202 In this thread: https://coreos.slack.com/archives/C01CQA76KMX/p1654051864855859?thread_ts=1653664318.699599&cid=C01CQA76KMX , JustinP pointed out that for at least one of those tests, the ErrImagePull was expected. Specifically, this: ns/e2e-deployment-2940 pod/webserver-deployment-57ccb67bb8-8tjcs node/ip-10-0-234-247.us-west-2.compute.internal uid/4cb7aeeb-6ffe-42a0-b1c6-5110ced25f07 container/httpd: reason/ContainerWait cause/ErrImagePull: rpc error: code = Unknown desc = reading manifest 404 in docker.io/library/webserver: errors: denied: requested access to the resource is denied unauthorized: authentication required is expected. The message was mis-interpreted as me but the message is trying to convey they the test is looking for webserver:404 which translates to docker.io/library/webserver:404 which is documented in the origin repo (look for webserver:404) as not an image that does not exist. So, I'm pivoting on this to modify the test to take this into account and not flag those ErrImagePull events as an error/failure; see https://github.com/openshift/origin/pull/27202 Tests now show either flaky or success and no longer fail https://testgrid.k8s.io/redhat-openshift-ocp-release-4.11-informing#periodic-ci-openshift-multiarch-master-nightly-4.11-upgrade-from-nightly-4.10-ocp-e2e-aws-arm64&include-filter-by-regex=should%20not%20encounter%20ErrImagePull%20in%20openshift%20namespace%20pods OCP is no longer using Bugzilla and this bug appears to have been left in an orphaned state. If the bug is still relevant, please open a new issue in the OCPBUGS Jira project: https://issues.redhat.com/projects/OCPBUGS/summary |