test: node client cert requests armoring: deny pod's access to /config/master API endpoint is failing frequently in CI, see search results: https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=node+client+cert+requests+armoring%3A+deny+pod%27s+access+to+%2Fconfig%2Fmaster+API+endpoint fail [github.com/openshift/origin/test/extended/csrapprover/csrapprover.go:48]: Unexpected error: <*errors.errorString | 0xc0002981c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-remote-libvirt-s390x-4.3/1293578108904935424
failure context ============= [It] deny pod's access to /config/master API endpoint [Suite:openshift/conformance/parallel] /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/csrapprover/csrapprover.go:36 Aug 12 17:31:23.259: INFO: Running 'oc --namespace=e2e-test-cluster-client-cert-bn47n --config=/tmp/configfile787210623 run get-bootstrap-creds --labels name=get-bootstrap-creds --image quay.io/fedora/fedora:32-x86_64 --restart Never --command -- /bin/bash -c sleep infinity' [AfterEach] node client cert requests armoring: /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/client.go:101 STEP: Collecting events from namespace "e2e-test-cluster-client-cert-bn47n". STEP: Found 5 events. Aug 12 17:34:25.311: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for get-bootstrap-creds: {default-scheduler } Scheduled: Successfully assigned e2e-test-cluster-client-cert-bn47n/get-bootstrap-creds to ci-op-pbbtjczd-416f4-lv9g6-worker-0-hbwd6 Aug 12 17:34:25.311: INFO: At 2020-08-12 17:31:26 +0000 UTC - event for get-bootstrap-creds: {kubelet ci-op-pbbtjczd-416f4-lv9g6-worker-0-hbwd6} Pulling: Pulling image "quay.io/fedora/fedora:32-x86_64" Aug 12 17:34:25.311: INFO: At 2020-08-12 17:31:38 +0000 UTC - event for get-bootstrap-creds: {kubelet ci-op-pbbtjczd-416f4-lv9g6-worker-0-hbwd6} Pulled: Successfully pulled image "quay.io/fedora/fedora:32-x86_64" Aug 12 17:34:25.311: INFO: At 2020-08-12 17:31:38 +0000 UTC - event for get-bootstrap-creds: {kubelet ci-op-pbbtjczd-416f4-lv9g6-worker-0-hbwd6} Created: Created container get-bootstrap-creds Aug 12 17:34:25.311: INFO: At 2020-08-12 17:31:38 +0000 UTC - event for get-bootstrap-creds: {kubelet ci-op-pbbtjczd-416f4-lv9g6-worker-0-hbwd6} Started: Started container get-bootstrap-creds Aug 12 17:34:25.451: INFO: POD NODE PHASE GRACE CONDITIONS Aug 12 17:34:25.451: INFO: get-bootstrap-creds ci-op-pbbtjczd-416f4-lv9g6-worker-0-hbwd6 Failed [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-08-12 17:31:24 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2020-08-12 17:31:24 +0000 UTC ContainersNotReady containers with unready status: [get-bootstrap-creds]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2020-08-12 17:31:24 +0000 UTC ContainersNotReady containers with unready status: [get-bootstrap-creds]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-08-12 17:31:24 +0000 UTC }] Aug 12 17:34:25.451: INFO: Aug 12 17:34:25.596: INFO: get-bootstrap-creds[e2e-test-cluster-client-cert-bn47n].container[get-bootstrap-creds].log standard_init_linux.go:211: exec user process caused "exec format error" Aug 12 17:34:25.731: INFO: skipping dumping cluster info - cluster too large Aug 12 17:34:25.934: INFO: Deleted {user.openshift.io/v1, Resource=users e2e-test-cluster-client-cert-bn47n-user}, err: <nil> Aug 12 17:34:26.152: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-cluster-client-cert-bn47n}, err: <nil> Aug 12 17:34:26.339: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens P8J7qchYTRC8PB-c4PbdZQAAAAAAAAAA}, err: <nil> [AfterEach] node client cert requests armoring: /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:152 Aug 12 17:34:26.339: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready STEP: Destroying namespace "e2e-test-cluster-client-cert-bn47n" for this suite. Aug 12 17:34:26.682: INFO: Running AfterSuite actions on all nodes Aug 12 17:34:26.682: INFO: Running AfterSuite actions on node 1 fail [github.com/openshift/origin/test/extended/csrapprover/csrapprover.go:48]: Unexpected error: <*errors.errorString | 0xc0002981c0>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred failed: (3m11s) 2020-08-12T17:34:26 "node client cert requests armoring: deny pod's access to /config/master API endpoint [Suite:openshift/conformance/parallel]" ============= in particular standard_init_linux.go:211: exec user process caused "exec format error" test suite is e2e-remote-libvirt-s390x-4.3 so this is s390x trying to exec a x86_64 binary
*** Bug 1868469 has been marked as a duplicate of this bug. ***
changed in 4.6 https://github.com/openshift/origin/pull/25087 backported in 4.5 https://bugzilla.redhat.com/show_bug.cgi?id=1846091 4.4 https://bugzilla.redhat.com/show_bug.cgi?id=1862171 4.3 https://bugzilla.redhat.com/show_bug.cgi?id=1867402 xref https://bugzilla.redhat.com/show_bug.cgi?id=1845792 Node team did backports to 4.4 and 4.3 in response to https://bugzilla.redhat.com/show_bug.cgi?id=1867613 but change originated with Cloud team.
Failing against all releases that run this test https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/?job=*e2e-remote-libvirt-s390x*
i don't think this bug is about the Cloud Compute component, it should probably be addressed to the node team.
Assigned to Cloud because https://bugzilla.redhat.com/show_bug.cgi?id=1845792, the change that introduced this break, was assign to Cloud and Alberto
ack, thanks Seth. i'll spend a little more time reviewing those.
i have added a PR[0] to address this issue to bz#1868760 with backports to 4.5/4.4/4.3 should we close this issue as duplicate or is there another change needed to fix this as well? [0] https://github.com/openshift/origin/pull/25480
That should work. $ skopeo inspect --override-arch=s390x docker://docker.io/fedora:32 { "Name": "docker.io/library/fedora", "Digest": "sha256:d6a6d60fda1b22b6d5fe3c3b2abe2554b60432b7b215adc11a2b5fae16f50188", "RepoTags": [ "20", "21", "22", "23", "24", "25", "26-modular", "26", "27", "28", "29", "30", "31", "32", "33", "34", "branched", "heisenbug", "latest", "modular", "rawhide" ], "Created": "2020-07-10T18:42:35.793370708Z", "DockerVersion": "18.09.7", "Labels": { "maintainer": "Clement Verna \u003ccverna\u003e" }, "Architecture": "s390x", "Os": "linux", "Layers": [ "sha256:258eddf3cf5180969401b06c6836e098764ef190af1afa5af6178521cbebbe83" ], "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "DISTTAG=f32container", "FGC=f32", "FBR=f32" ] } I vote for dup'ing.
thanks Seth! i am marking this closed as a duplicate of bz#1868760 *** This bug has been marked as a duplicate of bug 1868760 ***