Bug 1971540

Summary: Hive operator unable to use impersonate headers while applying role bindings
Product: OpenShift Container Platform Reporter: Osher De Paz <odepaz>
Component: HiveAssignee: Devan Goodwin <dgoodwin>
Status: CLOSED DUPLICATE QA Contact: wang lin <lwan>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.8CC: jmatthew, lwan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-15 20:02:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Hive operator logs
none
make deploy none

Description Osher De Paz 2021-06-14 10:06:46 UTC 
Created attachment 1790976 [details]
Hive operator logs

Description of problem:

Using a new ocp version (4.8.0-0.nightly-2021-06-12-174011) I'm unable to deploy hive from image registry.ci.openshift.org/openshift/hive-v4.0:hive

Logs of hive-operator pod are complaining about the following error:
time="2021-06-14T09:45:56Z" level=info msg="applying asset with GC" asset=config/rbac/hive_admin_role_binding.yaml controller=hive
time="2021-06-14T09:45:56Z" level=warning msg="running the apply command failed" controller=hive error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\"" stderr= stdout=
time="2021-06-14T09:45:56Z" level=error msg="error applying asset" asset=config/rbac/hive_admin_role_binding.yaml controller=hive error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\""
time="2021-06-14T09:45:56Z" level=error msg="error deploying Hive" controller=hive error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\""
time="2021-06-14T09:45:56Z" level=info msg="reconcile complete" controller=hive elapsedMillis=520 elapsedMillisGT=0 outcome=unspecified
time="2021-06-14T09:45:56Z" level=error msg="Reconciler error" _name=hive-controller error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\"" name=hive namespace=

Older ocp versions like 4.8.0-0.nightly-2021-06-03-221810 (probably everything before 2021-06-10) don't have this problem.

Version-Release number of selected component (if applicable):
tag registry.ci.openshift.org/openshift/hive-v4.0:hive which is updated regularly, but I'm pretty sure it's not version-related but a breakage because of ocp.

How reproducible: 100%


Steps to Reproduce:
1. Install a new nightly image of OCP (e.g. 4.8.0-0.nightly-2021-06-12-174011)
2. Use Makefile in hive repo: make deploy
3. Check logs of pod hive-operator in hive namespace

Actual results:

hive cannot complete installation, and controllers cannot start.

Expected results:

Full installation of hive operator, with controllers up and running.

Additional info:

Relevant builds
Passing build - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-assisted-service-master-e2e-metal-assisted-operator-disconnected-periodic/1402777674690072576
Failing build - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-assisted-service-master-e2e-metal-assisted-operator-disconnected-periodic/1403140177861283840

Unfortunately no relevant logs of hive in those builds, but we can inspect ocp changes there.
Comment 1 Osher De Paz 2021-06-14 10:07:19 UTC 
Created attachment 1790978 [details]
make deploy
Comment 4 Devan Goodwin 2021-06-15 17:28:43 UTC 
Installed registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-06-12-174011 and tried to install Hive on it.

From openshift-apiserver logs:

E0615 16:46:28.929685       1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin", Err:(*errors.errorString)(0xc003db3e30)}: Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-name"
E0615 16:46:30.311918       1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin", Err:(*errors.errorString)(0xc003db69e0)}: Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-name"

From node logs after editing APIServer.Spec.Audit.Profile=AllRequestBodies:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"aaa68735-044b-4e66-b537-121986243b18","stage":"ResponseComplete","requestURI":"/apis/authorization.openshift.io/v1/clusterrolebindings/hive-admin","verb":"get","user":{"username":"system:serviceaccount:hive:hive-operator","groups":["system:serviceaccounts","system:serviceaccounts:hive","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["hive-operator-5fb94c5b7b-clhc5"],"authentication.kubernetes.io/pod-uid":["3bbecc2b-48a4-42a3-b6d4-e46db944dbc9"]}},"sourceIPs":["10.0.173.255","10.129.0.1"],"userAgent":"Go-http-client/2.0","objectRef":{"resource":"clusterrolebindings","name":"hive-admin","apiGroup":"authorization.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","code":500},"requestReceivedTimestamp":"2021-06-15T16:46:28.924208Z","stageTimestamp":"2021-06-15T16:46:28.930387Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"hive-operator-rolebinding\" of ClusterRole \"hive-operator-role\" to ServiceAccount \"hive-operator/hive\""}}

Note: "extra":{"authentication.kubernetes.io/pod-name":["hive-operator-5fb94c5b7b-clhc5"]
Comment 5 Osher De Paz 2021-06-15 20:02:49 UTC 
Seems to not be an issue in hive, but a bug in ocp which has been surfaced with recent changes
Marking as duplicate

*** This bug has been marked as a duplicate of bug 1972383 ***