Created attachment 1790976 [details] Hive operator logs Description of problem: Using a new ocp version (4.8.0-0.nightly-2021-06-12-174011) I'm unable to deploy hive from image registry.ci.openshift.org/openshift/hive-v4.0:hive Logs of hive-operator pod are complaining about the following error: time="2021-06-14T09:45:56Z" level=info msg="applying asset with GC" asset=config/rbac/hive_admin_role_binding.yaml controller=hive time="2021-06-14T09:45:56Z" level=warning msg="running the apply command failed" controller=hive error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\"" stderr= stdout= time="2021-06-14T09:45:56Z" level=error msg="error applying asset" asset=config/rbac/hive_admin_role_binding.yaml controller=hive error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\"" time="2021-06-14T09:45:56Z" level=error msg="error deploying Hive" controller=hive error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\"" time="2021-06-14T09:45:56Z" level=info msg="reconcile complete" controller=hive elapsedMillis=520 elapsedMillisGT=0 outcome=unspecified time="2021-06-14T09:45:56Z" level=error msg="Reconciler error" _name=hive-controller error="error when retrieving current configuration of:\nResource: \"authorization.openshift.io/v1, Resource=clusterrolebindings\", GroupVersionKind: \"authorization.openshift.io/v1, Kind=ClusterRoleBinding\"\nName: \"hive-admin\", Namespace: \"\"\nfrom server for: \"object\": Get \"https://[fd02::1]:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin\": net/http: invalid header field name \"Impersonate-Extra-authentication.kubernetes.io/pod-name\"" name=hive namespace= Older ocp versions like 4.8.0-0.nightly-2021-06-03-221810 (probably everything before 2021-06-10) don't have this problem. Version-Release number of selected component (if applicable): tag registry.ci.openshift.org/openshift/hive-v4.0:hive which is updated regularly, but I'm pretty sure it's not version-related but a breakage because of ocp. How reproducible: 100% Steps to Reproduce: 1. Install a new nightly image of OCP (e.g. 4.8.0-0.nightly-2021-06-12-174011) 2. Use Makefile in hive repo: make deploy 3. Check logs of pod hive-operator in hive namespace Actual results: hive cannot complete installation, and controllers cannot start. Expected results: Full installation of hive operator, with controllers up and running. Additional info: Relevant builds Passing build - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-assisted-service-master-e2e-metal-assisted-operator-disconnected-periodic/1402777674690072576 Failing build - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-assisted-service-master-e2e-metal-assisted-operator-disconnected-periodic/1403140177861283840 Unfortunately no relevant logs of hive in those builds, but we can inspect ocp changes there.
Created attachment 1790978 [details] make deploy
Installed registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-06-12-174011 and tried to install Hive on it. From openshift-apiserver logs: E0615 16:46:28.929685 1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin", Err:(*errors.errorString)(0xc003db3e30)}: Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-name" E0615 16:46:30.311918 1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin", Err:(*errors.errorString)(0xc003db69e0)}: Get "https://172.30.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/hive-admin": net/http: invalid header field name "Impersonate-Extra-authentication.kubernetes.io/pod-name" From node logs after editing APIServer.Spec.Audit.Profile=AllRequestBodies: {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"aaa68735-044b-4e66-b537-121986243b18","stage":"ResponseComplete","requestURI":"/apis/authorization.openshift.io/v1/clusterrolebindings/hive-admin","verb":"get","user":{"username":"system:serviceaccount:hive:hive-operator","groups":["system:serviceaccounts","system:serviceaccounts:hive","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["hive-operator-5fb94c5b7b-clhc5"],"authentication.kubernetes.io/pod-uid":["3bbecc2b-48a4-42a3-b6d4-e46db944dbc9"]}},"sourceIPs":["10.0.173.255","10.129.0.1"],"userAgent":"Go-http-client/2.0","objectRef":{"resource":"clusterrolebindings","name":"hive-admin","apiGroup":"authorization.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","code":500},"requestReceivedTimestamp":"2021-06-15T16:46:28.924208Z","stageTimestamp":"2021-06-15T16:46:28.930387Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"hive-operator-rolebinding\" of ClusterRole \"hive-operator-role\" to ServiceAccount \"hive-operator/hive\""}} Note: "extra":{"authentication.kubernetes.io/pod-name":["hive-operator-5fb94c5b7b-clhc5"]
Seems to not be an issue in hive, but a bug in ocp which has been surfaced with recent changes Marking as duplicate *** This bug has been marked as a duplicate of bug 1972383 ***