Hide Forgot
Description of problem: $ oc edit pod prometheus-operator-579779cd5c-n8jqt E1022 20:36:04.786726 1 operator.go:278] Sync "openshift-monitoring/main" failed: updating statefulset failed: StatefulSet.apps "alertmanager-main" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. level=info ts=2018-10-22T20:36:21.018246178Z caller=operator.go:402 component=alertmanageroperator msg="sync alertmanager" key=openshift-monitoring/main E1022 20:36:21.041111 1 operator.go:278] Sync "openshift-monitoring/main" failed: updating statefulset failed: StatefulSet.apps "alertmanager-main" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. level=info ts=2018-10-22T20:36:47.988566504Z caller=operator.go:402 component=alertmanageroperator msg="sync alertmanager" key=openshift-monitoring/main E1022 20:36:48.008604 1 operator.go:278] Sync "openshift-monitoring/main" failed: updating statefulset failed: StatefulSet.apps "alertmanager-main" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. level=info ts=2018-10-22T20:37:16.90989681Z caller=operator.go:402 component=alertmanageroperator msg="sync alertmanager" key=openshift-monitoring/main E1022 20:37:16.924442 1 operator.go:278] Sync "openshift-monitoring/main" failed: updating statefulset failed: StatefulSet.apps "alertmanager-main" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. level=info ts=2018-10-22T20:38:29.030985292Z caller=operator.go:402 component=alertmanageroperator msg="sync alertmanager" key=openshift-monitoring/main E1022 20:38:29.049614 1 operator.go:278] Sync "openshift-monitoring/main" failed: updating statefulset failed: StatefulSet.apps "alertmanager-main" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden. Version-Release number of selected component (if applicable): v3.11.16
Created attachment 1496505 [details] statefulset content
There is another error "unable to retrieve auth token: invalid username/password" # oc -n openshift-monitoring describe pod node-exporter-vdj4z Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Failed 1m (x2 over 1m) kubelet, ip-172-31-49-167.ec2.internal Error: ErrImagePull Normal BackOff 1m (x2 over 1m) kubelet, ip-172-31-49-167.ec2.internal Back-off pulling image "registry.reg-aws.openshift.com:443/openshift3/prometheus-node-exporter:v3.11.28" Warning Failed 1m (x2 over 1m) kubelet, ip-172-31-49-167.ec2.internal Error: ImagePullBackOff Normal BackOff 1m (x2 over 1m) kubelet, ip-172-31-49-167.ec2.internal Back-off pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy:v3.11.28" Warning Failed 1m (x2 over 1m) kubelet, ip-172-31-49-167.ec2.internal Error: ImagePullBackOff Warning Failed 1m (x3 over 1m) kubelet, ip-172-31-49-167.ec2.internal Failed to pull image "registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy:v3.11.28": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password Normal Pulling 1m (x3 over 1m) kubelet, ip-172-31-49-167.ec2.internal pulling image "registry.reg-aws.openshift.com:443/openshift3/prometheus-node-exporter:v3.11.28" Warning Failed 1m (x3 over 1m) kubelet, ip-172-31-49-167.ec2.internal Failed to pull image "registry.reg-aws.openshift.com:443/openshift3/prometheus-node-exporter:v3.11.28": rpc error: code = Unknown desc = unable to retrieve auth token: invalid username/password Warning Failed 1m (x3 over 1m) kubelet, ip-172-31-49-167.ec2.internal Error: ErrImagePull Normal Pulling 1m (x3 over 1m) kubelet, ip-172-31-49-167.ec2.internal pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy:v3.11.28" ********************************************************************** # oc -n openshift-monitoring get pod -o wide | grep node-exporter-vdj4z node-exporter-vdj4z 0/2 ImagePullBackOff 0 4m 172.31.49.167 ip-172-31-49-167.ec2.internal <none>
find in the attached "statefulset content", I see registry.reg-aws.openshift.com:443/openshift3/ose-configmap-reloader:v3.11.7 registry.reg-aws.openshift.com:443/openshift3/ose-prometheus-config-reloader:v3.11.7 other version is v3.11.0-0.21.0
From # oc -n openshift-monitoring get deployment.apps/prometheus-operator -oyaml - --config-reloader-image=registry.reg-aws.openshift.com:443/openshift3/ose-configmap-reloader:v3.11.28 - --prometheus-config-reloader=registry.reg-aws.openshift.com:443/openshift3/ose-prometheus-config-reloader:v3.11.28 maybe it is the version issue caused the problem, we should make sure it uses the same version
Created attachment 1496593 [details] prometheus-operator deployment file
see https://github.com/kubernetes/kubernetes/issues/66137
This has been fixed in a newer version of the Prometheus Operator, so we should probably bump the version in the 3.11 release. For now what you can do is just delete the underlying StatefulSet.
Change back to MODIFIED, since free-int environment is still v3.11.69
$ oc -n openshift-monitoring logs $(oc -n openshift-monitoring get pod | grep prometheus-operator | awk '{print $1}') | grep Forbidden nothing returned no such issue now, payload: 4.0.0-0.nightly-2019-04-10-182914
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758