Created attachment 1739231 [details]
kube-rbac-proxy container logs

Description of problem:
check kube-rbac-proxy container logs for node-exporter pods, "[invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]" for some node-exporter pod's kube-rbac-proxy container, not for all
# for i in $(oc -n openshift-monitoring get po | grep node-exporter| awk '{print $1}'); do echo $i; oc -n openshift-monitoring logs $i -c kube-rbac-proxy | grep "error in cryptographic primitive" | head -n 5; echo -e "\n"; done
node-exporter-4r4bl
E1215 02:15:49.829960    2724 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:04.816119    2724 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:19.825445    2724 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:34.815418    2724 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:49.819390    2724 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]


node-exporter-7nmng
E1215 02:15:48.825964    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:03.759746    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:18.753322    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:33.753387    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
E1215 02:16:48.753577    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]


node-exporter-cv8vn


node-exporter-mj7jw


node-exporter-prbxw


node-exporter-ztnkd
****************************************
# oc -n openshift-monitoring get pod -o wide | grep -E "node-exporter-4r4bl|node-exporter-7nmng"
node-exporter-4r4bl                            2/2     Running   0          111m   10.0.1.5      whu47az2-62hg9-worker-usgovvirginia-lrf92   <none>           <none>
node-exporter-7nmng                            2/2     Running   0          124m   10.0.0.8      whu47az2-62hg9-master-2                     <none>           <none>

debug on one cluster
# oc debug node/whu47az2-62hg9-master-2
Creating debug namespace/openshift-debug-node-xgbsh ...
Starting pod/whu47az2-62hg9-master-2-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.0.8
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# journalctl | grep node-exporter-7nmng   
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.635086    1628 kubelet.go:1834] SyncLoop (ADD, "api"): "node-exporter-7nmng_openshift-monitoring(81e0d5de-9e17-4d91-8968-46fb0ba6d482)"
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.724433    1628 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "sys" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-sys") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.724498    1628 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.724537    1628 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "root" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-root") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.724570    1628 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "node-exporter-token-njg2s" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-token-njg2s") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.724669    1628 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "proc" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-proc") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.724742    1628 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "node-exporter-textfile" (UniqueName: "kubernetes.io/empty-dir/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-textfile") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825195    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "root" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-root") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825272    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-token-njg2s" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-token-njg2s") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825305    1628 operation_generator.go:663] MountVolume.SetUp succeeded for volume "root" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-root") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825317    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "proc" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-proc") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825362    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-textfile" (UniqueName: "kubernetes.io/empty-dir/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-textfile") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825368    1628 operation_generator.go:663] MountVolume.SetUp succeeded for volume "proc" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-proc") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825387    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "sys" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-sys") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825426    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.825473    1628 operation_generator.go:663] MountVolume.SetUp succeeded for volume "sys" (UniqueName: "kubernetes.io/host-path/81e0d5de-9e17-4d91-8968-46fb0ba6d482-sys") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: E1215 01:59:25.825591    1628 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls podName:81e0d5de-9e17-4d91-8968-46fb0ba6d482 nodeName:}" failed. No retries permitted until 2020-12-15 01:59:26.32556863 +0000 UTC m=+83.203733528 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls\") pod \"node-exporter-7nmng\" (UID: \"81e0d5de-9e17-4d91-8968-46fb0ba6d482\") : secret \"node-exporter-tls\" not found"
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.826141    1628 operation_generator.go:663] MountVolume.SetUp succeeded for volume "node-exporter-textfile" (UniqueName: "kubernetes.io/empty-dir/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-textfile") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:25 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:25.839502    1628 operation_generator.go:663] MountVolume.SetUp succeeded for volume "node-exporter-token-njg2s" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-token-njg2s") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:26 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:26.327367    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:26 whu47az2-62hg9-master-2 hyperkube[1628]: E1215 01:59:26.327672    1628 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls podName:81e0d5de-9e17-4d91-8968-46fb0ba6d482 nodeName:}" failed. No retries permitted until 2020-12-15 01:59:27.32763635 +0000 UTC m=+84.205801348 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls\") pod \"node-exporter-7nmng\" (UID: \"81e0d5de-9e17-4d91-8968-46fb0ba6d482\") : secret \"node-exporter-tls\" not found"
Dec 15 01:59:27 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:27.331515    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:27 whu47az2-62hg9-master-2 hyperkube[1628]: E1215 01:59:27.331852    1628 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls podName:81e0d5de-9e17-4d91-8968-46fb0ba6d482 nodeName:}" failed. No retries permitted until 2020-12-15 01:59:29.331816395 +0000 UTC m=+86.209981393 (durationBeforeRetry 2s). Error: "MountVolume.SetUp failed for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls\") pod \"node-exporter-7nmng\" (UID: \"81e0d5de-9e17-4d91-8968-46fb0ba6d482\") : secret \"node-exporter-tls\" not found"
Dec 15 01:59:29 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:29.340835    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:29 whu47az2-62hg9-master-2 hyperkube[1628]: E1215 01:59:29.341251    1628 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls podName:81e0d5de-9e17-4d91-8968-46fb0ba6d482 nodeName:}" failed. No retries permitted until 2020-12-15 01:59:33.341204661 +0000 UTC m=+90.219369859 (durationBeforeRetry 4s). Error: "MountVolume.SetUp failed for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls\") pod \"node-exporter-7nmng\" (UID: \"81e0d5de-9e17-4d91-8968-46fb0ba6d482\") : secret \"node-exporter-tls\" not found"
Dec 15 01:59:33 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:33.360496    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:33 whu47az2-62hg9-master-2 hyperkube[1628]: E1215 01:59:33.360732    1628 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls podName:81e0d5de-9e17-4d91-8968-46fb0ba6d482 nodeName:}" failed. No retries permitted until 2020-12-15 01:59:41.360708879 +0000 UTC m=+98.238873777 (durationBeforeRetry 8s). Error: "MountVolume.SetUp failed for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls\") pod \"node-exporter-7nmng\" (UID: \"81e0d5de-9e17-4d91-8968-46fb0ba6d482\") : secret \"node-exporter-tls\" not found"
Dec 15 01:59:41 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:41.396641    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:41 whu47az2-62hg9-master-2 hyperkube[1628]: E1215 01:59:41.396880    1628 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls podName:81e0d5de-9e17-4d91-8968-46fb0ba6d482 nodeName:}" failed. No retries permitted until 2020-12-15 01:59:57.39684631 +0000 UTC m=+114.275011308 (durationBeforeRetry 16s). Error: "MountVolume.SetUp failed for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls\") pod \"node-exporter-7nmng\" (UID: \"81e0d5de-9e17-4d91-8968-46fb0ba6d482\") : secret \"node-exporter-tls\" not found"
Dec 15 01:59:57 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:57.463999    1628 reconciler.go:269] operationExecutor.MountVolume started for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:57 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:57.477181    1628 operation_generator.go:663] MountVolume.SetUp succeeded for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/81e0d5de-9e17-4d91-8968-46fb0ba6d482-node-exporter-tls") pod "node-exporter-7nmng" (UID: "81e0d5de-9e17-4d91-8968-46fb0ba6d482")
Dec 15 01:59:57 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:57.483900    1628 kuberuntime_manager.go:425] No sandbox for pod "node-exporter-7nmng_openshift-monitoring(81e0d5de-9e17-4d91-8968-46fb0ba6d482)" can be found. Need to start a new one
Dec 15 01:59:57 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 01:59:57.484482455Z" level=info msg="Running pod sandbox: openshift-monitoring/node-exporter-7nmng/POD" id=f7d299a6-2dad-40cf-93b8-50fd3c26dfaf name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Dec 15 01:59:57 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 01:59:57.863315711Z" level=info msg="Ran pod sandbox 24fe725ae5dc2c63726f96b6b6bf991b88644ffd2d08a92be8c1a5bb5b7aa5e8 with infra container: openshift-monitoring/node-exporter-7nmng/POD" id=f7d299a6-2dad-40cf-93b8-50fd3c26dfaf name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Dec 15 01:59:58 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 01:59:58.056684    1628 kubelet.go:1872] SyncLoop (PLEG): "node-exporter-7nmng_openshift-monitoring(81e0d5de-9e17-4d91-8968-46fb0ba6d482)", event: &pleg.PodLifecycleEvent{ID:"81e0d5de-9e17-4d91-8968-46fb0ba6d482", Type:"ContainerStarted", Data:"24fe725ae5dc2c63726f96b6b6bf991b88644ffd2d08a92be8c1a5bb5b7aa5e8"}
Dec 15 01:59:59 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 01:59:59.809806361Z" level=info msg="Creating container: openshift-monitoring/node-exporter-7nmng/init-textfile" id=1ef2ad81-1c87-4eb8-b13d-98be89eb21dc name=/runtime.v1alpha2.RuntimeService/CreateContainer
Dec 15 01:59:59 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 01:59:59.946973273Z" level=info msg="Created container cdb9ed5fab7231ad764d7392948183d48a2a4ffb9f5e0f073a3a021c806d8740: openshift-monitoring/node-exporter-7nmng/init-textfile" id=1ef2ad81-1c87-4eb8-b13d-98be89eb21dc name=/runtime.v1alpha2.RuntimeService/CreateContainer
Dec 15 01:59:59 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 01:59:59.960819336Z" level=info msg="Started container cdb9ed5fab7231ad764d7392948183d48a2a4ffb9f5e0f073a3a021c806d8740: openshift-monitoring/node-exporter-7nmng/init-textfile" id=a3e443c3-5ed6-481f-ac90-e3f372b7b844 name=/runtime.v1alpha2.RuntimeService/StartContainer
Dec 15 02:00:00 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 02:00:00.061786    1628 kubelet.go:1872] SyncLoop (PLEG): "node-exporter-7nmng_openshift-monitoring(81e0d5de-9e17-4d91-8968-46fb0ba6d482)", event: &pleg.PodLifecycleEvent{ID:"81e0d5de-9e17-4d91-8968-46fb0ba6d482", Type:"ContainerStarted", Data:"cdb9ed5fab7231ad764d7392948183d48a2a4ffb9f5e0f073a3a021c806d8740"}
Dec 15 02:00:01 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 02:00:01.064148    1628 kubelet.go:1872] SyncLoop (PLEG): "node-exporter-7nmng_openshift-monitoring(81e0d5de-9e17-4d91-8968-46fb0ba6d482)", event: &pleg.PodLifecycleEvent{ID:"81e0d5de-9e17-4d91-8968-46fb0ba6d482", Type:"ContainerDied", Data:"cdb9ed5fab7231ad764d7392948183d48a2a4ffb9f5e0f073a3a021c806d8740"}
Dec 15 02:00:01 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 02:00:01.070878380Z" level=info msg="Creating container: openshift-monitoring/node-exporter-7nmng/node-exporter" id=e1725058-a3ad-4d16-a619-56812699c83f name=/runtime.v1alpha2.RuntimeService/CreateContainer
Dec 15 02:00:01 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 02:00:01.242726021Z" level=info msg="Created container 89d2a7eb9bff0b7764a5e6e2387d446647d576b1c44479487cc3e0f30f74e8af: openshift-monitoring/node-exporter-7nmng/node-exporter" id=e1725058-a3ad-4d16-a619-56812699c83f name=/runtime.v1alpha2.RuntimeService/CreateContainer
Dec 15 02:00:01 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 02:00:01.254698318Z" level=info msg="Started container 89d2a7eb9bff0b7764a5e6e2387d446647d576b1c44479487cc3e0f30f74e8af: openshift-monitoring/node-exporter-7nmng/node-exporter" id=a0b64187-af48-4885-aef5-5b6fa0d044be name=/runtime.v1alpha2.RuntimeService/StartContainer
Dec 15 02:00:01 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 02:00:01.275683540Z" level=info msg="Creating container: openshift-monitoring/node-exporter-7nmng/kube-rbac-proxy" id=459b33ad-b7c1-46a7-9ebc-b64852611034 name=/runtime.v1alpha2.RuntimeService/CreateContainer
Dec 15 02:00:01 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 02:00:01.454898556Z" level=info msg="Created container f0cff2d1bfcbda7a87523004ed81387332e25f46386eb76f1e519f761daaa528: openshift-monitoring/node-exporter-7nmng/kube-rbac-proxy" id=459b33ad-b7c1-46a7-9ebc-b64852611034 name=/runtime.v1alpha2.RuntimeService/CreateContainer
Dec 15 02:00:01 whu47az2-62hg9-master-2 crio[1590]: time="2020-12-15 02:00:01.472228943Z" level=info msg="Started container f0cff2d1bfcbda7a87523004ed81387332e25f46386eb76f1e519f761daaa528: openshift-monitoring/node-exporter-7nmng/kube-rbac-proxy" id=b4761e94-3946-4f55-86a1-fd529698ca18 name=/runtime.v1alpha2.RuntimeService/StartContainer
Dec 15 02:00:02 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 02:00:02.069091    1628 kubelet.go:1872] SyncLoop (PLEG): "node-exporter-7nmng_openshift-monitoring(81e0d5de-9e17-4d91-8968-46fb0ba6d482)", event: &pleg.PodLifecycleEvent{ID:"81e0d5de-9e17-4d91-8968-46fb0ba6d482", Type:"ContainerStarted", Data:"f0cff2d1bfcbda7a87523004ed81387332e25f46386eb76f1e519f761daaa528"}
Dec 15 02:00:02 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 02:00:02.069149    1628 kubelet.go:1872] SyncLoop (PLEG): "node-exporter-7nmng_openshift-monitoring(81e0d5de-9e17-4d91-8968-46fb0ba6d482)", event: &pleg.PodLifecycleEvent{ID:"81e0d5de-9e17-4d91-8968-46fb0ba6d482", Type:"ContainerStarted", Data:"89d2a7eb9bff0b7764a5e6e2387d446647d576b1c44479487cc3e0f30f74e8af"}
Dec 15 03:47:58 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 03:47:58.674952    1628 logs.go:317] Finish parsing log file "/var/log/pods/openshift-monitoring_node-exporter-7nmng_81e0d5de-9e17-4d91-8968-46fb0ba6d482/node-exporter/0.log"
Dec 15 03:47:58 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 03:47:58.901702    1628 logs.go:317] Finish parsing log file "/var/log/pods/openshift-monitoring_node-exporter-7nmng_81e0d5de-9e17-4d91-8968-46fb0ba6d482/kube-rbac-proxy/0.log"
Dec 15 03:52:53 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 03:52:53.671318    1628 logs.go:317] Finish parsing log file "/var/log/pods/openshift-monitoring_node-exporter-7nmng_81e0d5de-9e17-4d91-8968-46fb0ba6d482/kube-rbac-proxy/0.log"
Dec 15 03:53:34 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 03:53:34.304673    1628 logs.go:317] Finish parsing log file "/var/log/pods/openshift-monitoring_node-exporter-7nmng_81e0d5de-9e17-4d91-8968-46fb0ba6d482/node-exporter/0.log"
Dec 15 03:53:34 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 03:53:34.569126    1628 logs.go:317] Finish parsing log file "/var/log/pods/openshift-monitoring_node-exporter-7nmng_81e0d5de-9e17-4d91-8968-46fb0ba6d482/kube-rbac-proxy/0.log"
Dec 15 03:54:47 whu47az2-62hg9-master-2 hyperkube[1628]: I1215 03:54:47.744752    1628 logs.go:317] Finish parsing log file "/var/log/pods/openshift-monitoring_node-exporter-7nmng_81e0d5de-9e17-4d91-8968-46fb0ba6d482/kube-rbac-proxy/0.log"
sh-4.4# cat /var/log/pods/openshift-monitoring_node-exporter-7nmng_81e0d5de-9e17-4d91-8968-46fb0ba6d482/kube-rbac-proxy/0.log
2020-12-15T02:00:01.511761565+00:00 stderr F I1215 02:00:01.511612    5629 main.go:190] Valid token audiences: 
2020-12-15T02:00:01.511843272+00:00 stderr F I1215 02:00:01.511761    5629 main.go:278] Reading certificate files
2020-12-15T02:00:01.512203505+00:00 stderr F I1215 02:00:01.512160    5629 main.go:311] Starting TCP socket on [10.0.0.8]:9100
2020-12-15T02:00:01.513020280+00:00 stderr F I1215 02:00:01.512988    5629 main.go:318] Listening securely on [10.0.0.8]:9100
2020-12-15T02:15:48.826021176+00:00 stderr F E1215 02:15:48.825964    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:16:03.759799154+00:00 stderr F E1215 02:16:03.759746    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:16:18.753378224+00:00 stderr F E1215 02:16:18.753322    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:16:33.753445413+00:00 stderr F E1215 02:16:33.753387    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:16:48.753630729+00:00 stderr F E1215 02:16:48.753577    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:17:03.745744995+00:00 stderr F E1215 02:17:03.745514    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:17:18.757029704+00:00 stderr F E1215 02:17:18.756973    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:17:33.755012158+00:00 stderr F E1215 02:17:33.754963    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:17:48.759355879+00:00 stderr F E1215 02:17:48.759308    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:18:03.753719763+00:00 stderr F E1215 02:18:03.753670    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:18:18.756052230+00:00 stderr F E1215 02:18:18.756007    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:18:33.760305549+00:00 stderr F E1215 02:18:33.760256    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]
2020-12-15T02:18:48.756544804+00:00 stderr F E1215 02:18:48.756497    5629 proxy.go:73] Unable to authenticate the request due to an error: [invalid bearer token, square/go-jose: error in cryptographic primitive, token lookup failed]

Version-Release number of selected component (if applicable):
node_exporter version=1.0.1
4.7.0-0.nightly-2020-12-14-165231

How reproducible:
first time to see the error

Steps to Reproduce:
1. check kube-rbac-proxy container logs for node-exporter pods
2.
3.

Actual results:


Expected results:


Additional info: