Attachment 1809474 Details for Bug 1988518 – get events pods and pod-yaml

get events pods and pod-yaml

metrics-issue.txt (text/plain), 13.80 KB, created by Tom Dale on 2021-07-30 17:44:24 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Tom Dale

Created: 2021-07-30 17:44:24 UTC

Size: 13.80 KB

patch

obsolete

>
>#########################################
>
>% oc get events -n openshift-storage | grep Warning
>9m10s       Warning   ProvisioningFailed             persistentvolumeclaim/db-noobaa-db-pg-0                                   failed to provision volume with StorageClass "ocs-storagecluster-ceph-rbd": rpc error: code = Internal desc = pool not found: pool (ocs-storagecluster-cephblockpool) not found in Ceph cluster
>9m1s        Warning   FailedScheduling               pod/noobaa-db-pg-0                                                        0/9 nodes are available: 9 pod has unbound immediate PersistentVolumeClaims.
>6m54s       Warning   Unhealthy                      pod/noobaa-endpoint-bdbd4b96b-trxfq                                       Readiness probe failed: dial tcp 10.129.2.18:6001: connect: connection refused
>6m14s       Warning   FailedGetResourceMetric        horizontalpodautoscaler/noobaa-endpoint                                   failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
>6m14s       Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/noobaa-endpoint                                   invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
>3m58s       Warning   FailedGetResourceMetric        horizontalpodautoscaler/noobaa-endpoint                                   failed to get cpu utilization: did not receive metrics for any ready pods
>4m13s       Warning   FailedComputeMetricsReplicas   horizontalpodautoscaler/noobaa-endpoint                                   invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: did not receive metrics for any ready pods
>48m         Warning   Failed                         pod/ocs-metrics-exporter-545d7dc948-ksszj                                 Error: container has runAsNonRoot and image will run as root (pod: "ocs-metrics-exporter-545d7dc948-ksszj_openshift-storage(a75039ba-56c4-4efb-8cda-57de7029a3cb)", container: ocs-metrics-exporter)
>9m34s       Warning   ProbeError                     pod/ocs-operator-7b4c69454f-zq57c                                         Readiness probe error: HTTP probe failed with statuscode: 500
>9m44s       Warning   Unhealthy                      pod/ocs-operator-7b4c69454f-zq57c                                                 Readiness probe failed: HTTP probe failed with statuscode: 500
>45m         Warning   InstallCheckFailed             clusterserviceversion/ocs-operator.v4.8.0-175.ci                                  install timeout
>6m34s       Warning   InstallCheckFailed             clusterserviceversion/ocs-operator.v4.8.0-175.ci                                  install failed: deployment ocs-metrics-exporter not ready before timeout: deployment "ocs-metrics-exporter" exceeded its progress deadline
>10m         Warning   FailedScheduling               pod/rook-ceph-mon-a-65fdcfb76-czgpv                                               0/9 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity rules, 1 node(s) didn't match pod anti-affinity rules, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 5 node(s) didn't match Pod's node affinity/selector.
>11m         Warning   Failed                         pod/rook-ceph-mon-a-canary-655f559996-mrz8h                                       Error: CreateContainer failed as the sandbox was stopped: e8ae12d68a84cc6565c567da3843f479d76115a8479aa5f4b112eb2ffa27bf5f
>10m         Warning   Failed                         pod/rook-ceph-mon-b-canary-84f5fc9858-8pnkw                                       Error: cannot find volume "rook-config-override" to mount into container "mon"
>10m         Warning   Failed                         pod/rook-ceph-mon-b-canary-84f5fc9858-8pnkw                                       Error: cannot find volume "rook-config-override" to mount into container "log-collector"
>10m         Warning   Failed                         pod/rook-ceph-mon-c-canary-5cb54f47fb-df7ct                                       Error: cannot find volume "rook-config-override" to mount into container "mon"
>10m         Warning   Failed                         pod/rook-ceph-mon-c-canary-5cb54f47fb-df7ct                                       Error: cannot find volume "rook-config-override" to mount into container "log-collector"
>tdale@fedora [10:35:00 AM] [~/github/ocs-test/install] [main *]
>
>
>
>#########################################
>-> % oc get pod -n openshift-storage
>NAME                                                              READY   STATUS                       RESTARTS   AGE
>csi-cephfsplugin-dvqwb                                            3/3     Running                      0          12m
>csi-cephfsplugin-jcqrp                                            3/3     Running                      0          12m
>csi-cephfsplugin-jr86f                                            3/3     Running                      0          12m
>csi-cephfsplugin-lrd54                                            3/3     Running                      0          12m
>csi-cephfsplugin-provisioner-5dd599f584-ktl5q                     6/6     Running                      0          12m
>csi-cephfsplugin-provisioner-5dd599f584-vvq9r                     6/6     Running                      0          12m
>csi-cephfsplugin-rpqpp                                            3/3     Running                      0          12m
>csi-cephfsplugin-vvqkc                                            3/3     Running                      0          12m
>csi-rbdplugin-d4pcn                                               3/3     Running                      0          12m
>csi-rbdplugin-hg9rs                                               3/3     Running                      0          12m
>csi-rbdplugin-provisioner-85b4b68989-6vkxp                        6/6     Running                      0          12m
>csi-rbdplugin-provisioner-85b4b68989-78z84                        6/6     Running                      0          12m
>csi-rbdplugin-rt59p                                               3/3     Running                      0          12m
>csi-rbdplugin-vcbvc                                               3/3     Running                      0          12m
>csi-rbdplugin-w97pf                                               3/3     Running                      0          12m
>csi-rbdplugin-xsj4k                                               3/3     Running                      0          12m
>noobaa-core-0                                                     1/1     Running                      0          10m
>noobaa-db-pg-0                                                    1/1     Running                      0          10m
>noobaa-endpoint-bdbd4b96b-trxfq                                   1/1     Running                      0          8m17s
>noobaa-operator-56b677f4b5-p9nc6                                  1/1     Running                      0          51m
>ocs-metrics-exporter-545d7dc948-ksszj                             0/1     CreateContainerConfigError   0          51m
>ocs-operator-7b4c69454f-zq57c                                     1/1     Running                      0          51m
>rook-ceph-crashcollector-46f2f7dc37ea4e6575628df2bf0bcfed-4rxmg   1/1     Running                      0          10m
>rook-ceph-crashcollector-7c279826ac8045b1036de1f423451a0c-hcw77   1/1     Running                      0          10m
>rook-ceph-crashcollector-8a5e4c0f73199944f192adcfc406c873-zxnnv   1/1     Running                      0          10m
>rook-ceph-crashcollector-b4e36a96dda5fa8dab54430a572f51e0-hj8hb   1/1     Running                      0          10m
>rook-ceph-crashcollector-b77460c4211559249d50a6be43c8f117-md222   1/1     Running                      0          10m
>rook-ceph-crashcollector-f8d86b819b5fcefab7f433e43460038a-hxp4g   1/1     Running                      0          10m
>rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-5b6bb777v2z7m   2/2     Running                      0          9m56s
>rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-778dd79cwqx6x   2/2     Running                      0          9m54s
>rook-ceph-mgr-a-67f5594bbc-jm4xc                                  2/2     Running                      0          10m
>rook-ceph-mon-a-65fdcfb76-czgpv                                   2/2     Running                      0          12m
>rook-ceph-mon-b-cc5d9894f-xtzgf                                   2/2     Running                      0          11m
>rook-ceph-mon-c-6c457b879d-b7q6d                                  2/2     Running                      0          11m
>rook-ceph-operator-5bc684b684-gl52b                               1/1     Running                      0          51m
>rook-ceph-osd-0-797569d4b6-ztnww                                  2/2     Running                      0          10m
>rook-ceph-osd-1-7c5985944d-nhrg8                                  2/2     Running                      0          10m
>rook-ceph-osd-2-689866f87b-pmc56                                  2/2     Running                      0          10m
>rook-ceph-osd-3-5ccbf6845c-55k2m                                  2/2     Running                      0          10m
>rook-ceph-osd-4-9d7599d75-rqvjq                                   2/2     Running                      0          10m
>rook-ceph-osd-5-6465ddf5dc-crrjb                                  2/2     Running                      0          10m
>rook-ceph-osd-prepare-ocs-deviceset-local-sc-0-data-0lm67jzx5j7   0/1     Completed                    0          10m
>rook-ceph-osd-prepare-ocs-deviceset-local-sc-0-data-1gn7965bzwb   0/1     Completed                    0          10m
>rook-ceph-osd-prepare-ocs-deviceset-local-sc-1-data-0tzg5kp6lhl   0/1     Completed                    0          10m
>rook-ceph-osd-prepare-ocs-deviceset-local-sc-1-data-1g249bgbgxs   0/1     Completed                    0          10m
>rook-ceph-osd-prepare-ocs-deviceset-local-sc-2-data-05tfzzmt7tr   0/1     Completed                    0          10m
>rook-ceph-osd-prepare-ocs-deviceset-local-sc-2-data-1qzn8x487v4   0/1     Completed                    0          10m
>rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-7f6c5445pk8q   2/2     Running                      0          9m14s
>#########################################
>oc get pod ocs-metrics-exporter-545d7dc948-ksszj -o yaml
>...
>spec:
>  containers:
>  - args:
>    - --namespaces=openshift-storage
>    command:
>    - /usr/local/bin/metrics-exporter
>    env:
>    - name: OPERATOR_CONDITION_NAME
>      value: ocs-operator.v4.8.0-175.ci
>    image: quay.io/rhceph-dev/ocs-operator@sha256:03472c000e638ef051f68801dd6b9d176078211a5b6112fba3f6a5d0a5f2174a
>    imagePullPolicy: IfNotPresent
>    name: ocs-metrics-exporter
>    ports:
>    - containerPort: 8080
>      protocol: TCP
>    - containerPort: 8081
>      protocol: TCP
>    resources: {}
>    securityContext:
>      capabilities:
>        drop:
>        - MKNOD
>      runAsNonRoot: true
>    terminationMessagePath: /dev/termination-log
>    terminationMessagePolicy: File
>    volumeMounts:
>    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
>      name: kube-api-access-rkdnd
>      readOnly: true
>  dnsPolicy: ClusterFirst
>  enableServiceLinks: true
>  imagePullSecrets:
>  - name: ocs-metrics-exporter-dockercfg-jmg54
>  nodeName: worker-3.pok-92.ocptest.pok.stglabs.ibm.com
>  preemptionPolicy: PreemptLowerPriority
>  priority: 0
>  restartPolicy: Always
>  schedulerName: default-scheduler
>  securityContext:
>    seLinuxOptions:
>      level: s0:c26,c0
>  serviceAccount: ocs-metrics-exporter
>  serviceAccountName: ocs-metrics-exporter
>  terminationGracePeriodSeconds: 30
>  tolerations:
>  - effect: NoSchedule
>    key: node.ocs.openshift.io/storage
>    operator: Equal
>    value: "true"
>  - effect: NoExecute
>    key: node.kubernetes.io/not-ready
>    operator: Exists
>    tolerationSeconds: 300
>  - effect: NoExecute
>    key: node.kubernetes.io/unreachable
>    operator: Exists
>    tolerationSeconds: 300
>  volumes:
>  - name: kube-api-access-rkdnd
>    projected:
>      defaultMode: 420
>      sources:
>      - serviceAccountToken:
>          expirationSeconds: 3607
>          path: token
>      - configMap:
>          items:
>          - key: ca.crt
>            path: ca.crt
>          name: kube-root-ca.crt
>      - downwardAPI:
>          items:
>          - fieldRef:
>              apiVersion: v1
>              fieldPath: metadata.namespace
>            path: namespace
>      - configMap:
>          items:
>          - key: service-ca.crt
>            path: service-ca.crt
>          name: openshift-service-ca.crt
>status:
>  conditions:
>  - lastProbeTime: null
>    lastTransitionTime: "2021-07-30T13:44:36Z"
>    status: "True"
>    type: Initialized
>  - lastProbeTime: null
>    lastTransitionTime: "2021-07-30T13:44:36Z"
>    message: 'containers with unready status: [ocs-metrics-exporter]'
>    reason: ContainersNotReady
>    status: "False"
>    type: Ready
>  - lastProbeTime: null
>    lastTransitionTime: "2021-07-30T13:44:36Z"
>    message: 'containers with unready status: [ocs-metrics-exporter]'
>    reason: ContainersNotReady
>    status: "False"
>    type: ContainersReady
>  - lastProbeTime: null
>    lastTransitionTime: "2021-07-30T13:44:36Z"
>    status: "True"
>    type: PodScheduled
>  containerStatuses:
>  - image: quay.io/rhceph-dev/ocs-operator@sha256:03472c000e638ef051f68801dd6b9d176078211a5b6112fba3f6a5d0a5f2174a
>    imageID: ""
>    lastState: {}
>    name: ocs-metrics-exporter
>    ready: false
>    restartCount: 0
>    started: false
>    state:
>      waiting:
>        message: 'container has runAsNonRoot and image will run as root (pod: "ocs-metrics-exporter-545d7dc948-ksszj_openshift-storage(a75039ba-56c4-4efb-8cda-57de7029a3cb)",
>          container: ocs-metrics-exporter)'
>        reason: CreateContainerConfigError
>  hostIP: 192.168.79.27
>  phase: Pending
>  podIP: 10.128.2.85
>  podIPs:
>  - ip: 10.128.2.85
>  qosClass: BestEffort
>  startTime: "2021-07-30T13:44:36Z"

Actions: View

Attachments on bug 1988518: 1809474