Attachment 1454420 Details for Bug 1594907 – Basic listings / describe node & pods

Basic listings / describe node & pods

file_1594907.txt (text/plain), 18.15 KB, created by Justin Pierce on 2018-06-25 16:44:52 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Justin Pierce

Created: 2018-06-25 16:44:52 UTC

Size: 18.15 KB

patch

obsolete

>[root@free-stg-master-03fb6 ~]# oc get nodes | grep ip-172-31-69-53.us-east-2.compute.internal
>ip-172-31-69-53.us-east-2.compute.internal    Ready,SchedulingDisabled   compute,infra    235d      v1.10.0+b81c8f8
>
>
>[root@free-stg-master-03fb6 ~]# oc describe node ip-172-31-69-53.us-east-2.compute.internal
>Name:               ip-172-31-69-53.us-east-2.compute.internal
>Roles:              compute,infra
>Labels:             beta.kubernetes.io/arch=amd64
>                    beta.kubernetes.io/instance-type=r4.xlarge
>                    beta.kubernetes.io/os=linux
>                    failure-domain.beta.kubernetes.io/region=us-east-2
>                    failure-domain.beta.kubernetes.io/zone=us-east-2a
>                    hostname=free-stg-node-infra-70a4e
>                    kubernetes.io/hostname=ip-172-31-69-53.us-east-2.compute.internal
>                    logging-infra-fluentd=true
>                    node-role.kubernetes.io/compute=true
>                    node-role.kubernetes.io/infra=true
>                    region=us-east-2
>                    type=infra
>Annotations:        volumes.kubernetes.io/controller-managed-attach-detach=true
>CreationTimestamp:  Wed, 01 Nov 2017 20:37:33 +0000
>Taints:             <none>
>Unschedulable:      true
>Conditions:
>  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
>  ----             ------  -----------------                 ------------------                ------                       -------
>  OutOfDisk        False   Mon, 25 Jun 2018 16:40:33 +0000   Thu, 21 Jun 2018 14:50:06 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
>  MemoryPressure   False   Mon, 25 Jun 2018 16:40:33 +0000   Thu, 21 Jun 2018 14:50:06 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
>  DiskPressure     False   Mon, 25 Jun 2018 16:40:33 +0000   Thu, 21 Jun 2018 14:50:06 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
>  Ready            True    Mon, 25 Jun 2018 16:40:33 +0000   Thu, 21 Jun 2018 14:50:31 +0000   KubeletReady                 kubelet is posting ready status
>  PIDPressure      False   Mon, 25 Jun 2018 16:40:33 +0000   Wed, 30 May 2018 17:35:14 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
>Addresses:
>  InternalIP:   172.31.69.53
>  ExternalIP:   18.221.227.84
>  InternalDNS:  ip-172-31-69-53.us-east-2.compute.internal
>  ExternalDNS:  ec2-18-221-227-84.us-east-2.compute.amazonaws.com
>  Hostname:     ip-172-31-69-53.us-east-2.compute.internal
>Capacity:
> cpu:            4
> hugepages-1Gi:  0
> hugepages-2Mi:  0
> memory:         31231632Ki
> pods:           80
>Allocatable:
> cpu:            2500m
> hugepages-1Gi:  0
> hugepages-2Mi:  0
> memory:         29032080Ki
> pods:           80
>System Info:
> Machine ID:                 d52c597d0f1a42aeb01b5a7d71e63f24
> System UUID:                EC21CD35-6994-3DAE-6902-C271D4531E14
> Boot ID:                    a8ef807f-94ad-411b-8591-8c8afe202e16
> Kernel Version:             3.10.0-862.3.2.el7.x86_64
> OS Image:                   Red Hat Enterprise Linux
> Operating System:           linux
> Architecture:               amd64
> Container Runtime Version:  docker://1.13.1
> Kubelet Version:            v1.10.0+b81c8f8
> Kube-Proxy Version:         v1.10.0+b81c8f8
>ExternalID:                  i-092b3b2b317544d13
>ProviderID:                  aws:///us-east-2a/i-092b3b2b317544d13
>Non-terminated Pods:         (15 in total)
>  Namespace                  Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits
>  ---------                  ----                                        ------------  ----------  ---------------  -------------
>  default                    docker-registry-44-58ghd                    100m (4%)     0 (0%)      256Mi (0%)       0 (0%)
>  default                    router-470-5mspn                            100m (4%)     0 (0%)      256Mi (0%)       0 (0%)
>  eparis                     admin-rpc-2bhxt                             25m (1%)      500m (20%)  128Mi (0%)       256Mi (0%)
>  logging                    logging-curator-23-dbv4c                    25m (1%)      0 (0%)      512Mi (1%)       512Mi (1%)
>  logging                    logging-es-data-master-dm4qr1as-23-lxxkh    475m (19%)    0 (0%)      12544Mi (44%)    12544Mi (44%)
>  logging                    logging-fluentd-c8877                       100m (4%)     0 (0%)      512Mi (1%)       512Mi (1%)
>  logging                    logging-kibana-26-qp7kv                     50m (2%)      0 (0%)      1280Mi (4%)      1280Mi (4%)
>  openshift-devops-monitor   prometheus-node-exporter-4fn44              100m (4%)     200m (8%)   30Mi (0%)        50Mi (0%)
>  openshift-infra            hawkular-cassandra-1-x8rmb                  375m (15%)    0 (0%)      4Gi (14%)        4Gi (14%)
>  openshift-infra            hawkular-cassandra-2-lcj7b                  375m (15%)    0 (0%)      4Gi (14%)        4Gi (14%)
>  openshift-infra            hawkular-metrics-hbv8d                      100m (4%)     0 (0%)      3Gi (10%)        3Gi (10%)
>  openshift-monitoring       node-exporter-j9v5l                         10m (0%)      20m (0%)    20Mi (0%)        40Mi (0%)
>  openshift-node             sync-zc7wm                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)
>  openshift-sdn              ovs-zbrfz                                   100m (4%)     200m (8%)   300Mi (1%)       400Mi (1%)
>  openshift-sdn              sdn-d4r6k                                   100m (4%)     0 (0%)      200Mi (0%)       0 (0%)
>Allocated resources:
>  (Total limits may be over 100 percent, i.e., overcommitted.)
>  CPU Requests  CPU Limits  Memory Requests  Memory Limits
>  ------------  ----------  ---------------  -------------
>  2035m (81%)   920m (36%)  27302Mi (96%)    26858Mi (94%)
>Events:         <none>
>
>
>[root@free-stg-master-03fb6 ~]# oc adm manage-node --list-pods ip-172-31-69-53.us-east-2.compute.internal
>
>Listing matched pods on node: ip-172-31-69-53.us-east-2.compute.internal
>
>NAMESPACE                  NAME                                       READY     STATUS        RESTARTS   AGE
>default                    docker-registry-44-58ghd                   0/1       Terminating   0          4d
>default                    router-470-5mspn                           1/1       Terminating   0          4d
>eparis                     admin-rpc-2bhxt                            1/1       Running       17         60d
>logging                    logging-curator-23-dbv4c                   1/1       Terminating   0          4d
>logging                    logging-es-data-master-dm4qr1as-23-lxxkh   1/2       Terminating   0          4d
>logging                    logging-fluentd-c8877                      1/1       Running       0          4d
>logging                    logging-kibana-26-qp7kv                    1/2       Terminating   0          4d
>openshift-devops-monitor   prometheus-node-exporter-4fn44             1/1       Running       12         94d
>openshift-infra            hawkular-cassandra-1-x8rmb                 0/1       Terminating   0          4d
>openshift-infra            hawkular-cassandra-2-lcj7b                 0/1       Terminating   1          4d
>openshift-infra            hawkular-metrics-hbv8d                     0/1       Terminating   3          4d
>openshift-monitoring       node-exporter-j9v5l                        2/2       Running       16         16d
>openshift-node             sync-zc7wm                                 1/1       Terminating   1          4d
>openshift-sdn              ovs-zbrfz                                  1/1       Terminating   1          4d
>openshift-sdn              sdn-d4r6k                                  1/1       Terminating   1          4d
>
>
>[root@free-stg-master-03fb6 ~]# oc describe pod -n default docker-registry-44-58ghd
>Name:                      docker-registry-44-58ghd
>Namespace:                 default
>Node:                      ip-172-31-69-53.us-east-2.compute.internal/172.31.69.53
>Start Time:                Thu, 21 Jun 2018 15:02:46 +0000
>Labels:                    deployment=docker-registry-44
>                           deploymentconfig=docker-registry
>                           docker-registry=default
>Annotations:               openshift.io/deployment-config.latest-version=44
>                           openshift.io/deployment-config.name=docker-registry
>                           openshift.io/deployment.name=docker-registry-44
>                           openshift.io/scc=restricted
>Status:                    Terminating (lasts 2h)
>Termination Grace Period:  30s
>IP:                        10.130.0.187
>Controlled By:             ReplicationController/docker-registry-44
>Containers:
>  registry:
>    Container ID:   docker://724907755c9187504567459c1ad0376c1afa8da791bd0a3162aab7ab6ee08445
>    Image:          registry.reg-aws.openshift.com:443/openshift3/ose-docker-registry:v3.10.2
>    Image ID:       docker-pullable://registry.reg-aws.openshift.com:443/openshift3/ose-docker-registry@sha256:cd78ae27f6e9b50a5a64f906c63033ff2b5991e3f923e88d9c211b826969b029
>    Port:           5000/TCP
>    Host Port:      0/TCP
>    State:          Running
>      Started:      Thu, 21 Jun 2018 15:02:52 +0000
>    Ready:          False
>    Restart Count:  0
>    Requests:
>      cpu:      100m
>      memory:   256Mi
>    Liveness:   http-get https://:5000/healthz delay=10s timeout=5s period=10s #success=1 #failure=3
>    Readiness:  http-get https://:5000/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
>    Environment:
>      REGISTRY_HTTP_ADDR:                                     :5000
>      REGISTRY_HTTP_NET:                                      tcp
>      REGISTRY_HTTP_SECRET:                                   jCFjFgBacZv6cHwk7wb5lxWNrkeDpimNmLrPCdqwtUU=
>      REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA:  false
>      REGISTRY_HTTP_TLS_KEY:                                  /etc/secrets/registry.key
>      REGISTRY_CONFIGURATION_PATH:                            /etc/registry/config.yml
>      REGISTRY_HTTP_TLS_CERTIFICATE:                          /etc/secrets/registry.crt
>      REGISTRY_OPENSHIFT_REQUESTS_WRITE_MAXRUNNING:           256
>      REGISTRY_OPENSHIFT_REQUESTS_WRITE_MAXWAITINQUEUE:       2h
>      REGISTRY_OPENSHIFT_SERVER_ADDR:                         docker-registry.default.svc:5000
>    Mounts:
>      /etc/registry from docker-config (rw)
>      /etc/secrets from registry-certificates (rw)
>      /registry from registry-storage (rw)
>      /var/run/secrets/kubernetes.io/serviceaccount from registry-token-5tw0z (ro)
>Conditions:
>  Type           Status
>  Initialized    True 
>  Ready          False 
>  PodScheduled   True 
>Volumes:
>  registry-storage:
>    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
>    Medium:  
>  registry-certificates:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  registry-certificates
>    Optional:    false
>  docker-config:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  registry-config
>    Optional:    false
>  registry-token-5tw0z:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  registry-token-5tw0z
>    Optional:    false
>QoS Class:       Burstable
>Node-Selectors:  type=infra
>Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
>Events:
>  Type     Reason         Age                 From                                                 Message
>  ----     ------         ----                ----                                                 -------
>  Warning  FailedKillPod  30m (x495 over 2h)  kubelet, ip-172-31-69-53.us-east-2.compute.internal  error killing pod: [failed to "KillContainer" for "registry" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: Cannot stop container 724907755c9187504567459c1ad0376c1afa8da791bd0a3162aab7ab6ee08445: Cannot kill container 724907755c9187504567459c1ad0376c1afa8da791bd0a3162aab7ab6ee08445: rpc error: code = 14 desc = grpc: the connection is unavailable"
>, failed to "KillPodSandbox" for "284ed0af-7564-11e8-b830-0203ad7dfcd7" with KillPodSandboxError: "rpc error: code = Unknown desc = Error response from daemon: Cannot stop container 02418c484eda7a50b1308bdf5344044884111afbee88fe7f190d27885d03ac2a: Cannot kill container 02418c484eda7a50b1308bdf5344044884111afbee88fe7f190d27885d03ac2a: rpc error: code = 14 desc = grpc: the connection is unavailable"
>]
>  Normal  Killing  37s (x600 over 2h)  kubelet, ip-172-31-69-53.us-east-2.compute.internal  Killing container with id docker://registry:Need to kill Pod
>
>
>[root@free-stg-master-03fb6 ~]# oc describe pod -n openshift-sdn ovs-zbrfz
>Name:                      ovs-zbrfz
>Namespace:                 openshift-sdn
>Node:                      ip-172-31-69-53.us-east-2.compute.internal/172.31.69.53
>Start Time:                Thu, 21 Jun 2018 14:17:55 +0000
>Labels:                    app=ovs
>                           component=network
>                           controller-revision-hash=1906358623
>                           openshift.io/component=network
>                           pod-template-generation=41
>                           type=infra
>Annotations:               openshift.io/scc=privileged
>                           scheduler.alpha.kubernetes.io/critical-pod=
>Status:                    Terminating (lasts 2h)
>Termination Grace Period:  30s
>IP:                        172.31.69.53
>Controlled By:             DaemonSet/ovs
>Containers:
>  openvswitch:
>    Container ID:  docker://2607980ac992f674be4c1df4c84d1b4afb9a533b03d7ae30260ce54e94993c48
>    Image:         registry.reg-aws.openshift.com:443/openshift3/ose-node:v3.10.2
>    Image ID:      docker-pullable://registry.reg-aws.openshift.com:443/openshift3/ose-node@sha256:d03725dbc11e1aa33bfa1be5d1b09c60aa615e49ddab86d853b8a139834ce5fc
>    Port:          <none>
>    Host Port:     <none>
>    Command:
>      /bin/bash
>      -c
>      #!/bin/bash
>set -euo pipefail
>
># if another process is listening on the cni-server socket, wait until it exits
>trap 'kill $(jobs -p); exit 0' TERM
>retries=0
>while true; do
>  if /usr/share/openvswitch/scripts/ovs-ctl status &>/dev/null; then
>    echo "warning: Another process is currently managing OVS, waiting 15s ..." 2>&1
>    sleep 15 & wait
>    (( retries += 1 ))
>  else
>    break
>  fi
>  if [[ "${retries}" -gt 40 ]]; then
>    echo "error: Another process is currently managing OVS, exiting" 2>&1
>    exit 1
>  fi
>done
>
># launch OVS
>function quit {
>    /usr/share/openvswitch/scripts/ovs-ctl stop
>    exit 0
>}
>trap quit SIGTERM
>/usr/share/openvswitch/scripts/ovs-ctl start --system-id=random
>
># Restrict the number of pthreads ovs-vswitchd creates to reduce the
># amount of RSS it uses on hosts with many cores
># https://bugzilla.redhat.com/show_bug.cgi?id=1571379
># https://bugzilla.redhat.com/show_bug.cgi?id=1572797
>if [[ `nproc` -gt 12 ]]; then
>    ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=4
>    ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=10
>fi
>while true; do sleep 5; done
>
>    State:          Running
>      Started:      Thu, 21 Jun 2018 14:50:13 +0000
>    Last State:     Terminated
>      Reason:       Completed
>      Exit Code:    0
>      Started:      Thu, 21 Jun 2018 14:17:56 +0000
>      Finished:     Thu, 21 Jun 2018 14:48:48 +0000
>    Ready:          True
>    Restart Count:  1
>    Limits:
>      cpu:     200m
>      memory:  400Mi
>    Requests:
>      cpu:        100m
>      memory:     300Mi
>    Environment:  <none>
>    Mounts:
>      /etc/openvswitch from host-config-openvswitch (rw)
>      /lib/modules from host-modules (ro)
>      /run/openvswitch from host-run-ovs (rw)
>      /sys from host-sys (ro)
>      /var/run/openvswitch from host-run-ovs (rw)
>      /var/run/secrets/kubernetes.io/serviceaccount from sdn-token-m4rbm (ro)
>Conditions:
>  Type           Status
>  Initialized    True 
>  Ready          True 
>  PodScheduled   True 
>Volumes:
>  host-modules:
>    Type:          HostPath (bare host directory volume)
>    Path:          /lib/modules
>    HostPathType:  
>  host-run-ovs:
>    Type:          HostPath (bare host directory volume)
>    Path:          /run/openvswitch
>    HostPathType:  
>  host-sys:
>    Type:          HostPath (bare host directory volume)
>    Path:          /sys
>    HostPathType:  
>  host-config-openvswitch:
>    Type:          HostPath (bare host directory volume)
>    Path:          /etc/origin/openvswitch
>    HostPathType:  
>  sdn-token-m4rbm:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  sdn-token-m4rbm
>    Optional:    false
>QoS Class:       Burstable
>Node-Selectors:  <none>
>Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
>                 node.kubernetes.io/memory-pressure:NoSchedule
>                 node.kubernetes.io/not-ready:NoExecute
>                 node.kubernetes.io/unreachable:NoExecute
>Events:
>  Type     Reason         Age                From                                                 Message
>  ----     ------         ----               ----                                                 -------
>  Normal   Killing        7m (x601 over 2h)  kubelet, ip-172-31-69-53.us-east-2.compute.internal  Killing container with id docker://openvswitch:Need to kill Pod
>  Warning  FailedKillPod  2m (x618 over 2h)  kubelet, ip-172-31-69-53.us-east-2.compute.internal  error killing pod: [failed to "KillContainer" for "openvswitch" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: Cannot stop container 2607980ac992f674be4c1df4c84d1b4afb9a533b03d7ae30260ce54e94993c48: Cannot kill container 2607980ac992f674be4c1df4c84d1b4afb9a533b03d7ae30260ce54e94993c48: rpc error: code = 14 desc = grpc: the connection is unavailable"
>, failed to "KillPodSandbox" for "e3e81e77-755d-11e8-91a0-02306c0cdc4b" with KillPodSandboxError: "rpc error: code = Unknown desc = Error response from daemon: Cannot stop container 83653451dba9113fb643257ba7299161aa9ef545b82d322f8f6308eee6100317: Cannot kill container 83653451dba9113fb643257ba7299161aa9ef545b82d322f8f6308eee6100317: rpc error: code = 14 desc = grpc: the connection is unavailable"
>]

Actions: View

Attachments on bug 1594907: 1454420 | 1454464