Attachment 1456777 Details for Bug 1598445 – pod listings & exception from log

pod listings & exception from log

file_1598445.txt (text/plain), 14.84 KB, created by Justin Pierce on 2018-07-05 13:33:57 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Justin Pierce

Created: 2018-07-05 13:33:57 UTC

Size: 14.84 KB

patch

obsolete

>[root@starter-us-west-1-master-2514b ~]# oc get pods
>NAME                              READY     STATUS              RESTARTS   AGE
>bootstrap-autoapprover-0          1/1       Running             3          1d
>hawkular-cassandra-1-779rl        1/1       Running             0          20h
>hawkular-cassandra-2-49lsc        1/1       Running             0          20h
>hawkular-cassandra-3-dhgm5        1/1       Running             0          20h
>hawkular-cassandra-4-l8nq6        0/1       CrashLoopBackOff    232        20h
>hawkular-cassandra-5-rpn5r        0/1       ContainerCreating   0          20h
>hawkular-metrics-56fmx            1/1       Running             1          20h
>hawkular-metrics-6r9gz            1/1       Running             1          20h
>hawkular-metrics-schema-kzpjl     0/1       Completed           0          20h
>hawkular-metrics-xc5gn            1/1       Running             1          20h
>heapster-9fpg9                    1/1       Running             1          20h
>hibernation-21-mjrj9              1/1       Running             0          1d
>oso-console-extensions-11-5cj8b   1/1       Running             0          1d
>user-analytics-2-lck5r            1/1       Running             0          1d
>
>
>[root@starter-us-west-1-master-2514b ~]# oc describe pod hawkular-cassandra-4-l8nq6
>Name:           hawkular-cassandra-4-l8nq6
>Namespace:      openshift-infra
>Node:           ip-172-31-23-163.us-west-1.compute.internal/172.31.23.163
>Start Time:     Wed, 04 Jul 2018 17:02:18 +0000
>Labels:         metrics-infra=hawkular-cassandra
>                name=hawkular-cassandra-4
>                type=hawkular-cassandra
>Annotations:    <none>
>Status:         Running
>IP:             10.131.26.140
>Controlled By:  ReplicationController/hawkular-cassandra-4
>Containers:
>  hawkular-cassandra-4:
>    Container ID:  docker://f49169c09db4ef3e3611f7c95734323159d6eeb03baac96c0f625588f4283f21
>    Image:         registry.reg-aws.openshift.com:443/openshift3/metrics-cassandra:v3.10.9
>    Image ID:      docker-pullable://registry.reg-aws.openshift.com:443/openshift3/metrics-cassandra@sha256:2944062dca22a2cc9e71b040f77f3e06a04047fcfbbd0145110f20bea98a7d78
>    Ports:         9042/TCP, 9160/TCP, 7000/TCP, 7001/TCP
>    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
>    Command:
>      /opt/apache-cassandra/bin/cassandra-docker.sh
>      --cluster_name=hawkular-metrics
>      --data_volume=/cassandra_data
>      --internode_encryption=all
>      --require_node_auth=true
>      --enable_client_encryption=true
>      --require_client_auth=true
>    State:          Waiting
>      Reason:       CrashLoopBackOff
>    Last State:     Terminated
>      Reason:       Error
>      Exit Code:    3
>      Started:      Thu, 05 Jul 2018 13:26:53 +0000
>      Finished:     Thu, 05 Jul 2018 13:27:07 +0000
>    Ready:          False
>    Restart Count:  232
>    Limits:
>      memory:  8Gi
>    Requests:
>      cpu:      375m
>      memory:   8Gi
>    Readiness:  exec [/opt/apache-cassandra/bin/cassandra-docker-ready.sh] delay=0s timeout=1s period=10s #success=1 #failure=3
>    Environment:
>      CASSANDRA_MASTER:               false
>      CASSANDRA_DATA_VOLUME:          /cassandra_data
>      JVM_OPTS:                       -Dcassandra.commitlog.ignorereplayerrors=true
>      ENABLE_PROMETHEUS_ENDPOINT:     True
>      TRUSTSTORE_NODES_AUTHORITIES:   /hawkular-cassandra-certs/tls.peer.truststore.crt
>      TRUSTSTORE_CLIENT_AUTHORITIES:  /hawkular-cassandra-certs/tls.client.truststore.crt
>      TAKE_SNAPSHOT:                  False
>      POD_NAMESPACE:                  openshift-infra (v1:metadata.namespace)
>      MEMORY_LIMIT:                   8589934592 (limits.memory)
>      CPU_LIMIT:                      node allocatable (limits.cpu)
>    Mounts:
>      /cassandra_data from cassandra-data (rw)
>      /hawkular-cassandra-certs from hawkular-cassandra-certs (rw)
>      /var/run/secrets/kubernetes.io/serviceaccount from cassandra-token-stwjp (ro)
>Conditions:
>  Type           Status
>  Initialized    True 
>  Ready          False 
>  PodScheduled   True 
>Volumes:
>  cassandra-data:
>    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
>    ClaimName:  metrics-cassandra-4
>    ReadOnly:   false
>  hawkular-cassandra-certs:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  hawkular-cassandra-certs
>    Optional:    false
>  cassandra-token-stwjp:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  cassandra-token-stwjp
>    Optional:    false
>QoS Class:       Burstable
>Node-Selectors:  type=infra
>Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
>Events:
>  Type     Reason               Age                 From                                                  Message
>  ----     ------               ----                ----                                                  -------
>  Warning  FailedPostStartHook  1h (x116 over 19h)  kubelet, ip-172-31-23-163.us-west-1.compute.internal  (combined from similar events): Exec lifecycle hook ([/opt/apache-cassandra/bin/cassandra-poststart.sh]) for Container "hawkular-cassandra-4" in Pod "hawkular-cassandra-4-l8nq6_openshift-infra(025ac583-7fac-11e8-9519-06e7d92b1aa4)" failed - error: command '/opt/apache-cassandra/bin/cassandra-poststart.sh' exited with 137: Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected
>Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
>, message: "The Cassandra datavolume (/cassandra_data) is not empty. Checking if an update is required.\nWe are running a newer version of Cassandra and need to upgrade the existing data.\nWaiting for Cassandra to enter the up and normal state\nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nCould not get the Cassandra status. This may mean that the Cassandra instance is not up yet. Will try again\nCassandra not in the up and normal state. Current state is \nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\nnodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\nnodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\nnodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\nnodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\nnodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\n/opt/apache-cassandra/bin/cassandra-poststart.sh: line 33: [: =: unary operator expected\nPicked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss\n"
>  Normal   Pulling  1h (x222 over 20h)   kubelet, ip-172-31-23-163.us-west-1.compute.internal  pulling image "registry.reg-aws.openshift.com:443/openshift3/metrics-cassandra:v3.10.9"
>  Warning  BackOff  4m (x5437 over 20h)  kubelet, ip-172-31-23-163.us-west-1.compute.internal  Back-off restarting failed container
>
>
>Exception excerpt from log:
>INFO  [main] 2018-07-05 13:27:04,936 StorageService.java:575 - Cassandra version: 3.0.15
>INFO  [main] 2018-07-05 13:27:04,936 StorageService.java:576 - Thrift API version: 20.1.0
>INFO  [main] 2018-07-05 13:27:04,937 StorageService.java:577 - CQL supported versions: 3.4.0 (default: 3.4.0)
>INFO  [main] 2018-07-05 13:27:04,989 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 98 MB and a resize interval of 60 minutes
>INFO  [main] 2018-07-05 13:27:04,990 StorageService.java:596 - Loading persisted ring state
>INFO  [main] 2018-07-05 13:27:05,091 MessagingService.java:541 - Starting Encrypted Messaging Service on SSL port 7001
>INFO  [HANDSHAKE-/10.131.23.5] 2018-07-05 13:27:05,302 OutboundTcpConnection.java:564 - Handshaking version with /10.131.23.5
>INFO  [HANDSHAKE-/10.131.26.141] 2018-07-05 13:27:05,302 OutboundTcpConnection.java:564 - Handshaking version with /10.131.26.141
>INFO  [HANDSHAKE-/10.129.28.188] 2018-07-05 13:27:05,312 OutboundTcpConnection.java:564 - Handshaking version with /10.129.28.188
>Exception (java.lang.UnsupportedOperationException) encountered during startup: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
>java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
>	at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:532)
>	at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:732)
>	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:642)
>	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:570)
>	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346)
>	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569)
>	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697)
>ERROR [main] 2018-07-05 13:27:06,125 CassandraDaemon.java:710 - Exception encountered during startup
>java.lang.UnsupportedOperationException: Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true
>	at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:532) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:732) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:642) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:570) ~[apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:346) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:569) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:697) [apache-cassandra-3.0.15.redhat-1.jar:3.0.15.redhat-1]
>INFO  [StorageServiceShutdownHook] 2018-07-05 13:27:06,128 HintsService.java:212 - Paused hints dispatch
>WARN  [StorageServiceShutdownHook] 2018-07-05 13:27:06,131 Gossiper.java:1500 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
>INFO  [StorageServiceShutdownHook] 2018-07-05 13:27:06,131 MessagingService.java:816 - Waiting for messaging service to quiesce
>INFO  [ACCEPT-hawkular-cassandra-4-l8nq6/10.131.26.140] 2018-07-05 13:27:06,131 MessagingService.java:1115 - MessagingService has terminated the accept() thread
>INFO  [StorageServiceShutdownHook] 2018-07-05 13:27:06,348 HintsService.java:212 - Paused hints dispatch

Actions: View

Attachments on bug 1598445: 1456777