Bug 1740957 - Failed to upgrade ES pods from 4.1.4 to 4.2.0.
Summary: Failed to upgrade ES pods from 4.1.4 to 4.2.0.
Keywords:
Status: CLOSED DUPLICATE of bug 1741350
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.0
Assignee: Jeff Cantrill
QA Contact: Anping Li
URL:
Whiteboard:
Depends On: 1740447 1741350
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-14 01:17 UTC by Qiaoling Tang
Modified: 2019-08-14 20:57 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1740447
Environment:
Last Closed: 2019-08-14 20:57:37 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Qiaoling Tang 2019-08-14 01:17:29 UTC
Same issue when upgrading ES from 4.1.4 to 4.2.0

+++ This bug was initially created as a clone of Bug #1740447 +++

Description of problem:
Deploy logging 4.1.4, then try to upgrade logging to 4.1.11, the ES pods couldn't upgrade successfully due to `Error occurred while updating node elasticsearch-cdm-pkyex0nr-1: Deployment.apps \"elasticsearch-cdm-pkyex0nr-1\" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: \"600m\": must be less than or equal to cpu limit`

$ oc logs -n openshift-operators-redhat elasticsearch-operator-8d644c48-j4d2v
time="2019-08-13T02:56:13Z" level=info msg="Go Version: go1.10.8"
time="2019-08-13T02:56:13Z" level=info msg="Go OS/Arch: linux/amd64"
time="2019-08-13T02:56:13Z" level=info msg="operator-sdk Version: 0.0.7"
time="2019-08-13T02:56:13Z" level=info msg="Watching logging.openshift.io/v1, Elasticsearch, , 5000000000"
time="2019-08-13T02:56:42Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 5 shards in preparation for cluster restart"
time="2019-08-13T02:56:42Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-pkyex0nr-1: Deployment.apps \"elasticsearch-cdm-pkyex0nr-1\" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: \"600m\": must be less than or equal to cpu limit"
time="2019-08-13T02:56:44Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-pkyex0nr-2: Deployment.apps \"elasticsearch-cdm-pkyex0nr-2\" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: \"600m\": must be less than or equal to cpu limit"
time="2019-08-13T02:56:47Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 2 shards in preparation for cluster restart"
time="2019-08-13T02:56:47Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-pkyex0nr-3: Deployment.apps \"elasticsearch-cdm-pkyex0nr-3\" is invalid: spec.template.spec.containers[0].resources.requests: Invalid value: \"600m\": must be less than or equal to cpu limit"
time="2019-08-13T02:56:58Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 6 shards in preparation for cluster restart"
time="2019-08-13T02:57:10Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart"
time="2019-08-13T02:57:34Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 6 shards in preparation for cluster restart"
time="2019-08-13T02:57:47Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 2 shards in preparation for cluster restart"
time="2019-08-13T02:57:59Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 3 shards in preparation for cluster restart"
time="2019-08-13T02:58:10Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 4 shards in preparation for cluster restart"

$ oc get clusterlogging instance -oyaml
  logStore:
    elasticsearch:
      nodeCount: 3
      redundancyPolicy: SingleRedundancy
      resources:
        requests:
          cpu: 600m
          memory: 4Gi
      storage:
        size: 10Gi
        storageClassName: gp2
    type: elasticsearch


$ oc get elasticsearch elasticsearch -oyaml
spec:
  managementState: Managed
  nodeSpec:
    image: image-registry.openshift-image-registry.svc:5000/openshift/ose-logging-elasticsearch5:v4.1.11-201908122027
    resources:
      requests:
        cpu: 600m
        memory: 4Gi
  nodes:
  - genUUID: pkyex0nr
    nodeCount: 3
    resources: {}
    roles:
    - client
    - data
    - master
    storage:
      size: 10Gi
      storageClassName: gp2
  redundancyPolicy: SingleRedundancy


$ oc get deploy elasticsearch-cdm-pkyex0nr-1 -oyaml 

        image: registry.redhat.io/openshift4/ose-logging-elasticsearch5:v4.1.4-201906271212
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9300
          name: cluster
          protocol: TCP
        - containerPort: 9200
          name: restapi
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /usr/share/elasticsearch/probe/readiness.sh
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 30
        resources:
          limits:
            cpu: 600m
            memory: 4Gi
          requests:
            cpu: 600m
            memory: 4Gi

Version-Release number of selected component (if applicable):
from:
ose-cluster-logging-operator:v4.1.4-201906271212
ose-elasticsearch-operator:v4.1.4-201906271212

to:
ose-elasticsearch-operator:v4.1.11-201908122027
ose-cluster-logging-operator:v4.1.11-201908122027


How reproducible:
Always

Steps to Reproduce:
1. Deploy logging 4.1.4
2. upgrade it to 4.1.11
3.

Actual results:


Expected results:


Additional info:

Comment 1 Jeff Cantrill 2019-08-14 20:57:37 UTC

*** This bug has been marked as a duplicate of bug 1741350 ***


Note You need to log in before you can comment on or make changes to this bug.