Bug 1710657 - Elasticsearch pods have an OOTB cpu limit of 1
Summary: Elasticsearch pods have an OOTB cpu limit of 1
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.2.0
Assignee: ewolinet
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-16 01:27 UTC by Mike Fiedler
Modified: 2019-07-15 14:54 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-logging-operator pull 180 None None None 2019-05-28 11:20:51 UTC
Github openshift elasticsearch-operator pull 160 None None None 2019-06-21 14:53:55 UTC

Description Mike Fiedler 2019-05-16 01:27:56 UTC
Description of problem:   Creating the default clusterlogging CR from OperatorHub gives the following clusterlogging CR (note there are no requests or limits specified):

apiVersion: logging.openshift.io/v1                                                                                                                                                                                
kind: ClusterLogging                                                                                                                                                                                               
metadata:                                                                                                                                                                                                          
  generation: 1                                                                                                                                                                                                    
  name: instance                                                                                                                                                                                                   
  selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance                                                                                                                    
spec:                                                                                                                                                                                                              
  collection:                                                                                                                                                                                                      
    logs:                                                                                                                                                                                                          
      fluentd:                                                                                                                                                                                                     
        resources: null                                                                                                                                                                                            
      rsyslog:                                                                                                                                                                                                     
        resources: null                                                                                                                                                                                            
      type: fluentd                                                                                                                                                                                                
  curation:                                                                                                                                                                                                        
    curator:                                                                                                                                                                                                       
      resources: null                                                                                                                                                                                              
      schedule: 30 3 * * *                                                                                                                                                                                         
    type: curator                                                                                                                                                                                                  
  logStore:                                                                                                                                                                                                        
    elasticsearch:                                                                                                                                                                                                 
      nodeCount: 3                                                                                                                                                                                                 
      redundancyPolicy: SingleRedundancy                                                                                                                                                                           
      resources: null                                                                                                                                                                                              
      storage:                                                                                                                                                                                                     
        size: 50G                                                                                                                                                                                                  
        storageClassName: gp2                                                                                                                                                                                      
    type: elasticsearch                                                                                                                                                                                            
  managementState: Managed                                                                                                                                                                                         
  visualization:                                                                                                                                                                                                   
    kibana:                                                                                                                                                                                                        
      proxy:                                                                                                                                                                                                       
        resources: null                                                                                                                                                                                            
      replicas: 1                                                                                                                                                                                                  
      resources: null
    type: kibana

This in turn creates an ES deployment which has a cpu limit of 1.   Per @jcantrill in https://bugzilla.redhat.com/show_bug.cgi?id=1710404#c1 there should be no limit.   Here's the ES deployment yaml:

First, an excerpt of the request/limit section:

        resources:
          limits:
            cpu: "1"
            memory: 16Gi
          requests:
            cpu: "1"
            memory: 16Gi


Here's the full deployment YAML:


apiVersion: extensions/v1beta1                                                                                                                                                                                     
kind: Deployment                                                                                                                                                                                                   
metadata:                                                                                                                                                                                                          
  annotations:                                                                                                                                                                                                     
    deployment.kubernetes.io/revision: "1"                                                                                                                                                                         
  creationTimestamp: null                                                                                                                                                                                          
  generation: 1                                                                                                                                                                                                    
  labels:                                                                                                                                                                                                          
    cluster-name: elasticsearch                                                                                                                                                                                    
    component: elasticsearch                                                                                                                                                                                       
    es-node-client: "true"                                                                                                                                                                                         
    es-node-data: "true"                                                                                                                                                                                           
    es-node-master: "true"                                                                                                                                                                                         
    node-name: elasticsearch-cdm-n02vyeo6-1                                                                                                                                                                        
    tuned.openshift.io/elasticsearch: "true"                                                                                                                                                                       
  name: elasticsearch-cdm-n02vyeo6-1                                                                                                                                                                               
  ownerReferences:                                                                                                                                                                                                 
  - apiVersion: logging.openshift.io/v1                                                                                                                                                                            
    controller: true                                                                                                                                                                                               
    kind: Elasticsearch                                                                                                                                                                                            
    name: elasticsearch                                                                                                                                                                                            
    uid: 4ccaf313-7716-11e9-a70e-0221fa8d2a68                                                                                                                                                                      
  selfLink: /apis/extensions/v1beta1/namespaces/openshift-logging/deployments/elasticsearch-cdm-n02vyeo6-1                                                                                                         
spec:                                                                                                                                                                                                              
  paused: true                                                                                                                                                                                                     
  progressDeadlineSeconds: 1800                                                                                                                                                                                    
  replicas: 1                                                                                                                                                                                                      
  revisionHistoryLimit: 10                                                                                                                                                                                         
  selector:                                                                                                                                                                                                        
    matchLabels:
      cluster-name: elasticsearch
      es-node-client: "true"
      es-node-data: "true"
      es-node-master: "true"
      node-name: elasticsearch-cdm-n02vyeo6-1
  strategy:
    type: Recreate
  template:                                                                                                                                                                                              [100/1917]
    metadata:                                                                                                                                                                                                      
      creationTimestamp: null                                                                                                                                                                                      
      labels:                                                                                                                                                                                                      
        cluster-name: elasticsearch                                                                                                                                                                                
        component: elasticsearch                                                                                                                                                                                   
        es-node-client: "true"                                                                                                                                                                                     
        es-node-data: "true"                                                                                                                                                                                       
        es-node-master: "true"                                                                                                                                                                                     
        node-name: elasticsearch-cdm-n02vyeo6-1                                                                                                                                                                    
        tuned.openshift.io/elasticsearch: "true"                                                                                                                                                                   
    spec:                                                                                                                                                                                                          
      affinity:                                                                                                                                                                                                    
        podAntiAffinity:                                                                                                                                                                                           
          preferredDuringSchedulingIgnoredDuringExecution:                                                                                                                                                         
          - podAffinityTerm:                                                                                                                                                                                       
              labelSelector:                                                                                                                                                                                       
                matchExpressions:                                                                                                                                                                                  
                - key: es-node-client                                                                                                                                                                              
                  operator: In                                                                                                                                                                                     
                  values:                                                                                                                                                                                          
                  - "true"                                                                                                                                                                                         
                - key: es-node-data                                                                                                                                                                                
                  operator: In                                                                                                                                                                                     
                  values:                                                                                                                                                                                          
                  - "true"                                                                                                                                                                                         
                - key: es-node-master                                                                                                                                                                              
                  operator: In
                  values:
                  - "true"
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: DC_NAME
          value: elasticsearch-cdm-n02vyeo6-1
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: KUBERNETES_TRUST_CERT
          value: "true"
        - name: SERVICE_DNS
          value: elasticsearch-cluster
        - name: CLUSTER_NAME
          value: elasticsearch
        - name: INSTANCE_RAM
          value: 16Gi
        - name: HEAP_DUMP_LOCATION
          value: /elasticsearch/persistent/heapdump.hprof
        - name: RECOVER_AFTER_TIME
          value: 5m                                                                                                                                                                                                
        - name: READINESS_PROBE_TIMEOUT                                                                                                                                                                            
          value: "30"                                                                                                                                                                                              
        - name: POD_LABEL                                                                                                                                                                                          
          value: cluster=elasticsearch
        - name: IS_MASTER
          value: "true"
        - name: HAS_DATA
          value: "true"
        image: quay.io/openshift/origin-logging-elasticsearch5:latest
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9300
          name: cluster
          protocol: TCP
        - containerPort: 9200
          name: restapi
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /usr/share/elasticsearch/probe/readiness.sh
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 30
        resources:
          limits:
            cpu: "1"
            memory: 16Gi
          requests:
            cpu: "1"
            memory: 16Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /elasticsearch/persistent
          name: elasticsearch-storage
        - mountPath: /usr/share/java/elasticsearch/config
          name: elasticsearch-config
        - mountPath: /etc/openshift/elasticsearch/secret
          name: certificates
      - args:
        - --https-address=:60000
        - --provider=openshift
        - --upstream=https://127.0.0.1:9200
        - --tls-cert=/etc/proxy/secrets/tls.crt
        - --tls-key=/etc/proxy/secrets/tls.key
        - --upstream-ca=/etc/proxy/elasticsearch/admin-ca
        - --openshift-service-account=elasticsearch
        - '-openshift-sar={"resource": "namespaces", "verb": "get"}'
        - '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}'
        - --pass-user-bearer-token
        - --cookie-secret=9X8uME/4a3Ejshe+SuI52A==
        image: quay.io/openshift/origin-oauth-proxy:latest
        imagePullPolicy: IfNotPresent
        name: proxy
        ports:
        - containerPort: 60000
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/proxy/secrets
          name: elasticsearch-metrics
        - mountPath: /etc/proxy/elasticsearch
          name: certificates
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: elasticsearch
      serviceAccountName: elasticsearch
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node.kubernetes.io/disk-pressure
        operator: Exists
      volumes:
      - configMap:
          defaultMode: 420
          name: elasticsearch
        name: elasticsearch-config
      - name: elasticsearch-storage
        persistentVolumeClaim:
          claimName: elasticsearch-elasticsearch-cdm-n02vyeo6-1
      - name: certificates
        secret:
          defaultMode: 420
          secretName: elasticsearch
      - name: elasticsearch-metrics
        secret:
          defaultMode: 420
          secretName: elasticsearch-metrics

Comment 2 Qiaoling Tang 2019-06-28 09:01:22 UTC
This could be reproduced in ose-cluster-logging-operator-v4.2.0-201906271419 and ose-elasticsearch-operator-v4.2.0-201906271419

$ oc get clusterlogging -oyaml
apiVersion: v1
items:
- apiVersion: logging.openshift.io/v1
  kind: ClusterLogging
  metadata:
    creationTimestamp: "2019-06-28T08:54:10Z"
    generation: 19
    name: instance
    namespace: openshift-logging
    resourceVersion: "198604"
    selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance
    uid: 4b996a5b-9982-11e9-96f9-0ada222ba802
  spec:
    collection:
      logs:
        fluentd:
          resources: null
        rsyslog:
          resources: null
        type: fluentd
    curation:
      curator:
        resources: null
        schedule: 30 3 * * *
      type: curator
    logStore:
      elasticsearch:
        nodeCount: 3
        redundancyPolicy: SingleRedundancy
        resources: null
        storage:
          size: 200G
          storageClassName: gp2
      type: elasticsearch
    managementState: Managed
    visualization:
      kibana:
        proxy:
          resources: null
        replicas: 1
        resources: null
      type: kibana

$ oc get deploy elasticsearch-cdm-yy8yrtno-1 -oyaml  |grep -A 6 resources
        resources:
          limits:
            cpu: "1"
            memory: 16Gi
          requests:
            cpu: "1"
            memory: 16Gi
--
        resources:
          limits:
            memory: 64Mi
          requests:
            cpu: 100m
            memory: 64Mi
        terminationMessagePath: /dev/termination-log

$ oc get elasticsearch -oyaml
apiVersion: v1
items:
- apiVersion: logging.openshift.io/v1
  kind: Elasticsearch
  metadata:
    creationTimestamp: "2019-06-28T08:54:10Z"
    generation: 6
    name: elasticsearch
    namespace: openshift-logging
    ownerReferences:
    - apiVersion: logging.openshift.io/v1
      controller: true
      kind: ClusterLogging
      name: instance
      uid: 4b996a5b-9982-11e9-96f9-0ada222ba802
    resourceVersion: "199235"
    selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/elasticsearches/elasticsearch
    uid: 4bab0c6f-9982-11e9-8303-02bb6c1311a4
  spec:
    managementState: Managed
    nodeSpec:
      image: image-registry.openshift-image-registry.svc:5000/openshift/ose-logging-elasticsearch5:v4.2.0
      resources:
        limits:
          memory: 16Gi
        requests:
          cpu: "1"
          memory: 16Gi
    nodes:
    - genUUID: yy8yrtno
      nodeCount: 3

Comment 3 ewolinet 2019-07-12 21:19:11 UTC
I'm unable to recreate this with the latest images, can you please retest and verify this is still an issue?

Comment 4 Qiaoling Tang 2019-07-15 02:30:22 UTC
Verified in ose-elasticsearch-operator-v4.2.0-201907121819 and ose-cluster-logging-operator-v4.2.0-201907121819


Note You need to log in before you can comment on or make changes to this bug.