Bug 1710657
| Summary: | Elasticsearch pods have an OOTB cpu limit of 1 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Mike Fiedler <mifiedle> |
| Component: | Logging | Assignee: | ewolinet |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.1.0 | CC: | aos-bugs, jcantril, pweil, qitang, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 4.2.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-10-16 06:28:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This could be reproduced in ose-cluster-logging-operator-v4.2.0-201906271419 and ose-elasticsearch-operator-v4.2.0-201906271419
$ oc get clusterlogging -oyaml
apiVersion: v1
items:
- apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
creationTimestamp: "2019-06-28T08:54:10Z"
generation: 19
name: instance
namespace: openshift-logging
resourceVersion: "198604"
selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance
uid: 4b996a5b-9982-11e9-96f9-0ada222ba802
spec:
collection:
logs:
fluentd:
resources: null
rsyslog:
resources: null
type: fluentd
curation:
curator:
resources: null
schedule: 30 3 * * *
type: curator
logStore:
elasticsearch:
nodeCount: 3
redundancyPolicy: SingleRedundancy
resources: null
storage:
size: 200G
storageClassName: gp2
type: elasticsearch
managementState: Managed
visualization:
kibana:
proxy:
resources: null
replicas: 1
resources: null
type: kibana
$ oc get deploy elasticsearch-cdm-yy8yrtno-1 -oyaml |grep -A 6 resources
resources:
limits:
cpu: "1"
memory: 16Gi
requests:
cpu: "1"
memory: 16Gi
--
resources:
limits:
memory: 64Mi
requests:
cpu: 100m
memory: 64Mi
terminationMessagePath: /dev/termination-log
$ oc get elasticsearch -oyaml
apiVersion: v1
items:
- apiVersion: logging.openshift.io/v1
kind: Elasticsearch
metadata:
creationTimestamp: "2019-06-28T08:54:10Z"
generation: 6
name: elasticsearch
namespace: openshift-logging
ownerReferences:
- apiVersion: logging.openshift.io/v1
controller: true
kind: ClusterLogging
name: instance
uid: 4b996a5b-9982-11e9-96f9-0ada222ba802
resourceVersion: "199235"
selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/elasticsearches/elasticsearch
uid: 4bab0c6f-9982-11e9-8303-02bb6c1311a4
spec:
managementState: Managed
nodeSpec:
image: image-registry.openshift-image-registry.svc:5000/openshift/ose-logging-elasticsearch5:v4.2.0
resources:
limits:
memory: 16Gi
requests:
cpu: "1"
memory: 16Gi
nodes:
- genUUID: yy8yrtno
nodeCount: 3
I'm unable to recreate this with the latest images, can you please retest and verify this is still an issue? Verified in ose-elasticsearch-operator-v4.2.0-201907121819 and ose-cluster-logging-operator-v4.2.0-201907121819 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922 |
Description of problem: Creating the default clusterlogging CR from OperatorHub gives the following clusterlogging CR (note there are no requests or limits specified): apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: generation: 1 name: instance selfLink: /apis/logging.openshift.io/v1/namespaces/openshift-logging/clusterloggings/instance spec: collection: logs: fluentd: resources: null rsyslog: resources: null type: fluentd curation: curator: resources: null schedule: 30 3 * * * type: curator logStore: elasticsearch: nodeCount: 3 redundancyPolicy: SingleRedundancy resources: null storage: size: 50G storageClassName: gp2 type: elasticsearch managementState: Managed visualization: kibana: proxy: resources: null replicas: 1 resources: null type: kibana This in turn creates an ES deployment which has a cpu limit of 1. Per @jcantrill in https://bugzilla.redhat.com/show_bug.cgi?id=1710404#c1 there should be no limit. Here's the ES deployment yaml: First, an excerpt of the request/limit section: resources: limits: cpu: "1" memory: 16Gi requests: cpu: "1" memory: 16Gi Here's the full deployment YAML: apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "1" creationTimestamp: null generation: 1 labels: cluster-name: elasticsearch component: elasticsearch es-node-client: "true" es-node-data: "true" es-node-master: "true" node-name: elasticsearch-cdm-n02vyeo6-1 tuned.openshift.io/elasticsearch: "true" name: elasticsearch-cdm-n02vyeo6-1 ownerReferences: - apiVersion: logging.openshift.io/v1 controller: true kind: Elasticsearch name: elasticsearch uid: 4ccaf313-7716-11e9-a70e-0221fa8d2a68 selfLink: /apis/extensions/v1beta1/namespaces/openshift-logging/deployments/elasticsearch-cdm-n02vyeo6-1 spec: paused: true progressDeadlineSeconds: 1800 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: cluster-name: elasticsearch es-node-client: "true" es-node-data: "true" es-node-master: "true" node-name: elasticsearch-cdm-n02vyeo6-1 strategy: type: Recreate template: [100/1917] metadata: creationTimestamp: null labels: cluster-name: elasticsearch component: elasticsearch es-node-client: "true" es-node-data: "true" es-node-master: "true" node-name: elasticsearch-cdm-n02vyeo6-1 tuned.openshift.io/elasticsearch: "true" spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: es-node-client operator: In values: - "true" - key: es-node-data operator: In values: - "true" - key: es-node-master operator: In values: - "true" topologyKey: kubernetes.io/hostname weight: 100 containers: - env: - name: DC_NAME value: elasticsearch-cdm-n02vyeo6-1 - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: KUBERNETES_TRUST_CERT value: "true" - name: SERVICE_DNS value: elasticsearch-cluster - name: CLUSTER_NAME value: elasticsearch - name: INSTANCE_RAM value: 16Gi - name: HEAP_DUMP_LOCATION value: /elasticsearch/persistent/heapdump.hprof - name: RECOVER_AFTER_TIME value: 5m - name: READINESS_PROBE_TIMEOUT value: "30" - name: POD_LABEL value: cluster=elasticsearch - name: IS_MASTER value: "true" - name: HAS_DATA value: "true" image: quay.io/openshift/origin-logging-elasticsearch5:latest imagePullPolicy: IfNotPresent name: elasticsearch ports: - containerPort: 9300 name: cluster protocol: TCP - containerPort: 9200 name: restapi protocol: TCP readinessProbe: exec: command: - /usr/share/elasticsearch/probe/readiness.sh failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 30 resources: limits: cpu: "1" memory: 16Gi requests: cpu: "1" memory: 16Gi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /elasticsearch/persistent name: elasticsearch-storage - mountPath: /usr/share/java/elasticsearch/config name: elasticsearch-config - mountPath: /etc/openshift/elasticsearch/secret name: certificates - args: - --https-address=:60000 - --provider=openshift - --upstream=https://127.0.0.1:9200 - --tls-cert=/etc/proxy/secrets/tls.crt - --tls-key=/etc/proxy/secrets/tls.key - --upstream-ca=/etc/proxy/elasticsearch/admin-ca - --openshift-service-account=elasticsearch - '-openshift-sar={"resource": "namespaces", "verb": "get"}' - '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}' - --pass-user-bearer-token - --cookie-secret=9X8uME/4a3Ejshe+SuI52A== image: quay.io/openshift/origin-oauth-proxy:latest imagePullPolicy: IfNotPresent name: proxy ports: - containerPort: 60000 name: metrics protocol: TCP resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /etc/proxy/secrets name: elasticsearch-metrics - mountPath: /etc/proxy/elasticsearch name: certificates dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: elasticsearch serviceAccountName: elasticsearch terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: node.kubernetes.io/disk-pressure operator: Exists volumes: - configMap: defaultMode: 420 name: elasticsearch name: elasticsearch-config - name: elasticsearch-storage persistentVolumeClaim: claimName: elasticsearch-elasticsearch-cdm-n02vyeo6-1 - name: certificates secret: defaultMode: 420 secretName: elasticsearch - name: elasticsearch-metrics secret: defaultMode: 420 secretName: elasticsearch-metrics