Bug 1886856
| Summary: | The ES proxy container always restarts. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Periklis Tsirakidis <periklis> |
| Component: | Logging | Assignee: | Periklis Tsirakidis <periklis> |
| Status: | CLOSED ERRATA | QA Contact: | Qiaoling Tang <qitang> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.6 | CC: | aos-bugs |
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | logging-exploration | ||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-24 11:21:19 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1880960 | ||
|
Description
Periklis Tsirakidis
2020-10-09 14:28:21 UTC
The fix doesn't work, the default request memory of proxy container is still `64Mi`.
$ oc get cl instance -oyaml
logStore:
elasticsearch:
nodeCount: 3
redundancyPolicy: SingleRedundancy
resources:
requests:
memory: 2Gi
storage:
size: 20Gi
storageClassName: standard
retentionPolicy:
application:
maxAge: 1d
audit:
maxAge: 2w
infra:
maxAge: 3h
type: elasticsearch
$ oc get es elasticsearch -oyaml
nodeSpec:
proxyResources:
limits:
memory: 64Mi
requests:
cpu: 100m
memory: 64Mi
resources:
requests:
memory: 2Gi
Image: quay.io/openshift/origin-elasticsearch-operator@sha256:1641e7698155161b3fadc3079c0d4abdbdd7107cc89a4642bb8e58309b8d9700
I manually set the memory to 256Mi, the proxy container restarted after running for about 3 hours.
proxy:
Container ID: cri-o://90af77f73f240cfe3a1be7e3a840671a9cd8da932909aade141681574a9dfe89
Image: quay.io/openshift/origin-elasticsearch-proxy:4.7.0
Image ID: quay.io/openshift/origin-elasticsearch-proxy@sha256:f8aeb4d918dc4a6d77641af58586fc39ec339c1f95b167d1f62aec5164e5da1f
Ports: 60000/TCP, 60001/TCP
Host Ports: 0/TCP, 0/TCP
Args:
--listening-address=:60000
--tls-cert=/etc/proxy/elasticsearch/logging-es.crt
--tls-key=/etc/proxy/elasticsearch/logging-es.key
--tls-client-ca=/etc/proxy/elasticsearch/admin-ca
--metrics-listening-address=:60001
--metrics-tls-cert=/etc/proxy/secrets/tls.crt
--metrics-tls-key=/etc/proxy/secrets/tls.key
--upstream-ca=/etc/proxy/elasticsearch/admin-ca
--cache-expiry=60s
--auth-backend-role=admin_reader={"namespace": "default", "verb": "get", "resource": "pods/log"}
--auth-backend-role=prometheus={"verb": "get", "resource": "/metrics"}
--auth-backend-role=jaeger={"verb": "get", "resource": "/jaeger", "resourceAPIGroup": "elasticsearch.jaegertracing.io"}
--auth-backend-role=elasticsearch-operator={"namespace": "*", "verb": "*", "resource": "*", "resourceAPIGroup": "logging.openshift.io"}
--auth-backend-role=index-management={"namespace":"openshift-logging", "verb": "*", "resource": "indices", "resourceAPIGroup": "elasticsearch.openshift.io"}
--auth-admin-role=admin_reader
--auth-default-role=project_user
State: Running
Started: Mon, 12 Oct 2020 01:05:56 -0400
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Sun, 11 Oct 2020 21:52:49 -0400
Finished: Mon, 12 Oct 2020 01:05:55 -0400
Ready: True
Restart Count: 1
Limits:
memory: 256Mi
Requests:
cpu: 100m
memory: 256Mi
Environment:
LOG_LEVEL: info
Mounts:
/etc/proxy/elasticsearch from certificates (rw)
/etc/proxy/secrets from elasticsearch-metrics (rw)
/var/run/secrets/kubernetes.io/serviceaccount from elasticsearch-token-rfc2v (ro)
$ oc get pod
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-6b87bdfcd-2nzfc 1/1 Running 0 4h7m
elasticsearch-cdm-pwwtd9g6-1-564d87bdfd-9m7zh 2/2 Running 1 3h39m
elasticsearch-cdm-pwwtd9g6-2-98c975c59-2ptpp 2/2 Running 1 3h38m
elasticsearch-cdm-pwwtd9g6-3-599cdf5df6-9wcf9 2/2 Running 1 3h36m
The ES pods has been running for 13 hours, and the proxy container didn't restart.
$ oc get pod
NAME READY STATUS RESTARTS AGE
cluster-logging-operator-54d478856f-h2fnv 1/1 Running 0 13h
elasticsearch-cdm-djppeosj-1-5c9f6ff64-t7r8z 2/2 Running 0 13h
elasticsearch-cdm-djppeosj-2-7d75b56bff-xkbsg 2/2 Running 0 13h
elasticsearch-cdm-djppeosj-3-7f4f86bd86-s5t8r 2/2 Running 0 13h
elasticsearch-delete-app-1604448000-9wxpr 0/1 Completed 0 2m26s
elasticsearch-delete-audit-1604448000-crt2l 0/1 Completed 0 2m26s
elasticsearch-delete-infra-1604448000-xp2v2 0/1 Completed 0 2m26s
$ oc get csv
NAME DISPLAY VERSION REPLACES PHASE
clusterlogging.4.7.0-202011021919.p0 Cluster Logging 4.7.0-202011021919.p0 Succeeded
elasticsearch-operator.4.7.0-202011030448.p0 Elasticsearch Operator 4.7.0-202011030448.p0 Succeeded
The default value of proxy container requested memory is 256Mi now.
managementState: Managed
nodeSpec:
proxyResources:
limits:
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
So verified this bz.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Errata Advisory for Openshift Logging 5.0.0), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0652 |