Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1865364

Summary:	Elasticsearch operator does not alert for out of file descriptors
Product:	OpenShift Container Platform	Reporter:	Brett Jones <brejones>
Component:	Logging	Assignee:	Brett Jones <brejones>
Status:	CLOSED ERRATA	QA Contact:	Qiaoling Tang <qitang>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	4.6	CC:	aos-bugs, lvlcek
Target Milestone:	---
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-27 15:09:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brett Jones 2020-08-03 20:59:50 UTC

Description of problem:

Elasticsearch should send a Prometheus alert when the underlying filesystem is expected to run out of file descriptors. 

Version-Release number of selected component (if applicable):

4.6

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Qiaoling Tang 2020-08-05 03:03:04 UTC

I tested with elasticsearch-operator.4.6.0-202008040915.p0 and can find the 2 alerts in prometheusrule/elasticsearch-prometheus-rules.

Hi @Brett, do you have any suggestions about how to make these 2 alerts pending/firing?

Comment 4 Lukas Vlcek 2020-08-06 15:02:38 UTC

Hi Qiaoling,

We were looking at this with Brett and we have some suggestions:

--------------

For the alerts around disk space available we can try to create large dummy files on the Elasticsearch storage to eat most of the free disk space.
One idea is to ssh to ES pod and use command like

dd if=/dev/urandom of=<es_storage_drive>/<some_path/file.txt bs=1048576 count=5000

to generate dummy files (kudos to Brett).

--------------

As for opened file descriptors alert this gets more tricky because you would need to limit the number of opened file descriptors by executing command

ulimit -n ...

but first, you will have to be a root and second, this must happen before ES process starts, which means you would have to provide your own modified image (maybe there is better way... ?).

HTH,
Lukáš

Comment 5 Qiaoling Tang 2020-08-11 01:26:21 UTC

Thank you very much. Your suggestions are of great help to me.

Comment 9 errata-xmlrpc 2020-10-27 15:09:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.1 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4198