Bug 1347666 - huge amount of NFS locks generated by aggregated integrated logging
Summary: huge amount of NFS locks generated by aggregated integrated logging
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Luke Meyer
QA Contact: chunchen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-17 11:20 UTC by Miheer Salunke
Modified: 2019-11-14 08:25 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-14 13:46:50 UTC
Target Upstream Version:


Attachments (Terms of Use)
Result of "ls -R /usr/local/es-storage" (16.69 KB, text/plain)
2016-08-16 02:39 UTC, Xia Zhao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1933 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.3 Release Advisory 2016-09-27 13:24:36 UTC

Comment 8 Miheer Salunke 2016-06-28 10:38:09 UTC
o Also as per comment 3 where the project was deleted in a hope that the application logging to elastic search db will stop and thereby the elastic search db will close it's connection, but still the lock seems to persist. When the project is deleted, it seems that the file descriptors or locks are not cleared properly. I think deletion of project shall take care of this.The problematic behaviour we see here is same as deleting a file without closing it's open FD so that the FDs remain in memory.

o Also when we scale down the dc of the elastic search database the locks are released i.e the locks are removed when the elastic search db is removed.

The above 2 points seem to be main issues.

Comment 10 chunchen 2016-07-14 08:46:03 UTC
@Miheer Salunke
Sorry for late response. We tested NFS pv on RHEL for logging stack functions, but not focus on its performance.

Comment 23 Luke Meyer 2016-07-19 20:51:58 UTC
(In reply to Miheer Salunke from comment #8)
> o Also as per comment 3 where the project was deleted in a hope that the
> application logging to elastic search db will stop and thereby the elastic
> search db will close it's connection, but still the lock seems to persist.

Deleting the project in OpenShift is not reflected in any way in Elasticsearch - it retains all the same indices that were created previously. So this is not surprising.

> o Also when we scale down the dc of the elastic search database the locks
> are released i.e the locks are removed when the elastic search db is removed.

Yes, restarting the cluster should clear all the locks. This is a workaround, albeit a bit drastic. If you first shut down all the fluentd instances and then delete all the ES pods en masse (assuming they're backed by persistent storage) then you could actually recover with minimal interruption. I'd hate for this to be our customer recommendation though.

I think it would actually be safe in this instance to disable NFS file locking. Each instance of ES should have exclusive use of its storage so local file locks should suffice. If you were mounting NFS manually you could simply add "nolock" in the NFS mount options; however there's no way to do this on an NFS PV (ref. https://github.com/kubernetes/kubernetes/issues/17226). I wonder if this can be done from the server side? The only other alternative I can think of is manually NFS mounting the volume and then treating it as a hostmount (which has its own complications).

Aside from fixing the lock-leaking behavior, of course. We're investigating.

Comment 24 Rich Megginson 2016-07-19 21:11:03 UTC
(In reply to Luke Meyer from comment #23)
> (In reply to Miheer Salunke from comment #8)
> > o Also as per comment 3 where the project was deleted in a hope that the
> > application logging to elastic search db will stop and thereby the elastic
> > search db will close it's connection, but still the lock seems to persist.
> 
> Deleting the project in OpenShift is not reflected in any way in
> Elasticsearch - it retains all the same indices that were created
> previously. So this is not surprising.

In fact this is a feature, not a bug, of the logging solution - the ability to review historical logs.

> 
> > o Also when we scale down the dc of the elastic search database the locks
> > are released i.e the locks are removed when the elastic search db is removed.
> 
> Yes, restarting the cluster should clear all the locks. This is a
> workaround, albeit a bit drastic. If you first shut down all the fluentd
> instances and then delete all the ES pods en masse (assuming they're backed
> by persistent storage) then you could actually recover with minimal
> interruption. I'd hate for this to be our customer recommendation though.
> 
> I think it would actually be safe in this instance to disable NFS file
> locking. Each instance of ES should have exclusive use of its storage so
> local file locks should suffice. If you were mounting NFS manually you could
> simply add "nolock" in the NFS mount options; however there's no way to do
> this on an NFS PV (ref.
> https://github.com/kubernetes/kubernetes/issues/17226). I wonder if this can
> be done from the server side? The only other alternative I can think of is
> manually NFS mounting the volume and then treating it as a hostmount (which
> has its own complications).
> 
> Aside from fixing the lock-leaking behavior, of course. We're investigating.

Comment 26 Luke Meyer 2016-07-25 15:17:25 UTC
> I think it would actually be safe in this instance to disable NFS file
> locking. Each instance of ES should have exclusive use of its storage so
> local file locks should suffice. 
> ... I wonder if this can be done from the server side?

Under Linux, I believe it can. It looks like the /etc/exports file (http://linux.die.net/man/5/exports) supports the "insecure_locks" option which at least does not authenticate locking requests. I would imagine that means it doesn't track locks at all, but I have yet to set up a test rig for this to see what actually happens.

I don't know much about NetApp config but it doesn't appear to me that it provides anything similar (https://library.netapp.com/ecmdocs/ECMP1196979/html/man5/na_exports.5.html).

I don't have anything further from the Elasticsearch side yet.

Comment 37 Luke Meyer 2016-08-05 14:38:41 UTC
The docs PR merged. This will take some amount of time to make it into the enterprise docs, marking MODIFIED until then.

As far as solving the actual problem, I've created https://trello.com/c/AyGHFFVJ/375-enable-using-nfs-as-backing-storage-for-lucene

Comment 38 Luke Meyer 2016-08-10 21:23:20 UTC
Origin and Enterprise docs look to have been updated. I'm going to put this into ON_QA in case QE would like to add/run any test cases with respect to the updated docs at https://docs.openshift.com/enterprise/3.2/install_config/aggregate_logging.html#aggregated-elasticsearch

Comment 39 Xia Zhao 2016-08-11 09:01:00 UTC
Was blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1366137

Comment 42 Xia Zhao 2016-08-16 02:37:33 UTC
@lmeyer It looked like the log entries exist in /usr/local/es-storage, I'm attaching the result of "ls -R /usr/local/es-storage/",  it seemed ok? Thanks.

Comment 43 Xia Zhao 2016-08-16 02:39:30 UTC
Created attachment 1191040 [details]
Result of "ls -R /usr/local/es-storage"

Comment 44 Luke Meyer 2016-08-16 14:52:29 UTC
@xzhao I was looking on the wrong host, of course. Silly mistake.

Your test system has /usr/local/es-storage mounted from /nfs/storage/elasticsearch-1 however /nfs/storage isn't actually an NFS mount. Do the docs need to be clarified here? I thought it would be clear enough that the intent is to have NFS storage mounted there, so that the backing file is hosted on NFS.

Again, having a lot of locks on the host where ES is running isn't necessarily a problem. It seems to be standard operation. The problem would be if there were a lot of locks on the NFS server host due to locks held by the ES hosts.

Comment 45 Xia Zhao 2016-08-17 03:54:38 UTC
Encountered this https://bugzilla.redhat.com/show_bug.cgi?id=1367634 when attempting to delete the logging stacks mounted with the workaround provided in logging docs. So assigned it back. 

@lmeyer Yes. It will be good to mention as more as possible in the doc. --Isn't the workaround which I should test is to place a non-NFS mount storage on NFS server to overcome the issue about too many locks generated there on the NFS server?

In my testing I followed exactly the steps in the doc (after the yellow "warning" mark) to the nfs server installed on node:
/nfs/storage/elasticsearch-1 on /usr/local/es-storage type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
And then mounted it to the es and es-ops pod. Please correct me if anything is misunderstood. Thanks in advance!

Comment 47 Xia Zhao 2016-08-19 08:39:52 UTC
@lmeyer storage team decided not to include the fix to bug #1367634 in OSE 3.2.1. Therefore, do you think it necessary to mention this potential issue inside our logging doc?

Comment 48 Luke Meyer 2016-08-22 21:00:53 UTC
(In reply to Xia Zhao from comment #45)
> Encountered this https://bugzilla.redhat.com/show_bug.cgi?id=1367634 when
> attempting to delete the logging stacks mounted with the workaround provided
> in logging docs. So assigned it back.

I couldn't reproduce this. Pod termination does sometimes randomly seem to take a while, but I'm guessing you saw it stuck for a few minutes. I wonder if you hit some other race condition. Anyway I didn't see it.

> In my testing I followed exactly the steps in the doc (after the yellow
> "warning" mark) to the nfs server installed on node:
> /nfs/storage/elasticsearch-1 on /usr/local/es-storage type xfs
> (rw,relatime,seclabel,attr2,inode64,noquota)
> And then mounted it to the es and es-ops pod. Please correct me if anything
> is misunderstood. Thanks in advance!

It's not clear to me if you understood that /nfs/storage is supposed to be mounted from an NFS volume before beginning those steps. In any case I tried to clarify here:

https://github.com/openshift/openshift-docs/pull/2703/files

Comment 50 Luke Meyer 2016-09-14 13:46:50 UTC
Closing as the fix is in docs and the long-term fix is captured in a card.


Note You need to log in before you can comment on or make changes to this bug.