o Also as per comment 3 where the project was deleted in a hope that the application logging to elastic search db will stop and thereby the elastic search db will close it's connection, but still the lock seems to persist. When the project is deleted, it seems that the file descriptors or locks are not cleared properly. I think deletion of project shall take care of this.The problematic behaviour we see here is same as deleting a file without closing it's open FD so that the FDs remain in memory. o Also when we scale down the dc of the elastic search database the locks are released i.e the locks are removed when the elastic search db is removed. The above 2 points seem to be main issues.
@Miheer Salunke Sorry for late response. We tested NFS pv on RHEL for logging stack functions, but not focus on its performance.
(In reply to Miheer Salunke from comment #8) > o Also as per comment 3 where the project was deleted in a hope that the > application logging to elastic search db will stop and thereby the elastic > search db will close it's connection, but still the lock seems to persist. Deleting the project in OpenShift is not reflected in any way in Elasticsearch - it retains all the same indices that were created previously. So this is not surprising. > o Also when we scale down the dc of the elastic search database the locks > are released i.e the locks are removed when the elastic search db is removed. Yes, restarting the cluster should clear all the locks. This is a workaround, albeit a bit drastic. If you first shut down all the fluentd instances and then delete all the ES pods en masse (assuming they're backed by persistent storage) then you could actually recover with minimal interruption. I'd hate for this to be our customer recommendation though. I think it would actually be safe in this instance to disable NFS file locking. Each instance of ES should have exclusive use of its storage so local file locks should suffice. If you were mounting NFS manually you could simply add "nolock" in the NFS mount options; however there's no way to do this on an NFS PV (ref. https://github.com/kubernetes/kubernetes/issues/17226). I wonder if this can be done from the server side? The only other alternative I can think of is manually NFS mounting the volume and then treating it as a hostmount (which has its own complications). Aside from fixing the lock-leaking behavior, of course. We're investigating.
(In reply to Luke Meyer from comment #23) > (In reply to Miheer Salunke from comment #8) > > o Also as per comment 3 where the project was deleted in a hope that the > > application logging to elastic search db will stop and thereby the elastic > > search db will close it's connection, but still the lock seems to persist. > > Deleting the project in OpenShift is not reflected in any way in > Elasticsearch - it retains all the same indices that were created > previously. So this is not surprising. In fact this is a feature, not a bug, of the logging solution - the ability to review historical logs. > > > o Also when we scale down the dc of the elastic search database the locks > > are released i.e the locks are removed when the elastic search db is removed. > > Yes, restarting the cluster should clear all the locks. This is a > workaround, albeit a bit drastic. If you first shut down all the fluentd > instances and then delete all the ES pods en masse (assuming they're backed > by persistent storage) then you could actually recover with minimal > interruption. I'd hate for this to be our customer recommendation though. > > I think it would actually be safe in this instance to disable NFS file > locking. Each instance of ES should have exclusive use of its storage so > local file locks should suffice. If you were mounting NFS manually you could > simply add "nolock" in the NFS mount options; however there's no way to do > this on an NFS PV (ref. > https://github.com/kubernetes/kubernetes/issues/17226). I wonder if this can > be done from the server side? The only other alternative I can think of is > manually NFS mounting the volume and then treating it as a hostmount (which > has its own complications). > > Aside from fixing the lock-leaking behavior, of course. We're investigating.
> I think it would actually be safe in this instance to disable NFS file > locking. Each instance of ES should have exclusive use of its storage so > local file locks should suffice. > ... I wonder if this can be done from the server side? Under Linux, I believe it can. It looks like the /etc/exports file (http://linux.die.net/man/5/exports) supports the "insecure_locks" option which at least does not authenticate locking requests. I would imagine that means it doesn't track locks at all, but I have yet to set up a test rig for this to see what actually happens. I don't know much about NetApp config but it doesn't appear to me that it provides anything similar (https://library.netapp.com/ecmdocs/ECMP1196979/html/man5/na_exports.5.html). I don't have anything further from the Elasticsearch side yet.
The docs PR merged. This will take some amount of time to make it into the enterprise docs, marking MODIFIED until then. As far as solving the actual problem, I've created https://trello.com/c/AyGHFFVJ/375-enable-using-nfs-as-backing-storage-for-lucene
Origin and Enterprise docs look to have been updated. I'm going to put this into ON_QA in case QE would like to add/run any test cases with respect to the updated docs at https://docs.openshift.com/enterprise/3.2/install_config/aggregate_logging.html#aggregated-elasticsearch
Was blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1366137
@lmeyer It looked like the log entries exist in /usr/local/es-storage, I'm attaching the result of "ls -R /usr/local/es-storage/", it seemed ok? Thanks.
Created attachment 1191040 [details] Result of "ls -R /usr/local/es-storage"
@xzhao I was looking on the wrong host, of course. Silly mistake. Your test system has /usr/local/es-storage mounted from /nfs/storage/elasticsearch-1 however /nfs/storage isn't actually an NFS mount. Do the docs need to be clarified here? I thought it would be clear enough that the intent is to have NFS storage mounted there, so that the backing file is hosted on NFS. Again, having a lot of locks on the host where ES is running isn't necessarily a problem. It seems to be standard operation. The problem would be if there were a lot of locks on the NFS server host due to locks held by the ES hosts.
Encountered this https://bugzilla.redhat.com/show_bug.cgi?id=1367634 when attempting to delete the logging stacks mounted with the workaround provided in logging docs. So assigned it back. @lmeyer Yes. It will be good to mention as more as possible in the doc. --Isn't the workaround which I should test is to place a non-NFS mount storage on NFS server to overcome the issue about too many locks generated there on the NFS server? In my testing I followed exactly the steps in the doc (after the yellow "warning" mark) to the nfs server installed on node: /nfs/storage/elasticsearch-1 on /usr/local/es-storage type xfs (rw,relatime,seclabel,attr2,inode64,noquota) And then mounted it to the es and es-ops pod. Please correct me if anything is misunderstood. Thanks in advance!
@lmeyer storage team decided not to include the fix to bug #1367634 in OSE 3.2.1. Therefore, do you think it necessary to mention this potential issue inside our logging doc?
(In reply to Xia Zhao from comment #45) > Encountered this https://bugzilla.redhat.com/show_bug.cgi?id=1367634 when > attempting to delete the logging stacks mounted with the workaround provided > in logging docs. So assigned it back. I couldn't reproduce this. Pod termination does sometimes randomly seem to take a while, but I'm guessing you saw it stuck for a few minutes. I wonder if you hit some other race condition. Anyway I didn't see it. > In my testing I followed exactly the steps in the doc (after the yellow > "warning" mark) to the nfs server installed on node: > /nfs/storage/elasticsearch-1 on /usr/local/es-storage type xfs > (rw,relatime,seclabel,attr2,inode64,noquota) > And then mounted it to the es and es-ops pod. Please correct me if anything > is misunderstood. Thanks in advance! It's not clear to me if you understood that /nfs/storage is supposed to be mounted from an NFS volume before beginning those steps. In any case I tried to clarify here: https://github.com/openshift/openshift-docs/pull/2703/files
Closing as the fix is in docs and the long-term fix is captured in a card.