Bug 1546216 - logrotate not working out of the box for gluster client (mount plugin) logs with CNS
Summary: logrotate not working out of the box for gluster client (mount plugin) logs w...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: cns-ansible
Version: cns-3.6
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: John Mulligan
QA Contact: Prasanth
URL:
Whiteboard:
: 1546930 (view as bug list)
Depends On: 1546930
Blocks: 1573420 1622458 OCS-3.11.1-Engineering-Proposed-BZs OCS-3.11.1-devel-triage-done 1642792 1707226
TreeView+ depends on / blocked
 
Reported: 2018-02-16 15:55 UTC by daniel
Modified: 2019-09-06 16:21 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1546930 (view as bug list)
Environment:
Last Closed: 2019-09-06 16:21:02 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github kubernetes kubernetes issues 68050 'None' closed glusterfs log files may become very large if volumes mount failed 2020-03-30 01:51:24 UTC
Red Hat Knowledge Base (Solution) 3356271 None None None 2018-02-16 16:27:27 UTC

Description daniel 2018-02-16 15:55:21 UTC
Description of problem:
Install CNS 3.6 on top of OCP 3.6/7 and use e.g. dynamic provisioning for pods.
Logs get created below:
/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/<vol-name>/*.log

but normal glusterfs log-rotation (which gets automatically deployed) just take the following paths into consideration:
-/var/log/glusterfs/*.log 


Version-Release number of selected component (if applicable):
- OCP 3.7
- CNS 3.6
- glusterfs-3.8.4-54.el7rhgs.x86_64


How reproducible:


Steps to Reproduce:
1. Install OCP with CNS on top 
2. use gluster volumes
3. see logs growing in /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/<vol-name>/*.log

Actual results:
logs never get rotated and fs usage grows, so manual interaction is needed

Expected results:
- we should add a rule automatically so that these logs get rotated
- and we should menation this in the documentation 

Additional info:
possibly sth like 

~~~
/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/*/*.log {
  sharedscripts
  weekly
  rotate 52
  missingok
  compress
  delaycompress
  notifempty
  postrotate
  /usr/bin/killall -HUP glusterfs > /dev/null 2>&1 || true
  /usr/bin/killall -HUP glusterd > /dev/null 2>&1 || true
  endscript
~~~

seems to work, but I am pretty sure there are better solutions

- possibly someone can review my approach and make better suggestions so that I can file a kcs

Comment 2 Raghavendra Talur 2018-02-20 04:41:26 UTC
Daniel,

Thanks for the bug, the path "/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/" belongs to openshift volume plugins and I believe it should be logrotated by openshift/origin. I will clone this bug so that we can track it.

Talur

Comment 3 Raghavendra Talur 2018-02-20 08:10:33 UTC
Moving the bug to cns-deployment component. This will just be a tracker to the OCP bug that it depends on.

Comment 4 Humble Chirammal 2018-05-07 11:30:50 UTC
This ( how to do logrotate) can be documented for now, logs are in a common path for plugins. If this can be changed or not have to be discussed further and also may have to think about backward compatibility. 
Bipin/Daniel, can anyone of you please help to create KCS and verify it in a CNS setup?

Comment 13 Michael Adam 2018-09-20 22:10:31 UTC
From https://bugzilla.redhat.com/show_bug.cgi?id=1546930#c5 :

https://access.redhat.com/solutions/3356271

manual logrotate config as a workaround

Comment 15 Michael Adam 2018-09-20 22:16:28 UTC
*** Bug 1546930 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.