Bug 1470521

Prasanna, can you move the log location to "gluster-block" inside /var/log/glusterfs?

(In reply to Humble Chirammal from comment #2)
> Prasanna, can you move the log location to "gluster-block" inside
> /var/log/glusterfs?

If in future we are looking to have separate containers for gluster-blockd and glusterd, moving gluster-block inside /var/log/glusterfs may not be a good thing?

Vijay, iic, you are asking to make it configurable in gluster-block code, Isnt it ?

gluster-block logs are again logged at /var/log/gluster-block

Moving the bug to assigned.

sh-4.2# ps -aux | grep 'gluster-block'
root      2238  0.1  0.0 1539852 49092 ?       Ssl  10:16   0:22 /usr/bin/tcmu-runner --tcmu-log-dir=/var/log/gluster-block/
root     22500  0.0  0.0 202208  1996 ?        Ssl  13:00   0:00 /usr/sbin/gluster-blockd --glfs-lru-count 15 --log-level INFO
root     24559  0.0  0.0   9044   664 pts/0    S+   13:58   0:00 grep gluster-block
sh-4.2# cat /usr/lib/systemd/system/gluster-block
cat: /usr/lib/systemd/system/gluster-block: No such file or directory
sh-4.2# cat /usr/lib/systemd/system/gluster-blockd.service 
[Unit]
Description=Gluster block storage utility
BindsTo=tcmu-runner.service 
After=tcmu-runner.service 

[Service]
Type=simple
Environment="GB_GLFS_LRU_COUNT=15"
Environment="GB_LOG_LEVEL=INFO"
EnvironmentFile=-/etc/sysconfig/gluster-blockd
ExecStart=/usr/sbin/gluster-blockd --glfs-lru-count $GB_GLFS_LRU_COUNT --log-level $GB_LOG_LEVEL $GB_EXTRA_ARGS
KillMode=process

[Install]
WantedBy=multi-user.target

sh-4.2# ls -l /var/log/glusterfs/gluster-block
total 28
-rw-r--r--. 1 root root 26554 Sep  5 16:41 gluster-block-cli.log
sh-4.2# ls -l /var/log/gluster-block/
total 776
-rw-r--r--. 1 root root 505868 Sep 12 13:02 gluster-block-configshell.log
-rw-r--r--. 1 root root   4768 Sep 12 13:02 gluster-blockd.log
-rw-------. 1 root root 240379 Sep 12 13:56 tcmu-runner-glfs.log
-rw-------. 1 root root  16379 Sep 12 13:37 tcmu-runner.log

gluster-block logs are still logged at /var/log/gluster-block.

sh-4.2# ls -l /var/log/gluster-block/
total 1692
-rw-r--r--. 1 root root 1636352 Sep 14 14:01 gluster-block-configshell.log
-rw-------. 1 root root   10440 Sep 14 14:01 gluster-block-gfapi.log
-rw-r--r--. 1 root root   81810 Sep 14 14:01 gluster-blockd.log
sh-4.2# ls -l /var/log/glusterfs/gluster-block/
total 228
-rw-r--r--. 1 root root  19170 Sep 14 14:01 gluster-block-cli.log
-rw-------. 1 root root 208729 Sep 14 14:01 tcmu-runner-glfs.log
-rw-------. 1 root root    357 Sep 14 12:31 tcmu-runner.log

Moving the bug to assigned.

(In reply to krishnaram Karthick from comment #19)
> gluster-block logs are still logged at /var/log/gluster-block.
> 
> sh-4.2# ls -l /var/log/gluster-block/
> total 1692
> -rw-r--r--. 1 root root 1636352 Sep 14 14:01 gluster-block-configshell.log
> -rw-------. 1 root root   10440 Sep 14 14:01 gluster-block-gfapi.log
> -rw-r--r--. 1 root root   81810 Sep 14 14:01 gluster-blockd.log
> sh-4.2# ls -l /var/log/glusterfs/gluster-block/
> total 228
> -rw-r--r--. 1 root root  19170 Sep 14 14:01 gluster-block-cli.log
> -rw-------. 1 root root 208729 Sep 14 14:01 tcmu-runner-glfs.log
> -rw-------. 1 root root    357 Sep 14 12:31 tcmu-runner.log
> 
> Moving the bug to assigned.

We analysed this bug and we can see "CNS" container set ENV of GB_LOGDIR properly in the container, from there on gluster block would have picked up this following below code:

  logDir = getenv("GB_LOGDIR");
  if (!logDir) {
    logDir = GB_LOGDIR;
  }

  if (strlen(logDir) > PATH_MAX - GB_MAX_LOGFILENAME) {
    fprintf(stderr, "strlen of logDir Path > PATH_MAX: %s\n", logDir);
    return EXIT_FAILURE;
  }



Which has not happened, any way we can fix in some other way in CNS, considering the timeline of CNS release we are taking another path to fix this issue.

(In reply to Humble Chirammal from comment #20)
> (In reply to krishnaram Karthick from comment #19)
> > gluster-block logs are still logged at /var/log/gluster-block.
> > 
> > sh-4.2# ls -l /var/log/gluster-block/
> > total 1692
> > -rw-r--r--. 1 root root 1636352 Sep 14 14:01 gluster-block-configshell.log
> > -rw-------. 1 root root   10440 Sep 14 14:01 gluster-block-gfapi.log
> > -rw-r--r--. 1 root root   81810 Sep 14 14:01 gluster-blockd.log
> > sh-4.2# ls -l /var/log/glusterfs/gluster-block/
> > total 228
> > -rw-r--r--. 1 root root  19170 Sep 14 14:01 gluster-block-cli.log
> > -rw-------. 1 root root 208729 Sep 14 14:01 tcmu-runner-glfs.log
> > -rw-------. 1 root root    357 Sep 14 12:31 tcmu-runner.log
> > 
> > Moving the bug to assigned.
> 
> We analysed this bug and we can see "CNS" container set ENV of GB_LOGDIR
> properly in the container, from there on gluster block would have picked up
> this following below code:

I debugged this along with Raghavendra Talur and Saravana yesterday and we found that the environment didn't have GB_LOGDIR set for gluster-blockd process.

here is the output:
http://pastebin.test.redhat.com/516156

Snippet for gluster-blockd

root       7638  0.9  0.2 768272 129756 ?       Ssl  12:31   0:41 /usr/sbin/gluster-blockd --glfs-lru-count 15 --log-level INFO PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin GB_GLFS_LRU_COUNT=15 GB_LOG_LEVEL=INFO

It doesn't have environment variable set for gluster-blockd process hence it is not writing to the desired directory.

Based on this, gluster-blockd is working as expected and the comment below is invalid.

> 
>   logDir = getenv("GB_LOGDIR");
>   if (!logDir) {
>     logDir = GB_LOGDIR;
>   }
> 
>   if (strlen(logDir) > PATH_MAX - GB_MAX_LOGFILENAME) {
>     fprintf(stderr, "strlen of logDir Path > PATH_MAX: %s\n", logDir);
>     return EXIT_FAILURE;
>   }
> 
> 
> 
> Which has not happened, any way we can fix in some other way in CNS,
> considering the timeline of CNS release we are taking another path to fix
> this issue.

(In reply to Pranith Kumar K from comment #21)
> (In reply to Humble Chirammal from comment #20)
> > (In reply to krishnaram Karthick from comment #19)
> > > gluster-block logs are still logged at /var/log/gluster-block.
> > > 
> > > sh-4.2# ls -l /var/log/gluster-block/
> > > total 1692
> > > -rw-r--r--. 1 root root 1636352 Sep 14 14:01 gluster-block-configshell.log
> > > -rw-------. 1 root root   10440 Sep 14 14:01 gluster-block-gfapi.log
> > > -rw-r--r--. 1 root root   81810 Sep 14 14:01 gluster-blockd.log
> > > sh-4.2# ls -l /var/log/glusterfs/gluster-block/
> > > total 228
> > > -rw-r--r--. 1 root root  19170 Sep 14 14:01 gluster-block-cli.log
> > > -rw-------. 1 root root 208729 Sep 14 14:01 tcmu-runner-glfs.log
> > > -rw-------. 1 root root    357 Sep 14 12:31 tcmu-runner.log
> > > 
> > > Moving the bug to assigned.
> > 
> > We analysed this bug and we can see "CNS" container set ENV of GB_LOGDIR
> > properly in the container, from there on gluster block would have picked up
> > this following below code:
> 
> I debugged this along with Raghavendra Talur and Saravana yesterday and we
> found that the environment didn't have GB_LOGDIR set for gluster-blockd
> process.

The ENV in the container has that var set.



> It doesn't have environment variable set for gluster-blockd process hence it
> is not writing to the desired directory.
> 
> Based on this, gluster-blockd is working as expected and the comment below
> is invalid.

The comment is **not** invalid, the container has it set. If it has not been picked up we need to analyse further where things ( systemd, gluster-block or container runtime) are going wrong. Regardless of the analysis we have the fix in progress.

(In reply to Humble Chirammal from comment #22)
> (In reply to Pranith Kumar K from comment #21)
> > (In reply to Humble Chirammal from comment #20)
> > > (In reply to krishnaram Karthick from comment #19)
> > > > gluster-block logs are still logged at /var/log/gluster-block.
> > > > 
> > > > sh-4.2# ls -l /var/log/gluster-block/
> > > > total 1692
> > > > -rw-r--r--. 1 root root 1636352 Sep 14 14:01 gluster-block-configshell.log
> > > > -rw-------. 1 root root   10440 Sep 14 14:01 gluster-block-gfapi.log
> > > > -rw-r--r--. 1 root root   81810 Sep 14 14:01 gluster-blockd.log
> > > > sh-4.2# ls -l /var/log/glusterfs/gluster-block/
> > > > total 228
> > > > -rw-r--r--. 1 root root  19170 Sep 14 14:01 gluster-block-cli.log
> > > > -rw-------. 1 root root 208729 Sep 14 14:01 tcmu-runner-glfs.log
> > > > -rw-------. 1 root root    357 Sep 14 12:31 tcmu-runner.log
> > > > 
> > > > Moving the bug to assigned.
> > > 
> > > We analysed this bug and we can see "CNS" container set ENV of GB_LOGDIR
> > > properly in the container, from there on gluster block would have picked up
> > > this following below code:
> > 
> > I debugged this along with Raghavendra Talur and Saravana yesterday and we
> > found that the environment didn't have GB_LOGDIR set for gluster-blockd
> > process.
> 
> The ENV in the container has that var set.

Process can't read container's env. Process needs to have the env set as well.
The output I attached in previous comment is from krk's setup. The environment of the process doesn't have the env-variable set.

> 
> 
> 
> > It doesn't have environment variable set for gluster-blockd process hence it
> > is not writing to the desired directory.
> > 
> > Based on this, gluster-blockd is working as expected and the comment below
> > is invalid.
> 
> The comment is **not** invalid, the container has it set. If it has not been
> picked up we need to analyse further where things ( systemd, gluster-block
> or container runtime) are going wrong. Regardless of the analysis we have
> the fix in progress.

Could you add RC for the bug and the fix as a comment in this bz please.

(In reply to Pranith Kumar K from comment #23)
> (In reply to Humble Chirammal from comment #22)
> > (In reply to Pranith Kumar K from comment #21)
> > > (In reply to Humble Chirammal from comment #20)
> > > > (In reply to krishnaram Karthick from comment #19)
> > > > > gluster-block logs are still logged at /var/log/gluster-block.
> > > > > 
> > > > > sh-4.2# ls -l /var/log/gluster-block/
> > > > > total 1692
> > > > > -rw-r--r--. 1 root root 1636352 Sep 14 14:01 gluster-block-configshell.log
> > > > > -rw-------. 1 root root   10440 Sep 14 14:01 gluster-block-gfapi.log
> > > > > -rw-r--r--. 1 root root   81810 Sep 14 14:01 gluster-blockd.log
> > > > > sh-4.2# ls -l /var/log/glusterfs/gluster-block/
> > > > > total 228
> > > > > -rw-r--r--. 1 root root  19170 Sep 14 14:01 gluster-block-cli.log
> > > > > -rw-------. 1 root root 208729 Sep 14 14:01 tcmu-runner-glfs.log
> > > > > -rw-------. 1 root root    357 Sep 14 12:31 tcmu-runner.log
> > > > > 
> > > > > Moving the bug to assigned.
> > > > 
> > > > We analysed this bug and we can see "CNS" container set ENV of GB_LOGDIR
> > > > properly in the container, from there on gluster block would have picked up
> > > > this following below code:
> > > 
> > > I debugged this along with Raghavendra Talur and Saravana yesterday and we
> > > found that the environment didn't have GB_LOGDIR set for gluster-blockd
> > > process.
> > 
> > The ENV in the container has that var set.
> 
> Process can't read container's env. Process needs to have the env set as
> well.
> The output I attached in previous comment is from krk's setup. The
> environment of the process doesn't have the env-variable set.


This is what I was referring in my comments. 

Inside the container we can see:

sh-4.2# env|grep GB
GB_GLFS_LRU_COUNT=15
GB_LOGDIR=/var/log/glusterfs/gluster-block
sh-4.2# python
Python 2.7.5 (default, May  3 2017, 07:55:04) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import os
>>> os.getenv("GB_LOGDIR")
'/var/log/glusterfs/gluster-block'

^^^^^^ Notice above, this claim that "ENV" of that container got this variable set properly. From hereonwards, it has vanished or not picked up, thats what we want to analyse or I was pointing out by above comments.

(In reply to Humble Chirammal from comment #24)
> (In reply to Pranith Kumar K from comment #23)
> > (In reply to Humble Chirammal from comment #22)
> > > (In reply to Pranith Kumar K from comment #21)
> > > > (In reply to Humble Chirammal from comment #20)
> > > > > (In reply to krishnaram Karthick from comment #19)
> > > > > > gluster-block logs are still logged at /var/log/gluster-block.
> > > > > > 
> > > > > > sh-4.2# ls -l /var/log/gluster-block/
> > > > > > total 1692
> > > > > > -rw-r--r--. 1 root root 1636352 Sep 14 14:01 gluster-block-configshell.log
> > > > > > -rw-------. 1 root root   10440 Sep 14 14:01 gluster-block-gfapi.log
> > > > > > -rw-r--r--. 1 root root   81810 Sep 14 14:01 gluster-blockd.log
> > > > > > sh-4.2# ls -l /var/log/glusterfs/gluster-block/
> > > > > > total 228
> > > > > > -rw-r--r--. 1 root root  19170 Sep 14 14:01 gluster-block-cli.log
> > > > > > -rw-------. 1 root root 208729 Sep 14 14:01 tcmu-runner-glfs.log
> > > > > > -rw-------. 1 root root    357 Sep 14 12:31 tcmu-runner.log
> > > > > > 
> > > > > > Moving the bug to assigned.
> > > > > 
> > > > > We analysed this bug and we can see "CNS" container set ENV of GB_LOGDIR
> > > > > properly in the container, from there on gluster block would have picked up
> > > > > this following below code:
> > > > 
> > > > I debugged this along with Raghavendra Talur and Saravana yesterday and we
> > > > found that the environment didn't have GB_LOGDIR set for gluster-blockd
> > > > process.
> > > 
> > > The ENV in the container has that var set.
> > 
> > Process can't read container's env. Process needs to have the env set as
> > well.
> > The output I attached in previous comment is from krk's setup. The
> > environment of the process doesn't have the env-variable set.
> 
> 
> This is what I was referring in my comments. 
> 
> Inside the container we can see:
> 
> sh-4.2# env|grep GB
> GB_GLFS_LRU_COUNT=15
> GB_LOGDIR=/var/log/glusterfs/gluster-block
> sh-4.2# python
> Python 2.7.5 (default, May  3 2017, 07:55:04) 
> [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> 
> >>> import os
> >>> os.getenv("GB_LOGDIR")
> '/var/log/glusterfs/gluster-block'
> 
> ^^^^^^ Notice above, this claim that "ENV" of that container got this
> variable set properly. From hereonwards, it has vanished or not picked up,
> thats what we want to analyse or I was pointing out by above comments.

Cool. As long as the bug is not in gluster-block, we are good. That is what I am intending to communicate as well. Do let us know the RC, it would be interesting to know the reason.
Thanks!

This is fixed with cns-deploy v44, I am moving to ON_QA.

Fixed in cns-deploy >= v43.

sh-4.2# ls -l /var/log/glusterfs/gluster-block
total 12432
-rw-r--r--. 1 root root   150344 Sep 18 07:47 gluster-block-cli.log
-rw-r--r--. 1 root root 12260006 Sep 18 11:10 gluster-block-configshell.log
-rw-------. 1 root root    15584 Sep 18 08:41 gluster-block-gfapi.log
-rw-r--r--. 1 root root   169483 Sep 18 11:10 gluster-blockd.log
-rw-------. 1 root root    30195 Sep 18 11:09 tcmu-runner-glfs.log
-rw-------. 1 root root    93184 Sep 18 11:32 tcmu-runner.log

verified in build - cns-deploy-5.0.0-43.el7rhgs.x86_64. gluster-block logs are now located in /var/log/glusterfs/gluster-block directory.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:2881