Bug 1546932

Summary: systemd units does not stop all gluster daemons
Product: [Community] GlusterFS Reporter: Dmitry Melekhov <dm>
Component: glusterdAssignee: Vijay Bellur <vbellur>
Status: CLOSED WONTFIX QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: mainlineCC: amukherj, bugs, srakonde, vbellur
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-31 10:49:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dmitry Melekhov 2018-02-20 04:54:52 UTC
Description of problem:

Systemd units, provided by centos storage sig are not able to stop all gluster daemons,
I guess these units are developed by gluster developers


Version-Release number of selected component (if applicable):

3.12.6


How reproducible:

Always

[root@vmhost1 ~]# systemctl stop glusterfsd
[root@vmhost1 ~]# systemctl stop glusterd
[root@vmhost1 ~]# ps ax | grep glust
 2478 ?        Ssl    0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d63ed11ad6712579ba0116e69f0b5a90.socket --xlator-option *replicate*.node-uuid=0172d180-e203-45e6-b447-79117a4765f2
 2555 ?        Ssl    0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/bitd -p /var/run/gluster/bitd/bitd.pid -l /var/log/glusterfs/bitd.log -S /var/run/gluster/de223d0e1e02670bb0a176ebd143beff.socket --global-timer-wheel
 2595 ?        Ssl    0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/scrub -p /var/run/gluster/scrub/scrub.pid -l /var/log/glusterfs/scrub.log -S /var/run/gluster/1ceaa9fdc4ca79dcc6da7779af8407d4.socket --global-timer-wheel
 2952 pts/39   S+     0:00 grep --color=auto glust

Steps to Reproduce:

[root@vmhost1 ~]# systemctl stop glusterfsd
[root@vmhost1 ~]# systemctl stop glusterd
[root@vmhost1 ~]# ps ax | grep glust
 2478 ?        Ssl    0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d63ed11ad6712579ba0116e69f0b5a90.socket --xlator-option *replicate*.node-uuid=0172d180-e203-45e6-b447-79117a4765f2
 2555 ?        Ssl    0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/bitd -p /var/run/gluster/bitd/bitd.pid -l /var/log/glusterfs/bitd.log -S /var/run/gluster/de223d0e1e02670bb0a176ebd143beff.socket --global-timer-wheel
 2595 ?        Ssl    0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/scrub -p /var/run/gluster/scrub/scrub.pid -l /var/log/glusterfs/scrub.log -S /var/run/gluster/1ceaa9fdc4ca79dcc6da7779af8407d4.socket --global-timer-wheel
 2952 pts/39   S+     0:00 grep --color=auto glust

Actual results:

some daemons are running


Expected results:

all daemons are stopped

Additional info:

Comment 1 Gaurav Yadav 2018-02-27 16:25:51 UTC
"systemctl stop glusterd" command will stop only glusterd daemon

If you want to stop all gluster daemon, same can be achieve by using a script stop-all-gluster-processes.sh. This script is available in 
/usr/local/share/glusterfs/scripts this directory.


This is not a valid bug, hence I am closing this bug.

Comment 2 Dmitry Melekhov 2018-02-28 05:15:17 UTC
Well, I don't think that bug is invalid.

If processes are started by 
systemctl start glusterd
systemctl start glusterfsd

they have to be stopped by
systemctl stop glusterd
systemctl stop glusterfsd

just because administrator uses systemd for system management.
external script is good, but it breaks unification.

Also, looks like these processes are not restarted during upgrade.

Thank you!

Comment 3 Gaurav Yadav 2018-03-06 03:55:06 UTC

(In reply to Need Real Name from comment #2)
> Well, I don't think that bug is invalid.
> 
> If processes are started by 
> systemctl start glusterd
  This will start glusterd. 
> systemctl start glusterfsd
  we dont have any service called glusterfsd. glusterfsd will get started after volume creation followed by volume start.
> 
> they have to be stopped by
> systemctl stop glusterd
  It will stop glusterd alone. glusterd is a management daemon which manages the cluster configuration. To answer your query let use consider a case where IO's are going on and in between systemctl stop glusterd is executed, Now if glusterd kills glusterfsd, IO's will be hampered. This is as per design, I don't see an issue here.

Its not a valid bug, hence I am closing this bug

Comment 4 Dmitry Melekhov 2018-03-06 04:26:19 UTC
No, you have

 systemctl status glusterfsd
● glusterfsd.service - GlusterFS brick processes (stopping only)
   Loaded: loaded (/usr/lib/systemd/system/glusterfsd.service


It is valid bug, because , as you just admitted , there is no way to stop gluster services using systemd.

Thank you!

Comment 5 Dmitry Melekhov 2018-03-06 04:37:08 UTC
> Now if glusterd kills glusterfsd, IO's will be hampered. 

yes, and if glusterfsd will be killed while system reboot what will happen?
you just said that we need correct way to stop glusterfsd and it doesn't work.

Comment 6 Samikshan Bairagya 2018-03-08 04:15:01 UTC
(In reply to Need Real Name from comment #5)
> > Now if glusterd kills glusterfsd, IO's will be hampered. 
> 
> yes, and if glusterfsd will be killed while system reboot what will happen?
> you just said that we need correct way to stop glusterfsd and it doesn't
> work.

Brick processes killed during system reboot are restarted by glusterd once the system is up. You don't need to explicitly start the bricks. To kill all gluster services you could do a `pkill gluster`; followed by `systemctl start glusterd` to restart the services.

Comment 7 Dmitry Melekhov 2018-03-08 10:41:24 UTC
Yes, but, as I wrote before, rhel and derivatives ( centos ) are systemd based and there is no unit which can do this 'pkill gluster'.
Although most of other daemons are systemd-aware.
This is why I created this bug report.
Thank you!

Comment 8 Shyamsundar 2018-10-23 14:54:47 UTC
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.

Comment 9 Amar Tumballi 2019-06-14 09:38:56 UTC
> systemctl start glusterfsd

[root@localhost ~]# systemctl start glusterfsd
Failed to start glusterfsd.service: Unit glusterfsd.service not found.

with the later releases, glusterfsd is not allowed to be started directly, but always will start from glusterd. (and is recommended that way). With that, we are closing the issue.

Comment 10 Dmitry Melekhov 2019-06-14 09:46:03 UTC
I'm talking about stopping, not starting, this is not fixed!

Comment 11 Amar Tumballi 2019-06-14 09:49:45 UTC
Dmitry, my point is, from glusterfs project, we are not supplying glusterfsd.service file itself, so, fixing this issue is not valid in the project, because systemd only bothers about glusterd, and as we would like to cleanup all the processes started by glusterd too, glusterfsd too is stopped.


https://github.com/gluster/glusterfs/tree/master/extras/systemd

Comment 12 Dmitry Melekhov 2019-06-14 09:52:52 UTC
Please fix glusterd.service, so while stopping glusterd glusterfsd will be stopped too.
Thank you!

Comment 13 Atin Mukherjee 2019-07-08 04:29:39 UTC
Dmitry - as per the design, stopping glusterd process doesn't mean we need to necessary stop all glusterfsd services. The design consideration we have in glusterfs is that management and I/O plane to remain as seggregated, so at any given point of time if the glusterd daemon i.e. management service needs to be brought down the brick processes (glusterfsd) should remain up and running. I hope this justifies the reason for the bug closure.

Comment 14 Dmitry Melekhov 2019-07-10 02:49:48 UTC
"stopping glusterd process doesn't mean we need to necessary stop all glusterfsd services."

Not a problem, but there is no way to correctly stop glusterfsd in case of server shutdown.
Please provide it!

Comment 15 Atin Mukherjee 2019-07-15 03:07:48 UTC
Have you explored stop-all-gluster-processes.sh script which is packaged as part of glusterfs-server?

Comment 16 Dmitry Melekhov 2019-07-15 03:12:04 UTC
This script is not called by systemd during system shutdown...

Comment 17 Atin Mukherjee 2019-07-15 13:08:37 UTC
Could you clarify how does an ungraceful shutdown of glusterfsd process (if not a graceful through systemd) affect your application? We'd be interested to know the same.

Comment 18 Dmitry Melekhov 2019-07-15 13:12:01 UTC
Well, frankly, main problem here is not server reboot, but updating gluster, in this case you do not correctly restart it...

Anyway, any service/daemon have to be correctly stopped during system shutdown, or you disagree? Just killing process is OK?

Comment 19 Sanju 2019-09-19 11:29:47 UTC
We are not planning to work on this in the near future, So there is no point in keeping the bug open. Closing this as won't fix.

Thanks,
Sanju

Comment 20 Dmitry Melekhov 2019-09-19 11:40:16 UTC
But you have to fix this!

Comment 21 Sanju 2020-01-31 10:49:30 UTC
Based on above comments, closing this as won't fix.