Bug 1967916

Summary: cephfs-mirror asok files are regenerated due to the restart of the daemon which happens everyday at particular time by itself.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Hemanth Kumar <hyelloji>
Component: CephFSAssignee: Venky Shankar <vshankar>
Status: CLOSED ERRATA QA Contact: Hemanth Kumar <hyelloji>
Severity: high Docs Contact:
Priority: high    
Version: 5.0CC: ceph-eng-bugs, kdreyer, pdonnell, sweil, vereddy, vshankar
Target Milestone: ---   
Target Release: 5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-16.2.0-87.el8cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-30 08:31:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hemanth Kumar 2021-06-04 12:12:34 UTC
Description of problem:
------------------------

I am seeing that asok files for cephfs-mirror is recreating everyday even when 
 the service is idle and without enabling/disabling the daemon.

Is there any cron job running to recreate/restart mirror-daemon ?. It's recreating everyday at around 3. Let me know what logs are required here.

[ceph: root@magna026 ceph]# ls -ltr /var/run/ceph/
total 0
srwxr-xr-x. 1 ceph ceph 0 May 26 12:24 ceph-client.cephfs-mirror.magna026.wsauwr.7.93865262945600.asok
srwxr-xr-x. 1 ceph ceph 0 May 26 12:27 ceph-client.cephfs-mirror.magna026.wsauwr.7.93865277003072.asok
srwxr-xr-x. 1 ceph ceph 0 May 26 12:29 remote-site-client.mirror_remote.7.93865291191616.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:21 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94895486285120.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:21 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94895500277056.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:21 remote-site-client.mirror_remote.7.94895513482560.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:28 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94895558948160.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:38 secondary-ec-client.mirror_remote_ec.7.94895622354240.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 ceph-client.cephfs-mirror.magna026.cnrpzs.8.93847650121024.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 ceph-client.cephfs-mirror.magna026.cnrpzs.8.93847664170304.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 ceph-client.cephfs-mirror.magna026.cnrpzs.8.93847677629760.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 secondary-ec-client.mirror_remote_ec.8.93847691007296.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 remote-site-client.mirror_remote.8.93847704704320.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94164422980928.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94164436981056.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94164450489664.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 secondary-ec-client.mirror_remote_ec.7.94164463949120.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 remote-site-client.mirror_remote.7.94164477580608.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94832657110336.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94832671110464.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94832684619072.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 secondary-ec-client.mirror_remote_ec.7.94832698037568.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 remote-site-client.mirror_remote.7.94832711759168.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94739633935680.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94739647862080.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94739661346112.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 secondary-ec-client.mirror_remote_ec.7.94739674789184.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 remote-site-client.mirror_remote.7.94739688510784.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94375494833472.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94375508825408.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94375522284864.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 secondary-ec-client.mirror_remote_ec.7.94375535744320.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 remote-site-client.mirror_remote.7.94375549482304.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94416095401280.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94416109286720.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94416122918208.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 secondary-ec-client.mirror_remote_ec.7.94416136451392.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 remote-site-client.mirror_remote.7.94416150197568.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94170139399488.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94170153465152.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94170166990144.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 secondary-ec-client.mirror_remote_ec.7.94170180465984.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 remote-site-client.mirror_remote.7.94170193827136.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 ceph-client.cephfs-mirror.magna026.cnrpzs.7.93871593043264.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 ceph-client.cephfs-mirror.magna026.cnrpzs.7.93871607133504.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 ceph-client.cephfs-mirror.magna026.cnrpzs.7.93871620552000.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 secondary-ec-client.mirror_remote_ec.7.93871634101568.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 remote-site-client.mirror_remote.7.93871647454528.asok
[ceph: root@magna026 ceph]#


ceph version : ceph version 16.2.0-45.el8cp

Comment 1 Venky Shankar 2021-06-07 09:18:07 UTC
(In reply to Hemanth Kumar from comment #0)
> Description of problem:
> ------------------------
> 
> I am seeing that asok files for cephfs-mirror is recreating everyday even
> when 
>  the service is idle and without enabling/disabling the daemon.
> 
> Is there any cron job running to recreate/restart mirror-daemon ?. It's
> recreating everyday at around 3. Let me know what logs are required here.
> 

I don't think any cron job is responsible for this. And I don't see why would the admin socket get recreated each day. The mirror daemon does not do anything special with the admin sockets apart from registering some asok hooks (commands).

Can you enable asok debugging with "debug asok = 20" and provide mirror daemon logs? Let the mirror daemon run for a day or two until you see the asok path change. Also run the mirror daemon with `--debug-cephfs_mirror = 20`.

Comment 6 Venky Shankar 2021-06-22 04:31:57 UTC
Update after discussing with Hemanth: logrotate seems to be backing up the log file and around the same time a new asok file is generated which is very weird. logrotate sends a SIGHUP to ceph daemons, however, the mirror daemon is not included in the daemon list (we probably need to add that).

Hmeanth will check with the respective devs and update (Thanks, Hemanth!).

Cheers,
Venky

Comment 11 Venky Shankar 2021-06-24 05:11:50 UTC
https://github.com/ceph/ceph/pull/41988

Comment 21 errata-xmlrpc 2021-08-30 08:31:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294