1967916 – cephfs-mirror asok files are regenerated due to the restart of the daemon which happens everyday at particular time by itself.

Bug 1967916 - cephfs-mirror asok files are regenerated due to the restart of the daemon which happens everyday at particular time by itself.

Summary: cephfs-mirror asok files are regenerated due to the restart of the daemon whi...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	CephFS
Sub Component:
Version:	5.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	5.0
Assignee:	Venky Shankar
QA Contact:	Hemanth Kumar
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-06-04 12:12 UTC by Hemanth Kumar
Modified:	2021-08-30 08:31 UTC (History)
CC List:	6 users (show)
Fixed In Version:	ceph-16.2.0-87.el8cp
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-08-30 08:31:02 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Ceph Project Bug Tracker	51318	None	None	None	2021-06-24 05:11:48 UTC
Red Hat Issue Tracker	RHCEPH-1305	None	None	None	2021-08-30 00:23:37 UTC
Red Hat Product Errata	RHBA-2021:3294	None	None	None	2021-08-30 08:31:24 UTC

Description Hemanth Kumar 2021-06-04 12:12:34 UTC

Description of problem:
------------------------

I am seeing that asok files for cephfs-mirror is recreating everyday even when 
 the service is idle and without enabling/disabling the daemon.

Is there any cron job running to recreate/restart mirror-daemon ?. It's recreating everyday at around 3. Let me know what logs are required here.

[ceph: root@magna026 ceph]# ls -ltr /var/run/ceph/
total 0
srwxr-xr-x. 1 ceph ceph 0 May 26 12:24 ceph-client.cephfs-mirror.magna026.wsauwr.7.93865262945600.asok
srwxr-xr-x. 1 ceph ceph 0 May 26 12:27 ceph-client.cephfs-mirror.magna026.wsauwr.7.93865277003072.asok
srwxr-xr-x. 1 ceph ceph 0 May 26 12:29 remote-site-client.mirror_remote.7.93865291191616.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:21 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94895486285120.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:21 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94895500277056.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:21 remote-site-client.mirror_remote.7.94895513482560.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:28 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94895558948160.asok
srwxr-xr-x. 1 ceph ceph 0 May 27 17:38 secondary-ec-client.mirror_remote_ec.7.94895622354240.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 ceph-client.cephfs-mirror.magna026.cnrpzs.8.93847650121024.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 ceph-client.cephfs-mirror.magna026.cnrpzs.8.93847664170304.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 ceph-client.cephfs-mirror.magna026.cnrpzs.8.93847677629760.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 secondary-ec-client.mirror_remote_ec.8.93847691007296.asok
srwxr-xr-x. 1 ceph ceph 0 May 28 03:19 remote-site-client.mirror_remote.8.93847704704320.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94164422980928.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94164436981056.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94164450489664.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 secondary-ec-client.mirror_remote_ec.7.94164463949120.asok
srwxr-xr-x. 1 ceph ceph 0 May 29 03:37 remote-site-client.mirror_remote.7.94164477580608.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94832657110336.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94832671110464.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94832684619072.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 secondary-ec-client.mirror_remote_ec.7.94832698037568.asok
srwxr-xr-x. 1 ceph ceph 0 May 30 03:27 remote-site-client.mirror_remote.7.94832711759168.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94739633935680.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94739647862080.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94739661346112.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 secondary-ec-client.mirror_remote_ec.7.94739674789184.asok
srwxr-xr-x. 1 ceph ceph 0 May 31 03:27 remote-site-client.mirror_remote.7.94739688510784.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94375494833472.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94375508825408.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94375522284864.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 secondary-ec-client.mirror_remote_ec.7.94375535744320.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  1 03:15 remote-site-client.mirror_remote.7.94375549482304.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94416095401280.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94416109286720.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94416122918208.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 secondary-ec-client.mirror_remote_ec.7.94416136451392.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  2 03:51 remote-site-client.mirror_remote.7.94416150197568.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94170139399488.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94170153465152.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 ceph-client.cephfs-mirror.magna026.cnrpzs.7.94170166990144.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 secondary-ec-client.mirror_remote_ec.7.94170180465984.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  3 03:48 remote-site-client.mirror_remote.7.94170193827136.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 ceph-client.cephfs-mirror.magna026.cnrpzs.7.93871593043264.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 ceph-client.cephfs-mirror.magna026.cnrpzs.7.93871607133504.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 ceph-client.cephfs-mirror.magna026.cnrpzs.7.93871620552000.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 secondary-ec-client.mirror_remote_ec.7.93871634101568.asok
srwxr-xr-x. 1 ceph ceph 0 Jun  4 03:34 remote-site-client.mirror_remote.7.93871647454528.asok
[ceph: root@magna026 ceph]#


ceph version : ceph version 16.2.0-45.el8cp

Comment 1 Venky Shankar 2021-06-07 09:18:07 UTC

(In reply to Hemanth Kumar from comment #0)
> Description of problem:
> ------------------------
> 
> I am seeing that asok files for cephfs-mirror is recreating everyday even
> when 
>  the service is idle and without enabling/disabling the daemon.
> 
> Is there any cron job running to recreate/restart mirror-daemon ?. It's
> recreating everyday at around 3. Let me know what logs are required here.
> 

I don't think any cron job is responsible for this. And I don't see why would the admin socket get recreated each day. The mirror daemon does not do anything special with the admin sockets apart from registering some asok hooks (commands).

Can you enable asok debugging with "debug asok = 20" and provide mirror daemon logs? Let the mirror daemon run for a day or two until you see the asok path change. Also run the mirror daemon with `--debug-cephfs_mirror = 20`.

Comment 6 Venky Shankar 2021-06-22 04:31:57 UTC

Update after discussing with Hemanth: logrotate seems to be backing up the log file and around the same time a new asok file is generated which is very weird. logrotate sends a SIGHUP to ceph daemons, however, the mirror daemon is not included in the daemon list (we probably need to add that).

Hmeanth will check with the respective devs and update (Thanks, Hemanth!).

Cheers,
Venky

Comment 11 Venky Shankar 2021-06-24 05:11:50 UTC

https://github.com/ceph/ceph/pull/41988

Comment 21 errata-xmlrpc 2021-08-30 08:31:02 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294

Note You need to log in before you can comment on or make changes to this bug.