+++ This bug was initially created as a clone of Bug #1365265 +++
+++ This bug was initially created as a clone of Bug #1318591 +++
+++ This bug was initially created as a clone of Bug #1318000 +++
Description of problem:
Glusterd is not operational on one node in the cluster (lvp87) due to snapshot conflicting with nfs-ganesha "export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf" in "/var/lib/glusterd/snaps"
Glusterd logs:
[2016-03-15 07:21:49.063861] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/snaps/Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01/export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf/info. [Not a directory]
[2016-03-15 07:21:49.063869] D [MSGID: 0] [store.c:440:gf_store_handle_retrieve] 0-: Returning -1
[2016-03-15 07:21:49.063873] E [MSGID: 106200] [glusterd-store.c:2550:glusterd_store_update_volinfo] 0-management: volinfo handle is NULL
[2016-03-15 07:21:49.063878] E [MSGID: 106207] [glusterd-store.c:2848:glusterd_store_retrieve_volume] 0-management: Failed to update volinfo for export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf volume
[2016-03-15 07:21:49.063883] D [MSGID: 0] [glusterd-utils.c:893:glusterd_volume_brickinfos_delete] 0-management: Returning 0
[2016-03-15 07:21:49.063888] D [MSGID: 0] [store.c:461:gf_store_handle_destroy] 0-: Returning 0
[2016-03-15 07:21:49.063897] D [MSGID: 0] [glusterd-utils.c:937:glusterd_volinfo_delete] 0-management: Returning 0
[2016-03-15 07:21:49.063902] E [MSGID: 106201] [glusterd-store.c:3046:glusterd_store_retrieve_volumes] 0-management: Unable to restore volume: export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf
[2016-03-15 07:21:49.063927] D [MSGID: 0] [glusterd-store.c:3071:glusterd_store_retrieve_volumes] 0-management: Returning with -1
[2016-03-15 07:21:49.063937] E [MSGID: 106195] [glusterd-store.c:3439:glusterd_store_retrieve_snap] 0-management: Failed to retrieve snap volumes for snap Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01
[2016-03-15 07:21:49.063942] E [MSGID: 106043] [glusterd-store.c:3593:glusterd_store_retrieve_snaps] 0-management: Unable to restore snapshot: Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01
[2016-03-15 07:21:49.063948] D [MSGID: 0] [glusterd-store.c:3611:glusterd_store_retrieve_snaps] 0-management: Returning with -1
[2016-03-15 07:21:49.063953] D [MSGID: 0] [glusterd-store.c:4343:glusterd_restore] 0-management: Returning -1
[2016-03-15 07:21:49.063967] E [MSGID: 101019] [xlator.c:428:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2016-03-15 07:21:49.063973] E [graph.c:322:glusterfs_graph_init] 0-management: initializing translator failed
[2016-03-15 07:21:49.063977] E [graph.c:661:glusterfs_graph_activate] 0-graph: init failed
[2016-03-15 07:21:49.064238] D [logging.c:1764:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages
[2016-03-15 07:21:49.064249] D [logging.c:1767:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages
[2016-03-15 07:21:49.064417] W [glusterfsd.c:1236:cleanup_and_exit] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xfd) [0x7f0a7d57e2fd] -->/usr/sbin/glusterd(glusterfs_process_volfp+0x126) [0x7f0a7d57e1a6] -->/usr/sbin/glusterd(cleanup_and_exit+0x69) [0x7f0a7d57d789] ) 0-: received signum (0), shutting down
[2016-03-15 07:21:49.064435] D [glusterfsd-mgmt.c:2355:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
Volume status:
Status of volume: certsd
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_certsd/brick_certsd_lvp88 49160 0 Y 2005
NFS Server on localhost 2049 0 Y 3584
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume certsd
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: gluster_shared_storage
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/var/lib/glus
terd/ss_brick 49161 0 Y 3564
NFS Server on localhost 2049 0 Y 3584
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume gluster_shared_storage
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv01-sknd3
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv01-sknd3/brick_pv01-sknd3_lvp88 49154 0 Y 2006
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv01-sknd3
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv02-ddfe5
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv02-ddfe5/brick_pv02-ddfe5_lvp88 49155 0 Y 2016
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv02-ddfe5
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv03-ed6fc
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv03-ed6fc/brick_pv03-ed6fc_lvp88 49156 0 Y 2022
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv03-ed6fc
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv04-1fr6e
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv04-1fr6e/brick_pv04-1fr6e_lvp88 49157 0 Y 2028
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv04-1fr6e
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv05-ku56u
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv05-ku56u/brick_pv05-ku56u_lvp88 49158 0 Y 2030
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv05-ku56u
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: pv06-m6o8i
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_pv06-m6o8i/brick_pv06-m6o8i_lvp88 49159 0 Y 2045
Self-heal Daemon on localhost N/A N/A Y 3592
Task Status of Volume pv06-m6o8i
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: registry
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick sv-2000lvp88.paas.local:/bricks/thin_
brick_registry/brick_registry_lvp88 49153 0 Y 2052
Self-heal Daemon on localhost N/A N/A Y 3592
Volume info:
Volume Name: pv06-m6o8i
Type: Replicate
Volume ID: b9ecd956-a10d-427a-ad95-81f735c58050
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sv-2000lvp87.paas.local:/bricks/thin_brick_pv06-m6o8i/brick_pv06-m6o8i_lvp87
Brick2: sv-2000lvp88.paas.local:/bricks/thin_brick_pv06-m6o8i/brick_pv06-m6o8i_lvp88
Options Reconfigured:
performance.readdir-ahead: on
nfs.disable: true
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
Volume Name: certsd
Type: Replicate
Volume ID: 17839849-381f-4299-8088-a1e62765e09c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sv-2000lvp87.paas.local:/bricks/thin_brick_certsd/brick_certsd_lvp87
Brick2: sv-2000lvp88.paas.local:/bricks/thin_brick_certsd/brick_certsd_lvp88
Options Reconfigured:
performance.readdir-ahead: on
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
[root@sv-2000lvp88 ~]# gluster v info registry
Volume Name: registry
Type: Replicate
Volume ID: 1619f2f4-892d-48d2-9c7a-431a8b57a67e
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sv-2000lvp87.paas.local:/bricks/thin_brick_registry/brick_registry_lvp87
Brick2: sv-2000lvp88.paas.local:/bricks/thin_brick_registry/brick_registry_lvp88
Options Reconfigured:
features.barrier: disable
performance.readdir-ahead: on
nfs.disable: true
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
[root@sv-2000lvp88 ~]# gluster v info gluster_shared_storage
Volume Name: gluster_shared_storage
Type: Replicate
Volume ID: c160df0e-4472-47e0-80dd-e118b5dddc3f
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sv-2000lvp88.paas.local:/var/lib/glusterd/ss_brick
Brick2: sv-2000lvp87:/var/lib/glusterd/ss_brick
Options Reconfigured:
performance.readdir-ahead: on
snap-activate-on-create: enable
auto-delete: enable
cluster.enable-shared-storage: enable
Version-Release number of selected component (if applicable):
RHGS 3.1.2
RHEL 7.2
How reproducible:
Always
Steps to Reproduce:
* Stop glusterd on both the nodes in the cluster
* Removed nf-ganesha package on both nodes than removed the exported conf file "export.Scheduled-Job-registry-1-registry_GMT-2016.03.14-11.00.01.conf" from "/var/lib/glusterd/snaps"
* Start Glusterd on both the nodes
Actual results:
Due to conflict, glusterd is not operational.
Expected results:
There should be no conflict & glusterd should be operational.
Additional info:
--- Additional comment from Mukul Malhotra on 2016-03-15 14:18 EDT ---
--- Additional comment from Mukul Malhotra on 2016-03-15 14:19 EDT ---
--- Additional comment from Mukul Malhotra on 2016-03-16 10:35:49 EDT ---
Hello,
Also, please improve the logging related to nfs-ganesha in the logs so that the same should be visible & would help in troubleshooting the issue.
Thanks
Mukul
--- Additional comment from Vijay Bellur on 2016-03-17 10:02:15 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#1) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-17 10:02:19 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : Skip invalid entries in "vols" directory) posted (#1) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-18 05:36:22 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#2) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-18 06:37:47 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : Skip invalid entries in "vols" directory) posted (#2) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-24 06:45:01 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories inside vols) posted (#3) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-24 06:45:05 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#3) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Mike McCune on 2016-03-28 18:50:27 EDT ---
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
--- Additional comment from Vijay Bellur on 2016-03-29 07:11:14 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories inside vols) posted (#4) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-29 07:11:18 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#4) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-30 02:59:09 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories inside vols) posted (#5) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-03-30 02:59:12 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#5) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-06-23 05:00:28 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#6) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-06-23 13:17:20 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories inside vols) posted (#6) for review on master by Atin Mukherjee (amukherj)
--- Additional comment from Vijay Bellur on 2016-07-08 07:54:24 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories inside vols) posted (#7) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-07-08 08:11:16 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#7) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-07-08 08:18:21 EDT ---
REVIEW: http://review.gluster.org/13763 (snapshot : Copy the export configuration properly) posted (#8) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-08-05 05:05:27 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : read only directories inside vols) posted (#8) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-08-05 08:07:14 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : skip non directories inside /var/lib/glusterd/vols) posted (#9) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-08-08 02:48:55 EDT ---
REVIEW: http://review.gluster.org/13764 (glusterd : skip non directories inside /var/lib/glusterd/vols) posted (#10) for review on master by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-08-08 10:23:54 EDT ---
COMMIT: http://review.gluster.org/13764 committed in master by Atin Mukherjee (amukherj)
------
commit 720b63c24b07ee64e1338db28de602b9abbef0a1
Author: Jiffin Tony Thottan <jthottan>
Date: Thu Mar 17 18:53:13 2016 +0530
glusterd : skip non directories inside /var/lib/glusterd/vols
Right now glusterd won't come up if vols directory contains an invalid entry.
Instead of doing that with this change a message will be logged and then skip
that entry
Change-Id: I665b5c35291b059cf054622da0eec4db44ec5f68
BUG: 1318591
Signed-off-by: Jiffin Tony Thottan <jthottan>
Reviewed-on: http://review.gluster.org/13764
Reviewed-by: Prashanth Pai <ppai>
Reviewed-by: Atin Mukherjee <amukherj>
Smoke: Gluster Build System <jenkins.org>
CentOS-regression: Gluster Build System <jenkins.org>
NetBSD-regression: NetBSD Build System <jenkins.org>
--- Additional comment from Vijay Bellur on 2016-08-08 14:08:02 EDT ---
REVIEW: http://review.gluster.org/15113 (glusterd : skip non directories inside /var/lib/glusterd/vols) posted (#1) for review on release-3.8 by jiffin tony Thottan (jthottan)
--- Additional comment from Vijay Bellur on 2016-08-09 11:03:18 EDT ---
COMMIT: http://review.gluster.org/15113 committed in release-3.8 by Niels de Vos (ndevos)
------
commit 32b5667f017e3206bd9d3b65074eecb7ddb84244
Author: Jiffin Tony Thottan <jthottan>
Date: Thu Mar 17 18:53:13 2016 +0530
glusterd : skip non directories inside /var/lib/glusterd/vols
Right now glusterd won't come up if vols directory contains an invalid entry.
Instead of doing that with this change a message will be logged and then skip
that entry
Backport details:
>Change-Id: I665b5c35291b059cf054622da0eec4db44ec5f68
>BUG: 1318591
>Signed-off-by: Jiffin Tony Thottan <jthottan>
>Reviewed-on: http://review.gluster.org/13764
>Reviewed-by: Prashanth Pai <ppai>
>Reviewed-by: Atin Mukherjee <amukherj>
>Smoke: Gluster Build System <jenkins.org>
>CentOS-regression: Gluster Build System <jenkins.org>
>NetBSD-regression: NetBSD Build System <jenkins.org>
(cherry picked from commit 720b63c24b07ee64e1338db28de602b9abbef0a1)
Change-Id: I665b5c35291b059cf054622da0eec4db44ec5f68
BUG: 1365265
Signed-off-by: Jiffin Tony Thottan <jthottan>
Reviewed-on: http://review.gluster.org/15113
Smoke: Gluster Build System <jenkins.org>
NetBSD-regression: NetBSD Build System <jenkins.org>
Reviewed-by: Prashanth Pai <ppai>
CentOS-regression: Gluster Build System <jenkins.org>
Reviewed-by: Niels de Vos <ndevos>
REVIEW: http://review.gluster.org/15133 (glusterd : skip non directories inside /var/lib/glusterd/vols) posted (#1) for review on release-3.7 by jiffin tony Thottan (jthottan)
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life.
Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS.
If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.