1329118 – volume create fails with "Failed to store the Volume information" due to /var/lib/glusterd/vols missing with latest build

Bug 1329118 - volume create fails with "Failed to store the Volume information" due to /var/lib/glusterd/vols missing with latest build

Summary: volume create fails with "Failed to store the Volume information" due to /var...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	glusterd
Sub Component:
Version:	rhgs-3.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	RHGS 3.1.3
Assignee:	Milind Changire
QA Contact:	SATHEESARAN
Docs Contact:
URL:
Whiteboard:
Depends On:	1324820
Blocks:	1311817
TreeView+	depends on / blocked

Reported:	2016-04-21 09:06 UTC by Nag Pavan Chilakam
Modified:	2016-09-17 16:44 UTC (History)
CC List:	8 users (show)
Fixed In Version:	glusterfs-3.7.9-4
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-06-23 05:18:57 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)
screenshot of the iso image selected (222.80 KB, image/png) 2016-04-21 09:06 UTC, Nag Pavan Chilakam	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2016:1240	0	normal	SHIPPED_LIVE	Red Hat Gluster Storage 3.1 Update 3	2016-06-23 08:51:28 UTC

Description Nag Pavan Chilakam 2016-04-21 09:06:48 UTC

Created attachment 1149386 [details]
screenshot of the iso image selected

Description of problem:
======================
When I updated to the latest build after installing RHG3 ISO image with latest 3.1.2 image and then updating to latest 3.1.3 rpms using repo (3.7.9-2), volume create fails due to missing vols directory in /var/lib/glusterd

Updating to this build is actually deleting the vols dir under /var/lib/glusterd.


NOte: I directly updated without creating any vols in the 3.1.2 build
Also, note that the vols dir was existing in 3.1.2 build and was lost after updating to 3.1.3 ie 3.7.9-2


I created a new machine with pxe boot.
I chose 7.2 RHEL RHGS 3 as the ISO Installation and selected the latest 3.1.2 build availble in the iso list.
After the machine was setup, following was the information of versions:
[root@dhcp35-67 ~]# rpm -qa|grep gluste
glusterfs-libs-3.7.5-19.el7rhgs.x86_64
python-gluster-3.7.5-19.el7rhgs.noarch
glusterfs-3.7.5-19.el7rhgs.x86_64
glusterfs-api-3.7.5-19.el7rhgs.x86_64
glusterfs-fuse-3.7.5-19.el7rhgs.x86_64
glusterfs-rdma-3.7.5-19.el7rhgs.x86_64
gluster-nagios-addons-0.2.5-1.el7rhgs.x86_64
glusterfs-geo-replication-3.7.5-19.el7rhgs.x86_64
vdsm-gluster-4.16.30-1.3.el7rhgs.noarch
gluster-nagios-common-0.2.3-1.el7rhgs.noarch
glusterfs-client-xlators-3.7.5-19.el7rhgs.x86_64
glusterfs-cli-3.7.5-19.el7rhgs.x86_64
glusterfs-server-3.7.5-19.el7rhgs.x86_64
[root@dhcp35-67 ~]# cat /etc/redhat-*
cat: /etc/redhat-access-insights: Is a directory
Red Hat Enterprise Linux Server release 7.2 (Maipo)
Red Hat Gluster Storage Server 3.1 Update 2


I then copied the gluster-server repo file from http://download.eng.bos.redhat.com/rel-eng/RHGS/3.1-u3-RHEL-7/2016-04-20.1/Server-RH-Gluster-3-Server.repo to the machine and did an yum-update to update my gluster rpms to latest build which is as below

glusterfs-client-xlators-3.7.9-2.el7rhgs.x86_64
glusterfs-server-3.7.9-2.el7rhgs.x86_64
python-gluster-3.7.5-19.el7rhgs.noarch
gluster-nagios-addons-0.2.5-1.el7rhgs.x86_64
vdsm-gluster-4.16.30-1.3.el7rhgs.noarch
glusterfs-3.7.9-2.el7rhgs.x86_64
glusterfs-api-3.7.9-2.el7rhgs.x86_64
glusterfs-cli-3.7.9-2.el7rhgs.x86_64
glusterfs-geo-replication-3.7.9-2.el7rhgs.x86_64
gluster-nagios-common-0.2.3-1.el7rhgs.noarch
glusterfs-libs-3.7.9-2.el7rhgs.x86_64
glusterfs-fuse-3.7.9-2.el7rhgs.x86_64
glusterfs-rdma-3.7.9-2.el7rhgs.x86_64



Now after starting glusterd, when i tried to create a volume , it failed with saying "volume create: test: failed: Failed to store the Volume information
"
due to [2016-04-21 07:21:26.201964] E [MSGID: 101010] [store.c:33:gf_store_mkdir] 0-: mkdir() failed on path /var/lib/glusterd/vols/test. [No such file or directory]

I noticed that there is no vols directory under /var/lib/glusterd and this is causing the problem





Steps to Reproduce:
==============
1)intiate a new vm create and select rhel7.2 rhg3 
2)in that select the latest 3.1.2 image(for me the latest availble was RHGSS-  3.1.2-RHEL-7-20160216.n.0-RHGSS-x86_64(screenshot attched)
3)Now after the install is successful copy the repo of 3.1.3 build 3.7.9-2 availble @ http://download.eng.bos.redhat.com/rel-eng/RHGS/3.1-u3-RHEL-7/2016-04-20.1/Server-RH-Gluster-3-Server.repo
4)Now note that there exists vols dir under /var/lib/glusterd
5)Now, update gluster packs using yum update glusterfs-server
6)Now on update it can be seen that the vols dir is missing
7)Start glusterd and try creating a volume, it will fail saying volume create: test: failed: Failed to store the Volume information

if you check the etc-glusterfs-glusterd.vol.log log it will say as below
[2016-04-21 07:21:26.201964] E [MSGID: 101010] [store.c:33:gf_store_mkdir] 0-: mkdir() failed on path /var/lib/glusterd/vols/test. [No such file or directory]


###########################################################################
2016-04-20 12:30:33.162016] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-20 12:30:33.162039] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-20 12:30:33.164317] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory]
[2016-04-20 12:30:33.170341] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2016-04-20 12:30:33.170370] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2016-04-20 12:30:33.170377] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
[2016-04-20 12:30:33.170454] E [rpcsvc.c:1390:rpcsvc_program_unregister_portmap] 0-rpc-service: Could not unregister with portmap
[2016-04-20 12:30:33.170463] E [rpcsvc.c:1516:rpcsvc_program_unregister] 0-rpc-service: portmap unregistration of program failed
[2016-04-20 12:30:33.170479] E [rpcsvc.c:1554:rpcsvc_program_unregister] 0-rpc-service: Program unregistration failed: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
[2016-04-21 05:20:48.504116] E [MSGID: 106243] [glusterd.c:1610:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2016-04-21 05:20:50.815020] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:20:50.815086] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:35.711441] E [rpc-transport.c:292:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.7.9/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2016-04-21 05:48:35.711477] E [MSGID: 106243] [glusterd.c:1610:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2016-04-21 05:48:37.661146] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:37.661203] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:37.698250] E [rpc-transport.c:292:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.7.9/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2016-04-21 05:48:37.698282] E [MSGID: 106243] [glusterd.c:1610:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2016-04-21 05:48:39.558386] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:39.558426] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:39.685863] E [rpc-transport.c:292:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.7.9/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2016-04-21 05:48:39.685894] E [MSGID: 106243] [glusterd.c:1610:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2016-04-21 05:48:41.463188] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:41.463234] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:41.544096] E [MSGID: 106243] [glusterd.c:1610:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2016-04-21 05:48:43.379918] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 05:48:43.379971] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 06:06:04.731342] E [MSGID: 101032] [store.c:435:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2016-04-21 06:06:42.085279] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2016-04-21 06:06:45.421026] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2016-04-21 06:06:48.333598] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2016-04-21 06:06:52.797157] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2016-04-21 06:06:55.332727] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2016-04-21 06:50:16.693312] E [MSGID: 106033] [glusterd-hooks.c:357:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/create/pre [No such file or directory]
[2016-04-21 06:50:16.693993] E [MSGID: 101010] [store.c:33:gf_store_mkdir] 0-: mkdir() failed on path /var/lib/glusterd/vols/afr. [No such file or directory]
[2016-04-21 06:50:16.694085] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694164] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694228] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694284] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694339] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694404] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694480] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x21) [0x7ff91f418d21] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694545] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x2d) [0x7ff91f418d2d] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694614] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_create_volume+0xea4) [0x7ff91f460aa4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:50:16.694823] E [MSGID: 106033] [glusterd-store.c:1709:glusterd_store_delete_volume] 0-management: Failed to rename volume directory for volume afr [No such file or directory]
[2016-04-21 06:50:16.694853] E [MSGID: 106123] [glusterd-syncop.c:1411:gd_commit_op_phase] 0-management: Commit of operation 'Volume Create' failed on localhost : Failed to store the Volume information
[2016-04-21 06:51:37.486734] E [MSGID: 106451] [glusterd-utils.c:5995:glusterd_is_path_in_use] 0-management: /rhs/brick1/afr is already part of a volume [No data available]
[2016-04-21 06:51:37.486769] E [MSGID: 106301] [glusterd-syncop.c:1281:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : /rhs/brick1/afr is already part of a volume
[2016-04-21 06:51:41.440235] E [MSGID: 106033] [glusterd-hooks.c:357:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/create/pre [No such file or directory]
[2016-04-21 06:51:41.440815] E [MSGID: 101010] [store.c:33:gf_store_mkdir] 0-: mkdir() failed on path /var/lib/glusterd/vols/afr. [No such file or directory]
[2016-04-21 06:51:41.440893] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.440960] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441022] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441083] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441157] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441222] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441288] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x21) [0x7ff91f418d21] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441346] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x2d) [0x7ff91f418d2d] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441413] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_create_volume+0xea4) [0x7ff91f460aa4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:51:41.441433] E [MSGID: 106033] [glusterd-store.c:1709:glusterd_store_delete_volume] 0-management: Failed to rename volume directory for volume afr [No such file or directory]
[2016-04-21 06:51:41.441460] E [MSGID: 106123] [glusterd-syncop.c:1411:gd_commit_op_phase] 0-management: Commit of operation 'Volume Create' failed on localhost : Failed to store the Volume information
[2016-04-21 06:51:41.427678] E [MSGID: 106451] [glusterd-utils.c:5995:glusterd_is_path_in_use] 0-management: /rhs/brick1/afr is already part of a volume [No data available]
[2016-04-21 06:53:42.731082] E [MSGID: 106451] [glusterd-utils.c:5995:glusterd_is_path_in_use] 0-management: /rhs/brick1/afr is already part of a volume [No data available]
[2016-04-21 06:53:42.745175] E [MSGID: 106033] [glusterd-hooks.c:357:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/create/pre [No such file or directory]
[2016-04-21 06:53:42.745825] E [MSGID: 101010] [store.c:33:gf_store_mkdir] 0-: mkdir() failed on path /var/lib/glusterd/vols/afr. [No such file or directory]
[2016-04-21 06:53:42.745906] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.745969] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746027] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746096] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746172] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746231] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746296] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x21) [0x7ff91f418d21] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746350] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x2d) [0x7ff91f418d2d] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746418] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_create_volume+0xea4) [0x7ff91f460aa4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:53:42.746438] E [MSGID: 106033] [glusterd-store.c:1709:glusterd_store_delete_volume] 0-management: Failed to rename volume directory for volume afr [No such file or directory]
[2016-04-21 06:53:42.746531] E [MSGID: 106123] [glusterd-syncop.c:1411:gd_commit_op_phase] 0-management: Commit of operation 'Volume Create' failed on localhost : Failed to store the Volume information
[2016-04-21 06:55:06.716027] E [MSGID: 101010] [store.c:33:gf_store_mkdir] 0-: mkdir() failed on path /var/lib/glusterd/vols/afr. [No such file or directory]
[2016-04-21 06:55:06.716129] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716229] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716295] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716364] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716434] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716493] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x15) [0x7ff91f418d15] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_bricks_cleanup_tmp+0x3c) [0x7ff91f418c8c] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716550] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x21) [0x7ff91f418d21] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716606] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volume_cleanup_tmp+0x2d) [0x7ff91f418d2d] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716666] E [store.c:163:gf_store_unlink_tmppath] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_op_create_volume+0xea4) [0x7ff91f460aa4] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_store_volinfo+0x34) [0x7ff91f4192b4] -->/lib64/libglusterfs.so.0(gf_store_unlink_tmppath+0x1d3) [0x7ff92a8b2bc3] ) 0-store: invalid argument: shandle [Invalid argument]
[2016-04-21 06:55:06.716683] E [MSGID: 106033] [glusterd-store.c:1709:glusterd_store_delete_volume] 0-management: Failed to rename volume directory for volume afr [No such file or directory]
[2016-04-21 06:55:06.716712] E [MSGID: 106123] [glusterd-syncop.c:1411:gd_commit_op_phase] 0-management: Commit of operation 'Volume Create' failed on localhost : Failed to store the Volume information
[2016-04-21 06:56:22.211640] E [MSGID: 106033] [glusterd-hooks.c:357:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/create/post [No such file or directory]
[2016-04-21 06:56:26.940270] E [MSGID: 106033] [glusterd-hooks.c:357:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/start/pre [No such file or directory]
[2016-04-21 06:56:27.851130] E [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(+0xf68f2) [0x7ff91f4728f2] -->/usr/lib64/glusterfs/3.7.9/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x426) [0x7ff91f472306] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7ff92a8c93c5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=afr --first=yes --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
[2016-04-21 07:13:40.828380] E [MSGID: 106033] [glusterd-hooks.c:357:glusterd_hooks_run_hooks] 0-management: Failed to open dir /var/lib/glusterd/hooks/1/set/pre [No such file or directory]
[2016-04-21 07:14:00.809618] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nufa key
[2016-04-21 07:14:00.810779] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for switch key
[2016-04-21 07:14:00.815506] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for strict-readdir key
[2016-04-21 07:14:00.820185] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for readdir-failover key
[2016-04-21 07:14:00.828722] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for transport.socket.ssl-enabled key
[2016-04-21 07:14:00.829247] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for transport.socket.own-thread key
[2016-04-21 07:14:00.848624] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.transport-type key
[2016-04-21 07:14:00.856456] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for nfs.log-level key
[2016-04-21 07:14:00.864490] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key
[2016-04-21 07:14:00.866513] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key
[2016-04-21 07:14:00.866536] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for config key
[2016-04-21 07:14:00.867927] E [MSGID: 106332] [glusterd-utils.c:10647:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for op-version key
[2016-04-21 07:14:00.870885] E [MSGID: 106434] [glusterd-utils.c:10635:glusterd_get_value_for_vme_entry] 0-management: xlator_volopt_dynload error (-1)
###############################################################################


Actual results:


Expected results:


Additional info:

Comment 2 SATHEESARAN 2016-04-21 10:52:10 UTC

Update on this bug from the mail chain :

<snip>
Hi,

Thanks Pavan, for this issue. It is a variant of another bug raised with 3.7.9-1 build. 

You should not see this issue after upgrading from 3.7.9-1 to 3.7.9-2 . There was a bug on spec file which causes the directories to be orphan. Bugzilla [1], is in Assigned state and has all acks in place, should be available by next build.

It is not a test blocker, please upgrade system from 3.1.3 first build to 3.1.3 second build. Any one who has upgraded please confirm.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1324820

Thanks,
Rahul

</snip>

Comment 3 Byreddy 2016-04-21 16:20:06 UTC

i am not seeing glusterd restart after update, as per update procedure we have to restart once update done.

Comment 4 Atin Mukherjee 2016-04-26 11:29:19 UTC

Is there any reason why this bug can't be closed or made dependent on?

Comment 5 SATHEESARAN 2016-04-26 12:32:36 UTC

(In reply to Atin Mukherjee from comment #4)
> Is there any reason why this bug can't be closed or made dependent on?

Hi Atin, 

As per comment2 the cause for this issue is that few directories are now owned by glusterfs-server package.

This bug was raised in concern to the issue where vols directory was remove post update from RHGS 3.1.2 to the interim build which is glusterfs-3.7.9-2.el7rhgs.
This still remains a mystery, which the '/var/lib/glusterd/vols' directory got removed post update.

This bug could be the manifestation of the issue created by BZ 1324820.

We will try to reproduce this issue, by updating from RHGS 3.1.2 to the build where BZ 1324820 was fixed.

If there are no issues found, then this bug should be CLOSED as NOTABUG.

Comment 7 SATHEESARAN 2016-05-11 07:35:16 UTC

After updating from RHGS 3.1.2 ( glusterfs-3.7.5-19.el7rhgs ) to nightly build ( glusterfs-3.7.9-4.el7rhgs ), this issue is not seen

/var/lib/glusterd/vols directory was available post update.

Comment 9 errata-xmlrpc 2016-06-23 05:18:57 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240

Note You need to log in before you can comment on or make changes to this bug.