Bug 998514

Summary: RHS-C: Sync hooks: "GlusterHookListException: List gluster hook failed"
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Prasanth <pprakash>
Component: glusterfsAssignee: Amar Tumballi <amarts>
Status: CLOSED ERRATA QA Contact: Prasanth <pprakash>
Severity: high Docs Contact:
Priority: high    
Version: 2.1CC: amarts, barumuga, ckannan, dtsang, knarra, mmahoney, pprakash, rhs-bugs, shaines, ssampat, surs, vbellur, vraman
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.4.0.21rhs-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 999795 (view as bug list) Environment:
Last Closed: 2013-09-23 22:36:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 999795, 1028982    
Attachments:
Description Flags
vdsm.log none

Description Prasanth 2013-08-19 13:17:16 UTC
Description of problem:

Sync hooks fails with the following exception in the vdsm log:

--------------------------
Thread-331::DEBUG::2013-08-19 18:31:49,451::BindingXMLRPC::913::vds::(wrapper) client [10.70.36.27]::call hooksList with () {} flowID [2f5ba54e]
Thread-331::ERROR::2013-08-19 18:31:49,453::BindingXMLRPC::929::vds::(wrapper) vdsm exception occured
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 32, in wrapper
    rv = func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 221, in hooksList
    status = self.svdsmProxy.glusterHooksList()
  File "/usr/share/vdsm/supervdsm.py", line 76, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 67, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterHooksList
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
GlusterHookListException: List gluster hook failed
error: [Errno 2] No such file or directory: '/var/lib/glusterd/hooks/1/start/pre'
Thread-333::DEBUG::2013-08-19 18:31:52,161::BindingXMLRPC::913::vds::(wrapper) client [10.70.36.27]::call hostsList with () {}
Thread-333::DEBUG::2013-08-19 18:31:52,399::BindingXMLRPC::920::vds::(wrapper) return hostsList with {'status': {'message': 'Done', 'code': 0}, 'hosts': [{'status': 'CONNECTED', 'hostname': '10.70.36.75', 'uuid': '4d37d7f0-ab66-4f13-bd0f-9a5847ca3b2e'}]}
Thread-334::DEBUG::2013-08-19 18:31:52,405::BindingXMLRPC::913::vds::(wrapper) client [10.70.36.27]::call volumesList with () {}
Thread-334::DEBUG::2013-08-19 18:31:52,528::BindingXMLRPC::920::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'vol1': {'transportType': ['TCP'], 'uuid': 'edef90bb-fb07-4c7a-a777-98e77b496d48', 'bricks': ['vm03.lab.eng.blr.redhat.com:/tmp/1', 'vm03.lab.eng.blr.redhat.com:/tmp/11'], 'volumeName': 'vol1', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '2', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'auth.allow': '*', 'nfs.disable': 'off', 'user.cifs': 'enable'}}, 'vol2': {'transportType': ['TCP'], 'uuid': '0fdfc0d2-6ce1-45bd-bd9e-f5802e1289ad', 'bricks': ['vm03.lab.eng.blr.redhat.com:/tmp/111', 'vm03.lab.eng.blr.redhat.com:/tmp/1111'], 'volumeName': 'vol2', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '2', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'auth.allow': '*', 'nfs.disable': 'off', 'user.cifs': 'disable'}}}}
--------------------------

Version-Release number of selected component (if applicable):  Red Hat Storage Console Version: 2.1.0-0.bb9.el6rhs 


How reproducible: Always


Steps to Reproduce:
1. Create a cluster and add the latest RHS node RHS-2.1-20130814.n.0-RHS-x86_64-DVD1.iso) 
2. Select the cluster and click on "Gluster Hooks"
3. Click on "Sync" and watch the vdsm.log

Actual results: Sync gluster hooks is failing


Expected results: Sync gluster hooks shouldn't fail to fetch the hooks


Additional info: vdsm.log attached

Comment 1 Prasanth 2013-08-19 13:19:09 UTC
Created attachment 788057 [details]
vdsm.log

Comment 3 Prasanth 2013-08-20 06:52:33 UTC
On further debugging, I could see that the following directories are missing in the RHS ISO and since vdsm checks for the existence of these, hookList fails:

------
/var/lib/glusterd/hooks/1/start/pre
/var/lib/glusterd/hooks/1/stop/post
/var/lib/glusterd/hooks/1/gsync-create/pre
/var/lib/glusterd/hooks/1/set/pre
------

So we need to confirm if the missing of these directories were intentional in the latest build or a bug in rpmbuild itself. 


PS: If I manually creates those missing directories in the RHS nodes, Sync works fine!

Comment 4 Prasanth 2013-08-20 06:57:55 UTC
# pwd
/var/lib/glusterd/hooks/1
[root@rhs-client31 1]# ls -al *
gsync-create:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:23 post

set:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:23 post

start:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:34 post

stop:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:23 pre

Comment 5 Amar Tumballi 2013-08-20 08:49:54 UTC
Prashanth,

What does find /var/lib/glusterd/hooks/1/ show?

There should be 4 hooks file at the moment in RHS, and I guess the directories appropriate.

Regards,
Amar

Comment 6 Prasanth 2013-08-20 09:01:30 UTC
(In reply to Amar Tumballi from comment #5)
> Prashanth,
> 
> What does find /var/lib/glusterd/hooks/1/ show?
> 
> There should be 4 hooks file at the moment in RHS, and I guess the
> directories appropriate.


Amar,

I could see 6 hook files and its directories as given below:

----
# find /var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/start
/var/lib/glusterd/hooks/1/start/post
/var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
/var/lib/glusterd/hooks/1/stop
/var/lib/glusterd/hooks/1/stop/pre
/var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh
/var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh
/var/lib/glusterd/hooks/1/gsync-create
/var/lib/glusterd/hooks/1/gsync-create/post
/var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh
/var/lib/glusterd/hooks/1/set
/var/lib/glusterd/hooks/1/set/post
/var/lib/glusterd/hooks/1/set/post/S30samba-set.sh
----

But earlier (till few builds back), directories existed even if hook file was not present in it by default. But that doesn't seem to be the case now. So wanted to confirm if something changed in between?

-Prasanth

Comment 7 Amar Tumballi 2013-08-20 09:54:19 UTC
Bala, can we create the missing directories in %post part of glusterfs-server package?

Comment 8 Bala.FA 2013-08-20 11:17:13 UTC
Not in %post.  It will be installed by glusterfs-server.rpm

Comment 9 Amar Tumballi 2013-08-21 08:46:47 UTC
https://code.engineering.redhat.com/gerrit/#/c/11618

Comment 10 Prasanth 2013-08-21 11:22:34 UTC
Verified as fixed in glusterfs-3.4.0.21rhs-1

------------------
[root@vm03 ]# rpm -qa |grep glusterfs
glusterfs-fuse-3.4.0.21rhs-1.el6rhs.x86_64
samba-glusterfs-3.6.9-159.1.el6rhs.x86_64
glusterfs-libs-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-api-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.21rhs-1.el6rhs.x86_64


[root@vm03 ]# find /var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/create
/var/lib/glusterd/hooks/1/create/pre
/var/lib/glusterd/hooks/1/create/post
/var/lib/glusterd/hooks/1/set
/var/lib/glusterd/hooks/1/set/pre
/var/lib/glusterd/hooks/1/set/post
/var/lib/glusterd/hooks/1/set/post/S30samba-set.sh
/var/lib/glusterd/hooks/1/start
/var/lib/glusterd/hooks/1/start/pre
/var/lib/glusterd/hooks/1/start/post
/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
/var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
/var/lib/glusterd/hooks/1/remove-brick
/var/lib/glusterd/hooks/1/remove-brick/pre
/var/lib/glusterd/hooks/1/remove-brick/post
/var/lib/glusterd/hooks/1/delete
/var/lib/glusterd/hooks/1/delete/pre
/var/lib/glusterd/hooks/1/delete/post
/var/lib/glusterd/hooks/1/add-brick
/var/lib/glusterd/hooks/1/add-brick/pre
/var/lib/glusterd/hooks/1/add-brick/post
/var/lib/glusterd/hooks/1/gsync-create
/var/lib/glusterd/hooks/1/gsync-create/pre
/var/lib/glusterd/hooks/1/gsync-create/post
/var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh
/var/lib/glusterd/hooks/1/stop
/var/lib/glusterd/hooks/1/stop/pre
/var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh
/var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh
/var/lib/glusterd/hooks/1/stop/post
------------------

As a result, Sync now works fine in Console.

Comment 12 Scott Haines 2013-09-23 22:36:03 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html