Bug 998514 - RHS-C: Sync hooks: "GlusterHookListException: List gluster hook failed"
RHS-C: Sync hooks: "GlusterHookListException: List gluster hook failed"
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
2.1
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Amar Tumballi
Prasanth
:
Depends On:
Blocks: 999795 1028982
  Show dependency treegraph
 
Reported: 2013-08-19 09:17 EDT by Prasanth
Modified: 2013-12-18 19:09 EST (History)
13 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.21rhs-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 999795 (view as bug list)
Environment:
Last Closed: 2013-09-23 18:36:03 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm.log (497.45 KB, text/x-log)
2013-08-19 09:19 EDT, Prasanth
no flags Details

  None (edit)
Description Prasanth 2013-08-19 09:17:16 EDT
Description of problem:

Sync hooks fails with the following exception in the vdsm log:

--------------------------
Thread-331::DEBUG::2013-08-19 18:31:49,451::BindingXMLRPC::913::vds::(wrapper) client [10.70.36.27]::call hooksList with () {} flowID [2f5ba54e]
Thread-331::ERROR::2013-08-19 18:31:49,453::BindingXMLRPC::929::vds::(wrapper) vdsm exception occured
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 32, in wrapper
    rv = func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 221, in hooksList
    status = self.svdsmProxy.glusterHooksList()
  File "/usr/share/vdsm/supervdsm.py", line 76, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 67, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterHooksList
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
GlusterHookListException: List gluster hook failed
error: [Errno 2] No such file or directory: '/var/lib/glusterd/hooks/1/start/pre'
Thread-333::DEBUG::2013-08-19 18:31:52,161::BindingXMLRPC::913::vds::(wrapper) client [10.70.36.27]::call hostsList with () {}
Thread-333::DEBUG::2013-08-19 18:31:52,399::BindingXMLRPC::920::vds::(wrapper) return hostsList with {'status': {'message': 'Done', 'code': 0}, 'hosts': [{'status': 'CONNECTED', 'hostname': '10.70.36.75', 'uuid': '4d37d7f0-ab66-4f13-bd0f-9a5847ca3b2e'}]}
Thread-334::DEBUG::2013-08-19 18:31:52,405::BindingXMLRPC::913::vds::(wrapper) client [10.70.36.27]::call volumesList with () {}
Thread-334::DEBUG::2013-08-19 18:31:52,528::BindingXMLRPC::920::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {'vol1': {'transportType': ['TCP'], 'uuid': 'edef90bb-fb07-4c7a-a777-98e77b496d48', 'bricks': ['vm03.lab.eng.blr.redhat.com:/tmp/1', 'vm03.lab.eng.blr.redhat.com:/tmp/11'], 'volumeName': 'vol1', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '2', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'auth.allow': '*', 'nfs.disable': 'off', 'user.cifs': 'enable'}}, 'vol2': {'transportType': ['TCP'], 'uuid': '0fdfc0d2-6ce1-45bd-bd9e-f5802e1289ad', 'bricks': ['vm03.lab.eng.blr.redhat.com:/tmp/111', 'vm03.lab.eng.blr.redhat.com:/tmp/1111'], 'volumeName': 'vol2', 'volumeType': 'DISTRIBUTE', 'replicaCount': '1', 'brickCount': '2', 'distCount': '1', 'volumeStatus': 'ONLINE', 'stripeCount': '1', 'options': {'auth.allow': '*', 'nfs.disable': 'off', 'user.cifs': 'disable'}}}}
--------------------------

Version-Release number of selected component (if applicable):  Red Hat Storage Console Version: 2.1.0-0.bb9.el6rhs 


How reproducible: Always


Steps to Reproduce:
1. Create a cluster and add the latest RHS node RHS-2.1-20130814.n.0-RHS-x86_64-DVD1.iso) 
2. Select the cluster and click on "Gluster Hooks"
3. Click on "Sync" and watch the vdsm.log

Actual results: Sync gluster hooks is failing


Expected results: Sync gluster hooks shouldn't fail to fetch the hooks


Additional info: vdsm.log attached
Comment 1 Prasanth 2013-08-19 09:19:09 EDT
Created attachment 788057 [details]
vdsm.log
Comment 3 Prasanth 2013-08-20 02:52:33 EDT
On further debugging, I could see that the following directories are missing in the RHS ISO and since vdsm checks for the existence of these, hookList fails:

------
/var/lib/glusterd/hooks/1/start/pre
/var/lib/glusterd/hooks/1/stop/post
/var/lib/glusterd/hooks/1/gsync-create/pre
/var/lib/glusterd/hooks/1/set/pre
------

So we need to confirm if the missing of these directories were intentional in the latest build or a bug in rpmbuild itself. 


PS: If I manually creates those missing directories in the RHS nodes, Sync works fine!
Comment 4 Prasanth 2013-08-20 02:57:55 EDT
# pwd
/var/lib/glusterd/hooks/1
[root@rhs-client31 1]# ls -al *
gsync-create:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:23 post

set:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:23 post

start:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:34 post

stop:
total 12
drwxr-xr-x. 3 root root 4096 Aug 12 12:23 .
drwxr-xr-x. 6 root root 4096 Aug 12 12:23 ..
drwxr-xr-x. 2 root root 4096 Aug 12 12:23 pre
Comment 5 Amar Tumballi 2013-08-20 04:49:54 EDT
Prashanth,

What does find /var/lib/glusterd/hooks/1/ show?

There should be 4 hooks file at the moment in RHS, and I guess the directories appropriate.

Regards,
Amar
Comment 6 Prasanth 2013-08-20 05:01:30 EDT
(In reply to Amar Tumballi from comment #5)
> Prashanth,
> 
> What does find /var/lib/glusterd/hooks/1/ show?
> 
> There should be 4 hooks file at the moment in RHS, and I guess the
> directories appropriate.


Amar,

I could see 6 hook files and its directories as given below:

----
# find /var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/start
/var/lib/glusterd/hooks/1/start/post
/var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
/var/lib/glusterd/hooks/1/stop
/var/lib/glusterd/hooks/1/stop/pre
/var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh
/var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh
/var/lib/glusterd/hooks/1/gsync-create
/var/lib/glusterd/hooks/1/gsync-create/post
/var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh
/var/lib/glusterd/hooks/1/set
/var/lib/glusterd/hooks/1/set/post
/var/lib/glusterd/hooks/1/set/post/S30samba-set.sh
----

But earlier (till few builds back), directories existed even if hook file was not present in it by default. But that doesn't seem to be the case now. So wanted to confirm if something changed in between?

-Prasanth
Comment 7 Amar Tumballi 2013-08-20 05:54:19 EDT
Bala, can we create the missing directories in %post part of glusterfs-server package?
Comment 8 Bala.FA 2013-08-20 07:17:13 EDT
Not in %post.  It will be installed by glusterfs-server.rpm
Comment 9 Amar Tumballi 2013-08-21 04:46:47 EDT
https://code.engineering.redhat.com/gerrit/#/c/11618
Comment 10 Prasanth 2013-08-21 07:22:34 EDT
Verified as fixed in glusterfs-3.4.0.21rhs-1

------------------
[root@vm03 ]# rpm -qa |grep glusterfs
glusterfs-fuse-3.4.0.21rhs-1.el6rhs.x86_64
samba-glusterfs-3.6.9-159.1.el6rhs.x86_64
glusterfs-libs-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-geo-replication-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-server-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-api-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-3.4.0.21rhs-1.el6rhs.x86_64
glusterfs-rdma-3.4.0.21rhs-1.el6rhs.x86_64


[root@vm03 ]# find /var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/
/var/lib/glusterd/hooks/1/create
/var/lib/glusterd/hooks/1/create/pre
/var/lib/glusterd/hooks/1/create/post
/var/lib/glusterd/hooks/1/set
/var/lib/glusterd/hooks/1/set/pre
/var/lib/glusterd/hooks/1/set/post
/var/lib/glusterd/hooks/1/set/post/S30samba-set.sh
/var/lib/glusterd/hooks/1/start
/var/lib/glusterd/hooks/1/start/pre
/var/lib/glusterd/hooks/1/start/post
/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
/var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
/var/lib/glusterd/hooks/1/remove-brick
/var/lib/glusterd/hooks/1/remove-brick/pre
/var/lib/glusterd/hooks/1/remove-brick/post
/var/lib/glusterd/hooks/1/delete
/var/lib/glusterd/hooks/1/delete/pre
/var/lib/glusterd/hooks/1/delete/post
/var/lib/glusterd/hooks/1/add-brick
/var/lib/glusterd/hooks/1/add-brick/pre
/var/lib/glusterd/hooks/1/add-brick/post
/var/lib/glusterd/hooks/1/gsync-create
/var/lib/glusterd/hooks/1/gsync-create/pre
/var/lib/glusterd/hooks/1/gsync-create/post
/var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh
/var/lib/glusterd/hooks/1/stop
/var/lib/glusterd/hooks/1/stop/pre
/var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh
/var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh
/var/lib/glusterd/hooks/1/stop/post
------------------

As a result, Sync now works fine in Console.
Comment 12 Scott Haines 2013-09-23 18:36:03 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.