Bug 1008163 - (Intermittent) peer probing from a host in cluster, causes rhsc to fail detection newly probed host in cluster
Summary: (Intermittent) peer probing from a host in cluster, causes rhsc to fail detec...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: rhsc
Version: 2.1
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: Bala.FA
QA Contact: Dustin Tsang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-15 11:12 UTC by Dustin Tsang
Modified: 2015-07-13 04:39 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-11-01 16:50:15 UTC
Target Upstream Version:


Attachments (Terms of Use)
ovirt-engine log in IST (8.43 KB, text/x-log)
2013-09-15 11:36 UTC, Dustin Tsang
no flags Details

Description Dustin Tsang 2013-09-15 11:12:06 UTC
Description of problem:

(Intermittent) peer probing from a host in cluster, causes rhsc to fail detection of new host

Version-Release number of selected component (if applicable):
rhsc-bb10
rhs-21-20130910

How reproducible:
the issues is intermittent happens about 30% of the time


Steps to Reproduce:
1. using the guy, add a host to a 3.2 cluster
2. using the gluster cli, from the host from step 1, peer probe the new a new host


Actual results:
in the gui, event generated, error on host from step 1

Expected results:
new host is detected;
Import button displays in the general sub tab for the cluster


Additional info:

Comment 1 Dustin Tsang 2013-09-15 11:14:54 UTC
vdsm.log:
----------
hread-27::DEBUG::2013-09-11 00:21:03,839::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call hostsList with () {} flowID [5a6376a]
Thread-27::DEBUG::2013-09-11 00:21:04,119::BindingXMLRPC::920::vds::(wrapper) return hostsList with {'status': {'message': 'Done', 'code': 0}, 'hosts': [{'status': 'CONNECTED', 'hostname': '10.70.37.67', 'uuid': '8c7cbfe2-f0c1-4f11-b1fd-bfa18ff93c70'}]}
Thread-28::DEBUG::2013-09-11 00:21:04,127::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call volumesList with () {} flowID [5a6376a]
Thread-28::DEBUG::2013-09-11 00:21:04,269::BindingXMLRPC::920::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {}}
Thread-30::DEBUG::2013-09-11 00:21:09,291::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call hostsList with () {}
Thread-30::ERROR::2013-09-11 00:21:09,470::BindingXMLRPC::929::vds::(wrapper) vdsm exception occured
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 32, in wrapper
    rv = func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 204, in hostsList
    return {'hosts': self.svdsmProxy.glusterPeerStatus()}
  File "/usr/share/vdsm/supervdsm.py", line 76, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 67, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterPeerStatus
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
GlusterHostsListFailedException: Hosts list failed
error: (null)
return code: -22
Thread-32::DEBUG::2013-09-11 00:21:09,998::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call volumesList with () {} flowID [4f545bb8]
Thread-32::DEBUG::2013-09-11 00:21:10,338::BindingXMLRPC::920::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {}}
Thread-34::DEBUG::2013-09-11 00:21:15,404::task::579::TaskManager.Task::(_updateState) Task=`854c1b39-cf31-4839-a0ca-45d04c8ae242`::moving from state init -> state preparing
Thread-34::INFO::2013-09-11 00:21:15,404::logUtils::40::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2013-09-11 00:21:15,404::logUtils::42::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} 

super-vdsm.log
----------------
MainThread::DEBUG::2013-09-11 00:20:49,461::supervdsmServer::422::SuperVdsm.Server::(main) Started serving super vdsm object
MainProcess|Thread-30::ERROR::2013-09-11 00:21:09,466::supervdsmServer::81::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 79, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 363, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/__init__.py", line 31, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/cli.py", line 789, in peerStatus
    raise ge.GlusterHostsListFailedException(rc=e.rc, err=e.err)
GlusterHostsListFailedException: Hosts list failed
error: (null)
return code: -22

Comment 3 Dustin Tsang 2013-09-15 11:36:56 UTC
Created attachment 797880 [details]
ovirt-engine log in IST

Comment 4 Bala.FA 2013-10-17 05:24:57 UTC
Please add gluster logs too.

Comment 6 Dustin Tsang 2013-11-01 16:50:15 UTC
I can't reproduce in rhsc-cb5.


Note You need to log in before you can comment on or make changes to this bug.