Bug 1008163

Summary: (Intermittent) peer probing from a host in cluster, causes rhsc to fail detection newly probed host in cluster
Product: Red Hat Gluster Storage Reporter: Dustin Tsang <dtsang>
Component: rhscAssignee: Bala.FA <barumuga>
Status: CLOSED NOTABUG QA Contact: Dustin Tsang <dtsang>
Severity: high Docs Contact:
Priority: medium    
Version: 2.1CC: dpati, dtsang, knarra, mmahoney, mmccune, pprakash, rhs-bugs, ssampat
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-01 16:50:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
ovirt-engine log in IST none

Description Dustin Tsang 2013-09-15 11:12:06 UTC 
Description of problem:

(Intermittent) peer probing from a host in cluster, causes rhsc to fail detection of new host

Version-Release number of selected component (if applicable):
rhsc-bb10
rhs-21-20130910

How reproducible:
the issues is intermittent happens about 30% of the time


Steps to Reproduce:
1. using the guy, add a host to a 3.2 cluster
2. using the gluster cli, from the host from step 1, peer probe the new a new host


Actual results:
in the gui, event generated, error on host from step 1

Expected results:
new host is detected;
Import button displays in the general sub tab for the cluster


Additional info:
Comment 1 Dustin Tsang 2013-09-15 11:14:54 UTC 
vdsm.log:
----------
hread-27::DEBUG::2013-09-11 00:21:03,839::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call hostsList with () {} flowID [5a6376a]
Thread-27::DEBUG::2013-09-11 00:21:04,119::BindingXMLRPC::920::vds::(wrapper) return hostsList with {'status': {'message': 'Done', 'code': 0}, 'hosts': [{'status': 'CONNECTED', 'hostname': '10.70.37.67', 'uuid': '8c7cbfe2-f0c1-4f11-b1fd-bfa18ff93c70'}]}
Thread-28::DEBUG::2013-09-11 00:21:04,127::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call volumesList with () {} flowID [5a6376a]
Thread-28::DEBUG::2013-09-11 00:21:04,269::BindingXMLRPC::920::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {}}
Thread-30::DEBUG::2013-09-11 00:21:09,291::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call hostsList with () {}
Thread-30::ERROR::2013-09-11 00:21:09,470::BindingXMLRPC::929::vds::(wrapper) vdsm exception occured
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 918, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 32, in wrapper
    rv = func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/api.py", line 204, in hostsList
    return {'hosts': self.svdsmProxy.glusterPeerStatus()}
  File "/usr/share/vdsm/supervdsm.py", line 76, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 67, in <lambda>
    **kwargs)
  File "<string>", line 2, in glusterPeerStatus
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
GlusterHostsListFailedException: Hosts list failed
error: (null)
return code: -22
Thread-32::DEBUG::2013-09-11 00:21:09,998::BindingXMLRPC::913::vds::(wrapper) client [10.70.37.123]::call volumesList with () {} flowID [4f545bb8]
Thread-32::DEBUG::2013-09-11 00:21:10,338::BindingXMLRPC::920::vds::(wrapper) return volumesList with {'status': {'message': 'Done', 'code': 0}, 'volumes': {}}
Thread-34::DEBUG::2013-09-11 00:21:15,404::task::579::TaskManager.Task::(_updateState) Task=`854c1b39-cf31-4839-a0ca-45d04c8ae242`::moving from state init -> state preparing
Thread-34::INFO::2013-09-11 00:21:15,404::logUtils::40::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-34::INFO::2013-09-11 00:21:15,404::logUtils::42::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} 

super-vdsm.log
----------------
MainThread::DEBUG::2013-09-11 00:20:49,461::supervdsmServer::422::SuperVdsm.Server::(main) Started serving super vdsm object
MainProcess|Thread-30::ERROR::2013-09-11 00:21:09,466::supervdsmServer::81::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 79, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 363, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/__init__.py", line 31, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/gluster/cli.py", line 789, in peerStatus
    raise ge.GlusterHostsListFailedException(rc=e.rc, err=e.err)
GlusterHostsListFailedException: Hosts list failed
error: (null)
return code: -22
Comment 3 Dustin Tsang 2013-09-15 11:36:56 UTC 
Created attachment 797880 [details]
ovirt-engine log in IST
Comment 4 Bala.FA 2013-10-17 05:24:57 UTC 
Please add gluster logs too.
Comment 6 Dustin Tsang 2013-11-01 16:50:15 UTC 
I can't reproduce in rhsc-cb5.