Description of problem: Peer probe fails after adding hosts via RHEVM. On first restart after installation of the host, engine tried to get list of hosts using the verb glusterHostsList, however received a response "2012-09-26 20:14:05,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-34) [40594417] XML RPC error in command GlusterServersListVDS ( HostName = rhs-client2.lab.eng.blr.redhat.com ), the error was: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException, XmlRpcException: <type 'exceptions.Exception'>:method "glusterHostsList" is not supported" Interestingly, the command "vdsClient localhost glusterHostsList" seems to crash with following traceback: --------------------- # vdsClient localhost glusterHostsList Traceback (most recent call last): File "/usr/share/vdsm/vdsClient.py", line 2125, in <module> code, message = commands[command][0](commandArgs) File "/usr/share/vdsm/vdsClientGluster.py", line 207, in do_glusterHostsList status = self.s.glusterHostsList() File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1237, in request errcode, errmsg, headers = h.getreply() File "/usr/lib64/python2.6/httplib.py", line 1064, in getreply response = self._conn.getresponse() File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 391, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status line = self.fp.readline() File "/usr/lib64/python2.6/socket.py", line 433, in readline data = recv(1) error: [Errno 104] Connection reset by peer --------------------- This happens on both the servers where rhs downstream vdsm is installed. Installed gluster packages: --------- glusterfs-3.3.0rhsvirt1-5.el6rhs.x86_64.rpm glusterfs-geo-replication-3.3.0rhsvirt1-5.el6rhs.x86_64.rpm glusterfs-debuginfo-3.3.0rhsvirt1-5.el6rhs.x86_64.rpm glusterfs-rdma-3.3.0rhsvirt1-5.el6rhs.x86_64.rpm glusterfs-devel-3.3.0rhsvirt1-5.el6rhs.x86_64.rpm glusterfs-server-3.3.0rhsvirt1-5.el6rhs.x86_64.rpm glusterfs-fuse-3.3.0rhsvirt1-5.el6rhs.x86_64.rpm --------- Installed vdsm packages: --------- vdsm-debuginfo-4.9.6-17.el6.x86_64.rpm vdsm-hook-vhostmd-4.9.6-17.el6.noarch.rpm vdsm-4.9.6-17.el6.x86_64.rpm vdsm-debug-plugin-4.9.6-17.el6.noarch.rpm vdsm-python-4.9.6-17.el6.x86_64.rpm vdsm-bootstrap-4.9.6-17.el6.noarch.rpm vdsm-gluster-4.9.6-17.el6.noarch.rpm vdsm-reg-4.9.6-17.el6.noarch.rpm vdsm-cli-4.9.6-17.el6.noarch.rpm vdsm-hook-faqemu-4.9.6-17.el6.noarch.rpm --------- How reproducible: Always Steps to Reproduce: 1. Add 2 hosts one after the other via RHEVM in gluster only cluster. 2. After the bootstrapping, check the #gluster peer status output from any of the vdsm servers. Actual results: # gluster peer status No peers present Expected results: vdsm servers should be peer probed successfully. Additional info: Engine logs and vdsm logs attached. Please let me know if you need any additional details for further debugging.
Created attachment 617694 [details] vdsm logs
Created attachment 617695 [details] engine logs
This was happening because the vdsm-gluster package did not get installed as part of the bootstrapping. That happened because the engine had RHEV-M's version of the vds-bootstrap package that doesn't include code to install gluster related packages. The solution is to install the vds-bootstrap package from RHS downstream in the RHEV-M engine machine. This is required because the testing is happening with locally built RPMs and these patches are not yet merged in RHEV-M downstream.