Description of problem: Deploy HE failed on [Initialize lockspace volume] task. """ [ INFO ] changed: [localhost] [ INFO ] TASK [Initialize lockspace volume] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:05.558402", "end": "2018-06-11 11:11:36.677061", "msg": "non-zero return code", "rc": 1, "start": "2018-06-11 11:11:31.118659", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main\n \"__main__\", fname, loader, pkg_name)\n File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code\n exec code in run_globals\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request\n verbose=self.__verbose\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request\n self.send_content(h, request_body)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders\n self._send_output(message_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output\n self.send(msg)\n File \"/usr/lib64/python2.7/httplib.py\", line 843, in send\n self.connect()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect\n self.sock.connect(base64.b16decode(self.host))\n File \"/usr/lib64/python2.7/socket.py\", line 224, in meth\n return getattr(self._sock,name)(*args)\nsocket.error: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main", " \"__main__\", fname, loader, pkg_name)", " File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code", " exec code in run_globals", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request", " self.send_content(h, request_body)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders", " self._send_output(message_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output", " self.send(msg)", " File \"/usr/lib64/python2.7/httplib.py\", line 843, in send", " self.connect()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect", " self.sock.connect(base64.b16decode(self.host))", " File \"/usr/lib64/python2.7/socket.py\", line 224, in meth", " return getattr(self._sock,name)(*args)", "socket.error: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} """ Version-Release number of selected component (if applicable): rhvh-4.2.4.2-0.20180608.0+1 rhvm-appliance-4.2-20180601.0.el7.noarch ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.13-1.el7ev.noarch cockpit-ovirt-dashboard-0.11.28-1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy HE via cockpit Actual results: The same as the description Expected results: Deploy HE successfully from cockpit. Additional info:
Created attachment 1449863 [details] create_target_vm.log
Do you also have /var/log/ovirt-hosted-engine-ha/agent.log and /var/log/ovirt-hosted-engine-ha/broker.log ?
agent.log and broker.log http://pastebin.test.redhat.com/601470
Sorry, also vdsm log for the relevant time frame.
(In reply to Simone Tiraboschi from comment #4) > Sorry, also vdsm log for the relevant time frame. http://pastebin.test.redhat.com/601517
Is this only on RHVH? If so, I think this is the same bug as rhbz#1534197
(In reply to Ryan Barry from comment #6) > Is this only on RHVH? If so, I think this is the same bug as rhbz#1534197 Yes, with same versions about ovirt-hosted-engine-setup or ovirt-hosted-engine-setup on RHEL7.5, it works well.
Thanks, marking as duplicate of bug #1534197 *** This bug has been marked as a duplicate of bug 1534197 ***
Tested with rhvh-4.2.4.3-0.20180614.0+1 also met this issue: from cockpit : """ [ INFO ] changed: [localhost] [ INFO ] TASK [Initialize lockspace volume] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:05.555190", "end": "2018-06-15 15:22:10.067281", "msg": "non-zero return code", "rc": 1, "start": "2018-06-15 15:22:04.512091", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main\n \"__main__\", fname, loader, pkg_name)\n File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code\n exec code in run_globals\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request\n verbose=self.__verbose\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request\n self.send_content(h, request_body)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders\n self._send_output(message_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output\n self.send(msg)\n File \"/usr/lib64/python2.7/httplib.py\", line 843, in send\n self.connect()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect\n self.sock.connect(base64.b16decode(self.host))\n File \"/usr/lib64/python2.7/socket.py\", line 224, in meth\n return getattr(self._sock,name)(*args)\nsocket.error: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main", " \"__main__\", fname, loader, pkg_name)", " File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code", " exec code in run_globals", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request", " self.send_content(h, request_body)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders", " self._send_output(message_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output", " self.send(msg)", " File \"/usr/lib64/python2.7/httplib.py\", line 843, in send", " self.connect()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect", " self.sock.connect(base64.b16decode(self.host))", " File \"/usr/lib64/python2.7/socket.py\", line 224, in meth", " return getattr(self._sock,name)(*args)", "socket.error: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} """ Log url: http://pastebin.test.redhat.com/604553
Can you please attach vdsm.log?
To update, I can't reproduce this. Is this reproducible on the CLI?
(In reply to Ryan Barry from comment #11) > To update, I can't reproduce this. > > Is this reproducible on the CLI? Yes, can reproduce on the CLI. vdsm.log: http://pastebin.test.redhat.com/605603
Works well for me with the latest build: cockpit-ovirt-dashboard-0.11.28-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.14-1.el7ev.noarch ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch rhvm-appliance-4.2-20180615.0.el7.noarch rhvh-4.2.4.3-0.20180615.0+1 [root@ibm-x3650m5-06 ~]# imgbase w You are on rhvh-4.2.4.3-0.20180615.0+1 [root@ibm-x3650m5-06 ~]# hosted-engine --vm-status --== Host 1 status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : ibm-x3650m5-06.lab.eng.pek2.redhat.com Host ID : 1 Engine status : {"health": "good", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : ecd6d40e local_conf_timestamp : 8882 Host timestamp : 8882 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=8882 (Tue Jun 19 13:34:18 2018) host-id=1 score=3400 vm_conf_refresh_time=8882 (Tue Jun 19 13:34:18 2018) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False