Bug 1589650
| Summary: | Deploy HE failed on [Initialize lockspace volume] task from cockpit | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-hosted-engine-setup | Reporter: | Yihui Zhao <yzhao> | ||||
| Component: | Plugins.General | Assignee: | Simone Tiraboschi <stirabos> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | meital avital <mavital> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | --- | CC: | bugs, cshao, huzhao, jiaczhan, phbailey, qiyuan, rbarry, sbonazzo, stirabos, weiwang, yaniwang, ycui, yturgema, yzhao | ||||
| Target Milestone: | --- | Keywords: | Regression, TestBlocker | ||||
| Target Release: | --- | Flags: | rbarry:
blocker?
yzhao: testing_ack+ |
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-06-12 08:46:14 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Integration | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1449863 [details]
create_target_vm.log
Do you also have /var/log/ovirt-hosted-engine-ha/agent.log and /var/log/ovirt-hosted-engine-ha/broker.log ? agent.log and broker.log http://pastebin.test.redhat.com/601470 Sorry, also vdsm log for the relevant time frame. (In reply to Simone Tiraboschi from comment #4) > Sorry, also vdsm log for the relevant time frame. http://pastebin.test.redhat.com/601517 Is this only on RHVH? If so, I think this is the same bug as rhbz#1534197 (In reply to Ryan Barry from comment #6) > Is this only on RHVH? If so, I think this is the same bug as rhbz#1534197 Yes, with same versions about ovirt-hosted-engine-setup or ovirt-hosted-engine-setup on RHEL7.5, it works well. Thanks, marking as duplicate of bug #1534197 *** This bug has been marked as a duplicate of bug 1534197 *** Tested with rhvh-4.2.4.3-0.20180614.0+1 also met this issue:
from cockpit :
"""
[ INFO ] changed: [localhost]
[ INFO ] TASK [Initialize lockspace volume]
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:05.555190", "end": "2018-06-15 15:22:10.067281", "msg": "non-zero return code", "rc": 1, "start": "2018-06-15 15:22:04.512091", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main\n \"__main__\", fname, loader, pkg_name)\n File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code\n exec code in run_globals\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request\n verbose=self.__verbose\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request\n self.send_content(h, request_body)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders\n self._send_output(message_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output\n self.send(msg)\n File \"/usr/lib64/python2.7/httplib.py\", line 843, in send\n self.connect()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect\n self.sock.connect(base64.b16decode(self.host))\n File \"/usr/lib64/python2.7/socket.py\", line 224, in meth\n return getattr(self._sock,name)(*args)\nsocket.error: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main", " \"__main__\", fname, loader, pkg_name)", " File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code", " exec code in run_globals", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request", " self.send_content(h, request_body)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders", " self._send_output(message_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output", " self.send(msg)", " File \"/usr/lib64/python2.7/httplib.py\", line 843, in send", " self.connect()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect", " self.sock.connect(base64.b16decode(self.host))", " File \"/usr/lib64/python2.7/socket.py\", line 224, in meth", " return getattr(self._sock,name)(*args)", "socket.error: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []}
"""
Log url:
http://pastebin.test.redhat.com/604553
Can you please attach vdsm.log? To update, I can't reproduce this. Is this reproducible on the CLI? (In reply to Ryan Barry from comment #11) > To update, I can't reproduce this. > > Is this reproducible on the CLI? Yes, can reproduce on the CLI. vdsm.log: http://pastebin.test.redhat.com/605603 Works well for me with the latest build:
cockpit-ovirt-dashboard-0.11.28-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.14-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch
rhvm-appliance-4.2-20180615.0.el7.noarch
rhvh-4.2.4.3-0.20180615.0+1
[root@ibm-x3650m5-06 ~]# imgbase w
You are on rhvh-4.2.4.3-0.20180615.0+1
[root@ibm-x3650m5-06 ~]# hosted-engine --vm-status
--== Host 1 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Hostname : ibm-x3650m5-06.lab.eng.pek2.redhat.com
Host ID : 1
Engine status : {"health": "good", "vm": "up", "detail": "Up"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : ecd6d40e
local_conf_timestamp : 8882
Host timestamp : 8882
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=8882 (Tue Jun 19 13:34:18 2018)
host-id=1
score=3400
vm_conf_refresh_time=8882 (Tue Jun 19 13:34:18 2018)
conf_on_shared_storage=True
maintenance=False
state=EngineUp
stopped=False
|
Description of problem: Deploy HE failed on [Initialize lockspace volume] task. """ [ INFO ] changed: [localhost] [ INFO ] TASK [Initialize lockspace volume] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:05.558402", "end": "2018-06-11 11:11:36.677061", "msg": "non-zero return code", "rc": 1, "start": "2018-06-11 11:11:31.118659", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main\n \"__main__\", fname, loader, pkg_name)\n File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code\n exec code in run_globals\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request\n verbose=self.__verbose\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request\n self.send_content(h, request_body)\n File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders\n self._send_output(message_body)\n File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output\n self.send(msg)\n File \"/usr/lib64/python2.7/httplib.py\", line 843, in send\n self.connect()\n File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect\n self.sock.connect(base64.b16decode(self.host))\n File \"/usr/lib64/python2.7/socket.py\", line 224, in meth\n return getattr(self._sock,name)(*args)\nsocket.error: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main", " \"__main__\", fname, loader, pkg_name)", " File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code", " exec code in run_globals", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 283, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 135, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1233, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1591, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1273, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1301, in single_request", " self.send_content(h, request_body)", " File \"/usr/lib64/python2.7/xmlrpclib.py\", line 1448, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 1037, in endheaders", " self._send_output(message_body)", " File \"/usr/lib64/python2.7/httplib.py\", line 881, in _send_output", " self.send(msg)", " File \"/usr/lib64/python2.7/httplib.py\", line 843, in send", " self.connect()", " File \"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 52, in connect", " self.sock.connect(base64.b16decode(self.host))", " File \"/usr/lib64/python2.7/socket.py\", line 224, in meth", " return getattr(self._sock,name)(*args)", "socket.error: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} """ Version-Release number of selected component (if applicable): rhvh-4.2.4.2-0.20180608.0+1 rhvm-appliance-4.2-20180601.0.el7.noarch ovirt-hosted-engine-setup-2.2.22-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.13-1.el7ev.noarch cockpit-ovirt-dashboard-0.11.28-1.el7ev.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy HE via cockpit Actual results: The same as the description Expected results: Deploy HE successfully from cockpit. Additional info: