Created attachment 1711263 [details] logs from alma04 Description of problem: Failed to deploy HE over NFS storage "FileNotFoundError: [Errno 2] No such file or directory" [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:00.444477", "end": "2020-08-13 10:09:31.279098", "msg": "non-zero return code", "rc": 1, "start": "2020-08-13 10:09:30.834621", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n \"__main__\", mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request\n verbose=self.__verbose\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request\n http_conn = self.send_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request\n self.send_content(connection, request_body)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.6/http/client.py\", line 974, in send\n self.connect()\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect\n self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main", " \"__main__\", mod_spec)", " File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code", " exec(code, run_globals)", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request", " http_conn = self.send_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request", " self.send_content(connection, request_body)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders", " self._send_output(message_body, encode_chunked=encode_chunked)", " File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output", " self.send(msg)", " File \"/usr/lib64/python3.6/http/client.py\", line 974, in send", " self.connect()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect", " self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook Version-Release number of selected component (if applicable): rhvm-appliance-4.4-20200722.0.el8ev.x86_64 ovirt-hosted-engine-setup-2.4.6-1.el8ev.noarch ovirt-hosted-engine-ha-2.4.4-1.el8ev.noarch vdsm-4.40.25-1.el8ev.x86_64 Linux 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux release 8.2 (Ootpa) How reproducible: 100% Steps to Reproduce: 1.Deploy HE over NFS. Actual results: Fails to deploy. Expected results: Should succeed. Additional info: Logs attached from deployment.
Unfortunately I couldn't gather engine logs as after failed deployment engine's VM went down: alma04 ~]# virsh -r list --all Id Name State --------------------
*** Bug 1869173 has been marked as a duplicate of this bug. ***
From broker.log: ainThread::INFO::2020-08-13 10:07:37,570::monitor::63::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::ERROR::2020-08-13 10:07:37,570::config_file::60::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker.config.he_local::(load) Configuration file '/etc/ovirt-hosted-engine/hosted-engine.conf' not available [[Errno 13] Permission denied: '/etc/ovirt-hosted-engine/hosted-engine.conf'] MainThread::WARNING::2020-08-13 10:07:37,571::storage_broker::100::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None' MainThread::ERROR::2020-08-13 10:07:37,571::broker::69::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Failed initializing the broker: 'metadata_image_UUID can't be 'None' MainThread::ERROR::2020-08-13 10:07:37,572::broker::71::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 64, in run self._storage_broker_instance = self._get_storage_broker() File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 143, in _get_storage_broker return storage_broker.StorageBroker() File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 80, in __init__ raise_on_none=True), File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 87, in get "'{0} can't be '{1}'".format(key, val) ValueError: 'metadata_image_UUID can't be 'None'
QA still have latest bits from 2020-08-18. Waiting for the build to get consumed.
Deployed over NFS with these components: vdsm-4.40.26-1.el8ev.x86_64 ovirt-hosted-engine-setup-2.4.6-1.el8ev.noarch ovirt-hosted-engine-ha-2.4.4-1.el8ev.noarch rhvm-appliance-4.4-20200722.0.el8ev.x86_64 Linux 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux release 8.2 (Ootpa) Deployment still failed with latest packages received from bob, which included an old ovirt-ansible-hosted-engine-setup-1.1.7-1.el8ev.noarch: [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:00.444397", "end": "2020-08-23 21:05:08.806915", "msg": "non-zero return code", "rc": 1, "start": "2020-08-23 21:05:08.362518", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n \"__main__\", mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request\n verbose=self.__verbose\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request\n http_conn = self.send_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request\n self.send_content(connection, request_body)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.6/http/client.py\", line 974, in send\n self.connect()\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect\n self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main", " \"__main__\", mod_spec)", " File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code", " exec(code, run_globals)", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request", " http_conn = self.send_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request", " self.send_content(connection, request_body)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders", " self._send_output(message_body, encode_chunked=encode_chunked)", " File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output", " self.send(msg)", " File \"/usr/lib64/python3.6/http/client.py\", line 974, in send", " self.connect()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect", " self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
Hosted Engine successfully deployed Worked with these components: ovirt-ansible-hosted-engine-setup-1.1.8-1.el8ev.noarch ovirt-hosted-engine-setup-2.4.6-1.el8ev.noarch ovirt-hosted-engine-ha-2.4.4-1.el8ev.noarch Red Hat Enterprise Linux release 8.2 (Ootpa) Linux 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
This bugzilla is included in oVirt 4.4.2 release, published on September 17th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.2 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.