Bug 1871854

+++ This bug was initially created as a clone of Bug #1868571 +++

Description of problem:
Failed to deploy HE over NFS storage  "FileNotFoundError: [Errno 2] No such file or directory"

[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:00.444477", "end": "2020-08-13 10:09:31.279098", "msg": "non-zero return code", "rc": 1, "start": "2020-08-13 10:09:30.834621", "stderr": "Traceback (most recent call last):\n  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n    ha_cli.reset_lockspace(force)\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n    stats = broker.get_stats_from_storage()\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage\n    result = self._proxy.get_stats()\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__\n    return self.__send(self.__name, args)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request\n    verbose=self.__verbose\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request\n    return self.single_request(host, handler, request_body, verbose)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request\n    http_conn = self.send_request(host, handler, request_body, verbose)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request\n    self.send_content(connection, request_body)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content\n    connection.endheaders(request_body)\n  File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output\n    self.send(msg)\n  File \"/usr/lib64/python3.6/http/client.py\", line 974, in send\n    self.connect()\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect\n    self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", "  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main", "    \"__main__\", mod_spec)", "  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code", "    exec(code, run_globals)", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", "    ha_cli.reset_lockspace(force)", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", "    stats = broker.get_stats_from_storage()", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage", "    result = self._proxy.get_stats()", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__", "    return self.__send(self.__name, args)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request", "    verbose=self.__verbose", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request", "    return self.single_request(host, handler, request_body, verbose)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request", "    http_conn = self.send_request(host, handler, request_body, verbose)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request", "    self.send_content(connection, request_body)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content", "    connection.endheaders(request_body)", "  File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders", "    self._send_output(message_body, encode_chunked=encode_chunked)", "  File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output", "    self.send(msg)", "  File \"/usr/lib64/python3.6/http/client.py\", line 974, in send", "    self.connect()", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect", "    self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook


Version-Release number of selected component (if applicable):
rhvm-appliance-4.4-20200722.0.el8ev.x86_64
ovirt-hosted-engine-setup-2.4.6-1.el8ev.noarch
ovirt-hosted-engine-ha-2.4.4-1.el8ev.noarch
vdsm-4.40.25-1.el8ev.x86_64
Linux 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.2 (Ootpa)

How reproducible:
100%

Steps to Reproduce:
1.Deploy HE over NFS.

Actual results:
Fails to deploy.

Expected results:
Should succeed.

Additional info:
Logs attached from deployment.

--- Additional comment from Nikolai Sednev on 2020-08-13 07:49:05 UTC ---

Unfortunately I couldn't gather engine logs as after failed deployment engine's VM went down:
alma04 ~]# virsh -r list --all
 Id   Name   State
--------------------

--- Additional comment from Asaf Rachmani on 2020-08-17 07:37:08 UTC ---



--- Additional comment from Asaf Rachmani on 2020-08-18 12:17:10 UTC ---

From broker.log:

ainThread::INFO::2020-08-13 10:07:37,570::monitor::63::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors
MainThread::ERROR::2020-08-13 10:07:37,570::config_file::60::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker.config.he_local::(load) Configuration file '/etc/ovirt-hosted-engine/hosted-engine.conf' not available [[Errno 13] Permission denied: '/etc/ovirt-hosted-engine/hosted-engine.conf']
MainThread::WARNING::2020-08-13 10:07:37,571::storage_broker::100::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None' 
MainThread::ERROR::2020-08-13 10:07:37,571::broker::69::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Failed initializing the broker: 'metadata_image_UUID can't be 'None'
MainThread::ERROR::2020-08-13 10:07:37,572::broker::71::ovirt_hosted_engine_ha.broker.broker.Broker::(run) Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 64, in run
    self._storage_broker_instance = self._get_storage_broker()
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 143, in _get_storage_broker
    return storage_broker.StorageBroker()
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 80, in __init__
    raise_on_none=True),
  File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/env/config.py", line 87, in get
    "'{0} can't be '{1}'".format(key, val)
ValueError: 'metadata_image_UUID can't be 'None'

--- Additional comment from Nikolai Sednev on 2020-08-20 17:03:37 UTC ---

QA still have latest bits from 2020-08-18. Waiting for the build to get consumed.

--- Additional comment from Nikolai Sednev on 2020-08-23 18:08:01 UTC ---

Deployed over NFS with these components:
vdsm-4.40.26-1.el8ev.x86_64
ovirt-hosted-engine-setup-2.4.6-1.el8ev.noarch
ovirt-hosted-engine-ha-2.4.4-1.el8ev.noarch
rhvm-appliance-4.4-20200722.0.el8ev.x86_64
Linux 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux release 8.2 (Ootpa)

Deployment still failed with latest packages received from bob, which included an old ovirt-ansible-hosted-engine-setup-1.1.7-1.el8ev.noarch:

[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:00.444397", "end": "2020-08-23 21:05:08.806915", "msg": "non-zero return code", "rc": 1, "start": "2020-08-23 21:05:08.362518", "stderr": "Traceback (most recent call last):\n  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n    \"__main__\", mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n    ha_cli.reset_lockspace(force)\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n    stats = broker.get_stats_from_storage()\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage\n    result = self._proxy.get_stats()\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__\n    return self.__send(self.__name, args)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request\n    verbose=self.__verbose\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request\n    return self.single_request(host, handler, request_body, verbose)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request\n    http_conn = self.send_request(host, handler, request_body, verbose)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request\n    self.send_content(connection, request_body)\n  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content\n    connection.endheaders(request_body)\n  File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output\n    self.send(msg)\n  File \"/usr/lib64/python3.6/http/client.py\", line 974, in send\n    self.connect()\n  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect\n    self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", "  File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main", "    \"__main__\", mod_spec)", "  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code", "    exec(code, run_globals)", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", "    ha_cli.reset_lockspace(force)", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", "    stats = broker.get_stats_from_storage()", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage", "    result = self._proxy.get_stats()", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__", "    return self.__send(self.__name, args)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request", "    verbose=self.__verbose", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request", "    return self.single_request(host, handler, request_body, verbose)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request", "    http_conn = self.send_request(host, handler, request_body, verbose)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request", "    self.send_content(connection, request_body)", "  File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content", "    connection.endheaders(request_body)", "  File \"/usr/lib64/python3.6/http/client.py\", line 1249, in endheaders", "    self._send_output(message_body, encode_chunked=encode_chunked)", "  File \"/usr/lib64/python3.6/http/client.py\", line 1036, in _send_output", "    self.send(msg)", "  File \"/usr/lib64/python3.6/http/client.py\", line 974, in send", "    self.connect()", "  File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 74, in connect", "    self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []}
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook