Bug 1739881

Hosted Engine deploys fail as the below error information:
[ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": [{"address": "hp-dl388g9-05.lab.eng.pek2.redhat.com", "affinity_labels": [], "auto_numa_status": "unknown", "certificate": {"organization": "lab.eng.pek2.redhat.com", "subject": "O=lab.eng.pek2.redhat.com,CN=hp-dl388g9-05.lab.eng.pek2.redhat.com"}, "cluster": {"href": "/ovirt-engine/api/clusters/c608f04c-ca39-11e9-9f4d-5254003404b0", "id": "c608f04c-ca39-11e9-9f4d-5254003404b0"}, "comment": "", "cpu": {"speed": 0.0, "topology": {}}, "device_passthrough": {"enabled": false}, "devices": [], "external_network_provider_configurations": [], "external_status": "ok", "hardware_information": {"supported_rng_sources": []}, "hooks": [], "href": "/ovirt-engine/api/hosts/18230f52-037a-4022-b4cf-f3055f77d093", "id": "18230f52-037a-4022-b4cf-f3055f77d093", "katello_errata": [], "kdump_status": "unknown", "ksm": {"enabled": false}, "max_scheduling_memory": 0, "memory": 0, "name": "hp-dl388g9-05.lab.eng.pek2.redhat.com", "network_attachments": [], "nics": [], "numa_nodes": [], "numa_supported": false, "os": {"custom_kernel_cmdline": ""}, "permissions": [], "port": 54321, "power_management": {"automatic_pm_enabled": true, "enabled": false, "kdump_detection": true, "pm_proxies": []}, "protocol": "stomp", "se_linux": {}, "spm": {"priority": 5, "status": "none"}, "ssh": {"fingerprint": "SHA256:DyOGZj6XqW8UhH5pqEWbtlFrdEhcc5lfanKwuXEUWIo", "port": 22}, "statistics": [], "status": "install_failed", "storage_connection_extensions": [], "summary": {"total": 0}, "tags": [], "transparent_huge_pages": {"enabled": false}, "type": "ovirt_node", "unmanaged_networks": [], "update_available": false, "vgpu_placement": "consolidated"}]}, "attempts": 120, "changed": false}

ovirt-hosted-engine-setup-ansible-bootstrap_local_vm-xxxx.log
2019-08-29 16:52:13,724+0800 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Always revoke the SSO token', 'ansible_result': u"type: <type 'dict'>\nstr: {'_ansible_no_log': False, u'invocation': {u'module_args': {u'username': None, u'hostname': None, u'ca_file': None, u'url': None, u'insecure': None, u'kerberos': False, u'compress': True, u'token': None, u'headers': None, u'state': u'absent', u'ovirt_auth': {u'failed': False, u'attempts': 1, u'chang", 'task_duration': 1, 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml'}
2019-08-29 17:09:37,333+0800 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Wait for the host to be up', 'ansible_result': u"type: <type 'dict'>\nstr: {'attempts': 120, '_ansible_no_log': False, u'invocation': {u'module_args': {u'all_content': False, u'pattern': u'name=hp-dl388g9-05.lab.eng.pek2.**FILTERED**.com', u'fetch_nested': False, u'cluster_version': None, u'nested_attributes': []}}, u'changed': False, u'ansible_facts': {u'ovirt_hosts': [{u'comme", 'task_duration': 1037, 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml'}
2019-08-29 17:10:39,126+0800 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Notify the user about a failure', 'ansible_result': u"type: <type 'dict'>\nstr: {'msg': u'The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\\n', 'changed': False, '_ansible_no_log': False}", 'task_duration': 0, 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml'}


engine.log
2019-08-29 16:52:18,055+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [7374dfae] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host hp-dl388g9-05.lab.eng.pek2.redhat.com: Failed to execute stage 'Misc configuration': expected string or buffer.

2019-08-29 16:52:18,805+08 ERROR [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [7374dfae] SSH error running command root.eng.pek2.redhat.com:'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x &&  "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True': IOException: Command returned failure code 1 during SSH session 'root.eng.pek2.redhat.com'
2019-08-29 16:52:18,806+08 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [7374dfae] Error during host hp-dl388g9-05.lab.eng.pek2.redhat.com install
2019-08-29 16:52:18,811+08 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [7374dfae] Host installation failed for host '18230f52-037a-4022-b4cf-f3055f77d093', 'hp-dl388g9-05.lab.eng.pek2.redhat.com': Command returned failure code 1 during SSH session 'root.eng.pek2.redhat.com'

2019-08-29 16:52:18,866+08 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [7374dfae] EVENT_ID: VDS_INSTALL_FAILED(505), Host hp-dl388g9-05.lab.eng.pek2.redhat.com installation failed. Command returned failure code 1 during SSH session 'root.eng.pek2.redhat.com'.

Test Version
rhvh-4.3.6.2-0.20190828.0
cockpit-machines-ovirt-195-1.el7.noarch
cockpit-195-1.el7.x86_64
cockpit-dashboard-195-1.el7.x86_64
cockpit-bridge-195-1.el7.x86_64
cockpit-storaged-195-1.el7.noarch
cockpit-ovirt-dashboard-0.13.7-1.el7ev.noarch
cockpit-system-195-1.el7.noarch
cockpit-ws-195-1.el7.x86_64
ovirt-hosted-engine-ha-2.3.5-1.el7ev.noarch
ovirt-hosted-engine-setup-2.3.12-1.el7ev.noarch
ovirt-ansible-engine-setup-1.1.9-1.el7ev.noarch
ovirt-ansible-hosted-engine-setup-1.0.27-1.el7ev.noarch
ovirt-ansible-repositories-1.1.5-1.el7ev.noarch
rhvm-appliance-4.3-20190828.0.el7.x86_64

Test Steps
1. Deploy Hosted engine via cockpit-ovirt

Test Result:
Hosted engine deploy failed. (Logs are in attachment.)

Move bug to "ASSIGNED"

Created attachment 1609308 [details]
Log files

we tried to deploy hosted engine on RHVH-4.3.6-20190828 but failed with the same error in host deploy log:

2019-09-01 12:47:12,066+0300 DEBUG otopi.context context._executeMethod:127 Stage misc METHOD otopi.plugins.ovirt_host_deploy.kernel.kernel.Plugin._misc
2019-09-01 12:47:12,066+0300 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND       **%EventStart STAGE misc METHOD otopi.plugins.ovirt_host_deploy.kernel.kernel.Plugin._misc (None)
2019-09-01 12:47:12,066+0300 DEBUG otopi.context context._executeMethod:145 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-dkPCLXgxSR/pythonlib/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/tmp/ovirt-dkPCLXgxSR/otopi-plugins/ovirt-host-deploy/kernel/kernel.py", line 185, in _misc
    self._add_disk_uuid_in_fips()
  File "/tmp/ovirt-dkPCLXgxSR/otopi-plugins/ovirt-host-deploy/kernel/kernel.py", line 148, in _add_disk_uuid_in_fips
    fips_m = self._RE_FIPS.match(self.environment[e_key])
TypeError: expected string or buffer
2019-09-01 12:47:12,067+0300 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Misc configuration': expected string or buffer

I didn't tried RHVH but tried and succeeded (today) with CentOS-7 both on master and 4.3.
The failure you experience is not related to this bug.

The bug and its fix are related to the web-ui and not to the deployment process itself.
Moving back to ON_QA. Please open a new bug with a 'regression' keyword.

(In reply to Ido Rosenzwig from comment #4)
> I didn't tried RHVH but tried and succeeded (today) with CentOS-7 both on
> master and 4.3.
> The failure you experience is not related to this bug.
> 
> The bug and its fix are related to the web-ui and not to the deployment
> process itself.
> Moving back to ON_QA. Please open a new bug with a 'regression' keyword.

opened:
https://bugzilla.redhat.com/show_bug.cgi?id=1747787

Since RHVH detect same bug BZ-1747787, QE need to verify this bug with next build.

Test Version:
rhvh-4.3.6.2-0.20190904.0
cockpit-system-195-1.el7.noarch
cockpit-195-1.el7.x86_64
cockpit-machines-ovirt-195-1.el7.noarch
cockpit-ovirt-dashboard-0.13.8-1.el7ev.noarch
cockpit-dashboard-195-1.el7.x86_64
cockpit-bridge-195-1.el7.x86_64
cockpit-ws-195-1.el7.x86_64
cockpit-storaged-195-1.el7.noarch
ovirt-hosted-engine-ha-2.3.5-1.el7ev.noarch
ovirt-hosted-engine-setup-2.3.12-1.el7ev.noarch
rhvm-appliance-4.3-20190904.0.el7.x86_64

Test Steps:
1. Clean install RHVH
2. Deploy hosted engine via cockpit-ovirt

Result:
Hosted engine deploys successfully, and no error with UI

bug is fixed, move to "VERIFIED"

This bugzilla is included in oVirt 4.3.6 release, published on September 26th 2019.

Since the problem described in this bug report should be
resolved in oVirt 4.3.6 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.