Created attachment 1387825 [details] he setup log Description of problem: The HE setup fails while running on RHEL 7.5 and using gluster storage domain. Command used: ovirt-hosted-engine-setup --config-append=/var/lib/ovirt-hosted-engine-setup/answers/he-answer.conf --config-append=/usr/share/cockpit/ovirt-dashboard/gdeploy-templates/he-common.conf Error: [ INFO ] TASK [Set engine pub key as authorized key without validating the TLS/SSL certificates] [ INFO ] changed: [localhost] [ INFO ] TASK [Force host-deploy in offline mode] [ INFO ] ok: [localhost] [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Obtain SSO token using username/password credentials] [ INFO ] ok: [localhost] [ INFO ] TASK [Add host] [ INFO ] changed: [localhost] [ INFO ] TASK [Wait for the host to become non operational] [ ERROR ] Error: Failed to read response. [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 150, "changed": false, "msg": "Failed to read response."} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up Version-Release number of selected component (if applicable): How reproducible: Tried twice and faced same issue Additional info: HE setup log file attached
Sahina, was the first attempt on that host? If not it's exactly a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1539040
(In reply to Simone Tiraboschi from comment #1) > Sahina, was the first attempt on that host? > If not it's exactly a duplicate of > https://bugzilla.redhat.com/show_bug.cgi?id=1539040 So the first attempt timed out after trying to download the appliance, and then I reran the setup to complete the appliance download.
Is there a way to cleanup and try again to be sure?
(In reply to Sahina Bose from comment #4) > Is there a way to cleanup and try again to be sure? Manually delete the section between ## # Start of VDSM configuration ## and ## in /etc/sysconfig/libvirt-guests and restart libvirtd. *** This bug has been marked as a duplicate of bug 1539040 ***
Re-opening this as there was no section related to VDSM found in /etc/sysconfig/libvirt-guests The HE install is being tried on RHEL 7.5
Thanks to Simone, I progressed further on this. My initial issue was to do with the engine password being part of the FQDN. Changing that and retrying gets me to: 2018-01-30 21:16:36,361+05 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-4) [1cb6edf0] Running command: CreateUserSessionCommand internal: false. 2018-01-30 21:16:36,396+05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-4) [1cb6edf0] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from '192.168.122.1' using session 'JuGooYeg8VITZPiqwDRJ+bHdzx3H2Wuabg4IQKaMN2YNu1zI8XLaWRkAQ2zvwU/1wW2PZgq4BQjdxDScxHQ4xQ==' logged in. 2018-01-30 21:16:56,760+05 ERROR [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-7) [7772aab9-aa8a-4f60-8aed-e04f7e58917e] Failed to establish session with host 'rhsdev-grafton2-nic2.lab.eng.blr.redhat.com': SSH connection timed out connecting to 'root.eng.blr.redhat.com' 2018-01-30 21:16:56,762+05 WARN [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-7) [7772aab9-aa8a-4f60-8aed-e04f7e58917e] Validation of action 'AddVds' failed for user admin@internal-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server rhsdev-grafton2-nic2.lab.eng.blr.redhat.com,VDS_CANNOT_CONNECT_TO_SERVER 2018-01-30 21:16:56,786+05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-7) [] Operation Failed: [Cannot add Host. Connecting to host via SSH has failed, verify that the host is reachable (IP address, routable address etc.) You may refer to the engine.log file for further details.] If I login to engine VM and try, indeed the hostname is not reachable : [root@grafton-engine1 ~]# ping rhsdev-grafton2-nic2.lab.eng.blr.redhat.com PING rhsdev-grafton2-nic2.lab.eng.blr.redhat.com (10.70.40.34) 56(84) bytes of data. ^C --- rhsdev-grafton2-nic2.lab.eng.blr.redhat.com ping statistics --- 4 packets transmitted, 0 received, 100% packet loss, time 2999ms The He setup again fails with: 2018-01-30 21:22:27,229+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:75 {u'_ansible_parsed' : True, u'_ansible_no_log': False, u'changed': False, u'attempts': 150, u'invocation': {u'module_args': {u'pattern': u'name=rhsdev-gr afton2-nic2.lab.eng.blr.redhat.com', u'fetch_nested': False, u'nested_attributes': []}}, u'ansible_facts': {u'ovirt_hosts': []}} 2018-01-30 21:22:27,329+0530 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:79 fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": []}, "attempts": 150, "changed": false} 2018-01-30 21:22:27,531+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:75 PLAY RECAP [grafton-engine1.redhat.com] : ok: 15 changed: 8 unreachable: 0 skipped: 4 failed: 0 2018-01-30 21:22:27,631+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:75 PLAY RECAP [localhost] : ok: 26 changed: 12 unreachable: 0 skipped: 1 failed: 1 2018-01-30 21:22:27,732+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:161 ansible-playbook rc: 2 2018-01-30 21:22:27,732+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:168 ansible-playbook stdout: 2018-01-30 21:22:27,732+0530 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils.run:170 to retry, use: --limit @/usr/share/ovirt-hosted-engine-setup/ansible/bootstrap_local_vm.retry
(In reply to Sahina Bose from comment #7) > Thanks to Simone, I progressed further on this. My initial issue was to do > with the engine password being part of the FQDN. Changing that and retrying > gets me to: OK, let's keep this for a separate bug. Can you please open a new one?
This is substantially a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1519650 and https://github.com/ansible/ansible/issues/19278 ovirt_auth is substantially causing a failure in the tasks consuming it if the password is contained in API url. ovirt_auth from ansible 2.5 can use env variable that works also in that case.
(In reply to Simone Tiraboschi from comment #8) > (In reply to Sahina Bose from comment #7) > > Thanks to Simone, I progressed further on this. My initial issue was to do > > with the engine password being part of the FQDN. Changing that and retrying > > gets me to: > > OK, let's keep this for a separate bug. > Can you please open a new one? Done - Bug 1540451
ovirt-hosted-engine-setup installation fails on RHEL7.5: --> Finished Dependency Resolution Error: Package: sssd-common-1.15.2-50.el7_4.9.x86_64 (rhel-7.5.z) Requires: sssd-client(x86-64) = 1.15.2-50.el7_4.9 Installed: sssd-client-1.16.0-13.el7.x86_64 (@anaconda/7.5) sssd-client(x86-64) = 1.16.0-13.el7 Available: sssd-client-1.15.2-50.el7_4.9.x86_64 (rhel-7.5.z) sssd-client(x86-64) = 1.15.2-50.el7_4.9 Error: Package: libkadm5-1.15.1-8.el7.x86_64 (rhel-7.5.z) Requires: krb5-libs(x86-64) = 1.15.1-8.el7 Installed: krb5-libs-1.15.1-18.el7.x86_64 (@anaconda/7.5) krb5-libs(x86-64) = 1.15.1-18.el7 Available: krb5-libs-1.15.1-8.el7.x86_64 (rhel-7.5.z) krb5-libs(x86-64) = 1.15.1-8.el7 Error: Package: cyrus-sasl-2.1.26-21.el7.x86_64 (rhel-7.5.z) Requires: cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Installed: cyrus-sasl-lib-2.1.26-23.el7.x86_64 (@anaconda/7.5) cyrus-sasl-lib(x86-64) = 2.1.26-23.el7 Available: cyrus-sasl-lib-2.1.26-21.el7.x86_64 (rhel-7.5.z) cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Error: Package: device-mapper-multipath-0.4.9-111.el7_4.2.x86_64 (rhel-7.5.z) Requires: kpartx = 0.4.9-111.el7_4.2 Installed: kpartx-0.4.9-118.el7.x86_64 (@anaconda/7.5) kpartx = 0.4.9-118.el7 Available: kpartx-0.4.9-111.el7_4.2.x86_64 (rhel-7.5.z) Anton, please provide your input. kpartx = 0.4.9-111.el7_4.2 Error: Package: sssd-common-1.15.2-50.el7_4.9.x86_64 (rhel-7.5.z) Requires: libsss_idmap(x86-64) = 1.15.2-50.el7_4.9 Installed: libsss_idmap-1.16.0-13.el7.x86_64 (@anaconda/7.5) libsss_idmap(x86-64) = 1.16.0-13.el7 Available: libsss_idmap-1.15.2-50.el7_4.9.x86_64 (rhel-7.5.z) libsss_idmap(x86-64) = 1.15.2-50.el7_4.9 Error: Package: audit-libs-python-2.7.6-3.el7.x86_64 (rhel-7.5.z) Requires: audit-libs(x86-64) = 2.7.6-3.el7 Installed: audit-libs-2.8.1-3.el7.x86_64 (@anaconda/7.5) audit-libs(x86-64) = 2.8.1-3.el7 Available: audit-libs-2.7.6-3.el7.x86_64 (rhel-7.5.z) audit-libs(x86-64) = 2.7.6-3.el7 Error: Package: policycoreutils-python-2.5-17.1.el7.x86_64 (rhel-7.5.z) Requires: policycoreutils = 2.5-17.1.el7 Installed: policycoreutils-2.5-22.el7.x86_64 (@anaconda/7.5) policycoreutils = 2.5-22.el7 Available: policycoreutils-2.5-17.1.el7.x86_64 (rhel-7.5.z) policycoreutils = 2.5-17.1.el7 Error: Package: cyrus-sasl-scram-2.1.26-21.el7.x86_64 (rhel-7.5.z) Requires: cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Installed: cyrus-sasl-lib-2.1.26-23.el7.x86_64 (@anaconda/7.5) cyrus-sasl-lib(x86-64) = 2.1.26-23.el7 Available: cyrus-sasl-lib-2.1.26-21.el7.x86_64 (rhel-7.5.z) cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Error: Package: krb5-workstation-1.15.1-8.el7.x86_64 (rhel-7.5.z) Requires: krb5-libs(x86-64) = 1.15.1-8.el7 Installed: krb5-libs-1.15.1-18.el7.x86_64 (@anaconda/7.5) krb5-libs(x86-64) = 1.15.1-18.el7 Available: krb5-libs-1.15.1-8.el7.x86_64 (rhel-7.5.z) krb5-libs(x86-64) = 1.15.1-8.el7 Error: Package: cyrus-sasl-gssapi-2.1.26-21.el7.x86_64 (rhel-7.5.z) Requires: cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Installed: cyrus-sasl-lib-2.1.26-23.el7.x86_64 (@anaconda/7.5) cyrus-sasl-lib(x86-64) = 2.1.26-23.el7 Available: cyrus-sasl-lib-2.1.26-21.el7.x86_64 (rhel-7.5.z) cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Error: Package: dracut-fips-033-502.el7_4.1.x86_64 (rhel-7.5.z) Requires: dracut = 033-502.el7_4.1 Installed: dracut-033-523.el7.x86_64 (@anaconda/7.5) dracut = 033-523.el7 Available: dracut-033-502.el7_4.1.x86_64 (rhel-7.5.z) dracut = 033-502.el7_4.1 Error: Package: glibc-2.17-196.el7_4.2.i686 (rhel-7.5.z) Requires: glibc-common = 2.17-196.el7_4.2 Installed: glibc-common-2.17-220.el7.x86_64 (@anaconda/7.5) glibc-common = 2.17-220.el7 Available: glibc-common-2.17-196.el7_4.2.x86_64 (rhel-7.5.z) glibc-common = 2.17-196.el7_4.2
I've just tried to deploy SHE 4.2.2 on RHEL7.5 and got this dependency issue: Error: Package: libkadm5-1.15.1-8.el7.x86_64 (rhel-7.5.z) Requires: krb5-libs(x86-64) = 1.15.1-8.el7 Installed: krb5-libs-1.15.1-18.el7.x86_64 (@anaconda/7.5) krb5-libs(x86-64) = 1.15.1-18.el7 Available: krb5-libs-1.15.1-8.el7.x86_64 (rhel-7.5.z) krb5-libs(x86-64) = 1.15.1-8.el7 Error: Package: sssd-common-1.15.2-50.el7_4.11.x86_64 (rhel-7.5.z) Requires: sssd-client(x86-64) = 1.15.2-50.el7_4.11 Installed: sssd-client-1.16.0-13.el7.x86_64 (@anaconda/7.5) sssd-client(x86-64) = 1.16.0-13.el7 Available: sssd-client-1.15.2-50.el7_4.11.x86_64 (rhel-7.5.z) sssd-client(x86-64) = 1.15.2-50.el7_4.11 Error: Package: cyrus-sasl-2.1.26-21.el7.x86_64 (rhel-7.5.z) Requires: cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Installed: cyrus-sasl-lib-2.1.26-23.el7.x86_64 (@anaconda/7.5) cyrus-sasl-lib(x86-64) = 2.1.26-23.el7 Available: cyrus-sasl-lib-2.1.26-21.el7.x86_64 (rhel-7.5.z) cyrus-sasl-lib(x86-64) = 2.1.26-21.el7 Error: Package: device-mapper-multipath-0.4.9-111.el7_4.2.x86_64 (rhel-7.5.z) Requires: kpartx = 0.4.9-111.el7_4.2 Installed: kpartx-0.4.9-118.el7.x86_64 (@anaconda/7.5) kpartx = 0.4.9-118.el7 Available: kpartx-0.4.9-111.el7_4.2.x86_64 (rhel-7.5.z) kpartx = 0.4.9-111.el7_4.2 Error: Package: audit-libs-python-2.7.6-3.el7.x86_64 (rhel-7.5.z) Requires: audit-libs(x86-64) = 2.7.6-3.el7 Installed: audit-libs-2.8.1-3.el7.x86_64 (@anaconda/7.5) audit-libs(x86-64) = 2.8.1-3.el7 Available: audit-libs-2.7.6-3.el7.x86_64 (rhel-7.5.z) audit-libs(x86-64) = 2.7.6-3.el7 Error: Package: sssd-common-1.15.2-50.el7_4.11.x86_64 (rhel-7.5.z) Requires: libsss_idmap(x86-64) = 1.15.2-50.el7_4.11 Installed: libsss_idmap-1.16.0-13.el7.x86_64 (@anaconda/7.5) libsss_idmap(x86-64) = 1.16.0-13.el7 Available: libsss_idmap-1.15.2-50.el7_4.11.x86_64 (rhel-7.5.z) libsss_idmap(x86-64) = 1.15.2-50.el7_4.11
I've used part of FQDN of the engine as engine admin's password. Ansible deployment of SHE on RHEL7.5 over Gluster on these componets failed: ovirt-engine-setup-4.2.1.5-0.1.el7.noarch ovirt-hosted-engine-ha-2.2.5-1.el7ev.noarch ovirt-hosted-engine-setup-2.2.10-1.el7ev.noarch rhvm-appliance.noarch 2:4.2-20180202.0.el7 Red Hat Enterprise Linux Server release 7.5 Beta (Maipo) Linux 3.10.0-829.el7.x86_64 #1 SMP Tue Jan 9 23:06:01 EST 2018 x86_64 x86_64 x86_64 GNU/Linux [ INFO ] TASK [Get ovirtmgmt route table id] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "ip rule list | grep ovirtmgmt | sed s/\\\\[.*\\\\]\\ //g | awk '{ print $9 }'", "delta": "0:00:00.011787", "end": "2018-02-19 19:53:09.673841", "rc": 0, "start": "2018-02-19 19:53:09.662054", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Remove local vm dir] [ INFO ] changed: [localhost] [ INFO ] TASK [Notify the user about a failure] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [localhost] [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Remove local vm dir] [ INFO ] ok: [localhost] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180219195317.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180219193322-xb8oaq.log [root@alma03 ~]# date Mon Feb 19 19:54:11 IST 2018 I've opened a separate bug on that here: https://bugzilla.redhat.com/show_bug.cgi?id=1546839
Ansible deployment of SHE on RHEL7.5 over Gluster on these components passed after following this https://bugzilla.redhat.com/show_bug.cgi?id=1546839#c2 work around: ovirt-engine-setup-4.2.1.5-0.1.el7.noarch ovirt-hosted-engine-ha-2.2.5-1.el7ev.noarch ovirt-hosted-engine-setup-2.2.10-1.el7ev.noarch rhvm-appliance.noarch 2:4.2-20180202.0.el7 Red Hat Enterprise Linux Server release 7.5 Beta (Maipo) Linux 3.10.0-829.el7.x86_64 #1 SMP Tue Jan 9 23:06:01 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Moving to verified with exception of work around, as the root cause of this very bug was fixed.
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.