Description of problem: alma04 ~]# hosted-engine --deploy --ansible [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup During customization use CTRL-D to abort. Continuing will configure this host for serving as hypervisor and create a local VM with a running engine. The locally running engine will be used to configure a storage domain and create a VM there. At the end the disk of the local VM will be moved to the shared storage. Are you sure you want to continue? (Yes, No)[Yes]: It has been detected that this program is executed through an SSH connection without using screen. Continuing with the installation may lead to broken installation if the network connection fails. It is highly recommended to abort the installation and run it inside a screen session using command "screen". Do you want to continue anyway? (Yes, No)[No]: yes Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180304145426-0at70c.log Version: otopi-1.7.7 (otopi-1.7.7-1.el7ev) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization --== STORAGE CONFIGURATION ==-- --== HOST NETWORK CONFIGURATION ==-- Please indicate a pingable gateway IP address [10.35.95.254]: [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Get all active network interfaces] [ INFO ] TASK [filter bonds with bad naming] Please indicate a nic to set ovirtmgmt bridge on: (enp3s0f0) [enp3s0f0]: --== VM CONFIGURATION ==-- If you want to deploy with a custom engine appliance image, please specify the path to the OVA archive you would like to use (leave it empty to skip, the setup will use rhvm-appliance rpm installing it if missing): [ INFO ] Detecting host timezone. Please provide the FQDN you would like to use for the engine appliance. Note: This will be the FQDN of the engine VM you are now going to launch, it should not point to the base host or to any other existing machine. Engine VM FQDN: (leave it empty to skip): []: nsednev-he-2.qa.lab.tlv.redhat.com Please provide the domain name you would like to use for the engine appliance. Engine VM domain: [qa.lab.tlv.redhat.com] Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? Automatically restart the engine VM as a monitored service after engine-setup (Yes, No)[Yes]? Enter root password that will be used for the engine appliance: Confirm appliance root password: Enter ssh public key for the root user that will be used for the engine appliance (leave it empty to skip): [WARNING] Skipping appliance root ssh public key Do you want to enable ssh access for the root user (yes, no, without-password) [yes]: Please specify the number of virtual CPUs for the VM (Defaults to appliance OVF value): [4]: Please specify the memory size of the VM in MB (Defaults to appliance OVF value): [16384]: You may specify a unicast MAC address for the VM or accept a randomly generated default [00:16:3e:03:19:f3]: 00:16:3e:7b:b8:54 How should the engine VM network be configured (DHCP, Static)[DHCP]? Add lines for the appliance itself and for this host to /etc/hosts on the engine VM? Note: ensuring that this host could resolve the engine VM hostname is still up to you (Yes, No)[No] --== HOSTED ENGINE CONFIGURATION ==-- Please provide the name of the SMTP server through which we will send notifications [localhost]: Please provide the TCP port number of the SMTP server [25]: Please provide the email address from which notifications will be sent [root@localhost]: Please provide a comma-separated list of email addresses which will get notifications [root@localhost]: Enter engine admin password: Confirm engine admin password: [ INFO ] Stage: Setup validation [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Cleaning previous attempts [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Stop libvirt] [ INFO ] changed: [localhost] [ INFO ] TASK [Drop vdsm config statements] [ INFO ] TASK [Restore initial abrt config files] [ INFO ] TASK [Restart abrtd] [ INFO ] changed: [localhost] [ INFO ] TASK [Drop libvirt sasl2 configuration by vdsm] [ INFO ] changed: [localhost] [ INFO ] TASK [Stop and disable services] [ INFO ] TASK [Start libvirt] [ INFO ] changed: [localhost] [ INFO ] TASK [Check for leftover local engine VM] [ INFO ] changed: [localhost] [ INFO ] TASK [Destroy leftover local engine VM] [ INFO ] skipping: [localhost] [ INFO ] TASK [Check for leftover defined local engine VM] [ INFO ] changed: [localhost] [ INFO ] TASK [Undefine leftover local engine VM] [ INFO ] skipping: [localhost] [ INFO ] TASK [Remove eventually entries for the local VM from known_hosts file] [ INFO ] ok: [localhost] [ INFO ] Starting local VM [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Create dir for local vm] [ INFO ] changed: [localhost] [ INFO ] TASK [Set local vm dir path] [ INFO ] ok: [localhost] [ INFO ] TASK [Fix local vm dir permission] [ INFO ] changed: [localhost] [ INFO ] TASK [Start libvirt] [ INFO ] ok: [localhost] [ INFO ] TASK [Check status of default libvirt network] [ INFO ] changed: [localhost] [ INFO ] TASK [Activate default libvirt network] [ INFO ] skipping: [localhost] [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Install rhvm-appliance rpm] [ INFO ] changed: [localhost] [ INFO ] TASK [Parse appliance configuration for path] [ INFO ] changed: [localhost] [ INFO ] TASK [Parse appliance configuration for sha1sum] [ INFO ] changed: [localhost] [ INFO ] TASK [Get OVA path] [ INFO ] ok: [localhost] [ INFO ] TASK [Compute sha1sum] [ INFO ] ok: [localhost] [ INFO ] TASK [Compare sha1sum] [ INFO ] skipping: [localhost] [ INFO ] TASK [Register appliance PATH] [ INFO ] skipping: [localhost] [ INFO ] TASK [Extract appliance to local vm dir] [ INFO ] changed: [localhost] [ INFO ] TASK [Find the appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [Get appliance disk size] [ INFO ] changed: [localhost] [ INFO ] TASK [Parse qemu-img output] [ INFO ] ok: [localhost] [ INFO ] TASK [Create cloud init user-data and meta-data files] [ INFO ] TASK [Create iso disk] [ INFO ] changed: [localhost] [ INFO ] TASK [Create local vm] [ INFO ] changed: [localhost] [ INFO ] TASK [Get local vm ip] [ INFO ] changed: [localhost] [ INFO ] TASK [Remove eventually entries for the local VM from /etc/hosts] [ INFO ] ok: [localhost] [ INFO ] TASK [Create an entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [Wait for ssh to restart on the engine VM] [ INFO ] ok: [localhost -> localhost] [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Wait for the engine VM] [ INFO ] ok: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Add an entry for this host on /etc/hosts on the engine VM] [ INFO ] changed: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Set FDQN] [ INFO ] changed: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Force the engine VM FDQN to resolve on 127.0.0.1] [ INFO ] changed: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Restore sshd reverse DNS lookups] [ INFO ] changed: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Generate an answer file for engine-setup] [ INFO ] changed: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Include before engine-setup custom tasks files for the engine VM] [ INFO ] TASK [Execute engine-setup] [ INFO ] changed: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Include before engine-setup custom tasks files for the engine VM] [ INFO ] TASK [Configure LibgfApi support] [ INFO ] skipping: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Restart the engine for LibgfApi support] [ INFO ] skipping: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Mask services to speed up future bootstraps] [ INFO ] TASK [Clean up boostrap answer file] [ INFO ] changed: [nsednev-he-2.qa.lab.tlv.redhat.com] [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Wait for engine to start] [ INFO ] ok: [localhost] [ INFO ] TASK [Detect VLAN ID] [ INFO ] changed: [localhost] [ INFO ] TASK [Set engine pub key as authorized key without validating the TLS/SSL certificates] [ INFO ] changed: [localhost] [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Obtain SSO token using username/password credentials] [ INFO ] ok: [localhost] [ INFO ] TASK [Set VLAN ID at datacenter level] [ INFO ] skipping: [localhost] [ INFO ] TASK [Force host-deploy in offline mode] [ INFO ] changed: [localhost] [ INFO ] TASK [Add host] [ INFO ] changed: [localhost] [ INFO ] TASK [Wait for the engine to start host install process] [ INFO ] ok: [localhost] [ INFO ] TASK [Wait for the management bridge to appear on the host] [ INFO ] changed: [localhost] [ INFO ] TASK [Wait for the host to be up] [ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": []}, "attempts": 12, "changed": false} [ INFO ] TASK [Check host install result] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fix network configuration if the host is still not up"} [ INFO ] TASK [Get virbr0 routing configuration] [ INFO ] changed: [localhost] [ INFO ] TASK [Get management bridge route table id] [ INFO ] changed: [localhost] [ INFO ] TASK [Check network configuration] [ INFO ] changed: [localhost] [ INFO ] TASK [Clean network configuration] [ INFO ] skipping: [localhost] [ INFO ] TASK [Restore network configuration] [ INFO ] changed: [localhost] [ INFO ] TASK [Wait for the host to be up] [ INFO ] ok: [localhost] [ INFO ] TASK [Remove host-deploy configuration file] [ INFO ] changed: [localhost] Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: Please specify the nfs version you would like to use (auto, v3, v4, v4_1)[auto]: Please specify the full shared storage connection path to use (example: host:/path): yellow-vdsb.qa.lab.tlv.redhat.com:/Compute_NFS/nsednev_he_2 If needed, specify additional mount options for the connection to the hosted-engine storagedomain []: rsize=32768,wsize=32768 [ INFO ] Creating Storage Domain [ INFO ] TASK [Gathering Facts] [ INFO ] ok: [localhost] [ INFO ] TASK [include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [Obtain SSO token using username/password credentials] [ INFO ] ok: [localhost] [ INFO ] TASK [Fetch host facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Fetch cluster id] [ INFO ] ok: [localhost] [ INFO ] TASK [Fetch cluster facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Fetch datacenter facts] [ INFO ] ok: [localhost] [ INFO ] TASK [Fetch datacenter id] [ INFO ] ok: [localhost] [ INFO ] TASK [Fetch datacenter_name] [ INFO ] ok: [localhost] [ INFO ] TASK [Add nfs storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Cannot add storage server connection when Host status is not up]\". HTTP response code is 409."} Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs)[nfs]: Version-Release number of selected component (if applicable): ovirt-hosted-engine-setup-2.2.11-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.6-1.el7ev.noarch Linux 3.10.0-858.el7.x86_64 #1 SMP Tue Feb 27 08:59:23 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.5 (Maipo) How reproducible: 100% Steps to Reproduce: 1.Deploy SHE over NFS on RHEL7.5, using additional mount options 2. 3. Actual results: Deployment fails over NFS Expected results: Deployment should succeed. Additional info: Sosreport from host is attached.
Created attachment 1403774 [details] sosreport from alma04
Are you sure it's related to NFS and not to the previous failure: [ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": []}, "attempts": 12, "changed": false} [ INFO ] TASK [Check host install result] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fix network configuration if the host is still not up"} Any idea why it failed there?
(In reply to Yaniv Kaul from comment #2) > Are you sure it's related to NFS and not to the previous failure: > [ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": > []}, "attempts": 12, "changed": false} > [ INFO ] TASK [Check host install result] > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fix > network configuration if the host is still not up"} > > Any idea why it failed there? It might be related, I don't know. I've reported these two bugs separately, to cover both deployment flows separately for NFS and iSCSI. Network configuration was not changed, I really have no idea why I've got this error.
I think that what you got is just a side effect of https://bugzilla.redhat.com/show_bug.cgi?id=1549642 Nikolai, could you please try reproducing this on an host configured with a static IP?
Happens also on statically configured host's NIC: [ ERROR ] fatal: [localhost]: FAILED! => {"ansible_facts": {"ovirt_hosts": []}, "attempts": 12, "changed": false} [ INFO ] TASK [Check host install result] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fix network configuration if the host is still not up"} [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."}
Comment #5 was tested on these components: ovirt-hosted-engine-setup-2.2.12-1.el7ev.noarch ovirt-hosted-engine-ha-2.2.6-1.el7ev.noarch rhvm-appliance-4.2-20180202.0.el7.noarch Linux 3.10.0-858.el7.x86_64 #1 SMP Tue Feb 27 08:59:23 EST 2018 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.5 (Maipo)
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
The hosted engine was using DHCP, it does not matter how the host NIC is configured. This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1549642 *** This bug has been marked as a duplicate of bug 1549642 ***