Created attachment 1173303 [details] all_log Description of problem: Hosted Engine host installation failed during the first time activation, user need retry several times to active the host can make the host up, this is inconvenient. Make a selection from the options below: (1) Continue setup - oVirt-Engine installation is ready and ovirt-engine service is up (2) Abort setup (3) Power off and restart the VM (4) Destroy VM and abort setup (1, 2, 3, 4)[1]: 1 Checking for oVirt-Engine status at cshaohe.redhat.com... [ INFO ] Engine replied: DB Up!Welcome to Health Status! [ INFO ] Acquiring internal CA cert from the engine [ INFO ] The following CA certificate is going to be used, please immediately interrupt if not correct: [ INFO ] Issuer: C=US, O=redhat.com, CN=cshaohe.redhat.com.90033, Subject: C=US, O=redhat.com, CN=cshaohe.redhat.com.90033, Fingerprint (SHA-1): 387B9A7B4DC060B5D3B65C43AE385D13307730A9 [ INFO ] Connecting to the Engine Enter the name of the cluster to which you want to add the host (Default) [Default]: [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... [ INFO ] Still waiting for VDSM host to become operational... [ INFO ] Still waiting for VDSM host to become operational... The host hosted_engine_1 is in non-operational state. Please try to activate it via the engine webadmin UI. Retry checking host status or ignore this and continue (Retry, Ignore)[Retry]? [ INFO ] The VDSM Host is now operational [ INFO ] Saving hosted-engine configuration on the shared storage domain Please shutdown the VM allowing the system to launch it as a monitored service. The system will wait until the VM is down. Version-Release number of selected component (if applicable): rhev-hypervisor7-7.2-20160624.1 rhev-hypervisor7-7.2-20160627.3 ovirt-node-3.6.1-13.0.el7ev.noarch ovirt-hosted-engine-ha-1.3.5.7-1.el7ev.noarch ovirt-node-plugin-hosted-engine-0.3.0-7.el7ev.noarch ovirt-hosted-engine-setup-1.3.7.2-1.el7ev.noarch rhevm-appliance-20160620.0-1.el7ev.ova How reproducible: 100% Steps to Reproduce: 1. Install rhev-hypervisor7-7.2-20160624.1. 2. Deploy Host Engine step by step. 3. Enter the name of the cluster to which you want to add the host[Default]: Input default. 4. Focus on the process. Actual results: Hosted Engine host installation failed during the first time activation, user need retry several times to active the host can make the host up, Expected results: Hosted Engine host installation successful on the first time active. Additional info:
Created attachment 1173304 [details] he-first-time
Created attachment 1173305 [details] he-sereral-times
We did not encounter this issue in previous vintage RHEV-H 3.6.z released build, only on RHEV-H 3.6.7 build.
Hi Simone, Could you please see this report? Thanks
Based on the logs I can see that it is recovery isssue: Thread-16::DEBUG::2016-06-28 07:49:36,842::bindingxmlrpc::1269::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Recovering from crash or Initializing', 'code': 99}} This is exactly what we see in BZ #1350763. There is assumption that #1350763 and are related to #1348103. I would make this as duplicated of #1350763.
On regular RHEL7.2 HE deployment using rhevm-appliance-20160620.0-1.el7ev.noarch everything works like a charm: [root@alma03 ~]# [root@alma03 ~]# hosted-engine --deploy [ INFO ] Stage: Initializing [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup Continuing will configure this host for serving as hypervisor and create a VM where you have to install the engine afterwards. Are you sure you want to continue? (Yes, No)[Yes]: Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160629165957-u74dp3.log Version: otopi-1.4.2 (otopi-1.4.2-1.el7ev) It has been detected that this program is executed through an SSH connection without using screen. Continuing with the installation may lead to broken installation if the network connection fails. It is highly recommended to abort the installation and run it inside a screen session using command "screen". Do you want to continue anyway? (Yes, No)[No]: yes [ INFO ] Hardware supports virtualization [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Waiting for VDSM hardware info [ INFO ] Generating libvirt-spice certificates [ INFO ] Stage: Environment customization --== STORAGE CONFIGURATION ==-- During customization use CTRL-D to abort. Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: Please specify the full shared storage connection path to use (example: host:/path): ^[[A^[[A^[[A10.35.64.11:/vol/RHEV/Virt/nsednev_3_6_ [ ERROR ] Please specify value Please specify the full shared storage connection path to use (example: host:/path): 10.35.64.11:/vol/RHEV/Virt/nsednev_3_6_HE_2 [ INFO ] Installing on first host --== SYSTEM CONFIGURATION ==-- --== NETWORK CONFIGURATION ==-- Please indicate a nic to set ovirtmgmt bridge on: (enp3s0f1, eno1, eno2, enp3s0f0) [enp3s0f1]: enp3s0f0 iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]: Please indicate a pingable gateway IP address [10.35.117.254]: --== VM CONFIGURATION ==-- Booting from cdrom on RHEL7 is ISO image based only, as cdrom passthrough is disabled (BZ760885) Please specify the device to boot the VM from (choose disk for the oVirt engine appliance) (cdrom, disk, pxe) [disk]: [ INFO ] Detecting available oVirt engine appliances The following appliance have been found on your system: [1] - The RHEV-M Appliance image (OVA) - 20160620.0-1.el7ev [2] - Directly select an OVA file Please select an appliance (1, 2) [1]: [ INFO ] Verifying its sha1sum [ INFO ] Checking OVF archive content (could take a few minutes depending on archive size) [ INFO ] Checking OVF XML content (could take a few minutes depending on archive size) [WARNING] OVF does not contain a valid image description, using default. Would you like to use cloud-init to customize the appliance on the first boot (Yes, No)[Yes]? Would you like to generate on-fly a cloud-init ISO image (of no-cloud type) or do you have an existing one (Generate, Existing)[Generate]? Please provide the FQDN you would like to use for the engine appliance. Note: This will be the FQDN of the engine VM you are now going to launch, it should not point to the base host or to any other existing machine. Engine VM FQDN: (leave it empty to skip): []: nsednev-he-2.qa.lab.tlv.redhat.com Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? Automatically restart the engine VM as a monitored service after engine-setup (Yes, No)[Yes]? Please provide the domain name you would like to use for the engine appliance. Engine VM domain: [qa.lab.tlv.redhat.com] Enter root password that will be used for the engine appliance (leave it empty to skip): Confirm appliance root password: How should the engine VM network should be configured (DHCP, Static)[DHCP]? Add lines for the appliance itself and for this host to /etc/hosts on the engine VM? Note: ensuring that this host could resolve the engine VM hostname is still up to you (Yes, No)[No] yes The following CPU types are supported by this host: - model_SandyBridge: Intel SandyBridge Family - model_Westmere: Intel Westmere Family - model_Nehalem: Intel Nehalem Family - model_Penryn: Intel Penryn Family - model_Conroe: Intel Conroe Family Please specify the CPU type to be used by the VM [model_SandyBridge]: Please specify the number of virtual CPUs for the VM [Defaults to appliance OVF value: 2]: 4 You may specify a unicast MAC address for the VM or accept a randomly generated default [00:16:3e:1a:3b:97]: 00:16:3E:7B:BB:BB Please specify the memory size of the VM in MB [Defaults to appliance OVF value: 4096]: Please specify the console type you would like to use to connect to the VM (vnc, spice) [vnc]: --== HOSTED ENGINE CONFIGURATION ==-- Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_1]: alma03.qa.lab.tlv.redhat.com Enter 'admin@internal' user password that will be used for accessing the Administrator Portal: Confirm 'admin@internal' user password: Please provide the name of the SMTP server through which we will send notifications [localhost]: smtp.redhat.com Please provide the TCP port number of the SMTP server [25]: Please provide the email address from which notifications will be sent [root@localhost]: nsednevhe2 Please provide a comma-separated list of email addresses which will get notifications [root@localhost]: nsednev [ INFO ] Stage: Setup validation --== CONFIGURATION PREVIEW ==-- Bridge interface : enp3s0f0 Engine FQDN : nsednev-he-2.qa.lab.tlv.redhat.com Bridge name : ovirtmgmt Host address : alma03.qa.lab.tlv.redhat.com SSH daemon port : 22 Firewall manager : iptables Gateway address : 10.35.117.254 Host name for web application : alma03.qa.lab.tlv.redhat.com Storage Domain type : nfs3 Host ID : 1 Image size GB : 50 GlusterFS Share Name : hosted_engine_glusterfs GlusterFS Brick Provisioning : False Storage connection : 10.35.64.11:/vol/RHEV/Virt/nsednev_3_6_HE_2 Console type : vnc Memory size MB : 4096 MAC address : 00:16:3E:7B:BB:BB Boot type : disk Number of CPUs : 4 OVF archive (for disk boot) : /usr/share/ovirt-engine-appliance/rhevm-appliance-20160620.0-1.el7ev.ova Restart engine VM after engine-setup: True CPU Type : model_SandyBridge Please confirm installation settings (Yes, No)[Yes]: [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Configuring libvirt [ INFO ] Configuring VDSM [ INFO ] Starting vdsmd [ INFO ] Waiting for VDSM hardware info [ INFO ] Configuring the management bridge [ INFO ] Creating Storage Domain [ INFO ] Creating Storage Pool [ INFO ] Connecting Storage Pool [ INFO ] Verifying sanlock lockspace initialization [ INFO ] Creating Image for 'hosted-engine.lockspace' ... [ INFO ] Image for 'hosted-engine.lockspace' created successfully [ INFO ] Creating Image for 'hosted-engine.metadata' ... [ INFO ] Image for 'hosted-engine.metadata' created successfully [ INFO ] Creating VM Image [ INFO ] Extracting disk image from OVF archive (could take a few minutes depending on archive size) [ INFO ] Validating pre-allocated volume size [ INFO ] Uploading volume to data domain (could take a few minutes depending on archive size) [ INFO ] Image successfully imported from OVF [ INFO ] Destroying Storage Pool [ INFO ] Start monitoring domain [ INFO ] Configuring VM [ INFO ] Updating hosted-engine configuration [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Creating VM You can now connect to the VM with the following command: /bin/remote-viewer vnc://localhost:5900 Use temporary password "1461snBO" to connect to vnc console. Please note that in order to use remote-viewer you need to be able to run graphical applications. This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding). Otherwise you can run the command from a terminal in your preferred desktop environment. If you cannot run graphical applications you can connect to the graphic console from another host or connect to the serial console using the following command: socat UNIX-CONNECT:/var/run/ovirt-vmconsole-console/4e0218bc-4384-4ed9-86b2-ed595fda3c44.sock,user=ovirt-vmconsole STDIO,raw,echo=0,escape=1 Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation. Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way for more info. If you need to reboot the VM you will need to start it manually using the command: hosted-engine --vm-start You can then set a temporary password using the command: hosted-engine --add-console-password [ INFO ] Running engine-setup on the appliance |- [ INFO ] Stage: Initializing |- [ INFO ] Stage: Environment setup |- Configuration files: ['/etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf', '/etc/ovirt-engine-setup.conf.d/10-packaging.conf', '/root/ovirt-engine-answers', '/root/heanswers.conf'] |- Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20160629101226-ddchui.log |- Version: otopi-1.4.2 (otopi-1.4.2-1.el6ev) |- [ INFO ] Stage: Environment packages setup |- [ INFO ] Stage: Programs detection |- [ INFO ] Stage: Environment setup |- [ INFO ] Stage: Environment customization |- |- --== PRODUCT OPTIONS ==-- |- |- |- --== PACKAGES ==-- |- |- |- --== ALL IN ONE CONFIGURATION ==-- |- |- |- --== NETWORK CONFIGURATION ==-- |- |- [ INFO ] iptables will be configured as firewall manager. |- |- --== DATABASE CONFIGURATION ==-- |- |- |- --== OVIRT ENGINE CONFIGURATION ==-- |- |- |- --== STORAGE CONFIGURATION ==-- |- |- |- --== PKI CONFIGURATION ==-- |- |- |- --== APACHE CONFIGURATION ==-- |- |- |- --== SYSTEM CONFIGURATION ==-- |- |- |- --== MISC CONFIGURATION ==-- |- |- |- --== END OF CONFIGURATION ==-- |- |- [ INFO ] Stage: Setup validation |- [WARNING] Less than 16384MB of memory is available |- |- --== CONFIGURATION PREVIEW ==-- |- |- Application mode : virt |- Default SAN wipe after delete : False |- Firewall manager : iptables |- Update Firewall : True |- Host FQDN : nsednev-he-2.qa.lab.tlv.redhat.com |- Engine database secured connection : False |- Engine database host : localhost |- Engine database user name : engine |- Engine database name : engine |- Engine database port : 5432 |- Engine database host name validation : False |- Engine installation : True |- PKI organization : qa.lab.tlv.redhat.com |- Configure local Engine database : True |- Set application as default page : True |- Configure Apache SSL : True |- Configure VMConsole Proxy : True |- Engine Host FQDN : nsednev-he-2.qa.lab.tlv.redhat.com |- Configure WebSocket Proxy : True |- [ INFO ] Stage: Transaction setup |- [ INFO ] Stopping engine service |- [ INFO ] Stopping ovirt-fence-kdump-listener service |- [ INFO ] Stopping websocket-proxy service |- [ INFO ] Stage: Misc configuration |- [ INFO ] Stage: Package installation |- [ INFO ] Stage: Misc configuration |- [ INFO ] Initializing PostgreSQL |- [ INFO ] Creating PostgreSQL 'engine' database |- [ INFO ] Configuring PostgreSQL |- [ INFO ] Creating/refreshing Engine database schema |- [ INFO ] Creating/refreshing Engine 'internal' domain database schema |- [ INFO ] Upgrading CA |- [ INFO ] Creating CA |- [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts |- [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts |- [ INFO ] Configuring WebSocket Proxy |- [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' |- [ INFO ] Stage: Transaction commit |- [ INFO ] Stage: Closing up |- |- --== SUMMARY ==-- |- |- [WARNING] Less than 16384MB of memory is available |- SSH fingerprint: 3d:17:f1:b4:7a:5d:3d:fe:09:a7:85:67:43:eb:72:96 |- Internal CA 25:EC:68:11:9B:6B:3F:B2:A6:24:57:EA:AE:FF:BA:0F:39:56:3F:03 |- Note! If you want to gather statistical information you can install Reports and/or DWH: |- http://nsednev-he-2.qa.lab.tlv.redhat.com:80/ovirt-engine/docs/manual/en_US/html/Installation_Guide/chap-History_and_Reports.html |- Web access is enabled at: |- http://nsednev-he-2.qa.lab.tlv.redhat.com:80/ovirt-engine |- https://nsednev-he-2.qa.lab.tlv.redhat.com:443/ovirt-engine |- Please use the user 'admin@internal' and password specified in order to login |- |- --== END OF SUMMARY ==-- |- |- [ INFO ] Starting engine service |- [ INFO ] Restarting httpd |- [ INFO ] Restarting ovirt-vmconsole proxy service |- [ INFO ] Stage: Clean up |- Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20160629101226-ddchui.log |- [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20160629101457-setup.conf' |- [ INFO ] Stage: Pre-termination |- [ INFO ] Stage: Termination |- [ INFO ] Execution of setup completed successfully |- HE_APPLIANCE_ENGINE_SETUP_SUCCESS [ INFO ] Engine-setup successfully completed [ INFO ] Engine is still unreachable [ INFO ] Engine is still not reachable, waiting... [ INFO ] Engine is still unreachable [ INFO ] Engine is still not reachable, waiting... [ INFO ] Engine replied: DB Up!Welcome to Health Status! [ INFO ] Acquiring internal CA cert from the engine [ INFO ] The following CA certificate is going to be used, please immediately interrupt if not correct: [ INFO ] Issuer: C=US, O=qa.lab.tlv.redhat.com, CN=nsednev-he-2.qa.lab.tlv.redhat.com.69253, Subject: C=US, O=qa.lab.tlv.redhat.com, CN=nsednev-he-2.qa.lab.tlv.redhat.com.69253, Fingerprint (SHA-1): 25EC68119B6B3FB2A62457EAAEFFBA0F39563F03 [ INFO ] Connecting to the Engine [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... [ INFO ] Still waiting for VDSM host to become operational... [ INFO ] The VDSM Host is now operational [ INFO ] Saving hosted-engine configuration on the shared storage domain [ INFO ] Shutting down the engine VM [ INFO ] Enabling and starting HA services [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160629171754.conf' [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ INFO ] Hosted Engine successfully set up [root@alma03 ~]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : True Hostname : alma03.qa.lab.tlv.redhat.com Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "powering up"} Score : 3400 stopped : False Local maintenance : False crc32 : 2629d720 Host timestamp : 7167 [root@alma03 ~]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : True Hostname : alma03.qa.lab.tlv.redhat.com Host ID : 1 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "up", "detail": "powering up"} Score : 3400 stopped : False Local maintenance : False crc32 : 2629d720 Host timestamp : 7167 [root@alma03 ~]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : True Hostname : alma03.qa.lab.tlv.redhat.com Host ID : 1 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : 56837a45 Host timestamp : 7245 [root@alma03 ~]# [root@alma03 ~]# [root@alma03 ~]# [root@alma03 ~]#
Created attachment 1174001 [details] deployment of HE from host alma03
Host's components: ovirt-host-deploy-1.4.1-1.el7ev.noarch qemu-kvm-rhev-2.3.0-31.el7_2.17.x86_64 ovirt-vmconsole-1.0.2-2.el7ev.noarch vdsm-4.17.31-0.el7ev.noarch ovirt-vmconsole-host-1.0.2-2.el7ev.noarch ovirt-hosted-engine-ha-1.3.5.7-1.el7ev.noarch libvirt-client-1.2.17-13.el7_2.5.x86_64 sanlock-3.2.4-2.el7_2.x86_64 ovirt-hosted-engine-setup-1.3.7.2-1.el7ev.noarch mom-0.5.4-1.el7ev.noarch ovirt-setup-lib-1.0.1-1.el7ev.noarch Linux version 3.10.0-327.28.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon Jun 27 14:48:28 EDT 2016 Linux 3.10.0-327.28.2.el7.x86_64 #1 SMP Mon Jun 27 14:48:28 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) Engine: rhevm-websocket-proxy-3.6.7.5-0.1.el6.noarch rhevm-spice-client-x64-cab-3.6-7.el6.noarch rhevm-setup-plugins-3.6.5-1.el6ev.noarch rhevm-doc-3.6.7-1.el6eng.noarch rhevm-branding-rhev-3.6.0-10.el6ev.noarch rhevm-restapi-3.6.7.5-0.1.el6.noarch rhevm-3.6.7.5-0.1.el6.noarch rhevm-dependencies-3.6.0-1.el6ev.noarch rhevm-tools-backup-3.6.7.5-0.1.el6.noarch rhevm-spice-client-x86-cab-3.6-7.el6.noarch rhevm-tools-3.6.7.5-0.1.el6.noarch rhevm-vmconsole-proxy-helper-3.6.7.5-0.1.el6.noarch rhevm-sdk-python-3.6.7.0-1.el6ev.noarch rhevm-guest-agent-common-1.0.11-6.el6ev.noarch rhevm-image-uploader-3.6.0-1.el6ev.noarch rhevm-log-collector-3.6.1-1.el6ev.noarch rhevm-setup-plugin-ovirt-engine-common-3.6.7.5-0.1.el6.noarch rhevm-spice-client-x86-msi-3.6-7.el6.noarch rhevm-userportal-3.6.7.5-0.1.el6.noarch rhevm-backend-3.6.7.5-0.1.el6.noarch rhevm-setup-plugin-vmconsole-proxy-helper-3.6.7.5-0.1.el6.noarch rhev-release-3.6.7-6-001.noarch rhevm-lib-3.6.7.5-0.1.el6.noarch rhevm-setup-base-3.6.7.5-0.1.el6.noarch rhevm-webadmin-portal-3.6.7.5-0.1.el6.noarch rhevm-dbscripts-3.6.7.5-0.1.el6.noarch rhevm-setup-plugin-ovirt-engine-3.6.7.5-0.1.el6.noarch rhevm-extensions-api-impl-3.6.7.5-0.1.el6.noarch rhev-guest-tools-iso-3.6-6.el6ev.noarch rhevm-setup-plugin-websocket-proxy-3.6.7.5-0.1.el6.noarch rhevm-spice-client-x64-msi-3.6-7.el6.noarch rhevm-setup-3.6.7.5-0.1.el6.noarch rhevm-iso-uploader-3.6.0-1.el6ev.noarch rhevm-cli-3.6.2.1-1.el6ev.noarch Linux version 2.6.32-642.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Wed Apr 13 00:51:26 EDT 2016 Linux 2.6.32-642.el6.x86_64 #1 SMP Wed Apr 13 00:51:26 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.8 (Santiago)
Thanks Piotr, based on comment#5, closing this bug as duplicate. *** This bug has been marked as a duplicate of bug 1350763 ***
Test version: rhev-hypervisor6-6.8-20160630.2.iso ovirt-node-3.2.3-32.el6.noarch ovirt-hosted-engine-ha-1.2.10-1.el6ev.noarch ovirt-hosted-engine-setup-1.2.6.1-1.el6ev.noarch 20160222.0-1.3.5.ova RHEVM-3.5.8-0.1.el6ev Test result: 1. Still met this issue during deploy HE. 2. Deploy additional host always failed, it report "Fail to configure manager network on the host". I am not sure why rhev-h 6.8 build have this issue.