Bug 1379810

Summary: [ ERROR ] Engine setup failed on the appliance /var/lib/cloud/instance/scripts/runcmd: line 2: /usr/bin/engine-setup: No such file or directory
Product: [oVirt] ovirt-appliance Reporter: Martyn Hodgson <martyn.hodgson>
Component: GeneralAssignee: Fabian Deutsch <fdeutsch>
Status: CLOSED CURRENTRELEASE QA Contact: Pavel Stehlik <pstehlik>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: ---CC: bugs, didi, michal.skrivanek
Target Milestone: ---Flags: rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-28 08:24:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Martyn Hodgson 2016-09-27 17:38:39 UTC
Description of problem:
engine-setup fails:
# ovirt-hosted-engine-setup
[ ERROR ] Engine setup failed on the appliance


Version-Release number of selected component (if applicable): N/A


How reproducible:
Every time run ovirt-hosted-engine-setup


Steps to Reproduce:
1. Create server using node 4.0.3 or node 4.0.4
2. Create directory /vmengine and export
3. Install engine appliance: yum install -y ovirt-engine-appliance
4. Attempt to create appliance: ovirt-hosted-engine-setup

Actual results:
.
.
.
[ INFO  ] Running engine-setup on the appliance
          |- /var/lib/cloud/instance/scripts/runcmd: line 2: /usr/bin/engine-setup: No such file or directory
          |- HE_APPLIANCE_ENGINE_SETUP_FAIL
[ ERROR ] Engine setup failed on the appliance
[ ERROR ] Failed to execute stage 'Closing up': Engine setup failed on the appliance Please check its log on the appliance.
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160927095852.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160927095235-e0205w.log


Expected results:
[ INFO  ] Engine-setup successfully completed

[ INFO  ] Hosted Engine successfully deployed

Additional info:
# ovirt-hosted-engine-setup
/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py:15: DeprecationWarning: vdscli uses xmlrpc. since ovirt 3.6 xmlrpc is deprecated, please use vdsm.jsonrpcvdscli
  import vdsm.vdscli
[ INFO  ] Stage: Initializing
[ INFO  ] Generating a temporary VNC password.
[ INFO  ] Stage: Environment setup
          During customization use CTRL-D to abort.
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install the engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]:
          It has been detected that this program is executed through an SSH connection without using screen.
          Continuing with the installation may lead to broken installation if the network connection fails.
          It is highly recommended to abort the installation and run it inside a screen session using command "screen".
          Do you want to continue anyway? (Yes, No)[No]: yes
[ INFO  ] Hardware supports virtualization
          Configuration files: []
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160927095235-e0205w.log
          Version: otopi-1.5.2 (otopi-1.5.2-1.el7.centos)
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ INFO  ] Generating libvirt-spice certificates
[ INFO  ] Stage: Environment customization

          --== STORAGE CONFIGURATION ==--

          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: nfs4
          Please specify the full shared storage connection path to use (example: host:/path): xor-q-virt04:/vmengine
[ INFO  ] Installing on first host

          --== SYSTEM CONFIGURATION ==--


          --== NETWORK CONFIGURATION ==--

          iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]:
          Please indicate a pingable gateway IP address [10.14.50.1]:
          Please indicate a nic to set ovirtmgmt bridge on: (em4, em1, em3, em2) [em1]:

          --== VM CONFIGURATION ==--

          Booting from cdrom on RHEL7 is ISO image based only, as cdrom passthrough is disabled (BZ760885)
          Please specify the device to boot the VM from (choose disk for the oVirt engine appliance)
          (cdrom, disk, pxe) [disk]:
          Please specify the console type you would like to use to connect to the VM (vnc, spice) [vnc]:
[ INFO  ] Detecting available oVirt engine appliances
          The following appliance have been found on your system:
                [1] - The oVirt Engine Appliance image (OVA) - 4.0-20160926.1.el7.centos
                [2] - Directly select an OVA file
          Please select an appliance (1, 2) [1]:
[ INFO  ] Verifying its sha1sum
[ INFO  ] Checking OVF archive content (could take a few minutes depending on archive size)
[ INFO  ] Checking OVF XML content (could take a few minutes depending on archive size)
[WARNING] OVF does not contain a valid image description, using default.
          Would you like to use cloud-init to customize the appliance on the first boot (Yes, No)[Yes]?
          Would you like to generate on-fly a cloud-init ISO image (of no-cloud type)
          or do you have an existing one (Generate, Existing)[Generate]?
          Please provide the FQDN you would like to use for the engine appliance.
          Note: This will be the FQDN of the engine VM you are now going to launch,
          it should not point to the base host or to any other existing machine.
          Engine VM FQDN: (leave it empty to skip):  []: xor-q-virtmgr04.xpressbetonline.com
          Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]?
          Automatically restart the engine VM as a monitored service after engine-setup (Yes, No)[Yes]?
          Please provide the domain name you would like to use for the engine appliance.
          Engine VM domain: [xpressbetonline.com]
          Enter root password that will be used for the engine appliance (leave it empty to skip):
          Confirm appliance root password:
          The following CPU types are supported by this host:
                 - model_Haswell-noTSX: Intel Haswell-noTSX Family
                 - model_SandyBridge: Intel SandyBridge Family
                 - model_Westmere: Intel Westmere Family
                 - model_Nehalem: Intel Nehalem Family
                 - model_Penryn: Intel Penryn Family
                 - model_Conroe: Intel Conroe Family
          Please specify the CPU type to be used by the VM [model_Haswell-noTSX]:
          Please specify the number of virtual CPUs for the VM (Defaults to appliance OVF value): [4]:
[WARNING] Minimum requirements for disk size not met
          You may specify a unicast MAC address for the VM or accept a randomly generated default [00:16:3e:3b:7e:09]:
          Please specify the memory size of the VM in MB (Defaults to appliance OVF value): [16384]:
          How should the engine VM network be configured (DHCP, Static)[DHCP]? Static
          Please enter the IP address to be used for the engine VM [10.14.50.6]: 10.14.50.154
[ INFO  ] The engine VM will be configured to use 10.14.50.154/24
          Please provide a comma-separated list (max 3) of IP addresses of domain name servers for the engine VM
          Engine VM DNS (leave it empty to skip) [10.11.21.231]:
          Add lines for the appliance itself and for this host to /etc/hosts on the engine VM?
          Note: ensuring that this host could resolve the engine VM hostname is still up to you
          (Yes, No)[No] yes

          --== HOSTED ENGINE CONFIGURATION ==--

          Enter engine admin password:
          Confirm engine admin password:
          Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_1]: xor-q-virt04
          Please provide the name of the SMTP server through which we will send notifications [localhost]:
          Please provide the TCP port number of the SMTP server [25]:
          Please provide the email address from which notifications will be sent [root@localhost]:
          Please provide a comma-separated list of email addresses which will get notifications [root@localhost]:
[ INFO  ] Stage: Setup validation

          --== CONFIGURATION PREVIEW ==--

          Bridge interface                   : em1
          Engine FQDN                        : xor-q-virtmgr04.xpressbetonline.com
          Bridge name                        : ovirtmgmt
          Host address                       : xor-q-virt04.xpressbetonline.com
          SSH daemon port                    : 22
          Firewall manager                   : iptables
          Gateway address                    : 10.14.50.1
          Host name for web application      : xor-q-virt04
          Storage Domain type                : nfs4
          Host ID                            : 1
          Image size GB                      : 10
          Storage connection                 : xor-q-virt04:/vmengine
          Console type                       : vnc
          Memory size MB                     : 16384
          MAC address                        : 00:16:3e:3b:7e:09
          Boot type                          : disk
          Number of CPUs                     : 4
          OVF archive (for disk boot)        : /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.0-20160926.1.el7.centos.ova
          Appliance version                  : 4.0-20160926.1.el7.centos
          Restart engine VM after engine-setup: True
          CPU Type                           : model_Haswell-noTSX

          Please confirm installation settings (Yes, No)[Yes]:
[ INFO  ] Stage: Transaction setup
[ INFO  ] Stage: Misc configuration
[ INFO  ] Stage: Package installation
[ INFO  ] Stage: Misc configuration
[ INFO  ] Configuring libvirt
[ INFO  ] Configuring VDSM
[ INFO  ] Starting vdsmd
[ INFO  ] Configuring the management bridge
[ INFO  ] Creating Storage Domain
[ INFO  ] Creating Storage Pool
[ INFO  ] Connecting Storage Pool
[ INFO  ] Verifying sanlock lockspace initialization
[ INFO  ] Creating Image for 'hosted-engine.lockspace' ...
[ INFO  ] Image for 'hosted-engine.lockspace' created successfully
[ INFO  ] Creating Image for 'hosted-engine.metadata' ...
[ INFO  ] Image for 'hosted-engine.metadata' created successfully
[ INFO  ] Creating VM Image
[ INFO  ] Extracting disk image from OVF archive (could take a few minutes depending on archive size)
[ INFO  ] Validating pre-allocated volume size
[ INFO  ] Uploading volume to data domain (could take a few minutes depending on archive size)
[ INFO  ] Image successfully imported from OVF
[ INFO  ] Destroying Storage Pool
[ INFO  ] Start monitoring domain
[ INFO  ] Configuring VM
[ INFO  ] Updating hosted-engine configuration
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ INFO  ] Creating VM
          You can now connect to the VM with the following command:
                /bin/remote-viewer vnc://localhost:5900
          Use temporary password "0495oXbL" to connect to vnc console.
          Please note that in order to use remote-viewer you need to be able to run graphical applications.
          This means that if you are using ssh you have to supply the -Y flag (enables trusted X11 forwarding).
          Otherwise you can run the command from a terminal in your preferred desktop environment.
          If you cannot run graphical applications you can connect to the graphic console from another host or connect to the serial console using the following command:
          socat UNIX-CONNECT:/var/run/ovirt-vmconsole-console/c8d95ae4-a43a-44a6-8eed-a16e35874653.sock,user=ovirt-vmconsole STDIO,raw,echo=0,escape=1
          Please ensure that your Guest OS is properly configured to support serial console according to your distro documentation.
          Follow http://www.ovirt.org/Serial_Console_Setup#I_need_to_access_the_console_the_old_way for more info.
          If you need to reboot the VM you will need to start it manually using the command:
          hosted-engine --vm-start
          You can then set a temporary password using the command:
          hosted-engine --add-console-password
[ INFO  ] Running engine-setup on the appliance
          |- /var/lib/cloud/instance/scripts/runcmd: line 2: /usr/bin/engine-setup: No such file or directory
          |- HE_APPLIANCE_ENGINE_SETUP_FAIL
[ ERROR ] Engine setup failed on the appliance
[ ERROR ] Failed to execute stage 'Closing up': Engine setup failed on the appliance Please check its log on the appliance.
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20160927095852.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20160927095235-e0205w.log

Comment 1 Martyn Hodgson 2016-09-27 22:09:09 UTC
Looked at the rpms in resources.ovirt.org>pub>ovirt-4.0>rpm>el7>noarch
ovirt-engine-appliance-4.0-20160829.1.el7.centos.noarch.rpm is 1.0GB
ovirt-engine-appliance-4.0-20160926.1.el7.centos.noarch.rpm is 930.5MB
20160926 is somewhat smaller and probably missing scripts/programs. 
I downloaded 20160829 and did "rpm -ivh" and then ran ovirt-hosted-engine-setup and that created me a hosted engine.

Comment 2 Fabian Deutsch 2016-09-28 08:24:21 UTC
Thanks.

The appliance was broken due to an installation failure. We took measures to detect these installation errors on future builds.

And there is now a new build available which should work:
http://jenkins.ovirt.org/job/ovirt-appliance_ovirt-4.0_build-artifacts-el7-x86_64/40/