Bug 1254556 - Failed to deploy HE over RHEL7.2 since connection to storage server failed
Summary: Failed to deploy HE over RHEL7.2 since connection to storage server failed
Keywords:
Status: CLOSED DUPLICATE of bug 1253939
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup
Version: 3.6.0
Hardware: x86_64
OS: Linux
low
high
Target Milestone: ---
: 3.6.0
Assignee: Sandro Bonazzola
QA Contact: Nikolai Sednev
URL:
Whiteboard: integration
Depends On: 1256223
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-08-18 11:50 UTC by Nikolai Sednev
Modified: 2015-12-14 15:39 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-09-04 14:59:10 UTC
oVirt Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
all_logs from the deployment (136.96 KB, application/x-gzip)
2015-08-18 11:50 UTC, Nikolai Sednev
no flags Details
answers-20150825190518.conf (3.07 KB, text/plain)
2015-08-25 16:10 UTC, Nikolai Sednev
no flags Details
ovirt-hosted-engine-setup-20150825190115-68txgo.log (308.16 KB, text/plain)
2015-08-25 16:16 UTC, Nikolai Sednev
no flags Details
messages (443.89 KB, text/plain)
2015-08-25 16:19 UTC, Nikolai Sednev
no flags Details

Description Nikolai Sednev 2015-08-18 11:50:43 UTC
Created attachment 1064296 [details]
all_logs from the deployment

Description of problem:
[ ERROR ] Failed to execute stage 'Misc configuration': Connection to storage server failed
     
Failed to deploy HE over RHEL7.2 | ERROR otopi.context context._executeMethod:**FILTERED**64 Failed to execute stage 'Misc configuration' : Connection to storage server failed

[root@alma02 ~]# hosted-engine --deploy
[ INFO  ] Stage: Initializing          
[ INFO  ] Generating a temporary VNC password.
[ INFO  ] Stage: Environment setup            
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]:                                                                              
          Configuration files: []                                                                                                         
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20150818134341-uu51nq.log                                
          Version: otopi-1.4.0_master (otopi-1.4.0-0.0.2.master.el7ev)                                                                    
          It has been detected that this program is executed through an SSH connection without using screen.                              
          Continuing with the installation may lead to broken installation if the network connection fails.                               
          It is highly recommended to abort the installation and run it inside a screen session using command "screen".                   
          Do you want to continue anyway? (Yes, No)[No]: yes                                                                              
[ INFO  ] Hardware supports virtualization                                                                                                
[ INFO  ] Stage: Environment packages setup                                                                                               
[ INFO  ] Stage: Programs detection                                                                                                       
[ INFO  ] Stage: Environment setup                                                                                                        
[ INFO  ] Waiting for VDSM hardware info                                                                                                  
[ INFO  ] Generating libvirt-spice certificates                                                                                           
[WARNING] Cannot locate gluster packages, Hyper Converged setup support will be disabled.                                                 
[ INFO  ] Please abort the setup and install vdsm-gluster, gluster-server >= 3.7.2 and restart vdsmd service in order to gain Hyper Converged setup support.                                                                                                                                                          
[ INFO  ] Stage: Environment customization                                                                                                                 
                                                                                                                                                           
          --== STORAGE CONFIGURATION ==--                                                                                                                  
                                                                                                                                                           
          During customization use CTRL-D to abort.                                                                                                        
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]:                                                       
          Please specify the full shared storage connection path to use (example: host:/path): 10.35.160.108:/Virt/nsednev_HE_3_5_z_3                      
[ INFO  ] Installing on first host                                                                                                                         
          Please provide storage domain name. [hosted_storage]:                                                                                            
          Local storage datacenter name is an internal name                                                                                                
          and currently will not be shown in engine's admin UI.                                                                                            
          Please enter local datacenter name [hosted_datacenter]:                                                                                          
                                                                                                                                                           
          --== SYSTEM CONFIGURATION ==--                                                                                                                   
                                                                                                                                                           
                                                                                                                                                           
          --== NETWORK CONFIGURATION ==--                                                                                                                  
                                                                                                                                                           
          Please indicate a nic to set ovirtmgmt bridge on: (enp3s0f1, eno1, eno2, enp3s0f0) [enp3s0f1]: enp3s0f0                                          
          iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]:                                                       
          Please indicate a pingable gateway IP address [10.35.117.254]:                                                                                   
                                                                                                                                                           
          --== VM CONFIGURATION ==--                                                                                                                       
                                                                                                                                                           
          Please specify the device to boot the VM from (choose disk for the oVirt engine appliance)                                                       
          (cdrom, disk, pxe) [disk]: pxe                                                                                                                   
          Please specify an alias for the Hosted Engine image [hosted_engine]:                                                                             
          The following CPU types are supported by this host:                                                                                              
                 - model_SandyBridge: Intel SandyBridge Family                                                                                             
                 - model_Westmere: Intel Westmere Family                                                                                                   
                 - model_Nehalem: Intel Nehalem Family                                                                                                     
                 - model_Penryn: Intel Penryn Family                                                                                                       
                 - model_Conroe: Intel Conroe Family                                                                                                       
          Please specify the CPU type to be used by the VM [model_SandyBridge]:                                                                            
          Please specify the number of virtual CPUs for the VM [Defaults to minimum requirement: 2]:                                                       
          Please specify the disk size of the VM in GB [Defaults to minimum requirement: 25]:                                                              
          You may specify a unicast MAC address for the VM or accept a randomly generated default [00:16:3e:49:a1:37]: 00:16:3E:7B:B8:53                   
          Please specify the memory size of the VM in MB [Defaults to minimum requirement: 4096]:                                                          
          Please specify the console type you would like to use to connect to the VM (vnc, spice) [vnc]:                                                   
                                                                                                                                                           
          --== HOSTED ENGINE CONFIGURATION ==--                                                                                                            
                                                                                                                                                           
          Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_1]:                                       
          Enter 'admin@internal' user password that will be used for accessing the Administrator Portal:                                                   
          Confirm 'admin@internal' user password:                                                                                                          
          Please provide the FQDN for the engine you would like to use.                                                                                    
          This needs to match the FQDN that you will use for the engine installation within the VM.                                                        
          Note: This will be the FQDN of the VM you are now going to create,                                                                               
          it should not point to the base host or to any other existing machine.                                                                           
          Engine FQDN: nsednev-he-1.qa.lab.tlv.redhat.com                                                                                                  
          Please provide the name of the SMTP server through which we will send notifications [localhost]:                                                 
          Please provide the TCP port number of the SMTP server [25]:                                                                                      
          Please provide the email address from which notifications will be sent [root@localhost]:                                                         
          Please provide a comma-separated list of email addresses which will get notifications [root@localhost]:                                          
[ INFO  ] Stage: Setup validation                                                                                                                          
                                                                                                                                                           
          --== CONFIGURATION PREVIEW ==--                                                                                                                  
                                                                                                                                                           
          Bridge interface                   : enp3s0f0                                                                                                    
          Engine FQDN                        : nsednev-he-1.qa.lab.tlv.redhat.com                                                                          
          Bridge name                        : ovirtmgmt                                                                                                   
          SSH daemon port                    : 22                                                                                                          
          Firewall manager                   : iptables                                                                                                    
          Gateway address                    : 10.35.117.254                                                                                               
          Host name for web application      : hosted_engine_1                                                                                             
          Host ID                            : 1                                                                                                           
          Image alias                        : hosted_engine                                                                                               
          Image size GB                      : 25                                                                                                          
          GlusterFS Share Name               : hosted_engine_glusterfs                                                                                     
          GlusterFS Brick Provisioning       : False                                                                                                       
          Storage connection                 : 10.35.160.108:/Virt/nsednev_HE_3_5_z_3                                                                      
          Console type                       : vnc                                                                                                         
          Memory size MB                     : 4096                                                                                                        
          MAC address                        : 00:16:3E:7B:B8:53                                                                                           
          Boot type                          : pxe                                                                                                         
          Number of CPUs                     : 2                                                                                                           
          CPU Type                           : model_SandyBridge                                                                                           
                                                                                                                                                           
          Please confirm installation settings (Yes, No)[Yes]:                                                                                             
[ INFO  ] Stage: Transaction setup                                                                                                                         
[ INFO  ] Stage: Misc configuration                                                                                                                        
[ INFO  ] Stage: Package installation                                                                                                                      
[ INFO  ] Stage: Misc configuration                                                                                                                        
[ INFO  ] Configuring libvirt                                                                                                                              
[ INFO  ] Configuring VDSM                                                                                                                                 
[ INFO  ] Starting vdsmd                                                                                                                                   
[ INFO  ] Waiting for VDSM hardware info                                                                                                                   
[ INFO  ] Waiting for VDSM hardware info                                                                                                                   
[ INFO  ] Configuring the management bridge                                                                                                                
[ ERROR ] Failed to execute stage 'Misc configuration': Connection to storage server failed                                                                
[ INFO  ] Stage: Clean up                                                                                                                                  
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20150818135026.conf'                                                  
[ INFO  ] Stage: Pre-termination                                                                                                                           
[ INFO  ] Stage: Termination 

Version-Release number of selected component (if applicable):
Linux version 3.10.0-304.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Fri Aug 7 09:53:04 EDT 2015

How reproducible:
100%

Steps to Reproduce:
1.Install clean RHEL7.2 OS with all required repos.
2.Install ovirt-vmconsole-proxy
3.Install ovirt-hosted-engine-setup
4.hosted-engine --deploy

Actual results:
HE deployment fails over NFS volume.

Expected results:
HE deployment over NFS volume should pass.

Additional info:
Logs attached

Comment 1 Michal Skrivanek 2015-08-18 14:43:36 UTC
well, what about those name resolution failures?
puppet misconfig?

Comment 2 Nikolai Sednev 2015-08-19 06:37:06 UTC
(In reply to Michal Skrivanek from comment #1)
> well, what about those name resolution failures?
> puppet misconfig?

Which names?
If you're talking about nsednev FQDN, it's OK.
puppet has no connection to the issue, it worked OK before and I don't see any connection to it.
I have these FQDNS in usage for years, they're reserved in DHCP following their MAC addresses and DNS records.

Comment 4 Nikolai Sednev 2015-08-25 16:10:46 UTC
Created attachment 1066923 [details]
answers-20150825190518.conf

Comment 5 Nikolai Sednev 2015-08-25 16:12:24 UTC
Tried to redeploy the ovirt-hosted-engine-setup-1.3.0-0.4.beta.git42eb801.el7ev.noarch HE over fresh installation of the RHEL7.2 and failed again with: 
[ ERROR ] Failed to execute stage 'Misc configuration': Connection to storage server failed

Comment 6 Nikolai Sednev 2015-08-25 16:16:26 UTC
Created attachment 1066925 [details]
ovirt-hosted-engine-setup-20150825190115-68txgo.log

Comment 7 Nikolai Sednev 2015-08-25 16:18:20 UTC
Aug 25 19:05:16 alma02 daemonAdapter: libvirt: Network Driver error : Network not found: no network with matching name 'vdsm-ovirtmgmt'


See the attached /var/log/messages

Comment 8 Nikolai Sednev 2015-08-25 16:19:06 UTC
Created attachment 1066940 [details]
messages

Comment 9 Sandro Bonazzola 2015-09-04 14:59:10 UTC
Aug 25 19:05:11 alma02 puppet-agent[36400]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve file metadata for puppet://foreman-proxy.qa.lab.tlv.redhat.com/plugins: getaddrinfo: Name or service not known

still in attachment #1066940 [details].
Please try to reproduce on a host without puppet / foreman.

Lowering severity and priority and marking as unconfirmed since we can't reproduce on our systems.

Please reopen if you can reproduce on a system without foreman / puppet.

Comment 10 Nikolai Sednev 2015-09-06 12:34:27 UTC
Please close this one as duplicate to https://bugzilla.redhat.com/show_bug.cgi?id=1253939 

The problem is not connected to Foreman or Puppet, it's vdsm create ovirtmgmt bridge with DEFROUTE=no.
It's default route being removed from the rhevm-bridge during HE-deployment.

Comment 12 Sandro Bonazzola 2015-12-14 15:39:47 UTC
Closing duplicate according to comment #10

*** This bug has been marked as a duplicate of bug 1253939 ***


Note You need to log in before you can comment on or make changes to this bug.