Bug 1275571

Summary: Failed to deploy HE3.6 via TUI on RHEVH7.2 due to network management bridge creation failure.
Product: Red Hat Enterprise Virtualization Manager Reporter: Nikolai Sednev <nsednev>
Component: ovirt-hosted-engine-setupAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED DUPLICATE QA Contact: Ilanit Stein <istein>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.6.1CC: cshao, cwu, dfediuck, ecohen, fdeutsch, gklein, huiwa, lsurette, mavital, nsednev, stirabos, ycui
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: node
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-03 13:08:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1269176    
Attachments:
Description Flags
logs from the host (black)
none
logs from black 2_11_15
none
sosreport black 2_11_15 none

Description Nikolai Sednev 2015-10-27 09:30:25 UTC
Description of problem:
On clean installation on Red Hat Enterprise Virtualization Hypervisor release 7.2 (20151025.0.el7ev) I ran as shown bellow and failed with the error:

[ INFO  ] Stage: Initializing
[ INFO  ] Generating a temporary VNC password.
[ INFO  ] Stage: Environment setup
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]:           Configuration files: ['/tmp/tmpAM2ccM']
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20151026152224-k4x1eo.log
          Version: otopi-1.4.0 (otopi-1.4.0-1.el7ev)
[ INFO  ] Hardware supports virtualization
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ ERROR ] Failed to execute stage 'Environment setup': Failed to reconfigure libvirt for VDSM
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20151026152232.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
Something went wrong setting up hosted engine, or the setup process was cancelled.

Press any key to continue...

If I'm following these steps then:
[root@black-vdsb ~]# vdsm-tool configure --force

Checking configuration status...

Current revision of multipath.conf detected, preserving
libvirt is already configured for vdsm
SUCCESS: ssl configured to true. No conflicts

Running configure...
checking certs..
Reconfiguration of libvirt is done.

Done configuring modules to VDSM.

I'm getting the same results:
[ INFO  ] Stage: Initializing
[ INFO  ] Generating a temporary VNC password.
[ INFO  ] Stage: Environment setup
          Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards.
          Are you sure you want to continue? (Yes, No)[Yes]: 
          Configuration files: ['/tmp/tmpRaZH19']
          Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20151026153500-e9t177.log
          Version: otopi-1.4.0 (otopi-1.4.0-1.el7ev)
[ INFO  ] Hardware supports virtualization
[ INFO  ] Stage: Environment packages setup
[ INFO  ] Stage: Programs detection
[ INFO  ] Stage: Environment setup
[ ERROR ] Failed to execute stage 'Environment setup': Failed to start service 'vdsmd'
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20151026153512.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
Something went wrong setting up hosted engine, or the setup process was cancelled.

Press any key to continue...

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Cleanly install RHEVH7.2 (20151025.0.el7ev) on host.
2.Deploy HE on iSCSI via TUI.
3.

Actual results:
Network management bridge creation fails and customer gets disconnected.

Expected results:
deployment should pass successfully.

Additional info:

Please see the logs I'm attached.

Comment 1 Nikolai Sednev 2015-10-27 09:31:35 UTC
Created attachment 1086765 [details]
logs from the host (black)

Comment 2 Nikolai Sednev 2015-10-27 09:32:30 UTC
Created this bug to separate from the https://bugzilla.redhat.com/show_bug.cgi?id=1269176 issue.

Comment 3 Doron Fediuck 2015-10-28 09:17:13 UTC
Tagging tentatively as node.

1. Are you sure this is not a duplicate of bug 1275371?
2. The attached logs are irrelevant. We need the setup log file and the matching
   vdsm log file.

Comment 4 Fabian Deutsch 2015-11-02 13:37:06 UTC
Nikolai, can you please provide all the contents of /var/log?

Comment 5 Nikolai Sednev 2015-11-02 16:57:48 UTC
Failed with the error:
                [5]     3514f0c5a51600824       60GiB   XtremIO XtremApp
                        status: free, paths: 1 active
         
          Please select the destination LUN (1, 2, 3, 4, 5) [1]: 5
[ INFO  ] Installing on first host
          Please provide storage domain name. [hosted_storage]: 
          Local storage datacenter name is an internal name
          and currently will not be shown in engine's admin UI.
          Please enter local datacenter name [hosted_datacenter]: 
         
          --== SYSTEM CONFIGURATION ==--
         
         
          --== NETWORK CONFIGURATION ==--
         
          Please indicate a nic to set ovirtmgmt bridge on: (enp4s0, enp6s0) [enp4s0]: 
          iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]: 
          Please indicate a pingable gateway IP address [10.35.64.254]: 
         
          --== VM CONFIGURATION ==--
         
[ ERROR ] Failed to execute stage 'Environment customization': The provided boot type is not supported. Please try again
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20151102164451.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
Something went wrong setting up hosted engine, or the setup process was cancelled.

Press any key to continue...

I've checked the network interfaces and saw that network management bridge was not created:
[root@black-vdsb ~]# ifconfig
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.35.64.51  netmask 255.255.255.0  broadcast 10.35.64.255
        inet6 fe80::221:5eff:fe3f:dc80  prefixlen 64  scopeid 0x20<link>
        ether 00:21:5e:3f:dc:80  txqueuelen 1000  (Ethernet)
        RX packets 26208  bytes 23983401 (22.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10064  bytes 1313546 (1.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 3890  bytes 1843481 (1.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3890  bytes 1843481 (1.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Comment 6 Nikolai Sednev 2015-11-02 16:59:08 UTC
Created attachment 1088644 [details]
logs from black 2_11_15

Comment 7 Nikolai Sednev 2015-11-02 17:06:38 UTC
Created attachment 1088646 [details]
sosreport black 2_11_15

Comment 8 Nikolai Sednev 2015-11-02 17:08:41 UTC
Adding the components from RHEVH7.2 (20151025.0.el7ev):
ovirt-node-selinux-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-node-plugin-cim-logic-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-node-plugin-hosted-engine-0.3.0-1.el7ev.noarch
ovirt-node-plugin-cim-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-node-branding-rhev-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-host-deploy-1.4.0-1.el7ev.noarch
ovirt-vmconsole-1.0.0-1.el7ev.noarch
ovirt-node-lib-config-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-node-lib-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-node-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
libvirt-1.2.17-13.el7.x86_64
ovirt-node-plugin-snmp-logic-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
mom-0.5.1-1.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7.x86_64
vdsm-4.17.10-5.el7ev.noarch
ovirt-hosted-engine-setup-1.3.0-1.el7ev.noarch
ovirt-node-plugin-vdsm-0.6.1-1.el7ev.noarch
ovirt-node-plugin-snmp-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-vmconsole-host-1.0.0-1.el7ev.noarch
ovirt-node-lib-legacy-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
ovirt-hosted-engine-ha-1.3.1-1.el7ev.noarch
ovirt-host-deploy-offline-1.4.0-1.el7ev.x86_64
ovirt-node-plugin-rhn-3.3.0-0.18.20151022git82dc52c.el7ev.noarch
sanlock-3.2.4-1.el7.x86_64
ovirt-setup-lib-1.0.0-1.el7ev.noarch

Comment 9 Nikolai Sednev 2015-11-02 17:15:37 UTC
Looks like something wrong with the "Network Filter Driver error : Network filter not found: no nwfilter with matching name 'vdsm-no-mac-spoofing'".
I saw also these in /var/log/messages:
Nov  2 15:59:13 localhost virt-who: Using libvirt url: ""
Nov  2 15:59:13 localhost journal: libvirt version: 1.2.17, package: 13.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-10-08-09:11:06, x86-035.build.eng.bos.redhat.com)
Nov  2 16:00:30 localhost vdsmd_init_common.sh: libvirt is already configured for vdsm
Nov  2 16:00:45 localhost vdsmd_init_common.sh: libvirt: Network Filter Driver error : Network filter not found: no nwfilter with matching name 'vdsm-no-mac-spoofing'
Nov  2 16:01:02 localhost journal: libvirt version: 1.2.17, package: 13.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-10-08-09:11:06, x86-035.build.eng.bos.redhat.com)
Nov  2 16:04:20 localhost vdsmd_init_common.sh: libvirt is already configured for vdsm
Nov  2 15:59:12 localhost virt-who: No configurations found, using libvirt as backend
Nov  2 15:59:12 localhost virt-who: Using configuration "env/cmdline" ("libvirt" mode)

Comment 10 Nikolai Sednev 2015-11-02 17:18:27 UTC
Adding the status of libvirt:
# systemctl status libvirtd.service -l
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
           └─unlimited-core.conf
   Active: active (running) since Mon 2015-11-02 16:27:00 UTC; 50min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 2835 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─2835 /usr/sbin/libvirtd --listen

Nov 02 16:26:59 black-vdsb.qa.lab.tlv.redhat.com systemd[1]: Starting Virtualization daemon...
Nov 02 16:27:00 black-vdsb.qa.lab.tlv.redhat.com systemd[1]: Started Virtualization daemon.
Nov 02 17:02:33 black-vdsb.qa.lab.tlv.redhat.com libvirtd[2835]: libvirt version: 1.2.17, package: 13.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2015-10-08-09:11:06, x86-035.build.eng.bos.redhat.com)
Nov 02 17:02:33 black-vdsb.qa.lab.tlv.redhat.com libvirtd[2835]: End of file while reading data: Input/output error

Comment 11 Simone Tiraboschi 2015-11-03 11:31:32 UTC
(In reply to Nikolai Sednev from comment #5)
>           --== VM CONFIGURATION ==--
>          
> [ ERROR ] Failed to execute stage 'Environment customization': The provided
> boot type is not supported. Please try again

Here there was something wrong with the answerfile generated by node:
2015-11-02 16:44:51 DEBUG otopi.context context.dumpEnvironment:510 ENV OVEHOSTED_VM/vmBoot=str:'None'

But if I'm not wrong it has been recently fixed; looking for that.

The original bug seams different.

Comment 12 Simone Tiraboschi 2015-11-03 13:02:45 UTC
(In reply to Simone Tiraboschi from comment #11)
> (In reply to Nikolai Sednev from comment #5)
> >           --== VM CONFIGURATION ==--
> >          
> > [ ERROR ] Failed to execute stage 'Environment customization': The provided
> > boot type is not supported. Please try again
> 
> Here there was something wrong with the answerfile generated by node:
> 2015-11-02 16:44:51 DEBUG otopi.context context.dumpEnvironment:510 ENV
> OVEHOSTED_VM/vmBoot=str:'None'
> 
> But if I'm not wrong it has been recently fixed; looking for that:

Yes, we fixed it there:
https://bugzilla.redhat.com/show_bug.cgi?id=1271976

Still checking the original bug on opening comment.

Comment 13 Simone Tiraboschi 2015-11-03 13:08:38 UTC
(In reply to Simone Tiraboschi from comment #12)
> Still checking the original bug on opening comment.

Sorry, no other logs here for the original issue.
Feel free to reopen if you have logs with a different issue.

*** This bug has been marked as a duplicate of bug 1271976 ***