Bug 990517 - setupNetworks | Failed to create rhevm bridge when adding new host
setupNetworks | Failed to create rhevm bridge when adding new host
Product: ovirt-host-deploy
Classification: oVirt
Component: Plugins.VDSM (Show other bugs)
x86_64 Linux
medium Severity high (vote)
: ---
: 1.1.0
Assigned To: Assaf Muller
Meni Yakove
: Regression
Depends On:
  Show dependency treegraph
Reported: 2013-07-31 07:16 EDT by Meni Yakove
Modified: 2016-02-10 14:54 EST (History)
11 users (show)

See Also:
Fixed In Version: is9
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-01-21 17:18:23 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
pm-rhel: blocker+
lpeer: devel_ack+

Attachments (Terms of Use)
engine and vdsm logs (5.35 MB, application/x-xz)
2013-07-31 07:16 EDT, Meni Yakove
no flags Details

External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 17596 None None None Never

  None (edit)
Description Meni Yakove 2013-07-31 07:16:44 EDT
Created attachment 781041 [details]
engine and vdsm logs

Description of problem:
When adding new host to cluster the installation failed and the host become non-operational because setupNetworks fail to configure rhevm bridge on the host.
Attach rhevm to ethX from setupNetworks dialog solve the problem and the host is UP.

Version-Release number of selected component (if applicable):

Steps to Reproduce:
1.add new host to cluster

Actual results:
Host is  non-operational and rhevm bridge creation fails

Expected results:
Host is up with rhevm bridge
Comment 1 Assaf Muller 2013-08-01 03:28:49 EDT
It looks like this can only happen if VDSM is installed and running and with the latest version.
Comment 2 Dan Kenigsberg 2013-08-02 03:42:20 EDT
From rose09-2013073114031375268609/var/log/vdsm/supervdsm.log:
supervdsm was started on July 29. A day later, setupNetwork was called, and failed due to libvirtd's restart during that interval.

MainThread::DEBUG::2013-07-29 16:38:01,209::supervdsmServer::363::SuperVdsm.Server::(main) Making sure I'm root - SuperVdsm
MainThread::DEBUG::2013-07-29 16:38:01,212::libvirtconnection::124::libvirtconnection::(get) trying to connect libvirt
Thread-15::DEBUG::2013-07-30 15:32:55,803::supervdsmServer::88::SuperVdsm.ServerCallback::(wrapper) calling to setupNetworks with ({'rhevm': {'nic
': 'eth0', 'bootproto': 'dhcp', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}
MainProcess|Thread-15::ERROR::2013-07-30 15:32:55,809::libvirtconnection::94::libvirtconnection::(wrapper) connection to libvirt broken.  ecode: 1 edom: 7
MainProcess|Thread-15::ERROR::2013-07-30 15:32:55,809::libvirtconnection::96::libvirtconnection::(wrapper) taking calling process down.

Indeed, libvirtd was restarted 3 seconds earlier (note the UTC+3 tz of supervdsm.log):

2013-07-30 12:32:52.211+0000: 12320: info : libvirt version: 0.10.2, package: 18.el6_4.9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2013-06-20-16:56:19, x86-002.build.bos.redhat.com)

That restart was initiated by a failed ovirt-host-depoly meni-rhevm-33-2013073114031375268590/var/log/ovirt-engine/host-deploy/ovirt-20130730152902-rose08.qa.lab.tlv.redhat.com-14346118.log:

2013-07-30 15:29:01 DEBUG otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:441 execute-output: ('/usr/bin/vdsm-tool', 'libvirt-configure') stdout:
Starting configure libvirt to VDSM ...
=Done configuring libvirt=

The first ovirt-host-deployment failed due to an iptables startup problem. A following deployment has finished successfully.

I believe that the problem could have been averted if ovirt-host-deploy restarted supervdsd prior to configuring libvirt, just like it does to vdsmd itself. Note that before we made supervdsmd its own service, its restart was implicit by vdsmd's restart.
Comment 5 Meni Yakove 2013-08-09 04:39:22 EDT
Comment 6 Itamar Heim 2014-01-21 17:18:23 EST
Closing - RHEV 3.3 Released
Comment 7 Itamar Heim 2014-01-21 17:24:45 EST
Closing - RHEV 3.3 Released

Note You need to log in before you can comment on or make changes to this bug.