Bug 861701 - 3.2 - need to sync networks between vdsm and libvirt.
3.2 - need to sync networks between vdsm and libvirt.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: vdsm (Show other bugs)
3.2.0
x86_64 Linux
high Severity high
: ---
: 3.2.0
Assigned To: Antoni Segura Puimedon
Meni Yakove
network
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-30 02:38 EDT by Meni Yakove
Modified: 2016-02-10 14:54 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when a network device was removed from the system through a means that was not VDSM, VDSM recognized that the device was removed and it stopped reporting the device to the engine. But libvirt still held a reference to the removed device. If the user attempted to create the device again, the attempt failed because libvirt still holds a reference to the non-existent device. Broken networks are now removed from libvirt before new networks are set up. The re-creation of broken networks no longer causes exceptions.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-10 16:32:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm.log (673.48 KB, application/zip)
2012-09-30 02:38 EDT, Meni Yakove
no flags Details
vdsm.log (934.64 KB, text/x-log)
2012-10-03 08:39 EDT, Meni Yakove
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 10901 None None None Never

  None (edit)
Description Meni Yakove 2012-09-30 02:38:49 EDT
Created attachment 619333 [details]
vdsm.log

Description of problem:
When bridge on virsh and on the host (brctl) do not match the network can be used in SetupNetworks again (the network is not attached to any interface). and when trying to assing the network there is error from libvirt:

MainProcess|Thread-848::ERROR::2012-09-27 12:16:30,051::configNetwork::1368::setupNetworks::(setupNetworks) operation failed: network 'vdsm-vmnet1' already exists with uuid 205bfb07-9b78-8d00-a6a6-aafe0b90947b


Version-Release number of selected component (if applicable):
vdsm-4.9.6-36.0.el6_3.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Open SetupNetworks and attach vmnet1 to nic.
2.On the host delete the bridge - brctl delbr vmnet1.
3.Open setupNetworks and attach the network net1 to another nic.
  
Actual results:
operation failed with errors.

Expected results:
operation should finished wothout errors.

Additional info:
Comment 2 Meni Yakove 2012-10-03 08:38:37 EDT
Another scenario: 

1. Create network vmnet1 and attach it to the cluster.
2. On SetupNetworks attach vmnet1 to nic - Don't save network configuration.
3. Create VM and use vmnet1 as nic for this VM.
4. Start the VM
5. On the host restart VDSm service.

 
Actual results:
Because we didn't save network configuration VDSM rollback the network configuration and now vmnet1 is deleted but the VM is still using vmnet1 network.

 brctl show
bridge name     bridge id               STP enabled     interfaces
rhevm           8000.00145e17d5b0       no              eth2
                                                        vnet1
vmnet1          8000.fe1a4a236168       no              vnet0


virsh -r net-list
Name                 State      Autostart
-----------------------------------------
vdsm-rhevm           active     yes       
vdsm-vmnet1          active     yes   

vmnet1 is attached to vnet0 (the VM network) and vdsm-vmnet1 exist in libvirt.
When stopping the VM vmnet1 is still exist under brctl and in libvirt but without nic attache to it (on brctl).

bridge name     bridge id               STP enabled     interfaces
rhevm           8000.00145e17d5b0       no              eth2
vmnet1          8000.000000000000       no

Even when restarting VDSM service again (when the VM is not running) vmnet1 exist in brctl and in libvirt, the only way to return to normal stat is to delete the network from brctl and from libvirt.


Also now when opening SetupNetworks vmnet1 is not attached to any nic and can be use again but when trying to attach vmnet1 to nic there is error:

MainProcess|Thread-173::ERROR::2012-10-03 14:32:04,801::configNetwork::1368::setupNetworks::(setupNetworks) [Errno 2] No such file or directory: '/etc/sysconfig/network-scripts/ifcfg-vmnet1'
Traceback (most recent call last):
  File "/usr/share/vdsm/configNetwork.py", line 1310, in setupNetworks
    implicitBonding=False)
  File "/usr/share/vdsm/configNetwork.py", line 1069, in delNetwork
    configWriter.setNewMtu(network=network, bridged=bridged)
  File "/usr/share/vdsm/configNetwork.py", line 617, in setNewMtu
    currmtu = self._getConfigValue(cf, 'MTU')
  File "/usr/share/vdsm/configNetwork.py", line 541, in _getConfigValue
    with open(conffile) as f:
IOError: [Errno 2] No such file or directory: '/etc/sysconfig/network-scripts/ifcfg-vmnet1'
MainProcess|Thread-173::ERROR::2012-10-03 14:32:04,802::supervdsmServer::61::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 59, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 107, in setupNetworks
    return configNetwork.setupNetworks(networks, bondings, **options)
  File "/usr/share/vdsm/configNetwork.py", line 1310, in setupNetworks
    implicitBonding=False)
  File "/usr/share/vdsm/configNetwork.py", line 1069, in delNetwork
    configWriter.setNewMtu(network=network, bridged=bridged)
  File "/usr/share/vdsm/configNetwork.py", line 617, in setNewMtu
    currmtu = self._getConfigValue(cf, 'MTU')
  File "/usr/share/vdsm/configNetwork.py", line 541, in _getConfigValue
    with open(conffile) as f:
IOError: [Errno 2] No such file or directory: '/etc/sysconfig/network-scripts/ifcfg-vmnet1'
Thread-173::DEBUG::2012-10-03 14:32:04,803::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /usr/bin/kill -9 19518' (cwd None)
Thread-173::DEBUG::2012-10-03 14:32:04,826::__init__::1164::Storage.Misc.excCmd::(_log) SUCCESS: <err> = ''; <rc> = 0
Thread-173::DEBUG::2012-10-03 14:32:04,833::supervdsm::81::SuperVdsmProxy::(_launchSupervdsm) Launching Super Vdsm
Thread-173::DEBUG::2012-10-03 14:32:04,838::__init__::1164::Storage.Misc.excCmd::(_log) '/usr/bin/sudo -n /usr/bin/python /usr/share/vdsm/supervdsmServer.py 1c711907-f40d-41d7-bf5b-32d6fe524ba7 19465' (cwd None)
MainThread::DEBUG::2012-10-03 14:32:05,011::supervdsmServer::247::SuperVdsm.Server::(main) Making sure I'm root
MainThread::DEBUG::2012-10-03 14:32:05,011::supervdsmServer::251::SuperVdsm.Server::(main) Parsing cmd args
MainThread::DEBUG::2012-10-03 14:32:05,012::supervdsmServer::254::SuperVdsm.Server::(main) Creating PID file
MainThread::DEBUG::2012-10-03 14:32:05,012::supervdsmServer::258::SuperVdsm.Server::(main) Cleaning old socket
MainThread::DEBUG::2012-10-03 14:32:05,013::supervdsmServer::262::SuperVdsm.Server::(main) Setting up keep alive thread
MainThread::DEBUG::2012-10-03 14:32:05,013::supervdsmServer::267::SuperVdsm.Server::(main) Creating remote object manager
MainThread::DEBUG::2012-10-03 14:32:05,015::supervdsmServer::278::SuperVdsm.Server::(main) Started serving super vdsm object
Thread-176::DEBUG::2012-10-03 14:32:05,116::BindingXMLRPC::864::vds::(wrapper) client [10.35.97.119]::call ping with () {} flowID [7df120fa]
Thread-176::DEBUG::2012-10-03 14:32:05,117::BindingXMLRPC::870::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-177::DEBUG::2012-10-03 14:32:05,635::BindingXMLRPC::864::vds::(wrapper) client [10.35.97.119]::call ping with () {} flowID [7df120fa]
Thread-177::DEBUG::2012-10-03 14:32:05,636::BindingXMLRPC::870::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-178::DEBUG::2012-10-03 14:32:06,155::BindingXMLRPC::864::vds::(wrapper) client [10.35.97.119]::call ping with () {} flowID [7df120fa]
Thread-178::DEBUG::2012-10-03 14:32:06,155::BindingXMLRPC::870::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-179::DEBUG::2012-10-03 14:32:06,673::BindingXMLRPC::864::vds::(wrapper) client [10.35.97.119]::call ping with () {} flowID [7df120fa]
Thread-179::DEBUG::2012-10-03 14:32:06,674::BindingXMLRPC::870::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-173::DEBUG::2012-10-03 14:32:06,849::supervdsm::102::SuperVdsmProxy::(_connect) Trying to connect to Super Vdsm
Thread-173::ERROR::2012-10-03 14:32:06,853::BindingXMLRPC::879::vds::(wrapper) unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/BindingXMLRPC.py", line 869, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 346, in setupNetworks
    return api.setupNetworks(networks, bondings, options)
File "/usr/share/vdsm/API.py", line 1115, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/share/vdsm/supervdsm.py", line 62, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 57, in <lambda>
    callMethod = lambda : getattr(self._supervdsmProxy._svdsm, self._funcName)(*args, **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 725, in _callmethod
    conn.send((self._id, methodname, args, kwds))
IOError: [Errno 32] Broken pipe
Comment 3 Meni Yakove 2012-10-03 08:39:20 EDT
Created attachment 620822 [details]
vdsm.log
Comment 11 Meni Yakove 2013-03-24 04:13:36 EDT
Verified on:
rhevm-3.2.0-10.16.master.el6ev.noarch
vdsm-4.10.2-12.0.el6ev.x86_64
Comment 12 Meni Yakove 2013-03-24 04:17:36 EDT
Verified on:
rhevm-3.2.0-10.16.master.el6ev.noarch
vdsm-4.10.2-12.0.el6ev.x86_64
Comment 13 Cheryn Tan 2013-04-03 03:01:51 EDT
This bug is currently attached to errata RHBA-2012:14332. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.
Comment 15 errata-xmlrpc 2013-06-10 16:32:13 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0886.html

Note You need to log in before you can comment on or make changes to this bug.