Bug 857160

Summary: Unable to create bonded interface for the rhevm network
Product: Red Hat Enterprise Virtualization Manager Reporter: Marcelo Giles <mgiles>
Component: ovirt-engineAssignee: Nobody's working on this, feel free to take it <nobody>
Status: CLOSED NOTABUG QA Contact: meital avital <mavital>
Severity: high Docs Contact:
Priority: high    
Version: 3.1.0CC: clasohm, dkinon, dyasny, ecohen, iheim, lpeer, masayag, mavital, mpastern, Rhev-m-bugs, yeylon, ykaul
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: network
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-14 06:56:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Python code snippet for bond creation
none
RHEVM log snippet for bond creation
none
host network configuration
none
reproducer script
none
Java stack trace none

Description Marcelo Giles 2012-09-13 17:25:21 UTC
Created attachment 612526 [details]
Python code snippet for bond creation

Description of problem:

Testing the rhevm-sdk with RHEVM and RHEL hosts.

RHEL host has 6 NICs, trying to configure 3 bonds for: rhevm (bond0), storage (bond1) and data (bond2). All interfaces on hosts have static IP addresses.

<eth0>--<bond0>--<rhevm>
<eth1>

<eth2>--<bond1>--<storage>
<eth3>

<eth4>--<bond2>--<data>
<eth5>

Succeeded in creating bond1 and bond2. But when trying to create bond0, since the rhevm bridge is already using eth0, it fails with an error message stating that the interface is in use. 

2012-09-11 17:10:54,156 WARN  [org.ovirt.engine.core.bll.AddBondCommand] (http-0.0.0.0-8443-9) CanDoAction of action AddBond failed. Reasons:NETWORK_INTERFACE_NAME_ALREAY_IN_USE

2012-09-11 17:10:54,157 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (http-0.0.0.0-8443-9) Operation Failed: [Network interface is already in use.]

When this happens, no vdsm messages are logged on the hosts, it seems that the action is stopped at the RHEVM host. 

Version-Release number of selected component (if applicable):
RHEVM 3.0.7
rhevm-sdk-3.1.0.5-1
RHEL Server 6.3

Note: Also tested using RHEVM 3.0.5 and RHEL 6.2 hosts and with RHEVM 3.1 beta and RHEL 6.3 hosts. Same results.

How reproducible:
Run the Python script, it creates bond1 and bond2, assigns ip/netmask/gateway/boot_protocol to them but fails to create bond0.

Additional info:

After searching for supporting documentation online via Customer Portal, ovirt.org, official product documentation, bugzilla and elsewhere, it seems that this hasn't been tried before using the SDK, so it might be an untested functionality but, if it works using the Web UI, it should also work using the API/SDK.

Code snippet for bond creation attached to BZ.
RHEVM logs snippets attached to BZ.

Comment 1 Marcelo Giles 2012-09-13 17:26:49 UTC
Created attachment 612527 [details]
RHEVM log snippet for bond creation

Comment 2 Moti Asayag 2012-09-19 12:47:40 UTC
When an interface has a network attached, the interface should be updated via UpdateNetworkToVdsInterfaceCommand.

AddBondCommand should be in use for creating a bond over nics which have no network attached. 

This is the reason for succeeding in bond creation for nics with no network attached and failing to update the eth0 which rhevm network is already attached.

From API side, supporting the described scenario is problematic:
1. Creating a bond is done via POST method.
2. Modifying an existing interface is done as PUT.

But what actually required here is creating a new bond by modifying an existing entity (detach a network, assemble a bond with that nic as a slave + additional slaves and attach the network to the created bond). This doesn't fall into any of REST methods.

The webadmin is capable doing so, since it uses the UpdateNetworkToVdsInterfaceCommand to achieve the complex network configuration.
But REST will use UpdateNetworkToVdsInterfaceCommand only via PUT to update a nic/network attached to, and at the moment the list of slaves is not being in REST API to the backend in UpdateNetworkToVdsInterfaceCommand.

We could add that list of slaves for the PUT method, but we'll end up with a mixed entity representing partially the slave nic and partially the required bond, e.g:

<host_nic>
    <name>bond0</name>
    <network id="388a02c5-c733-499b-bc3a-24a5f0534a61"/>
    <ip gateway="10.1.1.2" netmask="255.255.255.0" address="10.1.1.1"/>
    <boot_protocol>static</boot_protocol>
    <bonding>
        <options>
            <option type="Active-Backup" value="1" name="mode"/>
            <option value="100" name="miimon"/>
        </options>
        <slaves>
            <host_nic id="0640fe2d-b462-4057-a1a8-afcc4469e551">
            </host_nic>
            <host_nic id="364a296b-44f6-4691-a441-3fe55068258d">
            </host_nic>
        </slaves>
    </bonding>
    <check_connectivity>true</check_connectivity>
</host_nic>

and the entity referred in the url is the nic that the network is attached to.

This functionality is provided via REST using the SetupNetworks api, introduced in 3.1.

Comment 4 Michael Pasternak 2012-09-19 12:58:30 UTC
this functionality was not supported in RHEVM 3.0.X api you are using

Comment 5 Michael Pasternak 2012-09-19 13:03:37 UTC
btw in 3.1 there is new action in api called setupnetworks [2], it should
do the job, you can try it in sdk against 3.1 beta [2].

[1] /api/hosts/xxx/nics/setupnetworks
[2]

1. nics = api.hosts.get(name=xxx).nics
2. nics_arr = nics.list()
   (change nics in nics_arr like you'd expect them to look like)
3. my_nics = params.HostNics(host_nic=[nics_arr[0],nics_arr[1],...])
   or
   my_nics = params.HostNics(host_nic=nics_arr)
4. nics.setupnetworks(action=params.Action(host_nics=my_nics))

Comment 6 Carsten Clasohm 2012-09-19 19:27:31 UTC
We have tried setupnetworks() with rhevm-sdk-3.1.0.8-1.el6ev and RHEV-M 3.1.0-14.el6ev.

Even if we just take the nics.list() result and call setupnetworks() with it, we get a Java NullPointerException exception on the RHEV-M server.

I'll attach a reproducer script, the current network configuration for the host we ran this against (taken from the API), and the Java stack trace.

Comment 7 Carsten Clasohm 2012-09-19 19:28:22 UTC
Created attachment 614564 [details]
host network configuration

Comment 8 Carsten Clasohm 2012-09-19 19:28:41 UTC
Created attachment 614565 [details]
reproducer script

Comment 9 Carsten Clasohm 2012-09-19 19:28:56 UTC
Created attachment 614571 [details]
Java stack trace

Comment 10 Carsten Clasohm 2012-09-19 22:26:24 UTC
After a few more trials, we managed to create bond0 with eth0/eth1 and the rhevm network using the setupnetworks() method. We did this by reverting the host's network configuration to rhevm on eth0, removing all other interfaces.

The NullPointerException seems to be related to the other networks we have on bond1 and bond2.

Comment 11 lpeer 2013-01-14 06:56:06 UTC
From the comments above it seems like when using sdk 3,1 with RHEV 3.1 the scenario worked and the bond was created.

Because this bugs encapsulates a lot of data which most of it is no longer relevant I would like to ask that if there is a specific issue with setupNetwork and bond creation in 3.1 please report a new bug with steps to reproduce.