Bug 1097480

Summary: [RHEVM-RHS] Error - "Failed to configure management network on the host", while adding RHSS Node to gluster cluster
Product: Red Hat Enterprise Virtualization Manager Reporter: SATHEESARAN <sasundar>
Component: ovirt-engine-webadmin-portalAssignee: Nobody <nobody>
Status: CLOSED DUPLICATE QA Contact: SATHEESARAN <sasundar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: acathrow, ecohen, gklein, iheim, masayag, myakove, rcyriac, Rhev-m-bugs, ssamanta, tkatarki, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: network
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
virt rhev integration
Last Closed: 2014-05-14 13:05:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Network RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
error message in engine.log none

Description SATHEESARAN 2014-05-14 01:45:56 UTC 
Description of problem:
======================
Tried to add the RHSS Node to the gluster-cluster. Installation failed with error message as follows "Failed to configure management network on the host."


Version-Release number of selected component (if applicable):
=============================================================

RHS 3.0 [ glusterfs-3.6.0-1.0.el6rhs ]
RHEV 3.4 [ av9 ] - 3.4.0-0.18.el6ev
JBOSS - http://download.devel.redhat.com/devel/candidates/JBEAP/JBEAP-6.3.0.ER2/compose/compose/Server/x86_64/os/

How reproducible:
=================
Always

Steps to Reproduce:
===================
1. Create a Datacenter of 3.4
2. Create gluster type of cluster with compatibility 3.4
3. Add a RHS 3.0 Node to that cluster

Actual results:
===============
Installation failed with error - "Failed to configure management network on the host"

Expected results:
================
Node should get installed in that cluster successfully

Additional info:
Comment 1 SATHEESARAN 2014-05-14 01:51:31 UTC 
I could see the following in engine.log

<snip>
2014-05-13 21:18:31,190 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] START, SetupNetworksVDS
Command(HostName = test-rhs, HostId = 85bf7ad3-4c28-4b87-9334-b70e809347c5, force=false, checkConnectivity=true, conectivityTimeout=120,
        networks=[rhevm {id=faeb1884-1dbd-4716-9345-a051c46253f8, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=f
alse, dataCenterId=8b3dd178-8d3e-4205-a7f1-447e072a027c, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display
=false, required=true, migration=false}, providedBy=null, label=null, qosId=null}],
        bonds=[],
        interfaces=[eth0 {id=18ac64cb-4372-4a40-9e98-3e6be53da642, vdsId=85bf7ad3-4c28-4b87-9334-b70e809347c5, macAddress=00:44:54:19:4b:1d, networkName=rhevm, vlanId=n
ull, bonded=null, bondName=null, bondOptions=null, bootProtocol=NONE, address=10.70.37.131, subnet=255.255.254.0, gateway=null, mtu=1500, bridged=false, speed=0, type=0
, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null}],
        removedNetworks=[],
        removedBonds=[]), log id: 7166fa58
2014-05-13 21:18:31,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] Failed in SetupNetworks
VDS method, for vds: test-rhs; host: 10.70.37.131
2014-05-13 21:18:31,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] Command SetupNetworksVD
SCommand(HostName = test-rhs, HostId = 85bf7ad3-4c28-4b87-9334-b70e809347c5, force=false, checkConnectivity=true, conectivityTimeout=120,
        networks=[rhevm {id=faeb1884-1dbd-4716-9345-a051c46253f8, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=8b3dd178-8d3e-4205-a7f1-447e072a027c, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true, migration=false}, providedBy=null, label=null, qosId=null}],
        bonds=[],
        interfaces=[eth0 {id=18ac64cb-4372-4a40-9e98-3e6be53da642, vdsId=85bf7ad3-4c28-4b87-9334-b70e809347c5, macAddress=00:44:54:19:4b:1d, networkName=rhevm, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=NONE, address=10.70.37.131, subnet=255.255.254.0, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null, qos=null qosOverridden=false, labels=null}],
        removedNetworks=[],
        removedBonds=[]) execution failed. Exception: NoSuchElementException: 
2014-05-13 21:18:31,195 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] FINISH, SetupNetworksVDSCommand, log id: 7166fa58
2014-05-13 21:18:31,195 ERROR [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] Command org.ovirt.engine.core.bll.network.host.SetupNetworksCommand throw exception: java.lang.NullPointerException
        at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.isDone(FutureVDSCommand.java:41) [vdsbroker.jar:]
        at org.ovirt.engine.core.bll.network.host.SetupNetworksCommand.pollInterruptively(SetupNetworksCommand.java:212) [bll.jar:]
        at org.ovirt.engine.core.bll.network.host.SetupNetworksCommand.executeCommand(SetupNetworksCommand.java:133) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1133) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1218) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1894) [bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1238) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:351) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:413) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:392) [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:341) [bll.jar:]
        at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) [:1.7.0_55]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
        at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) [jboss-as-ee.jar:7.4.0.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-10]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) [jboss-as-ee.jar:7.4.0.Final-redhat-10

<skipping-few-lines here>

2014-05-13 21:18:31,225 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] START, SetVdsStatusVDSCommand(HostName = test-rhs, HostId = 85bf7ad3-4c28-4b87-9334-b70e809347c5, status=NonOperational, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 40bbccd6
2014-05-13 21:18:31,245 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] FINISH, SetVdsStatusVDSCommand, log id: 40bbccd6
2014-05-13 21:18:31,259 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-4-thread-14) [7e3c0407] Correlation ID: b64b04, Job ID: d2b2e162-98f2-40de-b9ae-f0bcf567b5a9, Call Stack: null, Custom Event ID: -1, Message: Host test-rhs installation failed. Failed to configure management network on the host.

</snip>
Comment 2 SATHEESARAN 2014-05-14 02:08:08 UTC 
Created attachment 895305 [details]
error message in engine.log
Comment 3 SATHEESARAN 2014-05-14 02:50:17 UTC 
Added the same node to RHEV 3.3 [ GlusterFS Datacenter 3.3 / gluster cluster 3.3 ], but the installation failed with error,
"Host rhss1 is installed with VDSM version (4.14) and cannot join cluster gluster-cluster-1-3_3 which is compatible with VDSM versions [4.13, 4.9, 4.11, 4.12, 4.10]"

But the bridge was created successfully as shown below:

[root@rhss4 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
;vdsmdummy;             8000.000000000000       no
rhevm           8000.001a4ab72081       no              eth0

After that I retried adding this RHS 3.0 node to RHEV 3.4 [ datacenter 3.4/ gluster cluster 3.4 ], and that seemed to work successfully with the bridge created earlier while adding it to RHEV 3.3
Comment 4 SATHEESARAN 2014-05-14 05:27:03 UTC 
Network details on RHSS Node
=============================

<BEFORE_ADDING_TO_RHEV3.3> [ BEFORE BRIDGE CREATION ]
======================================================
[root@rhss4 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:1a:4a:b7:20:81 brd ff:ff:ff:ff:ff:ff
    inet 10.70.37.123/23 brd 10.70.37.255 scope global eth0
5: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 3e:81:5b:b4:24:e8 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: bond4: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
8: bond1: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: bond2: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: bond3: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff


<AFTER_ADDING_TO_RHEV3.3> [ AFTER BRIDGE CREATION ]
======================================================
[root@rhss4 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:1a:4a:b7:20:81 brd ff:ff:ff:ff:ff:ff
5: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 3e:81:5b:b4:24:e8 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
7: bond4: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
8: bond1: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: bond2: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: bond3: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
13: rhevm: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:1a:4a:b7:20:81 brd ff:ff:ff:ff:ff:ff
    inet 10.70.37.123/23 brd 10.70.37.255 scope global rhevm
Comment 5 SATHEESARAN 2014-05-14 10:35:18 UTC 
I couldn't reproduce the issue with the latest RHEVM [1] and latest nightly build of RHSS 3.0 [2]

[1] av9.1
[2] RHS 3.0 Nightly ( RHSS-3.0-20140513.n.0-RHS-x86_64-DVD1.iso )
http://download.eng.bos.redhat.com/composes/nightly/RHSS-3.0-20140513.n.0/

Feel free to close this bug.
Will re-open, if the issue is seen again
Comment 6 Moti Asayag 2014-05-14 13:05:36 UTC 

*** This bug has been marked as a duplicate of bug 1096715 ***