Description of problem: Adding a new network interface to a host fails Version-Release number of selected component (if applicable): ovirt-engine-4.1.1.5-0.1.el7.noarch rhevm-4.1.1.5-0.1.el7.noarch vdsm-4.19.9-1.el7ev.x86_64 How reproducible: Every time Steps to Reproduce: 1. Create 2 new networks - non dhcp, non required and non vm 2. Select the host and add the networks via Webadmin-> Hosts ->Setup hosts->Setup Host Networks - Press OK - fails with Only a singe default route network is allowed Actual results: Only a singe default route network is allowed Error while executing action HostSetupNetworks: Illegal Network parameters Expected results: Additional info: Engine Log ----------------------------------- 2017-03-20 18:56:41,014+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [5017de28-70e2-41b6-a42a-1b7ad8074702] Failed in 'HostSetupNetworksVDS' method 2017-03-20 18:56:41,074+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-7) [5017de28-70e2-41b6-a42a-1b7ad8074702] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Corre lation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM pluto-vdsb.eng.lab.tlv.redhat.com command HostSetupNetworksVDS failed: Only a singe default route network is allowed. 2017-03-20 18:56:41,075+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [5017de28-70e2-41b6-a42a-1b7ad8074702] Error: VDSGenericException: VDSErrorException: Fai led to HostSetupNetworksVDS, error = Only a singe default route network is allowed., code = 21 2017-03-20 18:56:41,075+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-7) [5017de28-70e2-41b6-a42a-1b7ad8074702] Exception: org.ovirt.engine.core.vdsbroker.vdsbrok er.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Only a singe default route network is allowed., code = 21 at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:76) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:222) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:192) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.FutureVDSCommand.get(FutureVDSCommand.java:70) [vdsbroker.jar:] at org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand.executeCommand(HostSetupNetworksCommand.java:290) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1251) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1391) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2055) [bll.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1451) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:397) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:511) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:493) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:446) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source) [:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl.jar:2.3.3.Final-redhat-1] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83) [wildfly-weld-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:374) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:243) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66) [wildfly-ejb3-7.0.5.GA-redhat-2.jar:7.0.5.GA-redhat-2] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
Created attachment 1264915 [details] server, engine, vdsm logs Adding logs
Kevin, it's quite likely that the Engine exception is far less interesting than the vdsm and super vdsm relevant logs - please quote it next time in the bug report.
yep, please include supervdsm.log as well.
Hi I would like to clarify something about this report. Kevin showed it to me yesterday and the first thing i noticed that the physical interfaces on the host were configured with bootproto=dhcp instead of none. Which i believe caused this errors as he tried to attach logical network on top of an interface with dhcp. It caused this issue with default route thing. I believe that the host was added to rhv-m in this state in which all of the interfaces were configured as dhcp and every attempt to set network on top of them, was causing to additional attempt to create default route. The main problem here, that once he failed, he can't perform any additional setup networks command and every attempt will through this error. - Note, that since morning i was trying to reproduce this report, but without any success. I can't reproduce it(but i saw it with my eyes).
Created attachment 1265665 [details] Added logs again with supervdsm Added logs again with supervdsm
We validate that only a single network has the defautlRoute set by examining the current setup request and the running config, if in all more than one exists, it will fail as seen here. The only reason I can think of to such a problem is a corrupted running config file. Other than that, I need to access the machine to check it. If the host on which this has happened still has this issue, please share the /var/run/vdsm/netconf content.
I ran into the same issue while upgrading an existing host to RHV4.0. I deleted every configured bridge to no avail. I then removed the bonds - to no avail. I had to remove /var/run/vdsm/netconf/bonds/* and /var/run/vdsm/netconf/nets/* in order to run the hosted-engine --deploy command. This at least is a workaround.
We have not been able to determine based on the provided information what caused this problem. It seems that the configuration includes two default routes and after the upgrade, we will not accept such condition and consider the configuration corrupted. Please recreate this scenario and allow us to connect to the problematic host while the problem exists.
(In reply to Edward Haas from comment #8) > We have not been able to determine based on the provided information what > caused this problem. > It seems that the configuration includes two default routes and after the > upgrade, we will not accept such condition and consider the configuration > corrupted. > > Please recreate this scenario and allow us to connect to the problematic > host while the problem exists. Seems to be a pure duplication of BZ - 1443292
I agree, we should continue handling this through BZ#1443292. *** This bug has been marked as a duplicate of bug 1443292 ***