Description of problem: After updating from RHEV 3.2 -> RHEV 3.3 attempting to create a new VM fails via both the API and the GUI. Version-Release number of selected component (if applicable): ovirt-host-deploy-1.1.3-1.el6ev.noarch ovirt-host-deploy-java-1.1.3-1.el6ev.noarch python-rhev-1.0-15.rc1.el6.noarch redhat-support-plugin-rhev-3.3.0-13.el6ev.noarch rhev-guest-tools-iso-3.3-10.noarch rhevm-3.3.0-0.45.el6ev.noarch rhevm-backend-3.3.0-0.45.el6ev.noarch rhevm-branding-rhev-3.3.0-1.4.el6ev.noarch rhevm-cli-3.3.0.13-1.el6ev.noarch rhevm-dbscripts-3.3.0-0.45.el6ev.noarch rhevm-dependencies-3.3.5-1.el6ev.noarch rhevm-doc-3.3.0-3.el6eng.noarch rhevm-dwh-3.3.0-28.el6ev.noarch rhevm-image-uploader-3.3.1-2.el6ev.noarch rhevm-iso-uploader-3.3.0-2.el6ev.noarch rhevm-lib-3.3.0-0.45.el6ev.noarch rhevm-log-collector-3.3.1-5.el6ev.noarch rhevm-reports-3.3.0-28.el6ev.noarch rhevm-restapi-3.3.0-0.45.el6ev.noarch rhevm-sdk-python-3.3.0.21-1.el6ev.noarch rhevm-setup-3.3.0-0.45.el6ev.noarch rhevm-setup-plugins-3.3.0-4.el6ev.noarch rhevm-spice-client-x64-cab-3.3-8.el6_5.noarch rhevm-spice-client-x64-msi-3.3-8.el6_5.noarch rhevm-spice-client-x86-cab-3.3-8.el6_5.noarch rhevm-spice-client-x86-msi-3.3-8.el6_5.noarch rhevm-tools-3.3.0-0.45.el6ev.noarch rhevm-userportal-3.3.0-0.45.el6ev.noarch rhevm-webadmin-portal-3.3.0-0.45.el6ev.noarch rhevm-websocket-proxy-3.3.0-0.45.el6ev.noarch How reproducible: 100% Steps to Reproduce: 1. Attempt to create VM via gui with default settings. Actual results: 2014-01-22 23:23:33,013 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [64996c8b] Lock Acquired to object EngineLock [exclusiveLocks= key: bharrington-testvm value: VM_NAME , sharedLocks= ] 2014-01-22 23:23:33,032 ERROR [org.ovirt.engine.core.bll.network.MacPoolManager] (ajp-/127.0.0.1:8702-3) [64996c8b] MacPoolManager(42ee7a0a): The MAC addresses pool is not initialized 2014-01-22 23:23:33,033 ERROR [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [64996c8b] Error during CanDoActionFailure.: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: MAC_POOL_NOT_INITIALIZED (Failed with error MAC_POOL_NOT_INITIALIZED and code 5011) at org.ovirt.engine.core.bll.network.MacPoolManager.getAvailableMacsCount(MacPoolManager.java:174) [bll.jar:] at org.ovirt.engine.core.bll.VmHandler.verifyAddVm(VmHandler.java:140) [bll.jar:] at org.ovirt.engine.core.bll.AddVmCommand.verifyAddVM(AddVmCommand.java:533) [bll.jar:] at org.ovirt.engine.core.bll.AddVmCommand.canAddVm(AddVmCommand.java:521) [bll.jar:] at org.ovirt.engine.core.bll.AddVmCommand.canAddVm(AddVmCommand.java:192) [bll.jar:] at org.ovirt.engine.core.bll.AddVmCommand.canDoAddVmCommand(AddVmCommand.java:249) [bll.jar:] at org.ovirt.engine.core.bll.AddVmCommand.canDoAction(AddVmCommand.java:347) [bll.jar:] at org.ovirt.engine.core.bll.AddVmFromScratchCommand.canDoAction(AddVmFromScratchCommand.java:145) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:741) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:356) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) [bll.jar:] at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:353) [bll.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor231.invoke(Unknown Source) [:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.3.0.Final-redhat-14] at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view9.RunAction(Unknown Source) [common.jar:] at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.RunAction(GenericApiGWTServiceImpl.java:131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:172) at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:233) at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:72) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] 2014-01-22 23:23:33,036 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [64996c8b] Lock freed to object EngineLock [exclusiveLocks= key: bharrington-testvm value: VM_NAME , sharedLocks= ] .invoke0(Native Method) [rt.jar:1.7.0_51] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:172) at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:233) at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:72) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] 2014-01-22 23:23:33,036 INFO [org.ovirt.engine.core.bll.AddVmFromScratchCommand] (ajp-/127.0.0.1:8702-3) [64996c8b] Lock freed to object EngineLock [exclusiveLocks= key: bharrington-testvm value: VM_NAME , sharedLocks= ] Additional info: This bug seems similar to bz987825.
In troubleshooting further I found the following values in the config: MacPoolRanges: 00:1A:4A:01:00:00-00:1A:4A:FF:FF:FF version: general MaxMacsCountInPool: 100000 version: general The MaxMacsCountInPool is *vastly* undersized for this range (as noted in https://access.redhat.com/site/solutions/236473). Potentially a more useful error message could be sent out? While there is no way to verify past bug 987825 I have a feeling this could have been the culprit. After verifying that there were no existing MAC addresses outside of the new range, changing the values to: # engine-config -s MacPoolRanges=00:1A:4A:01:00:00-00:1A:4A:02:FF:FF # engine-config -s MaxMacsCountInPool=132072 Solves the problem and VMs are now able to be created.
What I see is that there was indeed a behavioral change in 3.3. Prior to 3.3, if the range would exceed the MAX then the pool would be partially initialized. Now, the pool is not initialized by a partial range (you can have several ranges so only those not "fitting" in the pool would not be added). Not sure this merits a fix, but probably a release note should be included. Also, in the new behavior you would get a MAC_POOL_NOT_ENOUGH_MAC_ADDRESSES error and not a MAC_POOL_NOT_INITIALIZED for this scenario, but if you can attach the engine log from your environment perhaps there is another culprit which is unrelated to MaxMacsCountInPool but does cause this error to appear.
Created attachment 854512 [details] ovirt-engine log Here is the engine log as requested.
The change of behavior causes an out of memory error in case of a very big range. It's been decided to restore the original behavior, which will fix the out of memory error and also keep same beavior from previous versions.
verified with Red Hat Enterprise Virtualization Manager Version: 3.4.0-0.2.master.el6ev [root@mp-rhevm34 ~]# engine-config -g MacPoolRanges MacPoolRanges: 00:1A:4A:01:00:00-00:1A:4A:02:FF:FF version: general [root@mp-rhevm34 ~]# engine-config -g MaxMacsCountInPool MaxMacsCountInPool: 5 version: general 2014-02-18 13:22:43,125 WARN [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-1) [457de3c] CanDoAction of action AddVmInterface failed. Reasons:VAR__TYPE__INTERFACE,VAR__ACTION__ADD,MAC_POOL_NOT_ENOUGH_MAC_ADDRESSES 2014-02-18 13:22:43,197 WARN [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp-/127.0.0.1:8702-3) [1015dbce] CanDoAction of action AddVmInterface failed. Reasons:VAR__TYPE__INTERFACE,VAR__ACTION__ADD,MAC_POOL_NOT_ENOUGH_MAC_ADDRESSES
This bug is referenced in ovirt-engine-3.4.0-beta3 logs. Moving to ON_QA
verified with oVirt Engine Version: 3.4.0-0.11.beta3.el6 2014-02-19 14:51:14,329 WARN [org.ovirt.engine.core.bll.network.vm.AddVmInterfaceCommand] (ajp--127.0.0.1-8702-6) [163123e3] CanDoAction of action AddVmInterface failed. Reasons:VAR__TYPE__INTERFACE,VAR__ACTION__ADD,MAC_POOL_NOT_ENOUGH_MAC_ADDRESSES
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2014-0506.html