Created attachment 1109249 [details] macpoolerrors-engine.log Description of problem: Creating a new MAC Address Pool fails. On web admin interface shows: Error while executing action AddMacPool: Internal Engine Error In engine.log: 2015-12-24 17:13:09,252 INFO [org.ovirt.engine.core.bll.AddMacPoolCommand] (default task-8) [2280bb7b] Running command: AddMacPoolCommand internal: false. Entities affected : ID: aaa00000 -0000-0000-0000-123456789aaa Type: SystemAction group CREATE_MAC_POOL with role type ADMIN 2015-12-24 17:13:09,264 ERROR [org.ovirt.engine.core.bll.AddMacPoolCommand] (default task-8) [2280bb7b] Command 'org.ovirt.engine.core.bll.AddMacPoolCommand' failed: This instance was not y et initialized. 2015-12-24 17:13:09,265 ERROR [org.ovirt.engine.core.bll.AddMacPoolCommand] (default task-8) [2280bb7b] Exception: java.lang.IllegalStateException: This instance was not yet initialized. Version-Release number of selected component (if applicable): ovirt-engine-3.6.3-0.0.master.20151223191430.git7294d71.el7.centos.noarch How reproducible: Not sure, always on my setup Steps to Reproduce: 1. Choose DC 2. Edit 3. Choose MAC Address Pool 4. New 5. Enter some values 6. OK Actual results: Expected results: Additional info: Similar errors when trying Configure -> MAC Address Pools. Attached log includes several attempts.
ovirt-engine 3.6.3-0.0.master.20151223191430.git7294d71.el7.centos
Well, it might be something deeper. Now got a similar error when trying to create a vm. Looking further.
Seems to work well on a clean 3.6 setup. Previous attempt was on a setup upgraded from 3.5.
Didi, Could you please supply more info: * exact 3.5.z version you used * upgrade log
(In reply to Yevgeny Zaspitsky from comment #4) > Didi, > > Could you please supply more info: > * exact 3.5.z version you used This machine only had snapshot versions, both 3.5 and 3.6 ones: # grep ovirt-engine-3 $(ls -tr /var/log/yum.log*) /var/log/yum.log-20150101:Nov 12 12:21:40 Installed: ovirt-engine-setup-plugin-ovirt-engine-3.5.1-0.0.master.20141111182009.git2c24911.el7.centos.noarch /var/log/yum.log-20150101:Nov 12 12:21:40 Installed: ovirt-engine-3.5.1-0.0.master.20141111182009.git2c24911.el7.centos.noarch /var/log/yum.log-20150101:Nov 17 13:00:54 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.1-0.0.master.20141112061952.git2c24911.el7.centos.noarch /var/log/yum.log-20150101:Nov 17 13:01:29 Updated: ovirt-engine-3.5.1-0.0.master.20141112061952.git2c24911.el7.centos.noarch /var/log/yum.log-20150101:Nov 24 17:57:56 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.1-0.0.master.20141124122901.el7.centos.noarch /var/log/yum.log-20150101:Nov 24 18:00:40 Updated: ovirt-engine-3.5.1-0.0.master.20141124122901.el7.centos.noarch /var/log/yum.log-20150515:May 14 16:49:15 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.4-0.0.master.20150512172202.git739dc64.el7.centos.noarch /var/log/yum.log-20150515:May 14 17:06:12 Updated: ovirt-engine-3.5.4-0.0.master.20150512172202.git739dc64.el7.centos.noarch /var/log/yum.log:Jul 06 10:40:27 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.5-0.0.master.20150705172210.git74a1a6a.el7.centos.noarch /var/log/yum.log:Jul 06 10:52:42 Updated: ovirt-engine-3.5.5-0.0.master.20150705172210.git74a1a6a.el7.centos.noarch /var/log/yum.log:Sep 21 10:08:06 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.6.0-0.0.master.20150920175753.gite4183e5.el7.centos.noarch /var/log/yum.log:Sep 21 10:13:22 Updated: ovirt-engine-3.6.0-0.0.master.20150920175753.gite4183e5.el7.centos.noarch /var/log/yum.log:Nov 19 12:54:38 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.6.1-0.0.master.20151105190211.git9a4a2d1.el7.centos.noarch /var/log/yum.log:Nov 19 13:09:48 Updated: ovirt-engine-3.6.1-0.0.master.20151105190211.git9a4a2d1.el7.centos.noarch /var/log/yum.log:Dec 24 15:36:19 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.6.3-0.0.master.20151223191430.git7294d71.el7.centos.noarch /var/log/yum.log:Dec 24 16:51:21 Updated: ovirt-engine-3.6.3-0.0.master.20151223191430.git7294d71.el7.centos.noarch > * upgrade log
Created attachment 1110235 [details] ovirt-engine-setup-bz1294091.tar.gz All setup logs attached. The one upgrading to 3.6 is ovirt-engine-setup-20151224161203-60w3tx.log. Then I ran engine-cleanup and engine-setup.
Didi, can you find the range of the original mac pool? (pardon, the natural bug assignee is currently away)
(In reply to Alona Kaplan from comment #8) > Didi, can you find the range of the original mac pool? (pardon, the natural > bug assignee is currently away) I do not think the bug is related to network. See comment 2. Changing to "general" for now, still not sure where the problem is.
Hi, I'm not at work, so I'll try to help just by reading logs: Issue is related to uninitialized MacPoolPerDc. When any operation is called on uninitialized pool, given message is produced. There are few patches bringing MAC pool under CDI, so when they're merged, this will project in error message on app startup, rather than this late. But currently pool is initialized using this method: org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolPerDc#initialize which is called from here: org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean#create reading logs on, this seems to be correctly called, and initialize method as well. From provided message: MAC_POOL_INITIALIZATION_FAILED it seems, that initialization was called, but provided range does not contain sole unicast address — navigate here: org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolManagerRanges#createMacsStorage for historic reasons in this situation MAC_POOL_INITIALIZATION_FAILED is thrown, leaving pool in unitialized state, so any operation on it will fail as well. ——— can you please verify, whether your pool settings contain at least one unicast address? Also notice, that ranges are subject to trimming. If you have defined very large range, which start with like 2^32 multicast macs and then are some unicast, this trimming may be producing different range than you expect. So please verify you mac pool settings (select * from mac_pool_ranges), ideally post result here, if there's something strange, try to use simpler range, and restart engine. If immediately after engine startup you see in log: 2015-12-24 17:22:06,644 INFO [org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) [] Start initializing MacPoolManagerRanges 2015-12-24 17:22:06,688 ERROR [org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) [] Error in initializing MAC Addresses pool manager: EngineException: MAC_POOL_INITIALIZATION_FAILED (Failed with error MAC_POOL_INITIALIZATION_FAILED and code 5010) 2015-12-24 17:22:06,688 ERROR [org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolPerDc] (org.ovirt.thread.pool-8-thread-1) [] Error initializing: EngineException: MAC_POOL_NOT_INITIALIZED (Failed with error MAC_POOL_NOT_INITIALIZED and code 5011) something is wrong with pool setting.
Didi, referring to your comment 2: can you share the log of the failure when trying to create a VM? Martin, maybe you can look into Didi's DB dump for the mac pool range?
After cleanup and setup again in comment 6 the error no longer occurred. I now started the vm running this engine from a snapshot taken before comment 6, and again get the same error when trying to add a mac address pool. The engine no longer has hosts so can't try creating a VM. I am adding access credentials in a private comment.
In a private comment, Martin noted that mac_pool_ranges is empty. This gave me a lead to check the sequence of events leading to the failure in this bug, and now I think it's a duplicate of bug 1265136. At some point, a patch was merged (master [1], 3.6 [2]) that deleted the option MacPoolRanges before adding its content to the new table mac_pool_ranges. It was quickly reverted [3], but I happened to upgrade this machine to a nightly 3.6 snapshot created between them, thus was hit by this. Therefore, I think it's safe to close as duplicate of bug 1265136, which was opened to track this. Regarding bug 1265136 comment 5, which says it's not easy to reproduce, I agree that currently it's not easy, if using only existing builds or branch HEADs (i.e. not building from [1] or [2]). Internal builds to QE also do not seem to include it - rhevm-3.6.0-0.16.master.el6.noarch.rpm was built on Sep 16, a day before [1][2] were merged, and the next one -0.17 on Sep 24, two days after the fix [3] was merged. Leaving for Martin to review and close (and/or comment). [1] https://gerrit.ovirt.org/#/q/I51d7ce9359a9ad38a034d656fb4ac1c5bab2ff43,n,z [2] https://gerrit.ovirt.org/46289 [3] https://gerrit.ovirt.org/#/q/Ic7f4593c55cfb65084d94a96e92f93ec6adcf010,n,z
Let us close it then, and reopen if it ever reproduces with a formal build of Engine. *** This bug has been marked as a duplicate of bug 1265136 ***