Bug 1294091 - MAC Address Pool creation fails with: This instance was not yet initialized
MAC Address Pool creation fails with: This instance was not yet initialized
Status: CLOSED DUPLICATE of bug 1265136
Product: ovirt-engine
Classification: oVirt
Component: General (Show other bugs)
3.6.1
Unspecified Unspecified
high Severity medium (vote)
: ovirt-3.6.2
: ---
Assigned To: Martin Mucha
network
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-24 10:40 EST by Yedidyah Bar David
Modified: 2016-02-10 14:15 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-04 09:24:08 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
danken: ovirt‑3.6.z?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
macpoolerrors-engine.log (94.26 KB, text/plain)
2015-12-24 10:40 EST, Yedidyah Bar David
no flags Details
ovirt-engine-setup-bz1294091.tar.gz (2.05 MB, application/x-gzip)
2015-12-29 08:43 EST, Yedidyah Bar David
no flags Details

  None (edit)
Description Yedidyah Bar David 2015-12-24 10:40:56 EST
Created attachment 1109249 [details]
macpoolerrors-engine.log

Description of problem:

Creating a new MAC Address Pool fails.

On web admin interface shows:

Error while executing action AddMacPool: Internal Engine Error

In engine.log:

2015-12-24 17:13:09,252 INFO  [org.ovirt.engine.core.bll.AddMacPoolCommand] (default task-8) [2280bb7b] Running command: AddMacPoolCommand internal: false. Entities affected :  ID: aaa00000
-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_MAC_POOL with role type ADMIN
2015-12-24 17:13:09,264 ERROR [org.ovirt.engine.core.bll.AddMacPoolCommand] (default task-8) [2280bb7b] Command 'org.ovirt.engine.core.bll.AddMacPoolCommand' failed: This instance was not y
et initialized.
2015-12-24 17:13:09,265 ERROR [org.ovirt.engine.core.bll.AddMacPoolCommand] (default task-8) [2280bb7b] Exception: java.lang.IllegalStateException: This instance was not yet initialized.

Version-Release number of selected component (if applicable):

ovirt-engine-3.6.3-0.0.master.20151223191430.git7294d71.el7.centos.noarch

How reproducible:

Not sure, always on my setup

Steps to Reproduce:
1. Choose DC
2. Edit
3. Choose MAC Address Pool
4. New
5. Enter some values
6. OK

Actual results:


Expected results:


Additional info:

Similar errors when trying Configure -> MAC Address Pools.
Attached log includes several attempts.
Comment 1 Dan Kenigsberg 2015-12-26 15:49:46 EST
ovirt-engine 3.6.3-0.0.master.20151223191430.git7294d71.el7.centos
Comment 2 Yedidyah Bar David 2015-12-27 06:01:54 EST
Well, it might be something deeper. Now got a similar error when trying to create a vm. Looking further.
Comment 3 Yedidyah Bar David 2015-12-27 10:39:53 EST
Seems to work well on a clean 3.6 setup.

Previous attempt was on a setup upgraded from 3.5.
Comment 4 Yevgeny Zaspitsky 2015-12-29 08:18:58 EST
Didi,

Could you please supply more info:
* exact 3.5.z version you used
* upgrade log
Comment 5 Yedidyah Bar David 2015-12-29 08:41:08 EST
(In reply to Yevgeny Zaspitsky from comment #4)
> Didi,
> 
> Could you please supply more info:
> * exact 3.5.z version you used

This machine only had snapshot versions, both 3.5 and 3.6 ones:

# grep ovirt-engine-3 $(ls -tr /var/log/yum.log*)
/var/log/yum.log-20150101:Nov 12 12:21:40 Installed: ovirt-engine-setup-plugin-ovirt-engine-3.5.1-0.0.master.20141111182009.git2c24911.el7.centos.noarch
/var/log/yum.log-20150101:Nov 12 12:21:40 Installed: ovirt-engine-3.5.1-0.0.master.20141111182009.git2c24911.el7.centos.noarch
/var/log/yum.log-20150101:Nov 17 13:00:54 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.1-0.0.master.20141112061952.git2c24911.el7.centos.noarch
/var/log/yum.log-20150101:Nov 17 13:01:29 Updated: ovirt-engine-3.5.1-0.0.master.20141112061952.git2c24911.el7.centos.noarch
/var/log/yum.log-20150101:Nov 24 17:57:56 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.1-0.0.master.20141124122901.el7.centos.noarch
/var/log/yum.log-20150101:Nov 24 18:00:40 Updated: ovirt-engine-3.5.1-0.0.master.20141124122901.el7.centos.noarch
/var/log/yum.log-20150515:May 14 16:49:15 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.4-0.0.master.20150512172202.git739dc64.el7.centos.noarch
/var/log/yum.log-20150515:May 14 17:06:12 Updated: ovirt-engine-3.5.4-0.0.master.20150512172202.git739dc64.el7.centos.noarch
/var/log/yum.log:Jul 06 10:40:27 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.5.5-0.0.master.20150705172210.git74a1a6a.el7.centos.noarch
/var/log/yum.log:Jul 06 10:52:42 Updated: ovirt-engine-3.5.5-0.0.master.20150705172210.git74a1a6a.el7.centos.noarch
/var/log/yum.log:Sep 21 10:08:06 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.6.0-0.0.master.20150920175753.gite4183e5.el7.centos.noarch
/var/log/yum.log:Sep 21 10:13:22 Updated: ovirt-engine-3.6.0-0.0.master.20150920175753.gite4183e5.el7.centos.noarch
/var/log/yum.log:Nov 19 12:54:38 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.6.1-0.0.master.20151105190211.git9a4a2d1.el7.centos.noarch
/var/log/yum.log:Nov 19 13:09:48 Updated: ovirt-engine-3.6.1-0.0.master.20151105190211.git9a4a2d1.el7.centos.noarch
/var/log/yum.log:Dec 24 15:36:19 Updated: ovirt-engine-setup-plugin-ovirt-engine-3.6.3-0.0.master.20151223191430.git7294d71.el7.centos.noarch
/var/log/yum.log:Dec 24 16:51:21 Updated: ovirt-engine-3.6.3-0.0.master.20151223191430.git7294d71.el7.centos.noarch

> * upgrade log
Comment 6 Yedidyah Bar David 2015-12-29 08:43 EST
Created attachment 1110235 [details]
ovirt-engine-setup-bz1294091.tar.gz

All setup logs attached.

The one upgrading to 3.6 is ovirt-engine-setup-20151224161203-60w3tx.log. Then I ran engine-cleanup and engine-setup.
Comment 8 Alona Kaplan 2015-12-30 08:39:59 EST
Didi, can you find the range of the original mac pool? (pardon, the natural bug assignee is currently away)
Comment 9 Yedidyah Bar David 2015-12-30 09:59:26 EST
(In reply to Alona Kaplan from comment #8)
> Didi, can you find the range of the original mac pool? (pardon, the natural
> bug assignee is currently away)

I do not think the bug is related to network. See comment 2. Changing to "general" for now, still not sure where the problem is.
Comment 10 Martin Mucha 2015-12-30 16:25:57 EST
Hi, I'm not at work, so I'll try to help just by reading logs:

Issue is related to uninitialized MacPoolPerDc. When any operation is called on uninitialized pool, given message is produced. 

There are few patches bringing MAC pool under CDI, so when they're merged, this will project in error message on app startup, rather than this late.

But currently pool is initialized using this method:
org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolPerDc#initialize

which is called from here:
org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean#create

reading logs on, this seems to be correctly called, and initialize method as well. From provided message: MAC_POOL_INITIALIZATION_FAILED it seems, that initialization was called, but provided range does not contain sole unicast address — navigate here:

org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolManagerRanges#createMacsStorage

for historic reasons in this situation MAC_POOL_INITIALIZATION_FAILED is thrown, leaving pool in unitialized state, so any operation on it will fail as well.

———

can you please verify, whether your pool settings contain at least one unicast address? Also notice, that ranges are subject to trimming. If you have defined very large range, which start with like 2^32 multicast macs and then are some unicast, this trimming may be producing different range than you expect. So please verify you mac pool settings (select * from mac_pool_ranges), ideally post result here, if there's something strange, try to use simpler range, and restart engine. If immediately after engine startup you see in log:

2015-12-24 17:22:06,644 INFO  [org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) [] Start initializing MacPoolManagerRanges
2015-12-24 17:22:06,688 ERROR [org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolManagerRanges] (org.ovirt.thread.pool-8-thread-1) [] Error in initializing MAC Addresses pool manager: EngineException: MAC_POOL_INITIALIZATION_FAILED (Failed with error MAC_POOL_INITIALIZATION_FAILED and code 5010)
2015-12-24 17:22:06,688 ERROR [org.ovirt.engine.core.bll.network.macpoolmanager.MacPoolPerDc] (org.ovirt.thread.pool-8-thread-1) [] Error initializing: EngineException: MAC_POOL_NOT_INITIALIZED (Failed with error MAC_POOL_NOT_INITIALIZED and code 5011)

something is wrong with pool setting.
Comment 11 Dan Kenigsberg 2016-01-04 03:04:11 EST
Didi, referring to your comment 2: can you share the log of the failure when  trying to create a VM?

Martin, maybe you can look into Didi's DB dump for the mac pool range?
Comment 12 Yedidyah Bar David 2016-01-04 03:35:43 EST
After cleanup and setup again in comment 6 the error no longer occurred.

I now started the vm running this engine from a snapshot taken before comment 6, and again get the same error when trying to add a mac address pool.

The engine no longer has hosts so can't try creating a VM.

I am adding access credentials in a private comment.
Comment 15 Yedidyah Bar David 2016-01-04 09:16:18 EST
In a private comment, Martin noted that mac_pool_ranges is empty.

This gave me a lead to check the sequence of events leading to the failure in this bug, and now I think it's a duplicate of bug 1265136.

At some point, a patch was merged (master [1], 3.6 [2]) that deleted the option MacPoolRanges before adding its content to the new table mac_pool_ranges. It was quickly reverted [3], but I happened to upgrade this machine to a nightly 3.6 snapshot created between them, thus was hit by this.

Therefore, I think it's safe to close as duplicate of bug 1265136, which was opened to track this. Regarding bug 1265136 comment 5, which says it's not easy to reproduce, I agree that currently it's not easy, if using only existing builds or branch HEADs (i.e. not building from [1] or [2]). Internal builds to QE also do not seem to include it - rhevm-3.6.0-0.16.master.el6.noarch.rpm was built on Sep 16, a day before [1][2] were merged, and the next one -0.17 on Sep 24, two days after the fix [3] was merged.

Leaving for Martin to review and close (and/or comment).

[1] https://gerrit.ovirt.org/#/q/I51d7ce9359a9ad38a034d656fb4ac1c5bab2ff43,n,z
[2] https://gerrit.ovirt.org/46289
[3] https://gerrit.ovirt.org/#/q/Ic7f4593c55cfb65084d94a96e92f93ec6adcf010,n,z
Comment 16 Dan Kenigsberg 2016-01-04 09:24:08 EST
Let us close it then, and reopen if it ever reproduces with a formal build of Engine.

*** This bug has been marked as a duplicate of bug 1265136 ***

Note You need to log in before you can comment on or make changes to this bug.