Bug 1569665

Assigning it to Michael as he is an assignee for the upstream bug.

Analysis in upstream bug CONTROLLER-1815 has determined that this could (only) be caused by downstream somehow somewhere doing a featuresBoot= or "feature:install" of "jolokia" (wrong) instead of "odl-jolokia" (right).  Mike, is that the case?

FTR: Janki has confirmed in email "that neither TripleO nor Puppet-ODL uses jolokia (instead of odl-jolokia)".

Tomas, can you check if this is affecting the clustering monitoring tool?

For references I install these features
https://github.com/smalleni/openstack-templates/blob/master/RDU-Perf/13/Dell/opendaylight-transactions.yaml

Could that be causing this?

> I install these features
> Could that be causing this?

I had a look at this list (good idea!), just copy/paste for future reference:

parameter_defaults:
  OpenDaylightFeatures: ["odl-mdsal-trace","odl-netvirt-openstack","odl-netvirt-ui","odl-jolokia"]

and did another local test trying to install these 3 of these 4 features into a local build from upstream stable/oxygen, and with those still cannot reproduce this; with same conclusions reached in CONTROLLER-1815.

The feature I could not myself locally install is odl-netvirt-ui.  Wondering if that could be the culprit here?? But I'm very surprised that you still have "odl-netvirt-ui"; that is not available anymore now, and a "feature:install odl-netvirt-ui" fails in a local build from upstream stable/oxygen.  (That's the DLUX UI which we are no longer supporting upstream;  in the next ODL Fluorine that "odl-netvirt-ui" feature has therefore been completely removed from Git; in Oxygen it's still in netvirt's git, but no longer built, so not available anymore either.)

So do try if removing "odl-netvirt-ui" by any chance does the trick?

But either way, whether removing "odl-netvirt-ui" fixes this or even if not, I do want to understand how come you can still install a feature which shouldn't be available anymore at all. 

Tomas and Sai, could you both also attach the output of this Karaf Shell CLI command in your envs:

  karaf@root()> feature:list

Sai and Tomas, can I ask you to also attach the same information I've just asked for from JamO in CONTROLLER-1815 - the "feature:list -r" thing followed by xN a "feature:info -t ..." line.

Jamo is chasing this upstreamin CSIT; whereas my understanding is that you both are hitting this downstream, where we have discprecancies; I'm interested in information from both.

PFA
karaf@root()> feature:list
https://gist.github.com/smalleni/d2cc9caf60d06a0d38c3942876ff6926
feature:list -r
https://gist.github.com/smalleni/79c163bf80773fa60de372e7836ed302

feature:info -t did not return before my patience timed out :-)

Created attachment 1425702 [details]
from https://gist.githubusercontent.com/smalleni/79c163bf80773fa60de372e7836ed302/raw/1647350c7512569fe3ee474e98a02be8c22871e5/gistfile1.txt

Created attachment 1425703 [details]
https://gist.githubusercontent.com/smalleni/d2cc9caf60d06a0d38c3942876ff6926/raw/ebde37406d1467b3e7e68720d5ee568da1f22ee7/gistfile1.txt

> do want to understand how come you can still install a feature
> which shouldn't be available anymore at all. 

The "feature:list -r" output shows that "odl-netvirt-ui" was ignored, so forget that.

> feature:info -t did not return before my patience timed out :-)

That's very curious, because it (should) just print out a list, it runs really quick for me.  Never mind though, I already spotted something more interestgin:

But feature:list shows that feature "jolokia" (not odl-jolokia) 1.3.5 is Uninstalled.  So my idea in CONTROLLER-1815 that is a double installation of jolokia and odl-jolokia is not what is causing this here. (It would also cause it, but is not causing it here.)

Something else that I just noticed in CONTROLLER-1815 is that it could be not duplicate jolokia Karaf features, but bundles...

Sai or Thomas, could you please attach (a) the full Karaf log, not just the exception, and (b) the output of "bundle:list -s -l" from an environment where you hit this?

I've made a breakthrough in the upstream issue (CONTROLLER-1815), and now underestand exactly what is causing this.  The underlying issue got fixed in upstream master as well as upcoming Oxygen SP1; but through a "fix" for this particular problem, but as part of the odlparent and yangtools version bumps which included a Karaf version bump.  

It's however very far from trivial, if at all possible, to backpatch that downstream.  We'll get this fixed "for free" whenever we pull to rebase on future Oxygen SP1.  As this issue really is "just" a certainly confusing, but absolutely not core functionality impacting problem, and really just a "bad" log message (of which we have many many; and made great progress in reducing now, but have some left, such as this), I'm setting priority Low and Impact Low on this, and will let it linger here until we'll get it fixed by adopting Oxygen SP1. 

Moving to  Oxygen SP1 in the very short term just to get a fix for this small issue is, IMHO, absolutely not worth it, and could easily lead to other problems; so let that happen in a future cycle - we'll all live just fine with this confusing error log message until then.

> but through a "fix" for this particular problem, but as part of....

"but NOT through..."

Mike, this problem was actually fixed in Oxygen SR1 (or SR2, not sure; but months ago already, not recently).  Can you do the right/needful BZ actions?