Description of problem:
When deploying a 3 controller + 6 compute node deployment with ODLs in HA collocated with the OSP controllers, occasionally we see a deploy failing with the following error.

curl -k -o /dev/null --fail --silent --head -u admin:admin http://172.16.0.13:8081/restconf/operational/network-topology:network-topology/topology/netvirt:1 returned 22 instead of one of [0]"

172.16.0.13 is the VIP for ODL.

On further investigation by Tim Rozet, we found out that there were no networking issues. The curl would work from all 3 controllers and some of the computes but not all computes. 

The issue seems to be that ODL features are not loaded in the correct order sometimes  leading to a non-functional ODL (started but returning HTTP 404) on one of the controllers. So this seems to be an initialization race condition where Jersey needs to finish initialization before ODL starts. More details in the commit message here: https://git.opendaylight.org/gerrit/#/c/70979/

java.lang.RuntimeException: Error obtaining AAAShiroProvider 


               

Version-Release number of selected component (if applicable):
OSP 13

How reproducible:

Very ocassionally only during scale deploys mostly

Steps to Reproduce:
1. Deploy OSP + ODL setup with a lot of computes (6 in our case)
2.
3.

Actual results:
Deploy failed

Expected results:
Deploy should succeeed everytime

Additional info: