Description of problem:
We are working on developing a plugin into the RDO-Manager (OSP-Director) installer which will enable the installation of Nuage as a part of the installer code. As a part of the development process, we have followed this document to deploy undercloud and overcloud:

ftp://partners.redhat.com/b5d52ebbb756856c8e7e6f44880b4741/OpenStack/Documents/director-01jul-html/index.html

We have defined a few Nuage specific parameters and some updates to neutron.conf and nova.conf via heat templates and pass the same using the "-e" option to the "openstack overcloud deploy" CLI. 

What we have noticed is that after a deployment in HA mode (using overcloud_controller_pacemaker.pp) even though the hiera database gets updated, the value in the relevant configuration file is not being updated. 

For example, when we change the "core_plugin" parameter from ml2 to nuage after overcloud was deployed, in HA mode, the output of "hiera neutron::core_plugin" reflects "nuage" as the value, but the value in neutron.conf stays at ml2. 

This behavior is not seen with a single controller setup (using overcloud_controller.pp). 


Version-Release number of selected component (if applicable):

N/A

How reproducible:

Always (10/10 times)

Steps to Reproduce:
1. Deploy overcloud following the steps outlined in (for HA mode deploy 2 controllers) : ftp://partners.redhat.com/b5d52ebbb756856c8e7e6f44880b4741/OpenStack/Documents/director-01jul-html/index.html
2. Once the overcloud is deployed, try to expose core_plugin by modifying relevant files in /usr/share/openstack-tripleo-heat-templates/ on the undercloud. 
3. Once its exposed, set the neutron core plugin resource (say NeutronCorePlugin) to anything other than ml2.  
4. Re-run the openstack overcloud deploy" command with the -e option pointing to your yaml file which sets the  NeutronCorePlugin value.
5. Post deployment, go to the controller overcloud machine and in sudo mode, check the hiera value of core_plugin. If you set things up the right way, you will be able to see the value you set in the template file here. 
6. Now check neutron.conf and it continues to say "ml2".
7. This is true on both controllers in HA mode. 

Actual results:

core_plugin in neutron.conf is ml2.

Expected results:

core_plugin should be nuage

Additional info:

The workaround for this is currently to pull down the overcloud and re-run the overcloud deployment with the updated values. It takes extra time and effort to deploy overcloud over and over again (no pun intended :)).