Bug 1267855 - unable to update overcloud configuration from underclound node.
unable to update overcloud configuration from underclound node.
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-rdomanager-oscplugin (Show other bugs)
7.0 (Kilo)
All Linux
medium Severity medium
: y2
: 7.0 (Kilo)
Assigned To: Steve Baker
Amit Ugol
n1kv
: Unconfirmed, ZStream
: 1273222 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-01 04:49 EDT by Pratik Pravin Bandarkar
Modified: 2016-04-26 12:47 EDT (History)
19 users (show)

See Also:
Fixed In Version: python-rdomanager-oscplugin-0.0.10-16.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the base resource registry environment was included for all overcloud stack updates, which meant customizations may be lost unless all environment files are repeated in order when calling "openstack overcloud deploy". With this update, it is possible to call "openstack overcloud deploy" with no environments without losing customizations. If any environment files are specified, then all environment files must be specified again in the desired order.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-21 11:49:49 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs from one of the controllers (100.92 KB, text/plain)
2015-11-11 14:52 EST, Shiva Prasad Rao
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1504363 None None None Never
OpenStack gerrit 238231 None None None Never

  None (edit)
Description Pratik Pravin Bandarkar 2015-10-01 04:49:26 EDT
1. Proposed title of this feature request  
allow updating overcloud configuration without login to the overcloud nodes.

3. What is the nature and description of the request?  
If we want to update any overcloud configuration then, we need to login to overcloud nodes in order to make changes. But, there should be some way to update overcloud configuration from undercloud node without need to login overcloud nodes. 

For ex. if we want to add dns forwarder for network then, from overcloud node we need to update  "dnsmasq_dns_servers" parameter from file "/etc/neutron/dhcp_agent.ini". So, there should be some way to update such configuration changes from undercloud system itself(without need login overcloud nodes).
Comment 5 Zane Bitter 2015-10-05 09:25:55 EDT
(In reply to Pratik Pravin Bandarkar from comment #2)
> 1. If we manually update the overcloud node configuration after login to
> overcloud nodes and then we update the overcloud from director(`openstack
> overcloud update stack` / `openstack overcloud deploy` ) then will it
> replace the config changes to default from overcloud nodes ?

It should, yes. (Note, the command to do this is `openstack overcloud deploy`. `openstack overcloud update stack` is for updating packages on the overcloud nodes with yum.) In 7.1 there are some subtleties involving when the client will re-send changed files, so it's possible there is a bug here or at least some documentation needed.

> 2. If we update the overcloud nodes with the method mentioned in [1], the
> changes are permanent ? will it get replace back to default on further stack
> update ?

Yes.

> [1]https://access.redhat.com/documentation/en-US/
> Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html-single/
> Director_Installation_and_Usage/index.html#sect-
> Configuring_after_Overcloud_Creation
Comment 6 Zane Bitter 2015-10-05 09:30:24 EDT
Bug 1268415 may be related.
Comment 9 Zane Bitter 2015-10-06 12:20:59 EDT
"Message: No valid host was found. There are not enough hosts available., Code: 500"

This is due to Heat trying to create a server in Nova/Ironic and there being none available to do so. I'm not sure why anything that you're doing would be creating an additional server though. I don't see anything wrong with the template or what you're trying to do.
Comment 10 Pratik Pravin Bandarkar 2015-10-07 05:44:36 EDT
ok. Its progressing.  I am able to update the configuration. But its updating the config files from compute and controller nodes. How to restrict the changes to controller node ?

My reproducer details:

<snip>
Deployed the stack:

[stack@instack ~]$  openstack overcloud deploy  --templates  



- Created below yaml files to update 

$ cat post_config_mod.yaml 
resource_registry:
  OS::TripleO::NodeExtraConfigPost: /home/stack/templates1/dnsmasq_dns_servers_mod.yaml


$ cat /home/stack/templates1/dnsmasq_dns_servers_mod.yaml
heat_template_version: 2014-10-16

description: >
  Example file to update /etc/neutron/dhcp_agent.ini
parameters:
  servers:
    type: json

resources:

  ExtraConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config: {get_file: /home/stack/templates1/update.sh}


  ExtraDeployments:
    type: OS::Heat::SoftwareDeployments
    properties:
      servers:  {get_param: servers}
      config: {get_resource: ExtraConfig}
      actions: ['CREATE', 'UPDATE']


$ cat /home/stack/templates1/update.sh
#!/bin/bash
#update "/etc/neutron/dhcp_agent.ini"
openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_dns_servers 8.8.8.8

____

- Update the stack:
$ # openstack overcloud deploy  --templates   -e ~/templates/post_config_mod.yaml

- From controller node I can see:

# grep -i 8.8.8.8 /etc/neutron/dhcp_agent.ini
dnsmasq_dns_servers = 8.8.8.8  <==

Changes reflected on compute node as well:
[root@overcloud-compute-0 heat-admin]# grep -i 8.8.8.8 /etc/neutron/dhcp_agent.ini
dnsmasq_dns_servers = 8.8.8.8

====================================================

Can you help to restrict the changes to controller node ?
Comment 11 Zane Bitter 2015-10-07 09:49:12 EDT
NodeExtraConfigPost applies to all types of nodes. There is a ControllerNodesPostDeployment that applies only to Controller nodes, but I believe this is not intended to be used for user customisations.

There is a ControllerExtraConfigPre deployment which _is_ designed for user customisations, and which applies only to controllers, but it is deployed before rather than after the main puppet deployment, so it may not work for your purposes here.

In any event, this is a question for an actual TripleO expert.
Comment 12 Jan Provaznik 2015-10-07 10:06:03 EDT
I wonder if any of above changes might be set through puppet modules which takes care of generating config files related to OS services, if so, you can put puppet config into hieradata files:
http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/extra_config.html#making-configuration-changes
Comment 13 Pratik Pravin Bandarkar 2015-10-09 02:31:14 EDT
(In reply to Jan Provaznik from comment #12)
> I wonder if any of above changes might be set through puppet modules which
> takes care of generating config files related to OS services, if so, you can
> put puppet config into hieradata files:
> http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/
> extra_config.html#making-configuration-changes

Can you point me the exact content we need to put in controller.yaml file in below scenario?

1. update "dnsmasq_dns_servers" entry from "/etc/neutron/dhcp_agent.ini" file
----
I already tried updating "/usr/share/openstack-tripleo-heat-templates/puppet/hieradata/controller.yaml" as below:
[...]
# neutron
neutron::core_plugin: 'ml2'
neutron::service_plugins:
  - 'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin'
neutron::server::sync_db: true
neutron::agents::dhcp::dnsmasq_config_file: /etc/neutron/dnsmasq-neutron.conf
neutron::agents::dhcp::dnsmasq_dns_servers: 10.75.5.25  <===
[...]

[stack@instack hieradata]$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 98c7c0e4-884b-4c86-a4d5-0d631c771a5e | overcloud  | UPDATE_COMPLETE | 2015-10-07T15:18:39Z |
+--------------------------------------+------------+-----------------+----------------------+

___

But still it didnt updated the config file on controller node:

[stack@instack hieradata]$ ssh heat-admin@192.0.2.15

[root@overcloud-controller-0 heat-admin]# grep -i dnsmasq_dns_servers  /etc/neutron/dhcp_agent.ini |grep -v '#'
dnsmasq_dns_servers = 8.8.8.8  <==

----------



2. update cinder.conf file with enterprise storage configuration on the controller node.
for ex: "Example 2.4. Sample FC configuration" from http://docs.openstack.org/kilo/config-reference/content/dell-storagecenter-driver.html


For both scenario I dont see any good documentation which explain what format/syntax/values we should use while updating heiradata file.

can you shed some light on the same ? 
____________________________________________

Also, is it good practice to update "/usr/share/openstack-tripleo-heat-templates/puppet/hieradata/" if we want to make any changes specific to controller/compute/storage nodes ? I guess we should have alternate option.
Comment 14 Jan Provaznik 2015-10-09 05:03:18 EDT
(In reply to Pratik Pravin Bandarkar from comment #13)
> (In reply to Jan Provaznik from comment #12)
> > I wonder if any of above changes might be set through puppet modules which
> > takes care of generating config files related to OS services, if so, you can
> > put puppet config into hieradata files:
> > http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/
> > extra_config.html#making-configuration-changes
> 
> Can you point me the exact content we need to put in controller.yaml file in
> below scenario?
> 
> 1. update "dnsmasq_dns_servers" entry from "/etc/neutron/dhcp_agent.ini" file
> ----
> I already tried updating
> "/usr/share/openstack-tripleo-heat-templates/puppet/hieradata/controller.
> yaml" as below:
> [...]
> # neutron
> neutron::core_plugin: 'ml2'
> neutron::service_plugins:
>   - 'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin'
> neutron::server::sync_db: true
> neutron::agents::dhcp::dnsmasq_config_file: /etc/neutron/dnsmasq-neutron.conf
> neutron::agents::dhcp::dnsmasq_dns_servers: 10.75.5.25  <===
> [...]
> 
> [stack@instack hieradata]$ heat stack-list
> +--------------------------------------+------------+-----------------+------
> ----------------+
> | id                                   | stack_name | stack_status    |
> creation_time        |
> +--------------------------------------+------------+-----------------+------
> ----------------+
> | 98c7c0e4-884b-4c86-a4d5-0d631c771a5e | overcloud  | UPDATE_COMPLETE |
> 2015-10-07T15:18:39Z |
> +--------------------------------------+------------+-----------------+------
> ----------------+
> 
> ___
> 
> But still it didnt updated the config file on controller node:
> 
> [stack@instack hieradata]$ ssh heat-admin@192.0.2.15
> 
> [root@overcloud-controller-0 heat-admin]# grep -i dnsmasq_dns_servers 
> /etc/neutron/dhcp_agent.ini |grep -v '#'
> dnsmasq_dns_servers = 8.8.8.8  <==
> 
> ----------
> 
> 
> 
> 2. update cinder.conf file with enterprise storage configuration on the
> controller node.
> for ex: "Example 2.4. Sample FC configuration" from
> http://docs.openstack.org/kilo/config-reference/content/dell-storagecenter-
> driver.html
> 
> 
> For both scenario I dont see any good documentation which explain what
> format/syntax/values we should use while updating heiradata file.
> 
> can you shed some light on the same ? 

I'm not familiar with puppet modules - adding Emilien whose team takes care of puppet.

> ____________________________________________
> 
> Also, is it good practice to update
> "/usr/share/openstack-tripleo-heat-templates/puppet/hieradata/" if we want
> to make any changes specific to controller/compute/storage nodes ? I guess
> we should have alternate option.

In the doc I linked is a note about using a custom location for templates. It links this bit:
http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/template_deploy.html#custom-template-location
Comment 15 Steve Baker 2015-10-11 19:15:17 EDT
This may be related to https://bugs.launchpad.net/tripleo/+bug/1504363

I'm investigating now.
Comment 16 Steve Baker 2015-10-11 23:00:47 EDT
The following worked for me.

$ cat dnsmasq_dns_servers_env.yaml 
resource_registry:
  OS::TripleO::ControllerExtraConfigPre: dnsmasq_dns_servers_mod.yaml

$ cat dnsmasq_dns_servers_mod.yaml 
heat_template_version: 2014-10-16

description: >
  Example file to update /etc/neutron/dhcp_agent.ini
parameters:
  server:
    type: string

resources:

  DhcpAgentConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config: {get_file: update.sh}

  DhcpAgentDeployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      server:  {get_param: server}
      config: {get_resource: DhcpAgentConfig}

$ cat update.sh 
#!/bin/bash
#update "/etc/neutron/dhcp_agent.ini"
openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_dns_servers 8.8.8.8

Invoke by adding -e dnsmasq_dns_servers_env.yaml to the existing openstack overcloud deploy command.
Comment 17 Pratik Pravin Bandarkar 2015-10-12 02:32:01 EDT
(In reply to Steve Baker from comment #16)
> The following worked for me.
> 
> $ cat dnsmasq_dns_servers_env.yaml 
> resource_registry:
>   OS::TripleO::ControllerExtraConfigPre: dnsmasq_dns_servers_mod.yaml
> 
> $ cat dnsmasq_dns_servers_mod.yaml 
> heat_template_version: 2014-10-16
> 
> description: >
>   Example file to update /etc/neutron/dhcp_agent.ini
> parameters:
>   server:
>     type: string
> 
> resources:
> 
>   DhcpAgentConfig:
>     type: OS::Heat::SoftwareConfig
>     properties:
>       group: script
>       config: {get_file: update.sh}
> 
>   DhcpAgentDeployment:
>     type: OS::Heat::SoftwareDeployment
>     properties:
>       server:  {get_param: server}
>       config: {get_resource: DhcpAgentConfig}
> 
> $ cat update.sh 
> #!/bin/bash
> #update "/etc/neutron/dhcp_agent.ini"
> openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT
> dnsmasq_dns_servers 8.8.8.8
> 
> Invoke by adding -e dnsmasq_dns_servers_env.yaml to the existing openstack
> overcloud deploy command.

Thanks a lot Steve.

ControllerExtraConfigPre:
it seems to be its hook for site-specific additional pre-deployment config.


How to implement such changes while initial deployment itself ? I guess "ControllerExtraConfigPre" will not help here as, it will get executed before deplying the configuration. <== please correct me if I am wrong.
Comment 18 Pratik Pravin Bandarkar 2015-10-12 03:08:15 EDT
also need help on: point 2 from comment #13

2. update cinder.conf file with enterprise storage configuration on the controller node.
for ex: "Example 2.4. Sample FC configuration" from http://docs.openstack.org/kilo/config-reference/content/dell-storagecenter-driver.html

________

How to update cinder.conf after deploying and while deploying the overcloud ?
Comment 19 Steve Baker 2015-10-12 15:20:51 EDT
> Thanks a lot Steve.
> 
> ControllerExtraConfigPre:
> it seems to be its hook for site-specific additional pre-deployment config.
> 
> 
> How to implement such changes while initial deployment itself ? I guess
> "ControllerExtraConfigPre" will not help here as, it will get executed
> before deplying the configuration. <== please correct me if I am wrong.

This can be used during the initial overcloud deploy, or added later to the existing overcloud. It is executed before deploying the configuration but the puppet modules only set values on existing configuration files, they don't overwrite the entire config file every time.

I'll take a look at comment #13, and I'll also try and come up with an alternative to comment #16 which just sets hieradata.
Comment 20 Steve Baker 2015-10-12 18:01:38 EDT
(In reply to Jan Provaznik from comment #14)
> (In reply to Pratik Pravin Bandarkar from comment #13)
> > (In reply to Jan Provaznik from comment #12)

> > 2. update cinder.conf file with enterprise storage configuration on the
> > controller node.
> > for ex: "Example 2.4. Sample FC configuration" from
> > http://docs.openstack.org/kilo/config-reference/content/dell-storagecenter-
> > driver.html

Here is a cinder example for the netapp driver[1]. The cinder puppet module has support for the dell_iscsi backend[2] so you should be able to come up with an equivalent template. And you can use the multiple approach[3] to combine all your controller customisations.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/extraconfig/pre_deploy/controller/cinder-netapp.yaml
[2] https://github.com/openstack/puppet-cinder/blob/master/manifests/backend/dellsc_iscsi.pp
[3] https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/extraconfig/pre_deploy/controller/multiple.yaml
Comment 21 Mike Burns 2015-10-20 06:36:21 EDT
*** Bug 1273222 has been marked as a duplicate of this bug. ***
Comment 23 Steve Baker 2015-10-20 16:27:47 EDT
There needs to be documentation for each customisation scenario in the official docs:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html-single/Director_Installation_and_Usage/index.html#chap-Creating_Custom_Configuration

Could you please raise a documentation bug for every scenario where it is not obvious how to customise something? We can't cover every customisation scenario in this bug.
Comment 24 Steve Baker 2015-10-20 16:38:46 EDT
Lets use this bug to track the issue described in bug #1273227, from ggillies:

Hi,

If I make a change to my puppet manifests or hiera data on my undercloud templates, then do another overcloud deploy (in an attempt to update an existing stack), then the changes do not apply.

This is because of this bug which also affects RHOS

https://bugs.launchpad.net/tripleo/+bug/1504363

Please consider this an issue of the highest priority as it's critically broken the ability for us to update and customise our deployed environments.

Regards,

Graeme
Comment 25 Steve Baker 2015-10-20 16:44:57 EDT
from https://bugs.launchpad.net/tripleo/+bug/1504363/comments/5

I believe this particular issue is caused by the fix to bug #1498607. The root registry is only included during create now, so any changes within tripleo-heat-templates won't be picked up.

https://review.openstack.org/#/c/226306/
Comment 26 Steve Baker 2015-10-20 16:49:51 EDT
The workaround is to specify the base environment on subsequent calls to openstack overcloud deploy, plus all other customisation -e environments, ie:

  openstack overcloud deploy -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e ...

I'll discuss options for fixes in subsequent comments
Comment 27 Steve Baker 2015-10-20 17:05:26 EDT
Fix proposal 1:

On an openstack overcloud deploy which results in a stack-update, implicitly include overcloud-resource-registry-puppet.yaml if the user specifies any other custom environments. Do not include overcloud-resource-registry-puppet.yaml if the user specifies no extra environments.

This solution would be an improvement over the current situation, but it isn't perfect.

It does the right thing when:
- the user does an openstack overcloud with no environments specified
- the user does an openstack overcloud with all custom environments specified

It will sometimes do the wrong thing when:
- the user specifies a partial list of custom environments (like -e do_my_custom_oneoff_thing.yaml)
Comment 28 Steve Baker 2015-10-20 17:21:14 EDT
Fix proposal 2:

Split out the resources in overcloud-resource-registry-puppet.yaml which are for customisation resources that are noop by default. (lets say its called overcloud-resource-registry-custom-defaults.yaml)

What is left of overcloud-resource-registry-puppet.yaml will always be invoked implicitly for every stack operation (overcloud deploy, overcloud update). overcloud-resource-registry-custom-defaults.yaml will only be included implicitly on the first overcloud deploy.

This will allow the user to specify zero, all, or one environment on subsequent overcloud operations, and their existing customisations will never be overwritten with noops.

The only problem I see with this approach is if new customisation resource types are added, the user will need to invoke their own noop environment for the new types.
Comment 29 Graeme Gillies 2015-10-20 22:44:26 EDT
Hi,

I'm not sure if this is the place to add this, but another thing I noticed I want to make clear

The way we operators expect things to work is, when we fire off an overcloud deploy (which is essentially an update), we expect all puppet code to run again. That is, do a round of system convergence.

At the moment I find I have to edit one of the manifest files (usually just adding a comment) in order for an overcloud deploy to reapply puppet.

especially as we add customisations, an overcloud deploy should always run all puppet code

Regards,

Graeme
Comment 30 Steve Baker 2015-10-21 17:20:08 EDT
I've posted a fix to upstream tripleoclient, and I'd like to wait for that to land before I "backport" to python-rdomanager-oscplugin, so reviews on that would be appreciated.

https://review.openstack.org/#/c/238231/
Comment 31 Shiva Prasad Rao 2015-11-11 14:52:11 EST
I patched this fix on my setup but I am still not able to get the puppet to update the configuration. I see that the hieradata is updated correctly.

(Attaching the logs)
Comment 32 Shiva Prasad Rao 2015-11-11 14:52 EST
Created attachment 1092856 [details]
logs from one of the controllers
Comment 33 Mike Burns 2015-11-11 15:04:10 EST
Shiva, can you leave a comment on the upstream patch?
Comment 34 Shiva Prasad Rao 2015-11-18 19:52:54 EST
Will this be a part of y2?

https://review.openstack.org/#/c/244346/1

https://review.openstack.org/#/c/244350/2
Comment 39 Amit Ugol 2015-12-14 04:41:20 EST
The solution described here is working for me. I haven't tested all possible settings but for the use-cases described here it is working.
Comment 41 errata-xmlrpc 2015-12-21 11:49:49 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2650

Note You need to log in before you can comment on or make changes to this bug.