Bug 1312007

Summary: When updating overcloud, OS::TripleO::NodeExtraConfigPost: isn't reapplied
Product: Red Hat OpenStack Reporter: David Hill <dhill>
Component: documentationAssignee: Dan Macpherson <dmacpher>
Status: CLOSED CURRENTRELEASE QA Contact: RHOS Documentation Team <rhos-docs>
Severity: medium Docs Contact:
Priority: high    
Version: 7.0 (Kilo)CC: akaris, augol, broskos, dhill, dmacpher, hbrock, jliberma, jslagle, jstransk, mburns, mcornea, nlevinki, nmorell, nstephan, rhel-osp-director-maint, rybrown, sbaker, shardy, srevivo, vcojot
Target Milestone: asyncKeywords: Documentation
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The documentation for OS::TripleO::NodeExtraConfigPost [1] does not include a DeployIdentifier parameter, so the configuration will only be applied during initial stack creation (or when something in the configuration actually changes). This is desirable sometimes, but not when the OS::TripleO::NodeExtraConfigPost is overriding settings applied by puppet since these are run on every stack update. To ensure that the OS::TripleO::NodeExtraConfigPost is applied every time the stack is updated, do the following in the template: - Create a string parameter called DeployIdentifier - Reference that parameter in the input_values of the deployment resource, eg input_values: deploy_identifier: {get_param: DeployIdentifier} An example can be seen here[2]. The documentation[1] should be updated to explain the use of DeployIdentifier. [1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Configuring_after_Overcloud_Creation.html [2] https://review.openstack.org/#/c/296488/1/extraconfig/post_deploy/example_run_on_update.yaml
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-01 07:36:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description David Hill 2016-02-25 13:57:07 UTC
Description of problem:
When updating overcloud, OS::TripleO::NodeExtraConfigPost: isn't reapplied unless the yaml file is modified and the checksum changes.   If a user customizes configuration using this resource, it will be applied only on creation time and if the template itself is modified later on which is a problem if a scale out operation is done because all configuration will be re-applied to all the servers but the postconfig won't be reapplied.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Deploy an overcloud with NodeExtraConfigPost resource
2. Update it, scale it, do anything with it

Actual results:
NodeExtraConfigPost isn't reapplied

Expected results:
It should be reapplied because it's configured to be updated on CREATE and UPDATE

Additional info:
I can reproduce this on my side ... I have to try it with RHOSP8.

Comment 1 David Hill 2016-02-25 23:30:57 UTC
This is my postdeployment.yaml

And this is my network-environment.yaml

I'm lazy so I'm calling it with -e network-environment.yaml instead of having another environment file.

So here are my steps:
1) openstack overcloud deploy -e network-environment.yaml  (etc)
2) ls -latr /root/extra
3) rm -rf /root/extra
4) openstack overcloud deploy -e network-environment.yaml (etc)
5) ls -latr /root/extra (not recreated)

Comment 2 David Hill 2016-02-26 00:23:33 UTC
This is what I get:

-bash-4.2$  heat resource-type-show OS::Heat::StructuredDeployments |grep actions -A4
    "actions": {
      "description": "Which stack actions will result in this deployment being triggered.",
      "default": [

-bash-4.2$ heat resource-list -n3 overcloud|grep OS::TripleO::NodeExtraConfigPost
| ExtraConfig                                   | 3233aac1-1af7-42c0-a0d9-53712812291b          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T04:37:43Z | ObjectStorageNodesPostDeployment              |
| ExtraConfig                                   | 89c8f861-0ef7-4337-807c-d9b5c9895a2d          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T04:38:27Z | ComputeNodesPostDeployment                    |
| ExtraConfig                                   | 01d1cadf-680e-4e05-92ee-274fbe031c10          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T05:06:41Z | ControllerNodesPostDeployment                 |
| ExtraConfig                                   | 34bde646-81e6-4b16-867f-7b7f7c8a8623          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T05:08:24Z | CephStorageNodesPostDeployment                |

-bash-4.2$ heat stack-list --show-nested | grep 26b623c1-776d-4d2d-87e0-9a52e00eb74
| 26b623c1-776d-4d2d-87e0-9a52e00eb74b | overcloud-ControllerNodesPostDeployment-cwywbgdhhqfm-ExtraConfig-hn32x2vn4u4t-ExtraDeployments-6y4s2baxbeqt | UPDATE_COMPLETE | 2016-02-25T02:12:14Z | 01d1cadf-680e-4e05-92ee-274fbe031c10 |

-bash-4.2$ heat resource-list -n5 26b623c1-776d-4d2d-87e0-9a52e00eb74b
| resource_name | physical_resource_id                 | resource_type                  | resource_status | updated_time         | parent_resource  |
| 0             | b25f8aac-eead-443b-9039-63b5db91ca4c | OS::Heat::StructuredDeployment | CREATE_COMPLETE | 2016-02-25T02:12:14Z | ExtraDeployments |
| 1             | f056b88a-424f-4fc0-8317-56cefe64291e | OS::Heat::StructuredDeployment | CREATE_COMPLETE | 2016-02-25T02:12:14Z | ExtraDeployments |
| 2             | 0629f217-d140-4b96-94bd-39f289dd751a | OS::Heat::StructuredDeployment | CREATE_COMPLETE | 2016-02-25T02:12:14Z | ExtraDeployments |

-bash-4.2$ heat deployment-show b25f8aac-eead-443b-9039-63b5db91ca4c
  "status": "COMPLETE",
  "server_id": "3fd5e2a7-8dee-4d08-b543-6f891d5da90b",
  "config_id": "7463822f-761d-4167-8b85-20694713e476",
  "output_values": {
    "deploy_stdout": "",
    "deploy_stderr": "",
    "deploy_status_code": 0
  "creation_time": "2016-02-25T02:12:17Z",
  "updated_time": "2016-02-25T02:13:11Z",
  "input_values": {},
  "action": "CREATE",
  "status_reason": "Outputs received",
  "id": "b25f8aac-eead-443b-9039-63b5db91ca4c"

Comment 4 Ryan Brown 2016-02-26 16:54:36 UTC
I see what's going on here. This is my understanding of your issue:

1. The regular TripleO config process makes change X
2. Your postconfig changes X to X-prime
3. On a stack update (incl. scale-up) regular config runs and reverts state to X
4. Your postconfig only runs on the new node, not on existing ones

The reason for this is because the configuration SWDeployment resource takes a parameter, DeployIdentifier, that changes on every stack action to force the update. 

According to Heat, there's no relation between the standard configuration and your postconfig, because the dependency is somewhere in the pile of bash that, to heat, is an unparseable blob. Thus, when a new instance is added Heat doesn't know to re-run the configuration on existing nodes because the script in the postconfig hasn't actually changed. 

As you noticed, if you change the script it is re-run, because Heat can see that the inputs have changed for all servers, triggering the update. 

If you want your postconfig to run every time, you can do that by depending on the same deployID that the standard config does, defined at[1] and depended upon[2]. This will result in your config being overridden briefly, then fixed by postconfig so be aware of that. 

[1]: https://github.com/david-hill/rhosp7/blob/master/my-overcloud/overcloud-without-mergepy.yaml#L684-L689
[2]: https://github.com/david-hill/rhosp7/blob/master/my-overcloud/overcloud-without-mergepy.yaml#L1349-L1357

Comment 5 Steve Baker 2016-02-28 20:37:53 UTC
The ExtraConfig resource[1] has no parameter available to pass in the NodeConfigIdentifers, and we can't add one without breaking ExtraConfig templates in already in the field.

However the ControllerPostPuppet *does*[2], so I would suggest switching to that so the config can be applied every time.

Unfortunately for HA pacemaker managed controllers ControllerPostPuppet is mapped to post_puppet_pacemaker.yaml[3] so you'll need to make sure it is still invoked by defining another type in your environment which maps to [3] and defining a resource of that type in your postdeployment.yaml.

For RHOSP 8 I think we should deprecate OS::TripleO::NodeExtraConfigPost and add a new type which also allows passing in NodeConfigIdentifers.

[1] https://github.com/david-hill/rhosp7/blob/master/my-overcloud/puppet/controller-post-puppet.yaml#L122-L128
[2] https://github.com/david-hill/rhosp7/blob/master/my-overcloud/puppet/controller-post-puppet.yaml#L114-L120
[3] https://github.com/david-hill/rhosp7/blob/master/my-overcloud/extraconfig/tasks/post_puppet_pacemaker.yaml

Comment 11 Nathan Morell 2016-03-02 22:09:27 UTC
I felt it necessary to add the use case I'm building and why this matters. I'm currently running my service endpoints on non-standard ports, because of this I have to edit haproxy.cfg as part of my deployment. From deployment to deployment this config does not change but I'm forced to edit the template each time I want to add additional hardware because it wipes out the changes. This is a must have.

Comment 12 David Hill 2016-03-03 00:07:38 UTC
@Nathan Morell: At this time, what I could suggest you is to do something like this:

- Deploy the overcloud/update/do whatever you want to do
- Create a shell script that applies those changes
- Run that script after each openstack commands (from the undercloud)

Comment 18 Steven Hardy 2016-03-23 14:34:55 UTC
I agree with Ryan that using DeployIdentifier should resolve this problem.

I posted an example illustrating how this works:


I'll also follow up with a docs patch explaining how this works, but basically tripleoclient passes a unique identifier (timestamp) every update via the DeployIdentifer parameter.  Since we use parameter_defaults to pass it, any template can make use of it.

Comment 20 Andreas Karis 2016-03-23 14:52:27 UTC
Steven, is it only necessary to have DeployIdentifier as a parameter to this template, or does it need to be passed into the input_values or SoftwareDeployments and from there possibly even into the SoftwareConfig?

Comment 22 Steve Baker 2016-03-23 20:39:53 UTC
Ah, I've been working on the assumption that DeployIdentifier was set in parameters, but it is set in parameter_defaults. This makes my comment #7 irrelevant, the solution is already available in shardy's comment #18.

Andreas(In reply to Andreas Karis from comment #20)
> Steven, is it only necessary to have DeployIdentifier as a parameter to this
> template, or does it need to be passed into the input_values or
> SoftwareDeployments and from there possibly even into the SoftwareConfig?

DeployIdentifier changes every time a stack update occurs, and having any input_values change will trigger the deployment again, so in shardy's example the only lines really necessary are 15,16,38,39

Comment 23 Steve Baker 2016-03-23 20:43:13 UTC
This needs to become a documentation bug to include DeployIdentifier in the example in section 8.3


Comment 24 Steven Hardy 2016-03-24 15:54:56 UTC
(In reply to Andreas Karis from comment #20)
> Steven, is it only necessary to have DeployIdentifier as a parameter to this
> template, or does it need to be passed into the input_values or
> SoftwareDeployments and from there possibly even into the SoftwareConfig?

It must be passed into the SoftwareDeployment input_values, but it doesn't necessarily have to be actually used by the SoftwareConfig.

Just adding the DeployIdentifier parameter isn't enough, because the SoftwareDeployment will still see no change.

Comment 26 Amit Ugol 2016-06-21 12:37:25 UTC
Moved to Omri as he is responsible for updates.

Comment 27 Marius Cornea 2016-08-25 14:28:56 UTC
(In reply to Steve Baker from comment #23)
> This needs to become a documentation bug to include DeployIdentifier in the
> example in section 8.3
> https://access.redhat.com/documentation/en-US/
> Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/
> Director_Installation_and_Usage/sect-Configuring_after_Overcloud_Creation.
> html

Moving this to the documentation component.

I was able to verify this by using the example in comment #18. After 2 successive overcloud deploy runs I got the following extra_update file:

[root@overcloud-controller-0 ~]# cat /root/extra_update 
extra_update 1472132055
extra_update 1472133706

Comment 29 Amit Ugol 2018-02-01 07:29:43 UTC
LGTM, clearing needinfo state on this old thing.

Comment 30 Dan Macpherson 2018-02-01 07:36:12 UTC
Thanks, Amit!