Bug 1312007

This is my postdeployment.yaml
https://github.com/david-hill/rhosp7/blob/7.3/my-overcloud/extraconfig/post_deploy/postdeployment.yaml

And this is my network-environment.yaml
https://github.com/david-hill/rhosp7/blob/7.3/network-environment.yaml

I'm lazy so I'm calling it with -e network-environment.yaml instead of having another environment file.

So here are my steps:
1) openstack overcloud deploy -e network-environment.yaml  (etc)
2) ls -latr /root/extra
3) rm -rf /root/extra
4) openstack overcloud deploy -e network-environment.yaml (etc)
5) ls -latr /root/extra (not recreated)

This is what I get:

-bash-4.2$  heat resource-type-show OS::Heat::StructuredDeployments |grep actions -A4
    "actions": {
      "description": "Which stack actions will result in this deployment being triggered.",
      "default": [
        "CREATE",
        "UPDATE"
      ],


-bash-4.2$ heat resource-list -n3 overcloud|grep OS::TripleO::NodeExtraConfigPost
| ExtraConfig                                   | 3233aac1-1af7-42c0-a0d9-53712812291b          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T04:37:43Z | ObjectStorageNodesPostDeployment              |
| ExtraConfig                                   | 89c8f861-0ef7-4337-807c-d9b5c9895a2d          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T04:38:27Z | ComputeNodesPostDeployment                    |
| ExtraConfig                                   | 01d1cadf-680e-4e05-92ee-274fbe031c10          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T05:06:41Z | ControllerNodesPostDeployment                 |
| ExtraConfig                                   | 34bde646-81e6-4b16-867f-7b7f7c8a8623          | OS::TripleO::NodeExtraConfigPost                  | UPDATE_COMPLETE | 2016-02-25T05:08:24Z | CephStorageNodesPostDeployment                |

-bash-4.2$ heat stack-list --show-nested | grep 26b623c1-776d-4d2d-87e0-9a52e00eb74
| 26b623c1-776d-4d2d-87e0-9a52e00eb74b | overcloud-ControllerNodesPostDeployment-cwywbgdhhqfm-ExtraConfig-hn32x2vn4u4t-ExtraDeployments-6y4s2baxbeqt | UPDATE_COMPLETE | 2016-02-25T02:12:14Z | 01d1cadf-680e-4e05-92ee-274fbe031c10 |

-bash-4.2$ heat resource-list -n5 26b623c1-776d-4d2d-87e0-9a52e00eb74b
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+------------------+
| resource_name | physical_resource_id                 | resource_type                  | resource_status | updated_time         | parent_resource  |
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+------------------+
| 0             | b25f8aac-eead-443b-9039-63b5db91ca4c | OS::Heat::StructuredDeployment | CREATE_COMPLETE | 2016-02-25T02:12:14Z | ExtraDeployments |
| 1             | f056b88a-424f-4fc0-8317-56cefe64291e | OS::Heat::StructuredDeployment | CREATE_COMPLETE | 2016-02-25T02:12:14Z | ExtraDeployments |
| 2             | 0629f217-d140-4b96-94bd-39f289dd751a | OS::Heat::StructuredDeployment | CREATE_COMPLETE | 2016-02-25T02:12:14Z | ExtraDeployments |
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+------------------+


-bash-4.2$ heat deployment-show b25f8aac-eead-443b-9039-63b5db91ca4c
{
  "status": "COMPLETE",
  "server_id": "3fd5e2a7-8dee-4d08-b543-6f891d5da90b",
  "config_id": "7463822f-761d-4167-8b85-20694713e476",
  "output_values": {
    "deploy_stdout": "",
    "deploy_stderr": "",
    "deploy_status_code": 0
  },
  "creation_time": "2016-02-25T02:12:17Z",
  "updated_time": "2016-02-25T02:13:11Z",
  "input_values": {},
  "action": "CREATE",
  "status_reason": "Outputs received",
  "id": "b25f8aac-eead-443b-9039-63b5db91ca4c"

I see what's going on here. This is my understanding of your issue:

1. The regular TripleO config process makes change X
2. Your postconfig changes X to X-prime
3. On a stack update (incl. scale-up) regular config runs and reverts state to X
4. Your postconfig only runs on the new node, not on existing ones

The reason for this is because the configuration SWDeployment resource takes a parameter, DeployIdentifier, that changes on every stack action to force the update. 

According to Heat, there's no relation between the standard configuration and your postconfig, because the dependency is somewhere in the pile of bash that, to heat, is an unparseable blob. Thus, when a new instance is added Heat doesn't know to re-run the configuration on existing nodes because the script in the postconfig hasn't actually changed. 

As you noticed, if you change the script it is re-run, because Heat can see that the inputs have changed for all servers, triggering the update. 

If you want your postconfig to run every time, you can do that by depending on the same deployID that the standard config does, defined at[1] and depended upon[2]. This will result in your config being overridden briefly, then fixed by postconfig so be aware of that. 



[1]: https://github.com/david-hill/rhosp7/blob/master/my-overcloud/overcloud-without-mergepy.yaml#L684-L689
[2]: https://github.com/david-hill/rhosp7/blob/master/my-overcloud/overcloud-without-mergepy.yaml#L1349-L1357

The ExtraConfig resource[1] has no parameter available to pass in the NodeConfigIdentifers, and we can't add one without breaking ExtraConfig templates in already in the field.

However the ControllerPostPuppet *does*[2], so I would suggest switching to that so the config can be applied every time.

Unfortunately for HA pacemaker managed controllers ControllerPostPuppet is mapped to post_puppet_pacemaker.yaml[3] so you'll need to make sure it is still invoked by defining another type in your environment which maps to [3] and defining a resource of that type in your postdeployment.yaml.

For RHOSP 8 I think we should deprecate OS::TripleO::NodeExtraConfigPost and add a new type which also allows passing in NodeConfigIdentifers.

[1] https://github.com/david-hill/rhosp7/blob/master/my-overcloud/puppet/controller-post-puppet.yaml#L122-L128
[2] https://github.com/david-hill/rhosp7/blob/master/my-overcloud/puppet/controller-post-puppet.yaml#L114-L120
[3] https://github.com/david-hill/rhosp7/blob/master/my-overcloud/extraconfig/tasks/post_puppet_pacemaker.yaml

I felt it necessary to add the use case I'm building and why this matters. I'm currently running my service endpoints on non-standard ports, because of this I have to edit haproxy.cfg as part of my deployment. From deployment to deployment this config does not change but I'm forced to edit the template each time I want to add additional hardware because it wipes out the changes. This is a must have.

@Nathan Morell: At this time, what I could suggest you is to do something like this:

- Deploy the overcloud/update/do whatever you want to do
- Create a shell script that applies those changes
- Run that script after each openstack commands (from the undercloud)

I agree with Ryan that using DeployIdentifier should resolve this problem.

I posted an example illustrating how this works:

https://review.openstack.org/296488

I'll also follow up with a docs patch explaining how this works, but basically tripleoclient passes a unique identifier (timestamp) every update via the DeployIdentifer parameter.  Since we use parameter_defaults to pass it, any template can make use of it.

Steven, is it only necessary to have DeployIdentifier as a parameter to this template, or does it need to be passed into the input_values or SoftwareDeployments and from there possibly even into the SoftwareConfig?

Ah, I've been working on the assumption that DeployIdentifier was set in parameters, but it is set in parameter_defaults. This makes my comment #7 irrelevant, the solution is already available in shardy's comment #18.

Andreas(In reply to Andreas Karis from comment #20)
> Steven, is it only necessary to have DeployIdentifier as a parameter to this
> template, or does it need to be passed into the input_values or
> SoftwareDeployments and from there possibly even into the SoftwareConfig?

DeployIdentifier changes every time a stack update occurs, and having any input_values change will trigger the deployment again, so in shardy's example the only lines really necessary are 15,16,38,39

This needs to become a documentation bug to include DeployIdentifier in the example in section 8.3

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/sect-Configuring_after_Overcloud_Creation.html

(In reply to Andreas Karis from comment #20)
> Steven, is it only necessary to have DeployIdentifier as a parameter to this
> template, or does it need to be passed into the input_values or
> SoftwareDeployments and from there possibly even into the SoftwareConfig?

It must be passed into the SoftwareDeployment input_values, but it doesn't necessarily have to be actually used by the SoftwareConfig.

Just adding the DeployIdentifier parameter isn't enough, because the SoftwareDeployment will still see no change.

Moved to Omri as he is responsible for updates.

(In reply to Steve Baker from comment #23)
> This needs to become a documentation bug to include DeployIdentifier in the
> example in section 8.3
> 
> https://access.redhat.com/documentation/en-US/
> Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/
> Director_Installation_and_Usage/sect-Configuring_after_Overcloud_Creation.
> html

Moving this to the documentation component.

I was able to verify this by using the example in comment #18. After 2 successive overcloud deploy runs I got the following extra_update file:

[root@overcloud-controller-0 ~]# cat /root/extra_update 
extra_update 1472132055
extra_update 1472133706

Clearing out my backlog and I saw this BZ. This looks like it's implemented but might need one or two people to check it.

Here are the OSP11 versions:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/advanced_overcloud_customization/chap-configuration_hooks#sect-Customizing_Overcloud_PreConfiguration

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/advanced_overcloud_customization/chap-configuration_hooks#sect-Customizing_Overcloud_PreConfiguration_All

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/advanced_overcloud_customization/chap-configuration_hooks#sect-Customizing_Overcloud_PostConfiguration_All

Marius and Steve, any chance you guys could have a quick look and make sure everything looks okay?

LGTM, clearing needinfo state on this old thing.

Thanks, Amit!