Description of problem:

In RHOSP13, the OS::Triple::NodeUserDataor resource[1] (and its role specific version, OS::TripleO::{{role.name}}NodeUserData) is executed after baremetal instances are created and before any other configuration steps are executed.
 [1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/advanced_overcloud_customization/index#sect-Customizing_Configuration_on_First_Boot

This is because we use os-collect-config to apply any configurations defined in heat resources, and os-collect-config daemon is started after cloud-init completes all configurations because of the constraint defined in systemd service file.

~~~
$ sudo cat /usr/lib/systemd/system/os-collect-config.service
[Unit]
Description=Collect metadata and run hook commands.
After=cloud-final.service
Before=crond.service

...
~~~

However this behavior has been changed since we replaced os-collect-config by config-download in RHOSP14.
Currently there are no dependency set between cloud-init invocation and actual deployment tasks, thus any deployment tasks can be stated BEFORE cloud-init completes all defined tasks.

This change makes it very difficult to use NodeUserData resource because of unexpected ordering/collisions between configurations. For example if any of the subsequent configurations (like ExtraConfogPre) depend on something implemented in NodeUserData, the configuration doesn't work as expected.

IMO we should implement an ordering enforcement to revive the previous behavior (or enforce users to use ExtraConfigPre instead)


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Deploy overcloud with OS::Tripleo::NodeUserData resource
~~~
heat_template_version: 2014-10-16

description: >
  Extra hostname configuration

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: sleep_config}

  sleep_config:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        sleep 300

outputs:
  OS::stack_id:
    value: {get_resource: userdata}
~~~

Actual results:
Deployment steps start before cloud-init completes applying all settings

Expected results:
Deployment steps start after cloud-init completes applying all settings


Additional info: