Bug 1820268

Description of problem:

Using OSP16, and when trying to rotate the fernet token keys in the controller nodes it fails as described in the documentation.


Version-Release number of selected component (if applicable):

(undercloud) [stack@undercloud ~]$ rpm -qa | grep tripleo
python3-tripleo-common-11.3.3-0.20200206225551.18d4fbc.el8ost.noarch
openstack-tripleo-common-11.3.3-0.20200206225551.18d4fbc.el8ost.noarch
puppet-tripleo-11.4.1-0.20200205150840.71ff36d.el8ost.noarch
openstack-tripleo-image-elements-10.6.1-0.20191022065313.7338463.el8ost.noarch
openstack-tripleo-validations-11.3.2-0.20200206065551.1a9b92a.el8ost.noarch
tripleo-ansible-0.4.2-0.20200207140443.b750574.el8ost.noarch
openstack-tripleo-common-containers-11.3.3-0.20200206225551.18d4fbc.el8ost.noarch
openstack-tripleo-puppet-elements-11.2.2-0.20200128210949.d668f88.el8ost.noarch
ansible-role-tripleo-modify-image-1.1.1-0.20200122200932.58d7a5b.el8ost.noarch
python3-tripleoclient-heat-installer-12.3.2-0.20200130192329.78ac810.el8ost.noarch
python3-tripleoclient-12.3.2-0.20200130192329.78ac810.el8ost.noarch
ansible-tripleo-ipsec-9.2.0-0.20191022054642.ffe104c.el8ost.noarch
openstack-tripleo-heat-templates-11.3.2-0.20200211065546.d3d6dc3.el8ost.noarch

(undercloud) [stack@undercloud ~]$ rpm -qa | grep mistral
python3-mistralclient-3.10.0-0.20190920090831.dc246bf.el8ost.noarch
python3-mistral-lib-1.2.1-0.20191118120254.4bac2b2.el8ost.noarch
puppet-mistral-15.4.1-0.20191014143431.c733b8a.el8ost.noarch

How reproducible:

Follow instructions per the official doc: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html-single/deploy_fernet_on_the_overcloud/index#rotate_the_fernet_keys_using_mistral

Steps to Reproduce:
1. source stackrc
2. openstack workflow execution create tripleo.fernet_keys.v1.rotate_fernet_keys '{"container": "overcloud"}'
3. openstack workflow execution show ID_of_Execution_Workflow_Above

Actual results:
The workflow execution show State ERROR and output is the following:

|                    | Stderr: "ERROR! A playbook must be a list of plays, got a <class 'ansible.parsing.yaml.objects.AnsibleUnicode'> instead\n\nThe error appears to be in '/tmp/ansible-mis
tral-actionlsuvogvk/playbook.yaml': line 1, column 1, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n/usr/share/tripleo-an
sible/playbooks/rotate-keys.yaml\n^ here\n"', action_cls='<class 'mistral.actions.action_factory.AnsiblePlaybookAction'>', attributes='{}', params='{'hosts': 'keystone', 'inventory': '/var/l
ib/mistral/overcloud/tripleo-ansible-inventory.yaml', 'ssh_private_key': 'PRIVATE_KEY', 'extra_env_variables': {'ANSIBLE_HOST_KEY_CHECKING': 'False', 'TRIPLEO_PLAN_NAME': 'overcloud'}, 'verbosity': 0, 'remote_
user': 'tripleo-admin', 'become': True, 'extra_vars': {'fernet_keys': {'/etc/keystone/fernet-keys/0': {'content': '8Q6xpk6IaG5djjUHei0EAUgCpeZQW1v0casrRPhXgcc='}, '/etc/keystone/fernet-keys/
1': {'content': 'G4hDLNQB2RFJoXx6c_Y26MGbbonM3CKkvZdbR5vbp2g='}, '/etc/keystone/fernet-keys/2': {'content': 'NVcarcz3uboaobSY6xTMRzcMdCgxt08Xg7JPmuyfnt8='}, '/etc/keystone/fernet-keys/3': {'
content': 'xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM='}}}, 'use_openstack_credentials': True, 'playbook': '/usr/share/tripleo-ansible/playbooks/rotate-keys.yaml', 'execution_id': '14f058af
-c348-483c-b348-4c5829864f76'}'] |
|                    |     [wf_ex_id=66c1898e-6f9d-4561-8040-e0a3a0ab8876, idx=0]: Workflow failed due to message status. Status:FAILED Message:The action raised an exception [action_ex_id=d
722b9ae-f355-42e5-9c8f-d41921b5e23f, msg='Unexpected error while running command. 

Expected results:

Workflow should be executed successfully, and output should look like this:

+-------------------+-------------------------------------------+
| Field             | Value                                     |
+-------------------+-------------------------------------------+
| ID                | 58c9c664-b966-4f82-b368-af5ed8de5b47      |
| Workflow ID       | 78f0990a-3d34-4bf2-a127-10c149bb275c      |
| Workflow name     | tripleo.fernet_keys.v1.rotate_fernet_keys |
| Description       |                                           |
| Task Execution ID | <none>                                    |
| State             | SUCCESS                                   |
| State info        | None                                      |
| Created at        | 2020-04-02 11:13:50                       |
| Updated at        | 2020-04-02 11:15:00                       |
+-------------------+-------------------------------------------+


Additional info:

If I get the rotate-keys.yaml ansible playbook (which I couldn't find in the Train release upstream, only in Stein) and run the same command from director (besides I get the ssh key and store it in a file), the commands works and the fernet tokens are rotated as expected:

1. Get the rotate-keys.yaml
(undercloud) [stack@undercloud ~]$ curl -s https://raw.githubusercontent.com/openstack/tripleo-common/stable/stein/playbooks/rotate-keys.yaml -o rotate-keys.yaml

2. Get the parameters of the failed workflow, specially the ssh private key and store it with permissions 600
undercloud) [stack@undercloud ~]$ ansible-playbook-3 rotate-keys.yaml --become --extra-vars '{"fernet_keys": {"/etc/keystone/fernet-keys/0": {"content": "xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM="}, "/etc/keystone/fernet-keys/1": {"content": "G4hDLNQB2RFJoXx6c_Y26MGbbonM3CKkvZdbR5vbp2g="}, "/etc/keystone/fernet-keys/2": {"content": "NVcarcz3uboaobSY6xTMRzcMdCgxt08Xg7JPmuyfn
t8="}}}' --inventory-file /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml --private-key key-from-mistral-wf