Description of problem: Using OSP16, and when trying to rotate the fernet token keys in the controller nodes it fails as described in the documentation. Version-Release number of selected component (if applicable): (undercloud) [stack@undercloud ~]$ rpm -qa | grep tripleo python3-tripleo-common-11.3.3-0.20200206225551.18d4fbc.el8ost.noarch openstack-tripleo-common-11.3.3-0.20200206225551.18d4fbc.el8ost.noarch puppet-tripleo-11.4.1-0.20200205150840.71ff36d.el8ost.noarch openstack-tripleo-image-elements-10.6.1-0.20191022065313.7338463.el8ost.noarch openstack-tripleo-validations-11.3.2-0.20200206065551.1a9b92a.el8ost.noarch tripleo-ansible-0.4.2-0.20200207140443.b750574.el8ost.noarch openstack-tripleo-common-containers-11.3.3-0.20200206225551.18d4fbc.el8ost.noarch openstack-tripleo-puppet-elements-11.2.2-0.20200128210949.d668f88.el8ost.noarch ansible-role-tripleo-modify-image-1.1.1-0.20200122200932.58d7a5b.el8ost.noarch python3-tripleoclient-heat-installer-12.3.2-0.20200130192329.78ac810.el8ost.noarch python3-tripleoclient-12.3.2-0.20200130192329.78ac810.el8ost.noarch ansible-tripleo-ipsec-9.2.0-0.20191022054642.ffe104c.el8ost.noarch openstack-tripleo-heat-templates-11.3.2-0.20200211065546.d3d6dc3.el8ost.noarch (undercloud) [stack@undercloud ~]$ rpm -qa | grep mistral python3-mistralclient-3.10.0-0.20190920090831.dc246bf.el8ost.noarch python3-mistral-lib-1.2.1-0.20191118120254.4bac2b2.el8ost.noarch puppet-mistral-15.4.1-0.20191014143431.c733b8a.el8ost.noarch How reproducible: Follow instructions per the official doc: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html-single/deploy_fernet_on_the_overcloud/index#rotate_the_fernet_keys_using_mistral Steps to Reproduce: 1. source stackrc 2. openstack workflow execution create tripleo.fernet_keys.v1.rotate_fernet_keys '{"container": "overcloud"}' 3. openstack workflow execution show ID_of_Execution_Workflow_Above Actual results: The workflow execution show State ERROR and output is the following: | | Stderr: "ERROR! A playbook must be a list of plays, got a <class 'ansible.parsing.yaml.objects.AnsibleUnicode'> instead\n\nThe error appears to be in '/tmp/ansible-mis tral-actionlsuvogvk/playbook.yaml': line 1, column 1, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n/usr/share/tripleo-an sible/playbooks/rotate-keys.yaml\n^ here\n"', action_cls='<class 'mistral.actions.action_factory.AnsiblePlaybookAction'>', attributes='{}', params='{'hosts': 'keystone', 'inventory': '/var/l ib/mistral/overcloud/tripleo-ansible-inventory.yaml', 'ssh_private_key': 'PRIVATE_KEY', 'extra_env_variables': {'ANSIBLE_HOST_KEY_CHECKING': 'False', 'TRIPLEO_PLAN_NAME': 'overcloud'}, 'verbosity': 0, 'remote_ user': 'tripleo-admin', 'become': True, 'extra_vars': {'fernet_keys': {'/etc/keystone/fernet-keys/0': {'content': '8Q6xpk6IaG5djjUHei0EAUgCpeZQW1v0casrRPhXgcc='}, '/etc/keystone/fernet-keys/ 1': {'content': 'G4hDLNQB2RFJoXx6c_Y26MGbbonM3CKkvZdbR5vbp2g='}, '/etc/keystone/fernet-keys/2': {'content': 'NVcarcz3uboaobSY6xTMRzcMdCgxt08Xg7JPmuyfnt8='}, '/etc/keystone/fernet-keys/3': {' content': 'xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM='}}}, 'use_openstack_credentials': True, 'playbook': '/usr/share/tripleo-ansible/playbooks/rotate-keys.yaml', 'execution_id': '14f058af -c348-483c-b348-4c5829864f76'}'] | | | [wf_ex_id=66c1898e-6f9d-4561-8040-e0a3a0ab8876, idx=0]: Workflow failed due to message status. Status:FAILED Message:The action raised an exception [action_ex_id=d 722b9ae-f355-42e5-9c8f-d41921b5e23f, msg='Unexpected error while running command. Expected results: Workflow should be executed successfully, and output should look like this: +-------------------+-------------------------------------------+ | Field | Value | +-------------------+-------------------------------------------+ | ID | 58c9c664-b966-4f82-b368-af5ed8de5b47 | | Workflow ID | 78f0990a-3d34-4bf2-a127-10c149bb275c | | Workflow name | tripleo.fernet_keys.v1.rotate_fernet_keys | | Description | | | Task Execution ID | <none> | | State | SUCCESS | | State info | None | | Created at | 2020-04-02 11:13:50 | | Updated at | 2020-04-02 11:15:00 | +-------------------+-------------------------------------------+ Additional info: If I get the rotate-keys.yaml ansible playbook (which I couldn't find in the Train release upstream, only in Stein) and run the same command from director (besides I get the ssh key and store it in a file), the commands works and the fernet tokens are rotated as expected: 1. Get the rotate-keys.yaml (undercloud) [stack@undercloud ~]$ curl -s https://raw.githubusercontent.com/openstack/tripleo-common/stable/stein/playbooks/rotate-keys.yaml -o rotate-keys.yaml 2. Get the parameters of the failed workflow, specially the ssh private key and store it with permissions 600 undercloud) [stack@undercloud ~]$ ansible-playbook-3 rotate-keys.yaml --become --extra-vars '{"fernet_keys": {"/etc/keystone/fernet-keys/0": {"content": "xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM="}, "/etc/keystone/fernet-keys/1": {"content": "G4hDLNQB2RFJoXx6c_Y26MGbbonM3CKkvZdbR5vbp2g="}, "/etc/keystone/fernet-keys/2": {"content": "NVcarcz3uboaobSY6xTMRzcMdCgxt08Xg7JPmuyfn t8="}}}' --inventory-file /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml --private-key key-from-mistral-wf
I have a patch up that attempts to update the location of the playbook in the mistral workflow. https://review.opendev.org/#/c/717291/
I confirmed the patch makes the workflow point to the right path this time, however now it fails because the playbook is not performing correctly the validation if keystone is running on a container: PLAY [keystone] ****************************************************************************************************************************************************************************** TASK [Gathering Facts] *********************************************************************************************************************************************************************** ok: [overcloud2-ctrl01] TASK [Check for containerized keystone fernet repository] ************************************************************************************************************************************ ok: [overcloud2-ctrl01] TASK [populate service facts] **************************************************************************************************************************************************************** ok: [overcloud2-ctrl01] TASK [Set container facts] ******************************************************************************************************************************************************************* ok: [overcloud2-ctrl01] TASK [Set keystone facts] ******************************************************************************************************************************************************************** skipping: [overcloud2-ctrl01] TASK [Remove previous fernet keys] *********************************************************************************************************************************************************** skipping: [overcloud2-ctrl01] TASK [Persist fernet keys to repository] ***************************************************************************************************************************************************** skipping: [overcloud2-ctrl01] => (item={'key': '/etc/keystone/fernet-keys/0', 'value': {'content': 'xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM='}}) skipping: [overcloud2-ctrl01] => (item={'key': '/etc/keystone/fernet-keys/1', 'value': {'content': 'G4hDLNQB2RFJoXx6c_Y26MGbbonM3CKkvZdbR5vbp2g='}}) skipping: [overcloud2-ctrl01] => (item={'key': '/etc/keystone/fernet-keys/2', 'value': {'content': 'NVcarcz3uboaobSY6xTMRzcMdCgxt08Xg7JPmuyfnt8='}}) TASK [Set permissions to match container's user] ********************************************************************************************************************************************* skipping: [overcloud2-ctrl01] => (item={'key': '/etc/keystone/fernet-keys/0', 'value': {'content': 'xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM='}}) skipping: [overcloud2-ctrl01] => (item={'key': '/etc/keystone/fernet-keys/1', 'value': {'content': 'G4hDLNQB2RFJoXx6c_Y26MGbbonM3CKkvZdbR5vbp2g='}}) skipping: [overcloud2-ctrl01] => (item={'key': '/etc/keystone/fernet-keys/2', 'value': {'content': 'NVcarcz3uboaobSY6xTMRzcMdCgxt08Xg7JPmuyfnt8='}}) TASK [Restart keystone container with docker] ************************************************************************************************************************************************ skipping: [overcloud2-ctrl01] TASK [Restart keystone container] ************************************************************************************************************************************************************ skipping: [overcloud2-ctrl01] TASK [Remove previous fernet keys] *********************************************************************************************************************************************************** changed: [overcloud2-ctrl01] TASK [Persist fernet keys to repository] ***************************************************************************************************************************************************** failed: [overcloud2-ctrl01] (item={'key': '/etc/keystone/fernet-keys/0', 'value': {'content': 'xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM='}}) => {"ansible_loop_var": "item", "changed": fal se, "checksum": "3b412c11e3ccbaaf2236041a5cdf08f1325605f1", "item": {"key": "/etc/keystone/fernet-keys/0", "value": {"content": "xWTb5JYlZah2XPYY8HiewABw6kExXrWF5IRVJ_wv4LM="}}, "msg": "Dest ination directory /etc/keystone/fernet-keys does not exist"} Fortunately this has been patched upstream, in the master branch: https://review.opendev.org/#/c/711872/1 So I just cherry picked to Train branch: https://review.opendev.org/#/c/717495/ let's see how it goes. Thanks,
openstack workflow execution create tripleo.fernet_keys.v1.rotate_fernet_keys '{"container": "overcloud"}' openstack workflow execution show f87819fb-7a32-4fd9-95b4-bde14875e02a
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148