Bug 1639495
| Summary: | fernet token rotation fails to automatically get deployed to the overcloud | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jeremy Agee <jagee> |
| Component: | openstack-tripleo-common | Assignee: | Rabi Mishra <ramishra> |
| Status: | CLOSED ERRATA | QA Contact: | Pavan <pkesavar> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 14.0 (Rocky) | CC: | dmatthew, emacchi, hrybacki, jjoyce, jschluet, kbasil, mariel, mburns, nkinder, pkesavar, ramishra, rheslop, slinaber, tdunnon, tvignaud |
| Target Milestone: | z1 | Keywords: | TestOnly, Triaged, ZStream |
| Target Release: | 14.0 (Rocky) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-common-9.4.1-0.20181012010880.el7ost | Doc Type: | Bug Fix |
| Doc Text: |
Previously, fernet key rotation failed because the workflow task `tripleo.fernet_keys.v1.rotate_fernet_keys` generated keys but did not push them to Keystone nodes on the overcloud. This fix updates `openstack-tripleo-common-9.4.1-0.20181012010880` to push the keys to the Keystone nodes, and fernet key rotation is now successful.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-03-18 13:03:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I believe Fernet bugs are best handled by the Security DFG My initial investigation into this issue, it looks as though the failure is occurring during the call to the deploy_ssh_key task within the tripleo.fernet_keys.v1 workflow. >mistral execution-create tripleo.fernet_keys.v1.rotate_fernet_keys '{"container": "overcloud"}' The above fails with an error of : { "status": "FAILED", "message": "The action raised an exception [action_ex_id=658da071-6304-4940-8d52-83ae2c9d38a4, action_cls='<class 'mistral.actions.action_factory.OrchestrationDeployAction'>', attributes='{}', params='{u'group': u'script', u'server_id': u'402b99b1-4618-4452-9793-c34dfe09ff78', u'config': u'#!/bin/bash\\nif ! grep \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" /home/heat-admin/.ssh/authorized_keys; then\\n echo \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" >> /home/heat-admin/.ssh/authorized_keys\\nfi\\n', u'name': u'copy_ssh_key'}']\n ERROR: Concurrent transaction for deployments of server 402b99b1-4618-4452-9793-c34dfe09ff78", "result": "Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=857d5aa0-4651-4617-9405-65f309d5a222] -> The action raised an exception [action_ex_id=658da071-6304-4940-8d52-83ae2c9d38a4, action_cls='<class 'mistral.actions.action_factory.OrchestrationDeployAction'>', attributes='{}', params='{u'group': u'script', u'server_id': u'402b99b1-4618-4452-9793-c34dfe09ff78', u'config': u'#!/bin/bash\\nif ! grep \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" /home/heat-admin/.ssh/authorized_keys; then\\n echo \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" >> /home/heat-admin/.ssh/authorized_keys\\nfi\\n', u'name': u'copy_ssh_key'}']\n ERROR: Concurrent transaction for deployments of server 402b99b1-4618-4452-9793-c34dfe09ff78\n [action_ex_id=658da071-6304-4940-8d52-83ae2c9d38a4, idx=0]: The action raised an exception [action_ex_id=658da071-6304-4940-8d52-83ae2c9d38a4, action_cls='<class 'mistral.actions.action_factory.OrchestrationDeployAction'>', attributes='{}', params='{u'group': u'script', u'server_id': u'402b99b1-4618-4452-9793-c34dfe09ff78', u'config': u'#!/bin/bash\\nif ! grep \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" /home/heat-admin/.ssh/authorized_keys; then\\n echo \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" >> /home/heat-admin/.ssh/authorized_keys\\nfi\\n', u'name': u'copy_ssh_key'}']\n ERROR: Concurrent transaction for deployments of server 402b99b1-4618-4452-9793-c34dfe09ff78\n\n send_message [task_ex_id=9091d4f3-d922-496d-896b-cf1b4c754319] -> Workflow failed due to message status\n [wf_ex_id=9914dc14-9cdf-4ff5-b4a7-dc1c216f3568, idx=0]: Workflow failed due to message status\n" } It seems to me as if the call to copy the keys are failing. But when I call the task to rotate the keys directly "tripleo.parameters.rotate_fernet_keys" it works as expected. >mistral -v run-action tripleo.parameters.rotate_fernet_keys '{"container": "overcloud"}' {"result": {"/etc/keystone/fernet-keys/0": {"content": "Oz4I-7tj1s9FHiLitfcrPXbK0ymIfAcPDmwev4uNw2U="}, "/etc/keystone/fernet-keys/8": {"content": "hkS4OXuJDFaJxkLzFWODu8m3hvZ0nrY9OORxb_LTZqo="}, "/etc/keystone/fernet-keys/7": {"content": "m97pzb5LqrI9ttUhG2oa2LUiTgCnRbVEYRgm_HpJyIg="}, "/etc/keystone/fernet-keys/6": {"content": "NEORRLljxHfPCySvNyHmCVwgd8kV6PJ1DRjd2tkR0tM="}, "/etc/keystone/fernet-keys/9": {"content": "aCPwAcLrGcS5cLCet1Y4nZ57qjT5kI8wqO46ZYtZuAo="}}} Its looking like the key rotation will be performed with ether method. I tried both methods and you can see each time it did rotate keys. Nether method had the keys make it into the overclouds keystone container on the controller though. tested it on core_puddle_version 2018-10-30.6
(undercloud) [stack@undercloud-0 ~]$ mistral -v run-action tripleo.parameters.rotate_fernet_keys '{"container": "overcloud"}'
{"result": {"/etc/keystone/fernet-keys/0": {"content": "rbznGYJnXPDpciniO3WvZKKDI65G4lMNnnxz_PGyRiQ="}, "/etc/keystone/fernet-keys/1": {"content": "es1sN-JXeuCAuJeRN7kI3oQMyS-MSvHBCLq2TLp_PEo="}, "/etc/keystone/fernet-keys/2": {"content": "_x06A0PHhfJwtjdLaq-SHPxrXZYLytnrEyHBGm2RJOQ="}}}
(undercloud) [stack@undercloud-0 ~]$ openstack workflow execution create tripleo.fernet_keys.v1.rotate_fernet_keys '{"container": "overcloud"}'
+--------------------+-------------------------------------------+
| Field | Value |
+--------------------+-------------------------------------------+
| ID | 8c7f2089-5c6e-47b6-b69f-427895d933f8 |
| Workflow ID | fc1bec73-a07e-4c09-b44a-27cccffb8c32 |
| Workflow name | tripleo.fernet_keys.v1.rotate_fernet_keys |
| Workflow namespace | |
| Description | |
| Task Execution ID | <none> |
| Root Execution ID | <none> |
| State | RUNNING |
| State info | None |
| Created at | 2018-11-01 17:53:39 |
| Updated at | 2018-11-01 17:53:39 |
+--------------------+-------------------------------------------+
(undercloud) [stack@undercloud-0 ~]$ mistral task-list 8c7f2089-5c6e-47b6-b69f-427895d933f8
+--------------------------------------+----------------+-------------------------------------------+--------------------+--------------------------------------+---------+------------+---------------------+---------------------+
| ID | Name | Workflow name | Workflow namespace | Workflow Execution ID | State | State info | Created at | Updated at |
+--------------------------------------+----------------+-------------------------------------------+--------------------+--------------------------------------+---------+------------+---------------------+---------------------+
| 4bae173f-8dac-409d-b9d8-0b1161629786 | deploy_ssh_key | tripleo.fernet_keys.v1.rotate_fernet_keys | | 8c7f2089-5c6e-47b6-b69f-427895d933f8 | RUNNING | None | 2018-11-01 17:53:39 | <none> |
| f5d032cc-4763-4f12-a5b5-232fcd4d7e3f | rotate_keys | tripleo.fernet_keys.v1.rotate_fernet_keys | | 8c7f2089-5c6e-47b6-b69f-427895d933f8 | SUCCESS | None | 2018-11-01 17:53:39 | 2018-11-01 17:53:39 |
+--------------------------------------+----------------+-------------------------------------------+--------------------+--------------------------------------+---------+------------+---------------------+---------------------+
(undercloud) [stack@undercloud-0 ~]$ mistral task-get-result f5d032cc-4763-4f12-a5b5-232fcd4d7e3f
{
"/etc/keystone/fernet-keys/0": {
"content": "DA9ZWSFbvw4QQ6Ih9xkiN1IH7RMOMSRbiI1kX8ZHER4="
},
"/etc/keystone/fernet-keys/1": {
"content": "es1sN-JXeuCAuJeRN7kI3oQMyS-MSvHBCLq2TLp_PEo="
},
"/etc/keystone/fernet-keys/2": {
"content": "_x06A0PHhfJwtjdLaq-SHPxrXZYLytnrEyHBGm2RJOQ="
},
"/etc/keystone/fernet-keys/3": {
"content": "rbznGYJnXPDpciniO3WvZKKDI65G4lMNnnxz_PGyRiQ="
}
}
[root@controller-0 keystone]# ls -la /var/lib/config-data/keystone/etc/keystone/fernet-keys/
total 8
drwx------. 2 42425 42425 24 Nov 1 17:19 .
drwxr-x---. 4 root 42425 201 Nov 1 17:19 ..
-rw-------. 1 42425 42425 44 Nov 1 17:19 0
-rw-------. 1 42425 42425 44 Nov 1 17:19 1
[root@controller-0 keystone]# docker exec -it keystone ls -la /etc/keystone/fernet-keys/
total 8
drwx------. 2 keystone keystone 24 Nov 1 17:29 .
drwxr-x---. 1 root keystone 69 Nov 1 17:29 ..
-rw-------. 1 keystone keystone 44 Nov 1 17:19 0
-rw-------. 1 keystone keystone 44 Nov 1 17:19 1
I had a quick look into this. I don't understand what the Zaqar question is in comment 3. So I ignored that and focused on the information in the description and comment 2. In the original description it seems there is a heat timeout: "Timeout for heat deployment 'copy_ssh_key'" There is also Heat error here that I don't think anyone has looked into. Here is a breakdown of how I understand line in comment 2... [action_ex_id=658da071-6304-4940-8d52-83ae2c9d38a4, action_cls='<class 'mistral.actions.action_factory.OrchestrationDeployAction'>', attributes='{}', params='{u'group': u'script', u'server_id': u'402b99b1-4618-4452-9793-c34dfe09ff78', u'config': u'#!/bin/bash\\nif ! grep \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" /home/heat-admin/.ssh/authorized_keys; then\\n echo \"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4tCdKo9Q5UjF8jX+2yImf8WTBFjfz14OjWNYQW/P58UadjPJHB6rCr2euW/2XF6gPO4HCrdHJeMKRDuSju/NNg0ZSSE0xbK68aPSqWfvkFJvfgrll5uu8hSOmbbClmxOD2K/bHecal4Z7C5paMm8v4k1pS6y9R56++a3kmZ+pbIuepkqFkV8wv7h/E1R+2Mb80JOoXRYrudYVk/L3FTFdh7tcqkhrvsMouB5jSn4VAzDAFIJV6iNeZAUqUu4Uvo1QXXoct2fN/JFW3OMER9Hfwk9nUYuV+UZwv1rOMvpF23oWpfeAfeXDWXZlxWv0y4dSlidY9exU5P4hqQ0zxwXx Generated by TripleO\" >> /home/heat-admin/.ssh/authorized_keys\\nfi\\n', u'name': u'copy_ssh_key'}']\n ERROR: Concurrent transaction for deployments of server 402b99b1-4618-4452-9793-c34dfe09ff78", So we have the action execution ID, and then the action class is OrchestrationDeployAction. That means the error is coming from a Heat deploy. Skipping over the keys and other params and looking at the end we have: ERROR: Concurrent transaction for deployments of server 402b99b1-4618-4452-9793-c34dfe09ff78 I am not familiar with this error, but it seems to be coming from Heat, not sure where but I found this: https://github.com/openstack/heat/blob/07fa426a16c11e1de7a014eb8f9ca016dbe38d8f/heat/common/exception.py#L539-L540 To verify this theory, we should look at the Mistral logs and verify the trace back is coming from heatclient, and then I think we need to investigate the Heat logs. I do not think this is a workflow related issue. I'm not actually sure if toure is seeing the same failure as in the original description. They seem a little different (timeout vs concurrent deploy errors) (In reply to Jeremy Agee from comment #4) > Its looking like the key rotation will be performed with ether method. I > tried both methods and you can see each time it did rotate keys. Nether > method had the keys make it into the overclouds keystone container on the > controller though. tested it on core_puddle_version 2018-10-30.6 This is because the actual key generation/rotation happens as a part of the "rotate_keys" task, which is the first task that is executed. Once the rotation is complete, the "deploy_ssh_key" task is executed to set up the keys so we can get into the keystone container on the overcloud IIUC. The deployment of the SSH keys is what fails with a timeout, so we are never actually progressing through the workflow execution to the point where we put the keys into place. If you look at the output from "mistral task-list" in the initial bug description, the "deploy_ssh_key" task (really the "tripleo.validations.v1.copy_ssh_key" workflow) takes just over 5 minutes before it completes with the timeout error. I think Dougal is onto something with his investigation in https://bugzilla.redhat.com/show_bug.cgi?id=1639495#c5 I have created a new upstream rocky deployment, and this issue can be easily reproduced by executing the tripleo.fernet_keys.v1.rotate_fernet_keys workflow. The execution will be stuck in RUNNING state for just over 5 minutes before it fails with the same error attempting to run the same tripleo.validations.v1.copy_ssh_key task. This task ultimately fails running the tripleo.deployment.v1.deploy_on_server task that is defined here: https://github.com/openstack/tripleo-common/blob/1d17a9992528f3d70a59b10de6a53c2178eea998/workbooks/validations.yaml#L282-L295 I think we need to have someone help debug this from a heat perspective to continue. I did a bit more digging, and the timeout comes from here: https://github.com/openstack/tripleo-common/blob/b8bfff895b422f5c399a21945cd303af156823f5/tripleo_common/actions/deployment.py#L86-L121 I will provide a more detailed attachment showing my troubleshooting steps, but we basically get stuck waiting to find some data in a swift temp URL that never arrives. (line 107 in the code block referenced above) The problem here is definitely that we rely on os-collect-config on the undercloud nodes, as it is responsible for detecting, pulling down, and applying software config changes from heat. As can be seen in these upstream docs, os-collect-config is disabled by default now that config-download is used for most things: https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ansible_config_download_differences.html It appears that the fernet rotation workflow was an oversight when other features moved to using config-download. We either need to enable os-collect-config by default on the overcloud nodes, or we need to switch to using config-download for fernet key rotation. I've submitted patch https://review.openstack.org/#/c/618618/, which should fix this. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0446 |
Description of problem: when the workflow task tripleo.fernet_keys.v1.rotate_fernet_keys is created keys are generated but do not get pushed to the overcloud. Steps to Reproduce: 1. run the key rotation command on the undercloud (undercloud) [stack@undercloud-0 ~]$ openstack workflow execution create tripleo.fernet_keys.v1.rotate_fernet_keys '{"container": "overcloud"}' +--------------------+-------------------------------------------+ | Field | Value | +--------------------+-------------------------------------------+ | ID | b23a988a-6571-4fdb-b776-4897682f0ca0 | | Workflow ID | f7b7db4a-0e85-409a-9d35-7b5f66698d0d | | Workflow name | tripleo.fernet_keys.v1.rotate_fernet_keys | | Workflow namespace | | | Description | | | Task Execution ID | <none> | | Root Execution ID | <none> | | State | RUNNING | | State info | None | | Created at | 2018-10-15 20:24:47 | | Updated at | 2018-10-15 20:24:47 | +--------------------+-------------------------------------------+ 2. Wait for the task failures. (undercloud) [stack@undercloud-0 ~]$ mistral task-list b23a988a-6571-4fdb-b776-4897682f0ca0 +--------------------------------------+----------------+-------------------------------------------+--------------------+--------------------------------------+---------+------------------------------+---------------------+---------------------+ | ID | Name | Workflow name | Workflow namespace | Workflow Execution ID | State | State info | Created at | Updated at | +--------------------------------------+----------------+-------------------------------------------+--------------------+--------------------------------------+---------+------------------------------+---------------------+---------------------+ | 22f97f48-3fc7-425e-8e5e-224958f04e68 | rotate_keys | tripleo.fernet_keys.v1.rotate_fernet_keys | | b23a988a-6571-4fdb-b776-4897682f0ca0 | SUCCESS | None | 2018-10-15 20:24:47 | 2018-10-15 20:24:48 | | 1697f654-17f3-4602-89d9-6c614bd739d9 | deploy_ssh_key | tripleo.fernet_keys.v1.rotate_fernet_keys | | b23a988a-6571-4fdb-b776-4897682f0ca0 | ERROR | Failure caused by error i... | 2018-10-15 20:24:48 | 2018-10-15 20:30:00 | | 9266b23f-8252-44cd-8a96-3dd13c7c6cf8 | send_message | tripleo.fernet_keys.v1.rotate_fernet_keys | | b23a988a-6571-4fdb-b776-4897682f0ca0 | ERROR | Workflow failed due to me... | 2018-10-15 20:30:00 | 2018-10-15 20:30:02 | +--------------------------------------+----------------+-------------------------------------------+--------------------+--------------------------------------+---------+------------------------------+---------------------+---------------------+ (undercloud) [stack@undercloud-0 ~]$ mistral task-get-result 22f97f48-3fc7-425e-8e5e-224958f04e68 { "/etc/keystone/fernet-keys/0": { "content": "ASXqOrTA92CdO5Nbtg_PCqDlqDRmRr3QYwXiLQ5_XyI=" }, "/etc/keystone/fernet-keys/8": { "content": "ILWuQ409Pyx-UJEAh-20FZ20FTvwNt936mSFxXFrAhU=" }, "/etc/keystone/fernet-keys/10": { "content": "t44E6EP9SwkUbvgm79Ztz3BT9PDQhHLfZrjlgGvEUYU=" }, "/etc/keystone/fernet-keys/11": { "content": "E6efW2uo6H72QlNkZTlOZpaTf1Dg2ZHGhj3wMpNnoPw=" }, "/etc/keystone/fernet-keys/9": { "content": "j9CLuS0-qNrcdGbyCkyFnukOnxSqjcEfYHb3btNCH5Q=" } } (undercloud) [stack@undercloud-0 ~]$ mistral task-get-result 1697f654-17f3-4602-89d9-6c614bd739d9 { "pubkey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO35k4Qvrl8jAgT7z4h5YabijLuthm8RRap0kBxSvTEyyslEFpFvS52NfsxFIJLOBAYHO5xprOkQ31zYeTbGQ7ACpASuJ9sgYR1L9Nf1nm4rn5rqDgJc9VQIoYDOwoxlqG5yA1Kr5ygbSbuneVGKb7LqDVZLYD+dDJ59K5vY3py6XjC8toZdss84rsY4nVfbHURkQLoF0zeX5OAyt6Wn7Ud+pZNLzOC4QeCa/KV0DRpi3sTEyLxwQW4JTJCbAdCr7zYjyDWDOk5UoSyr4f+Nz8o/1PyZpYASDwKsN7mWDFRgxxfoXzJ7tnJ96p3V2WAf3b7KdiZQEPSWnAWoy8lXhb Generated by TripleO", "result": "Failure caused by error in tasks: deploy_ssh_key\n\n deploy_ssh_key [task_ex_id=bff41766-9b4d-4a9b-9b50-11c478801275] -> One or more actions had failed.\n [wf_ex_id=53959efe-ad90-474b-8d91-4dab1028e7f2, idx=0]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=b2e59fe8-1b91-4873-9320-a9e5cfb44bac] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=fa10cbff-c45c-4825-b8ce-fcdd1f9f443a, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=a60d20d6-6b77-4a6f-985a-7fd2fc87127b] -> Workflow failed due to message status\n [wf_ex_id=b3dd93ec-ad08-44a1-9e9b-364f4c340f96, idx=0]: Workflow failed due to message status\n\n [wf_ex_id=a384dba3-9db8-48d6-8f5f-90aa7a191032, idx=1]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=8a3c1637-8772-4ae8-ac48-e5a380163193] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=ebb61488-7ae7-4523-9217-1866c0e496f1, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=b630f9aa-c540-46bd-8bc3-0e5e379b3baa] -> Workflow failed due to message status\n [wf_ex_id=639e3c3d-f721-486e-9542-7b3165651651, idx=0]: Workflow failed due to message status\n\n", "servers": [ { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:61:98:29", "version": 4, "addr": "192.168.24.12", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000002", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "cb7772b9-e164-462f-bd04-af5e79915b93", "links": [ { "href": "https://192.168.24.2:13774/flavors/cb7772b9-e164-462f-bd04-af5e79915b93", "rel": "bookmark" } ] }, "id": "8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "cedb88c4-acb5-4b5b-b957-41d60687e123", "name": "compute-0", "created": "2018-10-12T18:54:27Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} }, { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:e2:ba:4a", "version": 4, "addr": "192.168.24.18", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000001", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "743af205-0968-4217-bdf9-cfa5ba44ca2f", "links": [ { "href": "https://192.168.24.2:13774/flavors/743af205-0968-4217-bdf9-cfa5ba44ca2f", "rel": "bookmark" } ] }, "id": "0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "fb0528da-b8de-4cd4-8659-735c44cd7d96", "name": "controller-0", "created": "2018-10-12T18:54:26Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} } ] } (undercloud) [stack@undercloud-0 ~]$ mistral task-get-result 9266b23f-8252-44cd-8a96-3dd13c7c6cf8 { "container": "overcloud-messages", "deployment_status_message": { "workflow_status": { "type": "tripleo.fernet_keys.v1.rotate_fernet_keys", "payload": { "status": "FAILED", "message": { "pubkey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO35k4Qvrl8jAgT7z4h5YabijLuthm8RRap0kBxSvTEyyslEFpFvS52NfsxFIJLOBAYHO5xprOkQ31zYeTbGQ7ACpASuJ9sgYR1L9Nf1nm4rn5rqDgJc9VQIoYDOwoxlqG5yA1Kr5ygbSbuneVGKb7LqDVZLYD+dDJ59K5vY3py6XjC8toZdss84rsY4nVfbHURkQLoF0zeX5OAyt6Wn7Ud+pZNLzOC4QeCa/KV0DRpi3sTEyLxwQW4JTJCbAdCr7zYjyDWDOk5UoSyr4f+Nz8o/1PyZpYASDwKsN7mWDFRgxxfoXzJ7tnJ96p3V2WAf3b7KdiZQEPSWnAWoy8lXhb Generated by TripleO", "result": "Failure caused by error in tasks: deploy_ssh_key\n\n deploy_ssh_key [task_ex_id=bff41766-9b4d-4a9b-9b50-11c478801275] -> One or more actions had failed.\n [wf_ex_id=53959efe-ad90-474b-8d91-4dab1028e7f2, idx=0]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=b2e59fe8-1b91-4873-9320-a9e5cfb44bac] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=fa10cbff-c45c-4825-b8ce-fcdd1f9f443a, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=a60d20d6-6b77-4a6f-985a-7fd2fc87127b] -> Workflow failed due to message status\n [wf_ex_id=b3dd93ec-ad08-44a1-9e9b-364f4c340f96, idx=0]: Workflow failed due to message status\n\n [wf_ex_id=a384dba3-9db8-48d6-8f5f-90aa7a191032, idx=1]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=8a3c1637-8772-4ae8-ac48-e5a380163193] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=ebb61488-7ae7-4523-9217-1866c0e496f1, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=b630f9aa-c540-46bd-8bc3-0e5e379b3baa] -> Workflow failed due to message status\n [wf_ex_id=639e3c3d-f721-486e-9542-7b3165651651, idx=0]: Workflow failed due to message status\n\n", "servers": [ { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:61:98:29", "version": 4, "addr": "192.168.24.12", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000002", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "cb7772b9-e164-462f-bd04-af5e79915b93", "links": [ { "href": "https://192.168.24.2:13774/flavors/cb7772b9-e164-462f-bd04-af5e79915b93", "rel": "bookmark" } ] }, "id": "8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "cedb88c4-acb5-4b5b-b957-41d60687e123", "name": "compute-0", "created": "2018-10-12T18:54:27Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} }, { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:e2:ba:4a", "version": 4, "addr": "192.168.24.18", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000001", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "743af205-0968-4217-bdf9-cfa5ba44ca2f", "links": [ { "href": "https://192.168.24.2:13774/flavors/743af205-0968-4217-bdf9-cfa5ba44ca2f", "rel": "bookmark" } ] }, "id": "0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "fb0528da-b8de-4cd4-8659-735c44cd7d96", "name": "controller-0", "created": "2018-10-12T18:54:26Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} } ] }, "execution": { "name": "tripleo.fernet_keys.v1.rotate_fernet_keys", "created_at": "2018-10-15 20:24:47", "updated_at": "2018-10-15 20:24:47", "id": "b23a988a-6571-4fdb-b776-4897682f0ca0", "params": { "namespace": "", "env": {} }, "input": { "queue_name": "tripleo", "container": "overcloud", "ansible_extra_env_variables": { "ANSIBLE_HOST_KEY_CHECKING": "False" } }, "spec": { "input": [ "container", { "queue_name": "tripleo" }, { "ansible_extra_env_variables": { "ANSIBLE_HOST_KEY_CHECKING": "False" } } ], "tasks": { "deploy_ssh_key": { "name": "deploy_ssh_key", "on-error": "send_message", "workflow": "tripleo.validations.v1.copy_ssh_key", "on-success": "get_privkey", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "type": "direct" }, "send_message": { "input": { "status": "<% $.status %>", "plan_name": "<% $.container %>", "queue_name": "<% $.queue_name %>", "message": "<% $.get('message', '') %>", "execution": "<% execution() %>", "type": "<% execution().name %>" }, "version": "2.0", "type": "direct", "name": "send_message", "workflow": "tripleo.messaging.v1.send" }, "deploy_keys": { "name": "deploy_keys", "on-error": "send_message", "on-success": "send_message", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "publish": { "status": "SUCCESS", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.ansible-playbook", "input": { "remote_user": "heat-admin", "use_openstack_credentials": true, "inventory": "/usr/bin/tripleo-ansible-inventory", "extra_vars": { "fernet_keys": "<% task(rotate_keys).result %>" }, "verbosity": 0, "extra_env_variables": "<% $.ansible_extra_env_variables + dict(TRIPLEO_PLAN_NAME=>$.container) %>", "hosts": "keystone", "ssh_private_key": "<% task(get_privkey).result %>", "become": true, "playbook": "/usr/share/tripleo-common/playbooks/rotate-keys.yaml" }, "type": "direct" }, "get_privkey": { "name": "get_privkey", "on-error": "send_message", "on-success": "deploy_keys", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.validations.get_privkey", "type": "direct" }, "rotate_keys": { "name": "rotate_keys", "on-error": "send_message", "on-success": "deploy_ssh_key", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.parameters.rotate_fernet_keys container=<% $.container %>", "type": "direct" } }, "name": "rotate_fernet_keys", "version": "2.0", "tags": [ "tripleo-common-managed" ] } }, "plan_name": "overcloud", "execution_id": "b23a988a-6571-4fdb-b776-4897682f0ca0", "deployment_status": null } }, "deployment_status": null }, "swift_message": { "type": "tripleo.fernet_keys.v1.rotate_fernet_keys", "payload": { "status": "FAILED", "message": { "pubkey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO35k4Qvrl8jAgT7z4h5YabijLuthm8RRap0kBxSvTEyyslEFpFvS52NfsxFIJLOBAYHO5xprOkQ31zYeTbGQ7ACpASuJ9sgYR1L9Nf1nm4rn5rqDgJc9VQIoYDOwoxlqG5yA1Kr5ygbSbuneVGKb7LqDVZLYD+dDJ59K5vY3py6XjC8toZdss84rsY4nVfbHURkQLoF0zeX5OAyt6Wn7Ud+pZNLzOC4QeCa/KV0DRpi3sTEyLxwQW4JTJCbAdCr7zYjyDWDOk5UoSyr4f+Nz8o/1PyZpYASDwKsN7mWDFRgxxfoXzJ7tnJ96p3V2WAf3b7KdiZQEPSWnAWoy8lXhb Generated by TripleO", "result": "Failure caused by error in tasks: deploy_ssh_key\n\n deploy_ssh_key [task_ex_id=bff41766-9b4d-4a9b-9b50-11c478801275] -> One or more actions had failed.\n [wf_ex_id=53959efe-ad90-474b-8d91-4dab1028e7f2, idx=0]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=b2e59fe8-1b91-4873-9320-a9e5cfb44bac] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=fa10cbff-c45c-4825-b8ce-fcdd1f9f443a, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=a60d20d6-6b77-4a6f-985a-7fd2fc87127b] -> Workflow failed due to message status\n [wf_ex_id=b3dd93ec-ad08-44a1-9e9b-364f4c340f96, idx=0]: Workflow failed due to message status\n\n [wf_ex_id=a384dba3-9db8-48d6-8f5f-90aa7a191032, idx=1]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=8a3c1637-8772-4ae8-ac48-e5a380163193] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=ebb61488-7ae7-4523-9217-1866c0e496f1, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=b630f9aa-c540-46bd-8bc3-0e5e379b3baa] -> Workflow failed due to message status\n [wf_ex_id=639e3c3d-f721-486e-9542-7b3165651651, idx=0]: Workflow failed due to message status\n\n", "servers": [ { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:61:98:29", "version": 4, "addr": "192.168.24.12", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000002", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "cb7772b9-e164-462f-bd04-af5e79915b93", "links": [ { "href": "https://192.168.24.2:13774/flavors/cb7772b9-e164-462f-bd04-af5e79915b93", "rel": "bookmark" } ] }, "id": "8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "cedb88c4-acb5-4b5b-b957-41d60687e123", "name": "compute-0", "created": "2018-10-12T18:54:27Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} }, { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:e2:ba:4a", "version": 4, "addr": "192.168.24.18", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000001", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "743af205-0968-4217-bdf9-cfa5ba44ca2f", "links": [ { "href": "https://192.168.24.2:13774/flavors/743af205-0968-4217-bdf9-cfa5ba44ca2f", "rel": "bookmark" } ] }, "id": "0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "fb0528da-b8de-4cd4-8659-735c44cd7d96", "name": "controller-0", "created": "2018-10-12T18:54:26Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} } ] }, "execution": { "name": "tripleo.fernet_keys.v1.rotate_fernet_keys", "created_at": "2018-10-15 20:24:47", "updated_at": "2018-10-15 20:24:47", "id": "b23a988a-6571-4fdb-b776-4897682f0ca0", "params": { "namespace": "", "env": {} }, "input": { "queue_name": "tripleo", "container": "overcloud", "ansible_extra_env_variables": { "ANSIBLE_HOST_KEY_CHECKING": "False" } }, "spec": { "input": [ "container", { "queue_name": "tripleo" }, { "ansible_extra_env_variables": { "ANSIBLE_HOST_KEY_CHECKING": "False" } } ], "tasks": { "deploy_ssh_key": { "name": "deploy_ssh_key", "on-error": "send_message", "workflow": "tripleo.validations.v1.copy_ssh_key", "on-success": "get_privkey", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "type": "direct" }, "send_message": { "input": { "status": "<% $.status %>", "plan_name": "<% $.container %>", "queue_name": "<% $.queue_name %>", "message": "<% $.get('message', '') %>", "execution": "<% execution() %>", "type": "<% execution().name %>" }, "version": "2.0", "type": "direct", "name": "send_message", "workflow": "tripleo.messaging.v1.send" }, "deploy_keys": { "name": "deploy_keys", "on-error": "send_message", "on-success": "send_message", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "publish": { "status": "SUCCESS", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.ansible-playbook", "input": { "remote_user": "heat-admin", "use_openstack_credentials": true, "inventory": "/usr/bin/tripleo-ansible-inventory", "extra_vars": { "fernet_keys": "<% task(rotate_keys).result %>" }, "verbosity": 0, "extra_env_variables": "<% $.ansible_extra_env_variables + dict(TRIPLEO_PLAN_NAME=>$.container) %>", "hosts": "keystone", "ssh_private_key": "<% task(get_privkey).result %>", "become": true, "playbook": "/usr/share/tripleo-common/playbooks/rotate-keys.yaml" }, "type": "direct" }, "get_privkey": { "name": "get_privkey", "on-error": "send_message", "on-success": "deploy_keys", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.validations.get_privkey", "type": "direct" }, "rotate_keys": { "name": "rotate_keys", "on-error": "send_message", "on-success": "deploy_ssh_key", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.parameters.rotate_fernet_keys container=<% $.container %>", "type": "direct" } }, "name": "rotate_fernet_keys", "version": "2.0", "tags": [ "tripleo-common-managed" ] } }, "plan_name": "overcloud", "execution_id": "b23a988a-6571-4fdb-b776-4897682f0ca0", "deployment_status": null } }, "result": "Workflow failed due to message status", "payload": { "status": "FAILED", "message": { "pubkey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO35k4Qvrl8jAgT7z4h5YabijLuthm8RRap0kBxSvTEyyslEFpFvS52NfsxFIJLOBAYHO5xprOkQ31zYeTbGQ7ACpASuJ9sgYR1L9Nf1nm4rn5rqDgJc9VQIoYDOwoxlqG5yA1Kr5ygbSbuneVGKb7LqDVZLYD+dDJ59K5vY3py6XjC8toZdss84rsY4nVfbHURkQLoF0zeX5OAyt6Wn7Ud+pZNLzOC4QeCa/KV0DRpi3sTEyLxwQW4JTJCbAdCr7zYjyDWDOk5UoSyr4f+Nz8o/1PyZpYASDwKsN7mWDFRgxxfoXzJ7tnJ96p3V2WAf3b7KdiZQEPSWnAWoy8lXhb Generated by TripleO", "result": "Failure caused by error in tasks: deploy_ssh_key\n\n deploy_ssh_key [task_ex_id=bff41766-9b4d-4a9b-9b50-11c478801275] -> One or more actions had failed.\n [wf_ex_id=53959efe-ad90-474b-8d91-4dab1028e7f2, idx=0]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=b2e59fe8-1b91-4873-9320-a9e5cfb44bac] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=fa10cbff-c45c-4825-b8ce-fcdd1f9f443a, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=a60d20d6-6b77-4a6f-985a-7fd2fc87127b] -> Workflow failed due to message status\n [wf_ex_id=b3dd93ec-ad08-44a1-9e9b-364f4c340f96, idx=0]: Workflow failed due to message status\n\n [wf_ex_id=a384dba3-9db8-48d6-8f5f-90aa7a191032, idx=1]: Failure caused by error in tasks: deploy_config, send_message\n\n deploy_config [task_ex_id=8a3c1637-8772-4ae8-ac48-e5a380163193] -> Timeout for heat deployment 'copy_ssh_key'\n [action_ex_id=ebb61488-7ae7-4523-9217-1866c0e496f1, idx=0]: Timeout for heat deployment 'copy_ssh_key'\n\n send_message [task_ex_id=b630f9aa-c540-46bd-8bc3-0e5e379b3baa] -> Workflow failed due to message status\n [wf_ex_id=639e3c3d-f721-486e-9542-7b3165651651, idx=0]: Workflow failed due to message status\n\n", "servers": [ { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:61:98:29", "version": 4, "addr": "192.168.24.12", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000002", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "cb7772b9-e164-462f-bd04-af5e79915b93", "links": [ { "href": "https://192.168.24.2:13774/flavors/cb7772b9-e164-462f-bd04-af5e79915b93", "rel": "bookmark" } ] }, "id": "8a93815f-4301-4a0a-ab7b-cf92eec4ff83", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "cedb88c4-acb5-4b5b-b957-41d60687e123", "name": "compute-0", "created": "2018-10-12T18:54:27Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} }, { "OS-EXT-STS:task_state": null, "addresses": { "ctlplane": [ { "OS-EXT-IPS-MAC:mac_addr": "52:54:00:e2:ba:4a", "version": 4, "addr": "192.168.24.18", "OS-EXT-IPS:type": "fixed" } ] }, "links": [ { "href": "https://192.168.24.2:13774/v2.1/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "self" }, { "href": "https://192.168.24.2:13774/servers/0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "rel": "bookmark" } ], "image": { "id": "c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "links": [ { "href": "https://192.168.24.2:13774/images/c47350ea-0d54-4d1c-a4a2-ddb4d8f0c46e", "rel": "bookmark" } ] }, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000001", "OS-SRV-USG:launched_at": "2018-10-12T18:57:26.000000", "flavor": { "id": "743af205-0968-4217-bdf9-cfa5ba44ca2f", "links": [ { "href": "https://192.168.24.2:13774/flavors/743af205-0968-4217-bdf9-cfa5ba44ca2f", "rel": "bookmark" } ] }, "id": "0713e7a5-55d5-4828-8d20-b1cfa7fbcde1", "security_groups": [ { "name": "default" } ], "user_id": "7c02abdef4264c77bdfd9128fe336b01", "OS-DCF:diskConfig": "MANUAL", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "True", "status": "ACTIVE", "updated": "2018-10-12T18:57:26Z", "hostId": "6556cddf99dcc6535a26b578b0841e92ab8cbb1bac7a439b2b313829", "OS-EXT-SRV-ATTR:host": "undercloud-0.localdomain", "OS-SRV-USG:terminated_at": null, "key_name": "default", "OS-EXT-SRV-ATTR:hypervisor_hostname": "fb0528da-b8de-4cd4-8659-735c44cd7d96", "name": "controller-0", "created": "2018-10-12T18:54:26Z", "tenant_id": "2af0d33e7418463489a3dd6b761c0459", "os-extended-volumes:volumes_attached": [], "metadata": {} } ] }, "execution": { "name": "tripleo.fernet_keys.v1.rotate_fernet_keys", "created_at": "2018-10-15 20:24:47", "updated_at": "2018-10-15 20:24:47", "id": "b23a988a-6571-4fdb-b776-4897682f0ca0", "params": { "namespace": "", "env": {} }, "input": { "queue_name": "tripleo", "container": "overcloud", "ansible_extra_env_variables": { "ANSIBLE_HOST_KEY_CHECKING": "False" } }, "spec": { "input": [ "container", { "queue_name": "tripleo" }, { "ansible_extra_env_variables": { "ANSIBLE_HOST_KEY_CHECKING": "False" } } ], "tasks": { "deploy_ssh_key": { "name": "deploy_ssh_key", "on-error": "send_message", "workflow": "tripleo.validations.v1.copy_ssh_key", "on-success": "get_privkey", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "type": "direct" }, "send_message": { "input": { "status": "<% $.status %>", "plan_name": "<% $.container %>", "queue_name": "<% $.queue_name %>", "message": "<% $.get('message', '') %>", "execution": "<% execution() %>", "type": "<% execution().name %>" }, "version": "2.0", "type": "direct", "name": "send_message", "workflow": "tripleo.messaging.v1.send" }, "deploy_keys": { "name": "deploy_keys", "on-error": "send_message", "on-success": "send_message", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "publish": { "status": "SUCCESS", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.ansible-playbook", "input": { "remote_user": "heat-admin", "use_openstack_credentials": true, "ssh_private_key": "<% task(get_privkey).result %>", "extra_vars": { "fernet_keys": "<% task(rotate_keys).result %>" }, "verbosity": 0, "extra_env_variables": "<% $.ansible_extra_env_variables + dict(TRIPLEO_PLAN_NAME=>$.container) %>", "hosts": "keystone", "inventory": "/usr/bin/tripleo-ansible-inventory", "become": true, "playbook": "/usr/share/tripleo-common/playbooks/rotate-keys.yaml" }, "type": "direct" }, "get_privkey": { "name": "get_privkey", "on-error": "send_message", "on-success": "deploy_keys", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.validations.get_privkey", "type": "direct" }, "rotate_keys": { "name": "rotate_keys", "on-error": "send_message", "on-success": "deploy_ssh_key", "publish-on-error": { "status": "FAILED", "message": "<% task().result %>" }, "version": "2.0", "action": "tripleo.parameters.rotate_fernet_keys container=<% $.container %>", "type": "direct" } }, "name": "rotate_fernet_keys", "version": "2.0", "tags": [ "tripleo-common-managed" ] } }, "plan_name": "overcloud", "execution_id": "b23a988a-6571-4fdb-b776-4897682f0ca0", "deployment_status": null } } 3. confirm the overcloud controller does not have the newly generated keys from the rotate_keys task. [root@controller-0 keystone]# ls -la /var/lib/config-data/puppet-generated/keystone/etc/keystone/fernet-keys/ total 28 drwx------. 2 42425 42425 70 Oct 15 19:34 . drwxr-x---. 4 root 42425 69 Oct 15 19:34 .. -rw-------. 1 42425 42425 44 Oct 15 19:34 0 -rw-------. 1 42425 42425 44 Oct 15 19:34 10 -rw-------. 1 42425 42425 44 Oct 15 19:34 7 -rw-------. 1 42425 42425 44 Oct 15 19:34 8 -rw-------. 1 42425 42425 44 Oct 15 19:34 9 The new token is not present. "/etc/keystone/fernet-keys/11": { "content": "E6efW2uo6H72QlNkZTlOZpaTf1Dg2ZHGhj3wMpNnoPw=" Expected results: The overcloud controller should have the new fernet tokens managed by mistral from the undercloud. Additional information: Current work around is to run a new overcloud deploy on the existing overcloud.