Bug 2097350

I might be wrong here, but I think we are missing the point. The problem I see is that Swift is unable to detect that it's handling an invalid token, and keeps using it until the expiration time, then it uses a new one and the problem "self heals". 

If Swift is able to resolve the situation in which a token is expired, shouldn't also handle more appropriately an invalid token, for example by requesting a new one upon the first 401 it receives and not after retrying for ~40 minutes?

This is the ansible task that is causing the token revocation:

~~~
2022-10-20 14:35:16,822 p=280722 u=mistral n=ansible | TASK [tripleo-keystone-resources : Async creation of Keystone user] ************
2022-10-20 14:35:16,822 p=280722 u=mistral n=ansible | Thursday 20 October 2022  14:35:16 +0200 (0:00:02.879)       0:18:20.701 ****** 
2022-10-20 14:35:17,387 p=280722 u=mistral n=ansible | changed: [undercloud] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
2022-10-20 14:35:17,389 p=280722 u=mistral n=ansible | changed: [undercloud] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}
2022-10-20 14:35:17,395 p=280722 u=mistral n=ansible | TASK [tripleo-keystone-resources : Check Keystone user status] *****************
2022-10-20 14:35:17,396 p=280722 u=mistral n=ansible | Thursday 20 October 2022  14:35:17 +0200 (0:00:00.573)       0:18:21.275 ****** 
2022-10-20 14:35:23,057 p=280722 u=mistral n=ansible | changed: [undercloud] => (item=swift) => {"ansible_job_id": "789898991559.931199", "ansible_loop_var": "tripleo_keystone_resources_user_async_result_item", "attempts": 2, "changed": true, "finished": 1, "tripleo_keystone_resources_user_async_result_item": {"ansible_job_id": "789898991559.931199", "ansible_loop_var": "tripleo_keystone_resources_data", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/789898991559.931199", "started": 1, "tripleo_keystone_resources_data": {"swift": {"password": "XEVRrkX7XnhA9NiYeUtVVKGq4"}}}, "user": {"default_project_id": null, "description": null, "domain_id": "default", "email": "swift@localhost", "enabled": true, "id": "536d28a39ccc43b486764b6761acec39", "name": "swift", "username": null}}
~~~

This task comes from /var/lib/mistral/overcloud/external_deploy_steps_tasks_step4.yaml, which imports the role "tripleo-keystone-resources". Specifically, the task is users.yaml, and I suppose the reason for the token revocation is the "update_password: always", here [0].


[0] https://github.com/openstack/tripleo-ansible/blob/stable/train/tripleo_ansible/roles/tripleo-keystone-resources/tasks/users.yml#L24

Oliver, Emilien,

I'm adding you because you might be knowledgeable about the background on this commit [0]. It seems that this password update on the swift user (even when the password doesn't change) causes a token revocation that in turns makes the external RGW fail until a new token is expired (possibly up to 1 hour). Is there a workaround we can provide to skip this task while we analyze and propose a fix to this issue?


[0] https://github.com/openstack/tripleo-ansible/commit/b8ff5517a1726a89a6adcae51a1c999fe2c88442

(In reply to Eric Nothen from comment #50)

> makes the external RGW fail until a new token is expired (possibly up to 1 hour). 

s/expired/requested/

(In reply to Eric Nothen from comment #49)
> This is the ansible task that is causing the token revocation:
> 
> (...)
> 
> This task comes from
> /var/lib/mistral/overcloud/external_deploy_steps_tasks_step4.yaml, which
> imports the role "tripleo-keystone-resources". Specifically, the task is
> users.yaml, and I suppose the reason for the token revocation is the
> "update_password: always", here [0].
> 

I can confirm that updating the password revokes the tokens, there was a similar issue, but in keystone-manager bootstrap in the past [1].

I have set up an environment to run just keystone-manager bootstrap, but didn't encounter the token revocation issue (the previous logs suggested that it might be an issue in the keystone-manager bootstrap), so it looks like you've found the root cause.

> 
> [0]
> https://github.com/openstack/tripleo-ansible/blob/stable/train/
> tripleo_ansible/roles/tripleo-keystone-resources/tasks/users.yml#L24

[1] https://bugs.launchpad.net/keystone/+bug/1647800

(In reply to Eric Nothen from comment #50)
> Oliver, Emilien,
> 
> I'm adding you because you might be knowledgeable about the background on
> this commit [0]. It seems that this password update on the swift user (even
> when the password doesn't change) causes a token revocation that in turns
> makes the external RGW fail until a new token is expired (possibly up to 1
> hour). Is there a workaround we can provide to skip this task while we
> analyze and propose a fix to this issue?
> 

I started drafting a patch [1], which tries to login with the new password and changes the update_password argument accordingly.
I'll probably have something more complete on Monday.

> 
> [0]
> https://github.com/openstack/tripleo-ansible/commit/
> b8ff5517a1726a89a6adcae51a1c999fe2c88442

[1] https://review.opendev.org/c/openstack/tripleo-ansible/+/862372

(In reply to Eric Nothen from comment #50)
> Oliver, Emilien,
> 
> I'm adding you because you might be knowledgeable about the background on
> this commit [0].
>  It seems that this password update on the swift user (even
> when the password doesn't change) causes a token revocation that in turns
> makes the external RGW fail until a new token is expired (possibly up to 1
> hour). Is there a workaround we can provide to skip this task while we
> analyze and propose a fix to this issue?
> 
> 
> [0]
> https://github.com/openstack/tripleo-ansible/commit/
> b8ff5517a1726a89a6adcae51a1c999fe2c88442

That was to preserve the existing behaviour when upgrading Ansible from 2.7 to 2.8.

Changing the component here to reflect that the change being delivered here is in tripleo-ansible, rather than ceph.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8794