1610210 – [UPGRADES][14] CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator

Bug 1610210 - [UPGRADES][14] CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator

Summary: [UPGRADES][14] CredentialEncryptionError: Credential could not be decrypted. ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	python-tripleoclient
Sub Component:
Version:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	beta
Target Release:	14.0 (Rocky)
Assignee:	Rabi Mishra
QA Contact:	Pavan
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-07-31 08:34 UTC by Yurii Prokulevych
Modified:	2019-01-11 11:51 UTC (History)
CC List:	14 users (show)
Fixed In Version:	python-tripleoclient-10.5.1-0.20180906012842.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-01-11 11:51:11 UTC
Target Upstream Version:
Embargoed:
Flags:	rmascena: needinfo-

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	591980	0	None	MERGED	Re-use old fernet and credential keys in containerized undercloud	2020-10-25 15:25:24 UTC
Red Hat Product Errata	RHEA-2019:0045	0	None	None	None	2019-01-11 11:51:19 UTC

Description Yurii Prokulevych 2018-07-31 08:34:35 UTC

Description of problem:
-----------------------
Attempt to prepare playbooks for upgrade failed:
openstack overcloud upgrade prepare --stack overcloud \
    --templates /usr/share/openstack-tripleo-heat-templates \
    -e /home/stack/virt/internal.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /home/stack/virt/network/network-environment.yaml \
    -e /home/stack/virt/enable-tls.yaml \
    -e /home/stack/virt/inject-trust-anchor.yaml \
    -e /home/stack/virt/public_vip.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
    -e /home/stack/virt/hostnames.yml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
    -e /home/stack/virt/debug.yaml \
    -e /home/stack/virt/nodes_data.yaml \
    -e /home/stack/virt/docker-images.yaml \
    -e /home/stack/cli_opts_params.yaml \
    -e /home/stack/virt/docker-images.yaml \
    --roles-file /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
...
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: CREATE_COMPLETE  state changed
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.NodeTLSCAData]: UPDATE_FAILED  resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected 
error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd.NodeTLSCAData]: UPDATE_FAILED  resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected $
rror prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: DELETE_IN_PROGRESS  state changed
2018-07-31 08:02:11Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf]: UPDATE_FAILED  Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An $
nexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd]: UPDATE_FAILED  Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An u$
expected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj-2-kn4lz2kbqg65.ControllerDeployment]: CREATE_COMPLETE  state changed
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj.0]: UPDATE_FAILED  resources.NodeTLSCAData: resources[0].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevente$
 the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f6a94725c925)
2018-07-31 08:02:11Z [overcloud-Controller-6luk43cdl2dj]: UPDATE_FAILED  Resource UPDATE failed: resources.NodeTLSCAData: resources[0].Resource DELETE failed: InternalServerError: resources.CADeployment: An une$
pected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-4f26-a022-f
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.CephStorageExtraConfigPre]: UPDATE_COMPLETE  state changed
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73.2]: UPDATE_FAILED  resources.NodeTLSCAData: resources[2].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevent$
d the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c4b3827b3027)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn.NodeTLSCAData]: UPDATE_FAILED  resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected $
rror prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Compute-pjxd24dovyp2-1-ampsmvt53hps.NovaComputeDeployment]: CREATE_COMPLETE  state changed
2018-07-31 08:02:12Z [overcloud-CephStorage-6saozupeon73]: UPDATE_FAILED  Resource UPDATE failed: resources.NodeTLSCAData: resources[2].Resource DELETE failed: InternalServerError: resources.CADeployment: An un$
xpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-4b06-b29a-c
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn]: UPDATE_FAILED  Resource UPDATE failed: resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An u$
expected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj.1]: UPDATE_FAILED  resources.NodeTLSCAData: resources[1].Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevente$
 the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4a85c34234fd)
2018-07-31 08:02:12Z [overcloud-Controller-6luk43cdl2dj]: UPDATE_FAILED  Resource UPDATE failed: resources.NodeTLSCAData: resources[1].Resource DELETE failed: InternalServerError: resources.CADeployment: An une$
pected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-4d6e-a288-4
2018-07-31 08:02:13Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn.SshHostPubKey]: DELETE_FAILED  InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected error prevente$
 the server from fulfilling your request. (HTTP 500) (Request-ID: req-ca402686-86b3-4c62-8b8f-e687dad29437)
2018-07-31 08:02:13Z [overcloud-Controller-6luk43cdl2dj-1-ir32unlimjmn]: UPDATE_FAILED  Resource DELETE failed: InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected erro$
 prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-ca402686-86b3-4c62-8b8f-e687dad29437)
2018-07-31 08:02:14Z [overcloud-Controller-6luk43cdl2dj-0-kkagf5ylzxpd.ControllerDeployment]: CREATE_COMPLETE  state changed
2018-07-31 08:02:14Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf.SshHostPubKey]: DELETE_FAILED  InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected error prevent$
d the server from fulfilling your request. (HTTP 500) (Request-ID: req-6d536dc8-180a-401a-a827-c4826e7a7085)
2018-07-31 08:02:14Z [overcloud-CephStorage-6saozupeon73-2-5qosfl6z2thf]: UPDATE_FAILED  Resource DELETE failed: InternalServerError: resources.SshHostPubKey.resources.SshHostPubKeyDeployment: An unexpected err$
r prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-6d536dc8-180a-401a-a827-c4826e7a7085)

 Stack overcloud/3dbb3a91-2bc8-434d-8db5-c73f27753bcb UPDATE_FAILED


openstack stack failures list overcloud
overcloud.Controller.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 2b118f9f-1b93-4ed6-841c-88144edc8dbf
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-10f28b72-9559-
4d6e-a288-4a85c34234fd)
overcloud.Controller.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: f9e905b3-d1fa-4b90-8471-2dcacd147c3a
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a580208a-c3a4-
4f26-a022-f6a94725c925)
overcloud.Controller.2.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: a59f6b62-d65e-4104-b626-96294f3965bc
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-d094ace2-c6c6-
4b8f-ae90-ed465680131f)
overcloud.Compute.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 0af3b7a3-e305-43d3-b950-af2f34d1f56a
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-3e94ec07-2d16-
4779-ac88-932e262c9e27)
overcloud.Compute.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: fda08616-77ba-4abd-80b6-726998db40b2
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-b82cba0a-a879-
47cd-b13d-59d8b84cf9d3)
overcloud.CephStorage.1.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: 6d74f773-2719-4bd8-86a4-bbd41725b9ef
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-db53f67b-16e9-
4327-a156-6c3c4d92a378)
overcloud.CephStorage.0.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: faf82cc2-f53d-452a-b91e-6fdcb3776c2c
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-07d16af3-9b24-
4676-a493-fbc915a1e50a)
overcloud.CephStorage.2.NodeTLSCAData:
  resource_type: OS::TripleO::NodeTLSCAData
  physical_resource_id: d8315964-d419-4659-bacc-7ae260b0eeb5
  status: UPDATE_FAILED
  status_reason: |
    resources.NodeTLSCAData: Resource DELETE failed: InternalServerError: resources.CADeployment: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-11bec0aa-69cf-
4b06-b29a-c4b3827b3027)

And from /var/log/containers/keystone/keystone.log
2018-07-31 04:01:47.036 175 DEBUG keystone.common.fernet_utils [req-a8e67edf-5e70-41e0-a02c-7c3b0129a167 - - - - -] Loaded 2 Fernet keys from /etc/keystone/fernet-keys, but `[fernet_tokens] max_active_keys = 5`;
 perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /usr/lib/python2.7/site-packages/keystone/common/fernet_utils.py:307
2018-07-31 04:01:47.044 172 ERROR keystone.credential.providers.fernet.core [req-b82cba0a-a879-47cd-b13d-59d8b84cf9d3 49498c2e31164edba109caeb5f31965f - d9acfcbcc4ca45ada06bfd03770c63b0 d9acfcbcc4ca45ada06bfd037
70c63b0 -] Credential could not be decrypted. Please contact the administrator: InvalidToken
2018-07-31 04:01:47.045 175 DEBUG keystone.server.flask.application [req-15bc1072-130e-458d-a693-6d1bf8ba76ba - - - - -] Dispatching back to Flask native app. __call__ /usr/lib/python2.7/site-packages/keystone/s
erver/flask/application.py:150
2018-07-31 04:01:47.045 175 DEBUG keystone.server.flask.application [req-15bc1072-130e-458d-a693-6d1bf8ba76ba - - - - -] SCRIPT_NAME: ``, PATH_INFO: `/` __call__ /usr/lib/python2.7/site-packages/keystone/server/
flask/application.py:161
2018-07-31 04:01:47.049 175 DEBUG keystone.server.flask.application [req-2618f8f6-a99f-4161-9b75-601667b49712 - - - - -] Dispatching back to Flask native app. __call__ /usr/lib/python2.7/site-packages/keystone/s
erver/flask/application.py:150
2018-07-31 04:01:47.049 175 DEBUG keystone.server.flask.application [req-2618f8f6-a99f-4161-9b75-601667b49712 - - - - -] SCRIPT_NAME: ``, PATH_INFO: `/` __call__ /usr/lib/python2.7/site-packages/keystone/server/
flask/application.py:161
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi [req-b82cba0a-a879-47cd-b13d-59d8b84cf9d3 49498c2e31164edba109caeb5f31965f - d9acfcbcc4ca45ada06bfd03770c63b0 d9acfcbcc4ca45ada06bfd03770c63b0 -] Credential could not be decrypted. Please contact the administrator: CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi Traceback (most recent call last):
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 148, in __call__
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     result = method(req, **params)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 57, in inner
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     self, f, check_function, request, None, *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 122, in protected_wrapper
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     check_function(self, request, prep_info, *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 131, in check_protection
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     *args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 150, in check_policy
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     _handle_member_from_driver(controller, policy_dict, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/authorization.py", line 76, in _handle_member_from_driver
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     ref = self.get_member_from_driver(kwargs[key])
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 103, in get_credential
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     return self._decrypt_credential(credential)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/credential/core.py", line 49, in _decrypt_credential
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     credential['encrypted_blob'],
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/credential/providers/fernet/core.py", line 112, in decrypt
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi     raise exception.CredentialEncryptionError(msg)
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator
2018-07-31 04:01:47.044 172 ERROR keystone.common.wsgi 



Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-keystone-14.0.0-0.20180717031603.67db074.el7ost.noarch

python-keystone-14.0.0-0.20180717031603.67db074.el7ost.noarch
python2-keystoneauth1-3.9.0-0.20180621113114.ebe781a.el7ost.noarch
python2-keystonemiddleware-5.1.0-0.20180614161553.83d0612.el7ost.noarch
puppet-keystone-13.1.1-0.20180716075228.4afabd9.el7ost.noarch
python2-keystoneclient-3.17.0-0.20180624203936.234ea50.el7ost.noarch

openstack-tripleo-common-containers-9.1.1-0.20180717062358.eee5526.el7ost.noarch
python2-tripleo-common-9.1.1-0.20180717062358.eee5526.el7ost.noarch
openstack-tripleo-common-9.1.1-0.20180717062358.eee5526.el7ost.noarch


How reproducible:
-----------------
100%


Steps to Reproduce:
-------------------
1. Install RHOS-13
2. Install RHOS-14 repos on uc/oc, prepare images for uc/oc upgrade
3. Upgrade uc to 14
4. Copy /usr/lib/heat/undercloud_heat_plugins to heat_engine container and restart it (dedicated bz 1610200 filed)
5. Run upgrade prepare command

Actual results:
---------------
Upgrade prepare fails

Expected results:
-----------------
Upgrade prepare succeeds

Additional info:
----------------
Virtual env: 3controllers + 2computes + 3ceph

Comment 3 Emilien Macchi 2018-08-14 16:35:15 UTC

I'll work on a patch this week.

Comment 4 Harry Rybacki 2018-08-14 19:51:39 UTC

Emilien, I spoke with Ozz and the fix merged but the bug wasn't updated. Moving bug to POST.

Comment 5 Yurii Prokulevych 2018-08-15 07:56:23 UTC

Harry, I tried that patch as fix for rhbz 1610246, which fixes issue when uc is already deployed with SSL.
But it doesn't fix '''CredentialEncryptionError: Credential could not be decrypted. Please contact the administrator''' error.

New sosreport from UC uploaded: 
sosreport-rhbz-1610210-undercloud-0-20180815032730.tar.xz

Comment 6 Harry Rybacki 2018-08-15 14:18:17 UTC

Thanks for the clarification, Yurii.

Comment 12 Harry Rybacki 2018-10-03 18:48:34 UTC

Changes merged in build python-tripleoclient-10.5.1-0.20180906012842.el7ost

Moving bug to MODIFIED.

Comment 17 errata-xmlrpc 2019-01-11 11:51:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045

Note You need to log in before you can comment on or make changes to this bug.