Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1404749

Summary: Update credentials for aodh service after 'controller and block storage' upgrade step
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: documentationAssignee: Dan Macpherson <dmacpher>
Status: CLOSED CURRENTRELEASE QA Contact: RHOS Documentation Team <rhos-docs>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: dmacpher, fbaudin, jcoufal, mandreou, pkilambi, sathlang, srevivo
Target Milestone: gaKeywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
During an upgrade from Red Hat OpenStack Platform (RHOSP) version 9 to version 10, credentials from RHOSP 9 are carried over until convergence, when the full upgrade is completed. This causes alarm evaluation to fail. Manually update options in '[service_credentials]' section: 1. Set auth_type to password: auth_type=password 2. os_* options are no longer valid. Remove os_* prefix from the following options: os_username - replace with username os_tenant_name - replace with project_name os_password - replace with password os_auth_url - replace with auth_url os_region_name - replace with region_name 3. Remove 'v2.0' version from auth_url auth_url=http://[fd00:fd00:fd00:2000::10]:5000/ 4. Restart the service: systemctl restart openstack-aodh-evaluator.service Aodh alarms will now be evaluated correctly.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-07 04:30:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yurii Prokulevych 2016-12-14 14:40:27 UTC 
Description of problem:
-----------------------

Credentials from osp9 are carried over until converge, causing alarm evaluation to fail with next traceback in aodh/evaluator.log:

2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold [-] alarm stats retrieval failed
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold Traceback (most recent call last):
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/aodh/evaluator/threshold.py", line 127, in _statistics
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     return self.cm_client.statistics.list(
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/aodh/evaluator/threshold.py", line 72, in cm_client
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     interface=auth_config.interface,
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/ceilometerclient/client.py", line 366, in get_client
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     return Client(version, endpoint, **kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/ceilometerclient/client.py", line 313, in Client
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     return client_class(*args, **client_kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/client.py", line 65, in __init__
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     self.alarm_client = self._get_alarm_client(**kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/ceilometerclient/v2/client.py", line 120, in _get_alarm_client
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     c.get("/")
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 187, in get
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     return self.request(url, 'GET', **kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/ceilometerclient/client.py", line 470, in request
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     **kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 344, in request
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 112, in request
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     return self.session.request(url, method, **kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     return wrapped(*args, **kwargs)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 484, in request
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     **endpoint_filter)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 770, in get_endpoint
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     auth = self._auth_required(auth, 'determine endpoint URL')
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold   File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 713, in _auth_required
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold     raise exceptions.MissingAuthPlugin(msg_fmt % msg)
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold MissingAuthPlugin: An auth plugin is required to determine endpoint URL
2016-12-14 13:56:51.990 6375 ERROR aodh.evaluator.threshold 


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
puppet-aodh-9.4.1-1.el7ost.noarch
openstack-aodh-evaluator-3.0.1-4.el7ost.noarch
python-aodh-3.0.1-4.el7ost.noarch
openstack-aodh-common-3.0.1-4.el7ost.noarch
openstack-aodh-api-3.0.1-4.el7ost.noarch
openstack-aodh-listener-3.0.1-4.el7ost.noarch
python-aodhclient-0.7.0-1.el7ost.noarch
openstack-aodh-notifier-3.0.1-4.el7ost.noarch


Steps to Reproduce:
-------------------
1. Upgrade from RHOS-9 to RHOS-10
2. After 'Controller And Block Storage Upgrade' step check aodh/evaluator.log


Actual results:
---------------
Traceback in log. Alarms ain't evaluated.


Expected results:
-----------------
Aodh is working through the upgrade

Additional info:
----------------
Workaround is to manually update options in '[service_credentials]' section:
1. Set auth_type to password:
   auth_type=password

2. os_* options ain't valid anymore, remove os_* prefix

os_username    -> username
os_tenant_name -> project_name
os_password    -> password
os_auth_url    -> auth_url
os_region_name -> region_name
 
3. Remove 'v2.0' version from auth_url
   auth_url=http://[fd00:fd00:fd00:2000::10]:5000/
Comment 1 Yurii Prokulevych 2016-12-14 14:40:57 UTC 
And after aforementioned changes restart openstack-aodh-evaluator.service
Comment 2 Sofer Athlan-Guyot 2016-12-14 15:06:15 UTC 
Oki, so this is in-transition error.

In the end, after the aodh migration we have this in the configuration:

    auth_type = password

    auth_url=http://192.0.2.6:35357

    [service_credentials]
    region_name = regionOne
    auth_type = password
    auth_url = http://172.16.2.4:5000
    project_name = service
    project_domain_id = default
    username = aodh
    user_domain_id = default
    password = zr8MzHpjWZH9A6sQhVUdtkZPC

(note that the os_* entries are still there, but it doesn't affect
operation)

So this is as expected and the aodh service is working.

The issue here is that after controller upgrade and after convergence
or aodh migration the service will be non-working.

Maybe could be solved by a doc warning ?
Comment 3 Yurii Prokulevych 2016-12-14 15:12:33 UTC 
Hi Sofer,

Totally agree this is transitory issue and it's 'fixed' after the whole upgrade process is finished.
But since it might take quite a long we need to document this limitation and a workaround to bypass it.

---
Yurii
Comment 5 Dan Macpherson 2018-08-07 04:30:21 UTC 
Closing this as a release note has been issued for this item.