Bug 1444979 - Running keystone in a separate node makes ceilometer-upgrade fail
Summary: Running keystone in a separate node makes ceilometer-upgrade fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 11.0 (Ocata)
Assignee: Pradeep Kilambi
QA Contact: Arik Chernetsky
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-24 16:24 UTC by Rodrigo Duarte
Modified: 2017-05-17 20:23 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-6.0.0-9.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 20:23:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1685828 0 None None None 2017-04-24 16:24:10 UTC
OpenStack gerrit 459756 0 None None None 2017-04-25 15:23:50 UTC
Red Hat Product Errata RHEA-2017:1245 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 Bug Fix and Enhancement Advisory 2017-05-17 23:01:50 UTC

Description Rodrigo Duarte 2017-04-24 16:24:10 UTC
The overcloud deploy will fail with the following error:

    Error: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]
    Error: /Stage[main]/Tripleo::Profile::Base::Ceilometer::Collector/Exec[ceilometer-db-upgrade]/returns: change from notrun to 0 failed: ceilometer-upgrade --skip-metering-database returned 1 instead of one of [0]

And the logs will have the following:

2017-04-24 14:23:16.274 41400 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-04-24 14:23:16.275 41400 DEBUG ceilometer.cmd.storage [-] Upgrading metering database upgrade /usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py:45
2017-04-24 14:23:16.275 41400 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:87
2017-04-24 14:23:16.304 41400 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('172.17.1.17', 27017), ('172.17.1.25', 27017), ('172.17.1.12', 27017)]
2017-04-24 14:23:16.338 41400 INFO ceilometer.cmd.storage [-] Skipping Gnocchi resource types upgrade
2017-04-24 14:26:54.495 57724 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-04-24 14:26:54.495 57724 DEBUG ceilometer.cmd.storage [-] Upgrading metering database upgrade /usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py:45
2017-04-24 14:26:54.496 57724 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/site-packages/ceilometer/storage/__init__.py:87
2017-04-24 14:26:54.513 57724 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('172.17.1.17', 27017), ('172.17.1.25', 27017), ('172.17.1.12', 27017)]
2017-04-24 14:26:54.526 57724 INFO ceilometer.cmd.storage [-] Skipping Gnocchi resource types upgrade
2017-04-24 14:31:33.629 78572 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-04-24 14:31:33.629 78572 INFO ceilometer.cmd.storage [-] Skipping metering database upgrade
2017-04-24 14:31:33.629 78572 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py:51
2017-04-24 14:31:33.676 78572 CRITICAL ceilometer [-] Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-cdc11fdf-6733-4f92-a74c-543ca0cce414)
2017-04-24 14:31:33.676 78572 ERROR ceilometer Traceback (most recent call last):
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2017-04-24 14:31:33.676 78572 ERROR ceilometer sys.exit(upgrade())
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 53, in upgrade
2017-04-24 14:31:33.676 78572 ERROR ceilometer gnocchi_client.upgrade_resource_types(conf)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/ceilometer/gnocchi_client.py", line 113, in upgrade_resource_types
2017-04-24 14:31:33.676 78572 ERROR ceilometer gnocchi.resource_type.get(name=name)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2017-04-24 14:31:33.676 78572 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line 37, in _get
2017-04-24 14:31:33.676 78572 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get
2017-04-24 14:31:33.676 78572 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line 35, in request
2017-04-24 14:31:33.676 78572 ERROR ceilometer **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
2017-04-24 14:31:33.676 78572 ERROR ceilometer return self.session.request(url, method, **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-04-24 14:31:33.676 78572 ERROR ceilometer return wrapped(*args, **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 491, in request
2017-04-24 14:31:33.676 78572 ERROR ceilometer auth_headers = self.get_auth_headers(auth)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 818, in get_auth_headers
2017-04-24 14:31:33.676 78572 ERROR ceilometer return auth.get_headers(self, **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers
2017-04-24 14:31:33.676 78572 ERROR ceilometer token = self.get_token(session)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token
2017-04-24 14:31:33.676 78572 ERROR ceilometer return self.get_access(session).auth_token
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access
2017-04-24 14:31:33.676 78572 ERROR ceilometer self.auth_ref = self.get_auth_ref(session)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 198, in get_auth_ref
2017-04-24 14:31:33.676 78572 ERROR ceilometer return self._plugin.get_auth_ref(session, **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 167, in get_auth_ref
2017-04-24 14:31:33.676 78572 ERROR ceilometer authenticated=False, log=False, **rkwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 766, in post
2017-04-24 14:31:33.676 78572 ERROR ceilometer return self.request(url, 'POST', **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-04-24 14:31:33.676 78572 ERROR ceilometer return wrapped(*args, **kwargs)
2017-04-24 14:31:33.676 78572 ERROR ceilometer File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 655, in request
2017-04-24 14:31:33.676 78572 ERROR ceilometer raise exceptions.from_response(resp, method, url)
2017-04-24 14:31:33.676 78572 ERROR ceilometer Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-cdc11fdf-6733-4f92-a74c-543ca0cce414)
2017-04-24 14:31:33.676 78572 ERROR ceilometer

Digging further it seems that the ceilometer user is not being created.

Fore more details, see [1] - all deployment jobs with keystone in a separate node are currently failing.

[1] https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS11/job/qe-DFG-security-poc-deployment-11-rhel-7.3-virt-3cont_3keystone_1comp-ipv4-vxlan-ussl-yes-ossl-yes/16/consoleFull

Comment 8 errata-xmlrpc 2017-05-17 20:23:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1245


Note You need to log in before you can comment on or make changes to this bug.