Bug 1342571 - Failed to run gnocchi-upgrade with ceph backend
Summary: Failed to run gnocchi-upgrade with ceph backend
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-puppet-modules
Version: 9.0 (Mitaka)
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ga
: 9.0 (Mitaka)
Assignee: Pradeep Kilambi
QA Contact: Yurii Prokulevych
URL:
Whiteboard:
Depends On: 1343144 1348147 1348454
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-03 14:39 UTC by Yurii Prokulevych
Modified: 2016-08-11 12:23 UTC (History)
12 users (show)

Fixed In Version: openstack-puppet-modules-8.1.1-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-11 12:23:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 325482 0 None MERGED ceph: allow to deploy python-cradox 2020-06-14 03:33:29 UTC
Red Hat Product Errata RHEA-2016:1597 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 9 Release Candidate Advisory 2016-08-11 16:06:52 UTC

Description Yurii Prokulevych 2016-06-03 14:39:25 UTC
Description of problem:
Attempt to create 'legacy' resources for ceilometer failed:
gnocchi-upgrade --debug --create-legacy-resource-types -v
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Exception in thread heartbeat:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 77, in _heartbeat
    while not self._stop_heartbeat.is_set():
AttributeError: 'CephStorage' object has no attribute '_stop_heartbeat'

CRITI [gnocchi] ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
Traceback (most recent call last):
  File "/bin/gnocchi-upgrade", line 10, in <module>
    sys.exit(upgrade())
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 58, in upgrade
    s = storage.get_driver(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 155, in get_driver
    return get_driver_class(conf)(conf.storage)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 73, in __init__
    raise ImportError("Your rados python module does not support "
ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 


Version-Release number of selected component (if applicable):
python-gnocchiclient-2.2.0-1.el7ost.noarch
openstack-gnocchi-common-2.1.1-1.el7ost.noarch
openstack-gnocchi-statsd-2.1.1-1.el7ost.noarch
python-gnocchi-2.1.1-1.el7ost.noarch
openstack-gnocchi-metricd-2.1.1-1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.1-1.el7ost.noarch
openstack-gnocchi-api-2.1.1-1.el7ost.noarch
openstack-gnocchi-carbonara-2.1.1-1.el7ost.noarch

python-rados-0.94.5-13.el7cp.x86_64
librados2-0.94.5-13.el7cp.x86_64


Steps to Reproduce:
1. Install overcloud.
2. Configure gnocchi to use ceph backend
3. Run gnocchi-upgrade --debug --create-legacy-resource-types

Comment 3 Mike Burns 2016-06-03 19:21:10 UTC
there are a couple potential issues that need to be solved

1.  the python-cradox library requires librados2 which exists only in Ceph repositories.  This means that people *must* have ceph repositories enabled on their nodes to get the package.  This means we need a doc bug.

2.  How is the package getting installed?  Are we adding an explicit dependency to gnocchi?  are we adding it to puppet to smartly detect/pull it in if we're using a ceph backend?  

3.  We need to ensure that images will get this package automatically as well.

Beyond those 2 questions, this bug should be split into 2 bugs.  1 to resolve #2 and 1 for the new component.

Comment 5 Jason Guiditta 2016-06-08 12:17:00 UTC
Awaiting merge upstream.

Comment 6 Jason Guiditta 2016-06-09 13:44:48 UTC
This is ready to go downstream today, can we get remaining acks on this?

Comment 15 Julien Danjou 2016-06-21 09:38:43 UTC
This is still failing as the cradox Python module cannot be opened by httpd.

It seems that the access is denied by SELinux because the file is wrongly labeled:

type=AVC msg=audit(1466501601.240:93660): avc:  denied  { read } for  pid=31052 comm="httpd" name="cradox.so" dev="sda2" ino=71512 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file


root@overcloud-controller-0 site-packages ]# pwd
/usr/lib64/python2.7/site-packages
[root@overcloud-controller-0 site-packages ]# ls -lZ *.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   acutilmodule.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _audit.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   auparse.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _cffi_backend.so
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 cradox.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _dbus_bindings.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _dbus_glib_bindings.so
[...]

Comment 20 Yurii Prokulevych 2016-07-20 12:23:51 UTC
Excerpt from gnocchi.conf:
--------------------------
[storage]
coordination_url = redis://:ACM2JaMWAYHKj3Jg6tTHzD9fw.2.23:6379/
driver = ceph
ceph_pool = metrics
ceph_username = openstack
ceph_keyring = /etc/ceph/ceph.client.openstack.keyring
ceph_conffile = /etc/ceph/ceph.conf

Supported resource types before upgrade:
----------------------------------------
gnocchi resource list-types
+---------------+--------------------------------------------+
| resource_type | resource_controller_url                    |
+---------------+--------------------------------------------+
| generic       | http://192.0.2.22:8041/v1/resource/generic |
+---------------+--------------------------------------------+

Upgrade:
--------

gnocchi-upgrade --debug --create-legacy-resource-types -v
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.

New resource types available:
-----------------------------
gnocchi resource list-types
+----------------------------+---------------------------------------------------------------+
| resource_type              | resource_controller_url                                       |
+----------------------------+---------------------------------------------------------------+
| instance                   | http://192.0.2.22:8041/v1/resource/instance                   |
| ceph_account               | http://192.0.2.22:8041/v1/resource/ceph_account               |
| network                    | http://192.0.2.22:8041/v1/resource/network                    |
| host_network_interface     | http://192.0.2.22:8041/v1/resource/host_network_interface     |
| instance_network_interface | http://192.0.2.22:8041/v1/resource/instance_network_interface |
| ipmi                       | http://192.0.2.22:8041/v1/resource/ipmi                       |
| image                      | http://192.0.2.22:8041/v1/resource/image                      |
| host_disk                  | http://192.0.2.22:8041/v1/resource/host_disk                  |
| swift_account              | http://192.0.2.22:8041/v1/resource/swift_account              |
| volume                     | http://192.0.2.22:8041/v1/resource/volume                     |
| generic                    | http://192.0.2.22:8041/v1/resource/generic                    |
| host                       | http://192.0.2.22:8041/v1/resource/host                       |
| instance_disk              | http://192.0.2.22:8041/v1/resource/instance_disk              |
| stack                      | http://192.0.2.22:8041/v1/resource/stack                      |
| identity                   | http://192.0.2.22:8041/v1/resource/identity                   |
+----------------------------+---------------------------------------------------------------+

Packages:
---------
openstack-gnocchi-indexer-sqlalchemy-2.1.3-2.el7ost.noarch
openstack-gnocchi-metricd-2.1.3-2.el7ost.noarch
openstack-gnocchi-common-2.1.3-2.el7ost.noarch
openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch
python-gnocchi-2.1.3-2.el7ost.noarch
openstack-gnocchi-api-2.1.3-2.el7ost.noarch
openstack-gnocchi-carbonara-2.1.3-2.el7ost.noarch
python-gnocchiclient-2.2.0-1.el7ost.noarch

python-rados-0.94.5-14.el7cp.x86_64
librados2-0.94.5-14.el7cp.x86_64

python-cradox-1.1.8-2.el7ost.x86_64

Comment 22 errata-xmlrpc 2016-08-11 12:23:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html


Note You need to log in before you can comment on or make changes to this bug.