Bug 1342571

Summary: Failed to run gnocchi-upgrade with ceph backend
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-puppet-modulesAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: high Docs Contact:
Priority: urgent    
Version: 9.0 (Mitaka)CC: apevec, fbaudin, jason.dobies, jdanjou, jguiditt, jjoyce, jschluet, lhh, mburns, morazi, srevivo, yprokule
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-puppet-modules-8.1.1-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 12:23:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1343144, 1348147, 1348454    
Bug Blocks:    

Description Yurii Prokulevych 2016-06-03 14:39:25 UTC
Description of problem:
Attempt to create 'legacy' resources for ceilometer failed:
gnocchi-upgrade --debug --create-legacy-resource-types -v
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Exception in thread heartbeat:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 77, in _heartbeat
    while not self._stop_heartbeat.is_set():
AttributeError: 'CephStorage' object has no attribute '_stop_heartbeat'

CRITI [gnocchi] ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
Traceback (most recent call last):
  File "/bin/gnocchi-upgrade", line 10, in <module>
    sys.exit(upgrade())
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 58, in upgrade
    s = storage.get_driver(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 155, in get_driver
    return get_driver_class(conf)(conf.storage)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 73, in __init__
    raise ImportError("Your rados python module does not support "
ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 


Version-Release number of selected component (if applicable):
python-gnocchiclient-2.2.0-1.el7ost.noarch
openstack-gnocchi-common-2.1.1-1.el7ost.noarch
openstack-gnocchi-statsd-2.1.1-1.el7ost.noarch
python-gnocchi-2.1.1-1.el7ost.noarch
openstack-gnocchi-metricd-2.1.1-1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.1-1.el7ost.noarch
openstack-gnocchi-api-2.1.1-1.el7ost.noarch
openstack-gnocchi-carbonara-2.1.1-1.el7ost.noarch

python-rados-0.94.5-13.el7cp.x86_64
librados2-0.94.5-13.el7cp.x86_64


Steps to Reproduce:
1. Install overcloud.
2. Configure gnocchi to use ceph backend
3. Run gnocchi-upgrade --debug --create-legacy-resource-types

Comment 3 Mike Burns 2016-06-03 19:21:10 UTC
there are a couple potential issues that need to be solved

1.  the python-cradox library requires librados2 which exists only in Ceph repositories.  This means that people *must* have ceph repositories enabled on their nodes to get the package.  This means we need a doc bug.

2.  How is the package getting installed?  Are we adding an explicit dependency to gnocchi?  are we adding it to puppet to smartly detect/pull it in if we're using a ceph backend?  

3.  We need to ensure that images will get this package automatically as well.

Beyond those 2 questions, this bug should be split into 2 bugs.  1 to resolve #2 and 1 for the new component.

Comment 5 Jason Guiditta 2016-06-08 12:17:00 UTC
Awaiting merge upstream.

Comment 6 Jason Guiditta 2016-06-09 13:44:48 UTC
This is ready to go downstream today, can we get remaining acks on this?

Comment 15 Julien Danjou 2016-06-21 09:38:43 UTC
This is still failing as the cradox Python module cannot be opened by httpd.

It seems that the access is denied by SELinux because the file is wrongly labeled:

type=AVC msg=audit(1466501601.240:93660): avc:  denied  { read } for  pid=31052 comm="httpd" name="cradox.so" dev="sda2" ino=71512 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file


root@overcloud-controller-0 site-packages ]# pwd
/usr/lib64/python2.7/site-packages
[root@overcloud-controller-0 site-packages ]# ls -lZ *.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   acutilmodule.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _audit.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   auparse.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _cffi_backend.so
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 cradox.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _dbus_bindings.so
-rwxr-xr-x. root root unconfined_u:object_r:lib_t:s0   _dbus_glib_bindings.so
[...]

Comment 20 Yurii Prokulevych 2016-07-20 12:23:51 UTC
Excerpt from gnocchi.conf:
--------------------------
[storage]
coordination_url = redis://:ACM2JaMWAYHKj3Jg6tTHzD9fw.2.23:6379/
driver = ceph
ceph_pool = metrics
ceph_username = openstack
ceph_keyring = /etc/ceph/ceph.client.openstack.keyring
ceph_conffile = /etc/ceph/ceph.conf

Supported resource types before upgrade:
----------------------------------------
gnocchi resource list-types
+---------------+--------------------------------------------+
| resource_type | resource_controller_url                    |
+---------------+--------------------------------------------+
| generic       | http://192.0.2.22:8041/v1/resource/generic |
+---------------+--------------------------------------------+

Upgrade:
--------

gnocchi-upgrade --debug --create-legacy-resource-types -v
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.

New resource types available:
-----------------------------
gnocchi resource list-types
+----------------------------+---------------------------------------------------------------+
| resource_type              | resource_controller_url                                       |
+----------------------------+---------------------------------------------------------------+
| instance                   | http://192.0.2.22:8041/v1/resource/instance                   |
| ceph_account               | http://192.0.2.22:8041/v1/resource/ceph_account               |
| network                    | http://192.0.2.22:8041/v1/resource/network                    |
| host_network_interface     | http://192.0.2.22:8041/v1/resource/host_network_interface     |
| instance_network_interface | http://192.0.2.22:8041/v1/resource/instance_network_interface |
| ipmi                       | http://192.0.2.22:8041/v1/resource/ipmi                       |
| image                      | http://192.0.2.22:8041/v1/resource/image                      |
| host_disk                  | http://192.0.2.22:8041/v1/resource/host_disk                  |
| swift_account              | http://192.0.2.22:8041/v1/resource/swift_account              |
| volume                     | http://192.0.2.22:8041/v1/resource/volume                     |
| generic                    | http://192.0.2.22:8041/v1/resource/generic                    |
| host                       | http://192.0.2.22:8041/v1/resource/host                       |
| instance_disk              | http://192.0.2.22:8041/v1/resource/instance_disk              |
| stack                      | http://192.0.2.22:8041/v1/resource/stack                      |
| identity                   | http://192.0.2.22:8041/v1/resource/identity                   |
+----------------------------+---------------------------------------------------------------+

Packages:
---------
openstack-gnocchi-indexer-sqlalchemy-2.1.3-2.el7ost.noarch
openstack-gnocchi-metricd-2.1.3-2.el7ost.noarch
openstack-gnocchi-common-2.1.3-2.el7ost.noarch
openstack-gnocchi-statsd-2.1.3-2.el7ost.noarch
python-gnocchi-2.1.3-2.el7ost.noarch
openstack-gnocchi-api-2.1.3-2.el7ost.noarch
openstack-gnocchi-carbonara-2.1.3-2.el7ost.noarch
python-gnocchiclient-2.2.0-1.el7ost.noarch

python-rados-0.94.5-14.el7cp.x86_64
librados2-0.94.5-14.el7cp.x86_64

python-cradox-1.1.8-2.el7ost.x86_64

Comment 22 errata-xmlrpc 2016-08-11 12:23:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html