Bug 1481993 - Failed to run gnocchi-upgrade with ceph backend in a container
Summary: Failed to run gnocchi-upgrade with ceph backend in a container
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-cradox
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 12.0 (Pike)
Assignee: Pradeep Kilambi
QA Contact: Sasha Smolyak
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-16 08:58 UTC by Attila Fazekas
Modified: 2018-02-05 19:12 UTC (History)
4 users (show)

Fixed In Version: python-cradox-1.3.2-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 21:52:15 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Attila Fazekas 2017-08-16 08:58:21 UTC
Description of problem:
 Failed to install containerized openstack with external ceph (just ceph credentials passed to tripleo, the ceph setup made in an another way).

From the stack failures: 

            "Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\"image\": \"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1\", \"environment\": [\"TRIPLEO_CONFIG_HASH=19b6aea103cbbdec15d3dada001b8933\"], \"command\": \"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c \\'/usr/bin/gnocchi-upgrade --sacks-number=128\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\", \"/var/log/containers/gnocchi:/var/log/gnocchi\", \"/etc/ceph:/etc/ceph:ro\"], \"net\": \"host\", \"detach\": false, \"privileged\": false}', '--env=TRIPLEO_CONFIG_HASH=19b6aea103cbbdec15d3dada001b8933', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '--volume=/etc/ceph:/etc/ceph:ro', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', \"'/usr/bin/gnocchi-upgrade\", \"--sacks-number=128'\"]. [1]", 


from controller-0/var/log/containers/gnocchi/gnocchi-metricd.log:

2017-08-16 00:20:10.883 664 DEBUG cotyledon._service [-] Run service processing(3) [664] wait_forever /usr/lib/python2.7/site-packages/cotyledon/_service.py:241
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils [-] Unhandled exception: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils Traceback (most recent call last):
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in exit_on_exception
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     yield
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in _run
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self.run()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 120, in run
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self._configure()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 171, in wrapped_f
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     return self.call(f, *args, **kw)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 248, in call
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     start_time=start_time)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 203, in iter
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     return fut.result()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 396, in result
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     return self.__get_result()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 251, in call
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     result = fn(*args, **kwargs)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 184, in _configure
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self.store = storage.get_driver(self.conf, self.coord)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     conf.storage, incoming, coord)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self.rados, self.ioctx = ceph.create_rados_connection(conf)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     raise ImportError("Your rados python module does not support "
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils 


The deployment without containers succeed.


Version-Release number of selected component (if applicable):
 2017-08-15.1

How reproducible:
 always


We had similar exception at the mitaka age:
https://bugzilla.redhat.com/show_bug.cgi?id=1342571

Comment 2 Pradeep Kilambi 2017-08-16 11:56:29 UTC
Yes know issue being investigated and fixed as part of bug#1480831


Issue is, the downstream containers have very old version of python-rados. We need to either rebase or replace with python-cradox. Jon rebuilt images with cradox. But the librados is still old, so we need to update that as well.

This is a non issue upstream as python-rados is up2date.

Comment 7 Sasha Smolyak 2017-11-13 13:06:22 UTC
The deployment passes without errors.

Comment 10 errata-xmlrpc 2017-12-13 21:52:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.