Bug 1481993

Summary: Failed to run gnocchi-upgrade with ceph backend in a container
Product: Red Hat OpenStack Reporter: Attila Fazekas <afazekas>
Component: python-cradoxAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: high Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: apevec, jjoyce, jschluet, lhh
Target Milestone: rcKeywords: AutomationBlocker, Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-cradox-1.3.2-2.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:52:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Attila Fazekas 2017-08-16 08:58:21 UTC
Description of problem:
 Failed to install containerized openstack with external ceph (just ceph credentials passed to tripleo, the ceph setup made in an another way).

From the stack failures: 

            "Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\"image\": \"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1\", \"environment\": [\"TRIPLEO_CONFIG_HASH=19b6aea103cbbdec15d3dada001b8933\"], \"command\": \"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c \\'/usr/bin/gnocchi-upgrade --sacks-number=128\\'\", \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\", \"/var/log/containers/gnocchi:/var/log/gnocchi\", \"/etc/ceph:/etc/ceph:ro\"], \"net\": \"host\", \"detach\": false, \"privileged\": false}', '--env=TRIPLEO_CONFIG_HASH=19b6aea103cbbdec15d3dada001b8933', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '--volume=/etc/ceph:/etc/ceph:ro', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', \"'/usr/bin/gnocchi-upgrade\", \"--sacks-number=128'\"]. [1]", 


from controller-0/var/log/containers/gnocchi/gnocchi-metricd.log:

2017-08-16 00:20:10.883 664 DEBUG cotyledon._service [-] Run service processing(3) [664] wait_forever /usr/lib/python2.7/site-packages/cotyledon/_service.py:241
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils [-] Unhandled exception: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils Traceback (most recent call last):
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in exit_on_exception
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     yield
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in _run
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self.run()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 120, in run
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self._configure()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 171, in wrapped_f
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     return self.call(f, *args, **kw)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 248, in call
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     start_time=start_time)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 203, in iter
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     return fut.result()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 396, in result
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     return self.__get_result()
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 251, in call
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     result = fn(*args, **kwargs)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 184, in _configure
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self.store = storage.get_driver(self.conf, self.coord)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     conf.storage, incoming, coord)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     self.rados, self.ioctx = ceph.create_rados_connection(conf)
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils     raise ImportError("Your rados python module does not support "
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
2017-08-16 00:20:10.887 664 ERROR cotyledon._utils 


The deployment without containers succeed.


Version-Release number of selected component (if applicable):
 2017-08-15.1

How reproducible:
 always


We had similar exception at the mitaka age:
https://bugzilla.redhat.com/show_bug.cgi?id=1342571

Comment 2 Pradeep Kilambi 2017-08-16 11:56:29 UTC
Yes know issue being investigated and fixed as part of bug#1480831


Issue is, the downstream containers have very old version of python-rados. We need to either rebase or replace with python-cradox. Jon rebuilt images with cradox. But the librados is still old, so we need to update that as well.

This is a non issue upstream as python-rados is up2date.

Comment 7 Sasha Smolyak 2017-11-13 13:06:22 UTC
The deployment passes without errors.

Comment 10 errata-xmlrpc 2017-12-13 21:52:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462