rhosp-director: OSP12 OC deployment fails: "Error running ['docker', 'run', '--name', 'gnocchi_db_sync' Environment: openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch instack-undercloud-7.2.1-0.20170729010706.el7ost.noarch openstack-tripleo-heat-templates-7.0.0-0.20170805163046.el7ost.noarch puppet-gnocchi-11.3.0-0.20170805095843.a7b430b.el7ost.noarch python-gnocchiclient-3.3.1-0.20170721134935.004bf71.el7ost.noarch Steps to reproduce: Attempt to deploy OC. Result: (undercloud) [stack@undercloud-0 ~]$ heat resource-list -n5 overcloud|grep -v COMPLE WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead +----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ | AllNodesDeploySteps | 58703cf5-c985-4b09-bbc4-89b8274c81cd | OS::TripleO::PostDeploySteps | CREATE_FAILED | 2017-08-11T23:58:47Z | overcloud | | ComputeDeployment_Step3 | a2752fb5-0fd0-4e78-b0cc-00b6f841caae | OS::Heat::StructuredDeploymentGroup | CREATE_FAILED | 2017-08-12T00:11:46Z | overcloud-AllNodesDeploySteps-frfdfvdnkj6y | | ControllerDeployment_Step3 | 78b4eefc-5050-4b25-933b-5235153c0f81 | OS::Heat::StructuredDeploymentGroup | CREATE_FAILED | 2017-08-12T00:11:46Z | overcloud-AllNodesDeploySteps-frfdfvdnkj6y | | 0 | f5dc279b-7174-4716-8d78-a2a090e11d25 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2017-08-12T00:30:29Z | overcloud-AllNodesDeploySteps-frfdfvdnkj6y-ControllerDeployment_Step3-qnfn3uhzjydl | +----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+ (undercloud) [stack@undercloud-0 ~]$ echo -e `heat deployment-show f5dc279b-7174-4716-8d78-a2a090e11d25`|grep -i error WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead \"Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\\"image\\": \\"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2\\", \\"environment\\": [\\"TRIPLEO_CONFIG_HASH=29f114217da4b2c5a8c583f6c31b9155\\"], \\"command\\": \\"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c /usr/bin/gnocchi-upgrade --sacks-number=128\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\\", \\"/var/log/containers/gnocchi:/var/log/gnocchi\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}', '--env=TRIPLEO_CONFIG_HASH=29f114217da4b2c5a8c583f6c31b9155', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', '/usr/bin/gnocchi-upgrade', '--sacks-number=128']. [1]\",
using the patch https://code.engineering.redhat.com/gerrit/115081 doesn't seem to help. Deployment failed with: \"Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\\"image\\": \\"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2\\", \\"environment\\": [\\"TRIPLEO_CONFIG_HASH=851453b9cfcff90dae31a5ca6e3c5e62\\"], \\"command\\": \\"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c \\'/usr/bin/gnocchi-upgrade --sacks-number=128\\'\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\\", \\"/var/log/containers/gnocchi:/var/log/gnocchi\\", \\"/etc/ceph:/etc/ceph:ro\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}', '--env=TRIPLEO_CONFIG_HASH=851453b9cfcff90dae31a5ca6e3c5e62', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '--volume=/etc/ceph:/etc/ceph:ro', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', \\"'/usr/bin/gnocchi-upgrade\\", \\"--sacks-number=128'\\"]. [1]\", \"stdout: \",
The error in comment #2 was seen with openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch
Looking at the startup command for gnocchi_db_sync , note the double space: /bin/docker run --name gnocchi_db_sync --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro --volume=/var/log/containers/gnocchi:/var/log/gnocchi --volume=/etc/ceph:/etc/ceph:ro --net=host --privileged=false 192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2 / u s r / b i n / b o o t s t r a p _ h o s t _ e x e c g n o c c h i _ a p i s u g n o c c h i - s / b i n / b a s h - c ' / u s r / b i n / g n o c c h i - u p g r a d e - - s a c k s - n u m b e r = 1 2 8 '
on overcloud-controller-0: [root@overcloud-controller-0 gnocchi]# grep -i error /var/log/containers/gnocchi/gnocchi-upgrade.log 2017-08-15 14:59:24.447 12 CRITICAL gnocchi [-] Unhandled error: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 14:59:24.447 12 ERROR gnocchi Traceback (most recent call last): 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/bin/gnocchi-upgrade", line 10, in <module> 2017-08-15 14:59:24.447 12 ERROR gnocchi sys.exit(upgrade()) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 66, in upgrade 2017-08-15 14:59:24.447 12 ERROR gnocchi s = storage.get_driver(conf) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver 2017-08-15 14:59:24.447 12 ERROR gnocchi conf.storage, incoming, coord) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__ 2017-08-15 14:59:24.447 12 ERROR gnocchi self.rados, self.ioctx = ceph.create_rados_connection(conf) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection 2017-08-15 14:59:24.447 12 ERROR gnocchi raise ImportError("Your rados python module does not support " 2017-08-15 14:59:24.447 12 ERROR gnocchi ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 14:59:24.447 12 ERROR gnocchi [root@overcloud-controller-0 gnocchi]# grep -i error /var/log/containers/gnocchi/gnocchi-upgrade.log 2017-08-15 14:59:24.447 12 CRITICAL gnocchi [-] Unhandled error: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 14:59:24.447 12 ERROR gnocchi Traceback (most recent call last): 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/bin/gnocchi-upgrade", line 10, in <module> 2017-08-15 14:59:24.447 12 ERROR gnocchi sys.exit(upgrade()) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 66, in upgrade 2017-08-15 14:59:24.447 12 ERROR gnocchi s = storage.get_driver(conf) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver 2017-08-15 14:59:24.447 12 ERROR gnocchi conf.storage, incoming, coord) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__ 2017-08-15 14:59:24.447 12 ERROR gnocchi self.rados, self.ioctx = ceph.create_rados_connection(conf) 2017-08-15 14:59:24.447 12 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection 2017-08-15 14:59:24.447 12 ERROR gnocchi raise ImportError("Your rados python module does not support " 2017-08-15 14:59:24.447 12 ERROR gnocchi ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 14:59:24.447 12 ERROR gnocchi
Yea python-rados is too old in downstream repos. Which is used by the base gnocchi container. We will need to rebase this package or override this with python-cradox. So there are mutiple issues being fixed in this bug. Its good to summarize: 1. Parsing issue with gnocchi-upgarde fixed 2. gnocchi ceph ordering issue, where ceph pools are created too late is fixed by adjusting the steps. 3. ceilometer-upgrade race condition fixed with a retry loop in ceilometer containers so it doesnt fail while apache is being restarted. 4. mounts Ceph config data into the container which seems needed for the gnocchi-upgrade command when configured to use Ceph. 5. <downstream only> Fix the rados dependency issue with an override downstream only so gnocchi base container images are using cradox instead. (Ideally we want to rebase rados. But thats a cross repo dependency).
The stack deployment failure still persists despite having the rpms in images as shown below: "Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\\"image\\": \\"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1\\", \\"environment\\": [\\"TRIPLEO_CONFIG_HASH=1377f3a1e7a334851823692a2ba02d0e\\"], \\"command\\": \\"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c \\'/usr/bin/gnocchi-upgrade --sacks-number=128\\'\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\\", \\"/var/log/containers/gnocchi:/var/log/gnocchi\\", \\"/etc/ceph:/etc/ceph:ro\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}', '--env=TRIPLEO_CONFIG_HASH=1377f3a1e7a334851823692a2ba02d0e', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '--volume=/etc/ceph:/etc/ceph:ro', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', \\"'/usr/bin/gnocchi-upgrade\\", \\"--sacks-number=128'\\"]. [1]\", [root@overcloud-controller-2 ~]# docker exec -u root -it gnocchi_metricd bash -c "rpm -qa|grep python-cradox" python-cradox-1.3.2-2.el7ost.x86_64 [root@overcloud-controller-0 ~]# docker run --name test1 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-08-15.1 bash -c "rpm -qa|grep python-cradox" python-cradox-1.3.2-2.el7ost.x86_64 However we still see the error for both gnocchi-metricd and gnocchi-statsd [root@overcloud-controller-2 ~]# tail -n 20 /var/log/containers/gnocchi/gnocchi-metricd.log |grep -i error 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils [-] Unhandled exception: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils Traceback (most recent call last): 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in exit_on_exception 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils yield 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils File "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in _run 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils self.run() 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 120, in run 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils self._configure() 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 115, in _configure 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils self.store = storage.get_driver(self.conf) 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils conf.storage, incoming, coord) 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__ 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils self.rados, self.ioctx = ceph.create_rados_connection(conf) 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils raise ImportError("Your rados python module does not support " 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils 2017-08-15 22:16:24.015 1 CRITICAL gnocchi [-] Unhandled error: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 22:16:24.015 1 ERROR gnocchi Traceback (most recent call last): 2017-08-15 22:16:24.015 1 ERROR gnocchi File "/usr/bin/gnocchi-statsd", line 10, in <module> 2017-08-15 22:16:24.015 1 ERROR gnocchi sys.exit(statsd()) 2017-08-15 22:16:24.015 1 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 102, in statsd 2017-08-15 22:16:24.015 1 ERROR gnocchi statsd_service.start() 2017-08-15 22:16:24.015 1 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 171, in start 2017-08-15 22:16:24.015 1 ERROR gnocchi stats = Stats(conf) 2017-08-15 22:16:24.015 1 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 38, in __init__ 2017-08-15 22:16:24.015 1 ERROR gnocchi self.storage = storage.get_driver(self.conf) 2017-08-15 22:16:24.015 1 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver 2017-08-15 22:16:24.015 1 ERROR gnocchi conf.storage, incoming, coord) 2017-08-15 22:16:24.015 1 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__ 2017-08-15 22:16:24.015 1 ERROR gnocchi self.rados, self.ioctx = ceph.create_rados_connection(conf) 2017-08-15 22:16:24.015 1 ERROR gnocchi File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection 2017-08-15 22:16:24.015 1 ERROR gnocchi raise ImportError("Your rados python module does not support " 2017-08-15 22:16:24.015 1 ERROR gnocchi ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 2017-08-15 22:16:24.015 1 ERROR gnocchi gnocchi_statd is restarting: [root@overcloud-controller-1 ~]# docker ps |grep gnocchi bd33de98210c 192.168.24.1:8787/rhosp12/openstack-gnocchi-metricd-docker:2017-08-15.1 "kolla_start" About an hour ago Up About an hour gnocchi_metricd 548a47e2ee61 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-08-15.1 "kolla_start" About an hour ago Restarting (1) 6 minutes ago gnocchi_statsd
The issue still exists because the cradox module we included still fails: $ docker exec -ti gnocchi_metricd python; Python 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import cradox Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /usr/lib64/python2.7/site-packages/cradox.so: undefined symbol: rados_inconsistent_pg_list Sasha worked around it by upgrading librados2 version in the base container from sasha: " Managed to overcome this by updating librados2 in container built from 192.168.24.1:8787/rhosp12/openstack-gnocchi-metricd-docker:2017-08-15.1 from librados2-0.94.5-2.el7.x86_64 to librados2-10.2.7-28.el7cp.x86_64 "
Verified: Environment: openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch The reported issue doesn't reproduce.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462