Bug 1480831 - rhosp-director: OSP12 OC deployment fails: "Error running ['docker', 'run', '--name', 'gnocchi_db_sync'
rhosp-director: OSP12 OC deployment fails: "Error running ['docker', 'run', '...
Status: VERIFIED
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
12.0 (Pike)
Unspecified Unspecified
urgent Severity urgent
: beta
: 12.0 (Pike)
Assigned To: Pradeep Kilambi
Alexander Chuzhoy
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-11 21:33 EDT by Alexander Chuzhoy
Modified: 2017-10-27 11:23 EDT (History)
13 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170805163047.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1709630 None None None 2017-08-14 18:02 EDT
OpenStack gerrit 491826 None None None 2017-08-13 09:04 EDT

  None (edit)
Description Alexander Chuzhoy 2017-08-11 21:33:00 EDT
rhosp-director: OSP12 OC deployment fails: "Error running ['docker', 'run', '--name', 'gnocchi_db_sync'

Environment:
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch
instack-undercloud-7.2.1-0.20170729010706.el7ost.noarch
openstack-tripleo-heat-templates-7.0.0-0.20170805163046.el7ost.noarch
puppet-gnocchi-11.3.0-0.20170805095843.a7b430b.el7ost.noarch
python-gnocchiclient-3.3.1-0.20170721134935.004bf71.el7ost.noarch


Steps to reproduce:
Attempt to deploy OC.

Result:
(undercloud) [stack@undercloud-0 ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| resource_name                                | physical_resource_id                                                                                                                                                                 | resource_type                                                                                                                  | resource_status | updated_time         | stack_name                                                                                                                                               |
+----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
| AllNodesDeploySteps                          | 58703cf5-c985-4b09-bbc4-89b8274c81cd                                                                                                                                                 | OS::TripleO::PostDeploySteps                                                                                                   | CREATE_FAILED   | 2017-08-11T23:58:47Z | overcloud                                                                                                                                                |
| ComputeDeployment_Step3                      | a2752fb5-0fd0-4e78-b0cc-00b6f841caae                                                                                                                                                 | OS::Heat::StructuredDeploymentGroup                                                                                            | CREATE_FAILED   | 2017-08-12T00:11:46Z | overcloud-AllNodesDeploySteps-frfdfvdnkj6y                                                                                                               |
| ControllerDeployment_Step3                   | 78b4eefc-5050-4b25-933b-5235153c0f81                                                                                                                                                 | OS::Heat::StructuredDeploymentGroup                                                                                            | CREATE_FAILED   | 2017-08-12T00:11:46Z | overcloud-AllNodesDeploySteps-frfdfvdnkj6y                                                                                                               |
| 0                                            | f5dc279b-7174-4716-8d78-a2a090e11d25                                                                                                                                                 | OS::Heat::StructuredDeployment                                                                                                 | CREATE_FAILED   | 2017-08-12T00:30:29Z | overcloud-AllNodesDeploySteps-frfdfvdnkj6y-ControllerDeployment_Step3-qnfn3uhzjydl                                                                       |
+----------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+






(undercloud) [stack@undercloud-0 ~]$ echo -e `heat deployment-show f5dc279b-7174-4716-8d78-a2a090e11d25`|grep -i error
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
 \"Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step3', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\\"image\\": \\"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2\\", \\"environment\\": [\\"TRIPLEO_CONFIG_HASH=29f114217da4b2c5a8c583f6c31b9155\\"], \\"command\\": \\"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c /usr/bin/gnocchi-upgrade --sacks-number=128\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\\", \\"/var/log/containers/gnocchi:/var/log/gnocchi\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}', '--env=TRIPLEO_CONFIG_HASH=29f114217da4b2c5a8c583f6c31b9155', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', '/usr/bin/gnocchi-upgrade', '--sacks-number=128']. [1]\",
Comment 2 Alexander Chuzhoy 2017-08-14 15:24:28 EDT
using the patch https://code.engineering.redhat.com/gerrit/115081 doesn't seem to help.


Deployment failed with:
 \"Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\\"image\\": \\"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2\\", \\"environment\\": [\\"TRIPLEO_CONFIG_HASH=851453b9cfcff90dae31a5ca6e3c5e62\\"], \\"command\\": \\"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c \\'/usr/bin/gnocchi-upgrade --sacks-number=128\\'\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\\", \\"/var/log/containers/gnocchi:/var/log/gnocchi\\", \\"/etc/ceph:/etc/ceph:ro\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}', '--env=TRIPLEO_CONFIG_HASH=851453b9cfcff90dae31a5ca6e3c5e62', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '--volume=/etc/ceph:/etc/ceph:ro', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', \\"'/usr/bin/gnocchi-upgrade\\", \\"--sacks-number=128'\\"]. [1]\", 
 \"stdout: \",
Comment 3 Alexander Chuzhoy 2017-08-14 15:30:21 EDT
The error in comment #2 was seen with openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch
Comment 5 Alexander Chuzhoy 2017-08-14 17:22:54 EDT
Looking at the startup command for gnocchi_db_sync , note the double space:

/bin/docker run --name gnocchi_db_sync --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro --volume=/var/log/containers/gnocchi:/var/log/gnocchi --volume=/etc/ceph:/etc/ceph:ro --net=host --privileged=false 192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-11.2 / u s r / b i n / b o o t s t r a p _ h o s t _ e x e c   g n o c c h i _ a p i   s u   g n o c c h i   - s   / b i n / b a s h   - c   ' / u s r / b i n / g n o c c h i - u p g r a d e   - - s a c k s - n u m b e r = 1 2 8 '
Comment 6 Alexander Chuzhoy 2017-08-15 11:36:58 EDT
on overcloud-controller-0:

[root@overcloud-controller-0 gnocchi]# grep -i error /var/log/containers/gnocchi/gnocchi-upgrade.log 
2017-08-15 14:59:24.447 12 CRITICAL gnocchi [-] Unhandled error: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0
2017-08-15 14:59:24.447 12 ERROR gnocchi Traceback (most recent call last):
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/bin/gnocchi-upgrade", line 10, in <module>
2017-08-15 14:59:24.447 12 ERROR gnocchi     sys.exit(upgrade())
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 66, in upgrade
2017-08-15 14:59:24.447 12 ERROR gnocchi     s = storage.get_driver(conf)
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver
2017-08-15 14:59:24.447 12 ERROR gnocchi     conf.storage, incoming, coord)
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__
2017-08-15 14:59:24.447 12 ERROR gnocchi     self.rados, self.ioctx = ceph.create_rados_connection(conf)
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection
2017-08-15 14:59:24.447 12 ERROR gnocchi     raise ImportError("Your rados python module does not support "
2017-08-15 14:59:24.447 12 ERROR gnocchi ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
2017-08-15 14:59:24.447 12 ERROR gnocchi 





[root@overcloud-controller-0 gnocchi]# grep -i error /var/log/containers/gnocchi/gnocchi-upgrade.log 
2017-08-15 14:59:24.447 12 CRITICAL gnocchi [-] Unhandled error: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0
2017-08-15 14:59:24.447 12 ERROR gnocchi Traceback (most recent call last):
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/bin/gnocchi-upgrade", line 10, in <module>
2017-08-15 14:59:24.447 12 ERROR gnocchi     sys.exit(upgrade())
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 66, in upgrade
2017-08-15 14:59:24.447 12 ERROR gnocchi     s = storage.get_driver(conf)
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver
2017-08-15 14:59:24.447 12 ERROR gnocchi     conf.storage, incoming, coord)
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__
2017-08-15 14:59:24.447 12 ERROR gnocchi     self.rados, self.ioctx = ceph.create_rados_connection(conf)
2017-08-15 14:59:24.447 12 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection
2017-08-15 14:59:24.447 12 ERROR gnocchi     raise ImportError("Your rados python module does not support "
2017-08-15 14:59:24.447 12 ERROR gnocchi ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
2017-08-15 14:59:24.447 12 ERROR gnocchi
Comment 7 Pradeep Kilambi 2017-08-15 12:31:14 EDT
Yea python-rados is too old in downstream repos. Which is used by the base gnocchi container. We will need to rebase this package or override this with python-cradox.


So there are mutiple issues being fixed in this bug. Its good to summarize:

1. Parsing issue with gnocchi-upgarde fixed

2. gnocchi ceph ordering issue, where ceph pools are created too late is fixed by adjusting the steps.

3. ceilometer-upgrade race condition fixed with a retry loop in ceilometer containers so it doesnt fail while apache is being restarted.

4.  mounts Ceph config data into the container which seems needed for the gnocchi-upgrade command when configured to use Ceph.

5. <downstream only> Fix the rados dependency issue with an override downstream only so gnocchi base container images are using cradox instead. (Ideally we want to rebase rados. But thats a cross repo dependency).
Comment 8 Alexander Chuzhoy 2017-08-15 18:52:39 EDT
The stack deployment failure still persists despite having the rpms in images as shown below:


"Error running ['docker', 'run', '--name', 'gnocchi_db_sync', '--label', 'config_id=tripleo_step4', '--label', 'container_name=gnocchi_db_sync', '--label', 'managed_by=paunch', '--label', 'config_data={\\"image\\": \\"192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1\\", \\"environment\\": [\\"TRIPLEO_CONFIG_HASH=1377f3a1e7a334851823692a2ba02d0e\\"], \\"command\\": \\"/usr/bin/bootstrap_host_exec gnocchi_api su gnocchi -s /bin/bash -c \\'/usr/bin/gnocchi-upgrade --sacks-number=128\\'\\", \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro\\", \\"/var/log/containers/gnocchi:/var/log/gnocchi\\", \\"/etc/ceph:/etc/ceph:ro\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": false}', '--env=TRIPLEO_CONFIG_HASH=1377f3a1e7a334851823692a2ba02d0e', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/gnocchi/etc/gnocchi/:/etc/gnocchi/:ro', '--volume=/var/log/containers/gnocchi:/var/log/gnocchi', '--volume=/etc/ceph:/etc/ceph:ro', '192.168.24.1:8787/rhosp12/openstack-gnocchi-api-docker:2017-08-15.1', '/usr/bin/bootstrap_host_exec', 'gnocchi_api', 'su', 'gnocchi', '-s', '/bin/bash', '-c', \\"'/usr/bin/gnocchi-upgrade\\", \\"--sacks-number=128'\\"]. [1]\",





[root@overcloud-controller-2 ~]# docker exec -u root -it gnocchi_metricd bash -c "rpm -qa|grep python-cradox"
python-cradox-1.3.2-2.el7ost.x86_64


[root@overcloud-controller-0 ~]# docker run --name test1 192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-08-15.1 bash -c "rpm -qa|grep python-cradox"
python-cradox-1.3.2-2.el7ost.x86_64



However we still see the error for both gnocchi-metricd and gnocchi-statsd

[root@overcloud-controller-2 ~]# tail -n 20 /var/log/containers/gnocchi/gnocchi-metricd.log |grep -i error
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils [-] Unhandled exception: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils Traceback (most recent call last):
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in exit_on_exception
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils     yield
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in _run
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils     self.run()
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 120, in run
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils     self._configure()
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 115, in _configure
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils     self.store = storage.get_driver(self.conf)
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils     conf.storage, incoming, coord)
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils     self.rados, self.ioctx = ceph.create_rados_connection(conf)
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils   File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils     raise ImportError("Your rados python module does not support "
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
2017-08-15 21:47:31.470 4828 ERROR cotyledon._utils 






2017-08-15 22:16:24.015 1 CRITICAL gnocchi [-] Unhandled error: ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0
2017-08-15 22:16:24.015 1 ERROR gnocchi Traceback (most recent call last):
2017-08-15 22:16:24.015 1 ERROR gnocchi   File "/usr/bin/gnocchi-statsd", line 10, in <module>
2017-08-15 22:16:24.015 1 ERROR gnocchi     sys.exit(statsd())
2017-08-15 22:16:24.015 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 102, in statsd
2017-08-15 22:16:24.015 1 ERROR gnocchi     statsd_service.start()
2017-08-15 22:16:24.015 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 171, in start
2017-08-15 22:16:24.015 1 ERROR gnocchi     stats = Stats(conf)
2017-08-15 22:16:24.015 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 38, in __init__
2017-08-15 22:16:24.015 1 ERROR gnocchi     self.storage = storage.get_driver(self.conf)
2017-08-15 22:16:24.015 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 162, in get_driver
2017-08-15 22:16:24.015 1 ERROR gnocchi     conf.storage, incoming, coord)
2017-08-15 22:16:24.015 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 46, in __init__
2017-08-15 22:16:24.015 1 ERROR gnocchi     self.rados, self.ioctx = ceph.create_rados_connection(conf)
2017-08-15 22:16:24.015 1 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/storage/common/ceph.py", line 53, in create_rados_connection
2017-08-15 22:16:24.015 1 ERROR gnocchi     raise ImportError("Your rados python module does not support "
2017-08-15 22:16:24.015 1 ERROR gnocchi ImportError: Your rados python module does not support omap feature. Install 'cradox' (recommended) or upgrade 'python-rados' >= 9.1.0 
2017-08-15 22:16:24.015 1 ERROR gnocchi 





gnocchi_statd is restarting:
[root@overcloud-controller-1 ~]# docker ps |grep gnocchi
bd33de98210c        192.168.24.1:8787/rhosp12/openstack-gnocchi-metricd-docker:2017-08-15.1           "kolla_start"            About an hour ago   Up About an hour                                   gnocchi_metricd
548a47e2ee61        192.168.24.1:8787/rhosp12/openstack-gnocchi-statsd-docker:2017-08-15.1            "kolla_start"            About an hour ago   Restarting (1) 6 minutes ago                       gnocchi_statsd
Comment 9 Pradeep Kilambi 2017-08-16 07:50:51 EDT
The issue still exists because the cradox module we included still fails:

$ docker exec -ti gnocchi_metricd python;
Python 2.7.5 (default, May  3 2017, 07:55:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cradox
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /usr/lib64/python2.7/site-packages/cradox.so: undefined
symbol: rados_inconsistent_pg_list

Sasha worked around it by upgrading librados2 version in the base container

from sasha:

"
Managed to overcome this by updating librados2 in container built from 192.168.24.1:8787/rhosp12/openstack-gnocchi-metricd-docker:2017-08-15.1
from  librados2-0.94.5-2.el7.x86_64
to librados2-10.2.7-28.el7cp.x86_64

"
Comment 14 Alexander Chuzhoy 2017-08-22 11:15:31 EDT
Verified:

Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch

The reported issue doesn't reproduce.

Note You need to log in before you can comment on or make changes to this bug.