Created attachment 1448650 [details]
gnocchi-upgrade-out.log

At least gnocchi-upgrade command, gnocchi db sync, can hit issue in handling exceptions,
when using cradox inside ceph storage backend code,
such that cradox (in code aliased as 'rados') does not have InvalidArgumentError class

> 2018-06-06 08:05:23,307 [21] INFO     gnocchi.cli.manage: Upgrading indexer SQLAlchemyIndexer: mysql+pymysql://gnocchi:VzYR8Qsf34X4Rr8HsnCmEHkjD.1.19/gnocchi?read_default_group=tripleo&read_default_file=/etc/my.cnf.d/tripleo.cnf
> 2018-06-06 08:05:23,585 [21] INFO     gnocchi.common.ceph: Ceph storage backend use 'cradox' python library
> 2018-06-06 08:05:23,633 [21] ERROR    gnocchi.utils: Unable to initialize storage driver
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
>     result = fn(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 117, in get_driver
>     conf.storage, coord)
>   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 48, in __init__
>     self.rados, self.ioctx = ceph.create_rados_connection(conf)
>   File "/usr/lib/python2.7/site-packages/gnocchi/common/ceph.py", line 70, in create_rados_connection
>     except rados.InvalidArgumentError:
> AttributeError: 'module' object has no attribute 'InvalidArgumentError'
> 2018-06-06 08:05:24,636 [21] INFO     gnocchi.common.ceph: Ceph storage backend use 'cradox' python library
> 2018-06-06 08:05:24,668 [21] ERROR    gnocchi.utils: Unable to initialize storage driver
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
>     result = fn(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 117, in get_driver
>     conf.storage, coord)
>   File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 48, in __init__
>     self.rados, self.ioctx = ceph.create_rados_connection(conf)
>   File "/usr/lib/python2.7/site-packages/gnocchi/common/ceph.py", line 70, in create_rados_connection
>     except rados.InvalidArgumentError:
> AttributeError: 'module' object has no attribute 'InvalidArgumentError'
> 2018-06-06 08:05:26,671 [21] INFO     gnocchi.common.ceph: Ceph storage backend use 'cradox' python library

// Other thing would be that even such internal code bug seems to keep gnocchi-upgrade cli in a loop ... but maybe there is some other reason for it?

Seems that this was modified in cradox already,
it's InvalidArgument class was renamed to InvalidArgumentError about a year ago
in https://github.com/sileht/pycradox/commit/2677c9ce907240b2bb4587cc80b3b87e4efb14b4#diff-a6287cd9d3fe6b277b02b08dd09a0c37L310
based on github ui seems it's tagged in 2.0(.7) versions.

But in gnocchi currently set minimal version is cradox>=1.2.0
https://github.com/gnocchixyz/gnocchi/blob/c43eac1ff19f6ae4a08b7386f9eeb5e41d4a448d/setup.cfg#L78

Possibly, solutions could be to in gnochhi bump required cradox version >2,
or add extra handling of possible miscompatibilities (at least alias InvalidArgument as InvalidArgumentError?),
or also backport of this renaming in cradox itself (but i have no idea about policies for such thing neither the scope of it)?

Observed with Red Hat OSP (13), when another bug caused missing keyring files.
> python-gnocchi-4.2.3-3.el7ost.noarch
> python-cradox-1.3.2-2.el7ost.x86_64.rpm