Bug 1109402

+++ This bug was initially created as a clone of Bug #1104239 +++

+++ This bug was initially created as a clone of Bug #1093305 +++

Description of problem:

Using RDO I3 i noticed errors in the logs about Deprecation amqp
While we use rabbitMQ

Version-Release number of selected component (if applicable):

openstack-nova-conductor-2014.1-2.el7.noarch


May  1 11:20:28 cougar16 nova-conductor: /usr/lib/python2.7/site-packages/amqp/channel.py:599: DeprecationWarning: auto_delete exchanges has been deprecated
May  1 11:20:28 cougar16 nova-conductor: 'auto_delete exchanges has been deprecated'))
May  1 11:20:28 cougar16 nova-conductor: /usr/lib/python2.7/site-packages/amqp/channel.py:599: DeprecationWarning: auto_delete exchanges has been deprecated
May  1 11:20:28 cougar16 nova-conductor: 'auto_delete exchanges has been deprecated'))

How reproducible:


Steps to Reproduce:
1.install RDO All in one with RabbitMq and neutron
2.check /var/log/messages 
3.

--- Additional comment from Stephen Gordon on 2014-06-03 11:16:55 EDT ---

Apparently these are frequent enough to be flooding the logs:

https://gist.github.com/jtaleric/015c8842a9e11355849a

--- Additional comment from Joe Talerico on 2014-06-03 13:10:40 EDT ---

To be clear, the gist above is from a RHEL OSP5 install on RHEL7. 

Nova packages:
openstack-nova-conductor-2014.1-2.el7ost.noarch
openstack-nova-common-2014.1-2.el7ost.noarch
openstack-nova-novncproxy-2014.1-2.el7ost.noarch
openstack-nova-api-2014.1-2.el7ost.noarch
openstack-nova-scheduler-2014.1-2.el7ost.noarch
openstack-nova-console-2014.1-2.el7ost.noarch
openstack-nova-cert-2014.1-2.el7ost.noar

--- Additional comment from Brent Eagles on 2014-06-03 16:25:35 EDT ---

Apparently the increase in the visibility of the deprecation message is deliberate. See: 

https://github.com/celery/kombu/issues/296

and

https://pypi.python.org/pypi/amqp/1.4.5

A discussion of auto_delete in rabbitmq and the way we use it:

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-May/036031.html

A little contradictory in that it claims the feature isn't going away.

The amqp lib in pypi seems to be the source of this warning. What is interesting is that it shouldn't be showing up if you are using python 2.7 (which seems to be the case above) as the DeprecationWarning has been defaulted to "ignore" in the warnings module in 2.7. Maybe a bogus py[co] file kicking around? If not then there is probably another issue to be reported against the python packages. (setting NEEDINFO to get it checked out)

It *is* a nuisance for folks stuck on python 2.6 though since it kicks up an exception. The workaround would be to use the "warnings" module's filtering mechanism to suppress the exception.

--- Additional comment from Brent Eagles on 2014-06-04 13:47:05 EDT ---

As a heads up, we don't get this with 6.5 because it uses amqplib, not amqp.py.

--- Additional comment from Brent Eagles on 2014-06-05 08:21:29 EDT ---

As it so often turns out, the amqp.py thing is not the entire story. On a RHEL 6.5 system with python 2.6.6, I hacked up my amqplib.py to see if it would display the deprecation warning and it did. I tried disabling it with a call to warnings.simplefilter() and it still appeared. A simple check reveals that the python environment on the host exhibiting the issue has the warning disabled by default, so that was not the source of the issue. It remained then to hunt down what actually was enabling it. Hacking warnings.py to print a stack trace when the filter manipulators were called revealed that

/usr/lib/python2.6/site-packages/migrate/changeset/__init__.py 

contains:

warnings.simplefilter('always', DeprecationWarning)

This is arguably a bug in that migrate module. Instead of restrict the filter to its own modules, it applies it globally. 

Full stack trace causing import follows:

always <type 'exceptions.DeprecationWarning'>
  File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 483, in run_service
    service.start()
  File "/usr/lib/python2.6/site-packages/nova/service.py", line 168, in start
    self.host, self.binary)
  File "/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 193, in service_get_by_args
    binary=binary)
  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 966, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 441, in service_get_all_by
    result = self.db.service_get_by_args(context, host, binary)
  File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 134, in service_get_by_args
    return IMPL.service_get_by_args(context, host, binary)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/api.py", line 147, in __getattr__
    self._load_backend()
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/api.py", line 142, in _load_backend
    backend_mod = importutils.import_module(backend_path)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/importutils.py", line 57, in import_module
    __import__(import_str)
  File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 57, in <module>
    from nova.openstack.common.db.sqlalchemy import utils as sqlalchemyutils
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/utils.py", line 22, in <module>
    from migrate.changeset import UniqueConstraint
  File "/usr/lib/python2.6/site-packages/migrate/__init__.py", line 31, in <module>
    from migrate.changeset import *
  File "/usr/lib/python2.6/site-packages/migrate/changeset/__init__.py", line 13, in <module>
    warnings.simplefilter('always', DeprecationWarning)
  File "/usr/lib64/python2.6/warnings.py", line 76, in simplefilter

--- Additional comment from Nikola Dipanov on 2014-06-12 07:08:03 EDT ---

Seeing that there is a root cause identified - I assume we can patch this in RHOS for the 5.0 release and do the proper fix upstream and drop the patch once that's in?

--- Additional comment from Brent Eagles on 2014-06-12 09:34:11 EDT ---

Nikola - I agree. We are moving to fix this at the "source" (i.e. sqlalchemy-migrate) instead of the previously posted workarounds. This should satisfy the concerns posted against the original approach. So to be clear, we'd be patching python-sqlalchemy and not nova or oslo, etc.

--- Additional comment from Brent Eagles on 2014-06-12 10:17:02 EDT ---

Indicated wrong package in Comment 7 - should have been python-migrate.

--- Additional comment from Brent Eagles on 2014-06-13 11:09:23 EDT ---

To verify:

Scenario A: existing openstack deployment

- Before updating packages, inspect /var/log/messages and check for reports similar to the ones indicated in this bug report (e.g. auto_delete exchanges are deprecated, etc). 
- Shutdown openstack services (openstack-service stop)
- Verify there are no openstack services running (ps ax | grep python, etc)
- Install updated python-migrate package. You can update all RPMs, but for the sake of direct verification it would be better to update only this package.
- Start openstack services (openstack-service start)
- Inspect or tail /var/log/messages. You should not see any new warnings about auto_delete exchanges.

Scenario B: new openstack deployment on fresh system
- Verify openstack services are running
- Inspect and/or tail /var/log/messages. You should not see deprecation warnings about auto_delete exchanges.

It is not clear that additional errata information is required as this was an internally reported issue and would not otherwise impact users unless they were dependent on DeprecationWarnings being "enabled-by-side-effect" in their apps.