Bug 1104239 - nova-conductor Deprecation Warning
Summary: nova-conductor Deprecation Warning
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-migrate
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 5.0 (RHEL 7)
Assignee: Brent Eagles
QA Contact: Toure Dunnon
URL:
Whiteboard:
Depends On: 1093305
Blocks: 1109402
TreeView+ depends on / blocked
 
Reported: 2014-06-03 15:16 UTC by Stephen Gordon
Modified: 2019-09-09 13:07 UTC (History)
8 users (show)

Fixed In Version: python-migrate-0.8.2-2.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of: 1093305
: 1109402 (view as bug list)
Environment:
Last Closed: 2014-07-08 15:14:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1328647 0 None None None Never
OpenStack gerrit 99657 0 None None None Never
Red Hat Product Errata RHEA-2014:0845 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Enhancement - Runtime Components 2014-07-08 19:11:27 UTC

Description Stephen Gordon 2014-06-03 15:16:17 UTC
+++ This bug was initially created as a clone of Bug #1093305 +++

Description of problem:

Using RDO I3 i noticed errors in the logs about Deprecation amqp
While we use rabbitMQ

Version-Release number of selected component (if applicable):

openstack-nova-conductor-2014.1-2.el7.noarch


May  1 11:20:28 cougar16 nova-conductor: /usr/lib/python2.7/site-packages/amqp/channel.py:599: DeprecationWarning: auto_delete exchanges has been deprecated
May  1 11:20:28 cougar16 nova-conductor: 'auto_delete exchanges has been deprecated'))
May  1 11:20:28 cougar16 nova-conductor: /usr/lib/python2.7/site-packages/amqp/channel.py:599: DeprecationWarning: auto_delete exchanges has been deprecated
May  1 11:20:28 cougar16 nova-conductor: 'auto_delete exchanges has been deprecated'))

How reproducible:


Steps to Reproduce:
1.install RDO All in one with RabbitMq and neutron
2.check /var/log/messages 
3.

Comment 1 Stephen Gordon 2014-06-03 15:16:55 UTC
Apparently these are frequent enough to be flooding the logs:

https://gist.github.com/jtaleric/015c8842a9e11355849a

Comment 2 Joe Talerico 2014-06-03 17:10:40 UTC
To be clear, the gist above is from a RHEL OSP5 install on RHEL7. 

Nova packages:
openstack-nova-conductor-2014.1-2.el7ost.noarch
openstack-nova-common-2014.1-2.el7ost.noarch
openstack-nova-novncproxy-2014.1-2.el7ost.noarch
openstack-nova-api-2014.1-2.el7ost.noarch
openstack-nova-scheduler-2014.1-2.el7ost.noarch
openstack-nova-console-2014.1-2.el7ost.noarch
openstack-nova-cert-2014.1-2.el7ost.noar

Comment 3 Brent Eagles 2014-06-03 20:25:35 UTC
Apparently the increase in the visibility of the deprecation message is deliberate. See: 

https://github.com/celery/kombu/issues/296

and

https://pypi.python.org/pypi/amqp/1.4.5

A discussion of auto_delete in rabbitmq and the way we use it:

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-May/036031.html

A little contradictory in that it claims the feature isn't going away.

The amqp lib in pypi seems to be the source of this warning. What is interesting is that it shouldn't be showing up if you are using python 2.7 (which seems to be the case above) as the DeprecationWarning has been defaulted to "ignore" in the warnings module in 2.7. Maybe a bogus py[co] file kicking around? If not then there is probably another issue to be reported against the python packages. (setting NEEDINFO to get it checked out)

It *is* a nuisance for folks stuck on python 2.6 though since it kicks up an exception. The workaround would be to use the "warnings" module's filtering mechanism to suppress the exception.

Comment 4 Brent Eagles 2014-06-04 17:47:05 UTC
As a heads up, we don't get this with 6.5 because it uses amqplib, not amqp.py.

Comment 5 Brent Eagles 2014-06-05 12:21:29 UTC
As it so often turns out, the amqp.py thing is not the entire story. On a RHEL 6.5 system with python 2.6.6, I hacked up my amqplib.py to see if it would display the deprecation warning and it did. I tried disabling it with a call to warnings.simplefilter() and it still appeared. A simple check reveals that the python environment on the host exhibiting the issue has the warning disabled by default, so that was not the source of the issue. It remained then to hunt down what actually was enabling it. Hacking warnings.py to print a stack trace when the filter manipulators were called revealed that

/usr/lib/python2.6/site-packages/migrate/changeset/__init__.py 

contains:

warnings.simplefilter('always', DeprecationWarning)

This is arguably a bug in that migrate module. Instead of restrict the filter to its own modules, it applies it globally. 

Full stack trace causing import follows:

always <type 'exceptions.DeprecationWarning'>
  File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 483, in run_service
    service.start()
  File "/usr/lib/python2.6/site-packages/nova/service.py", line 168, in start
    self.host, self.binary)
  File "/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 193, in service_get_by_args
    binary=binary)
  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 966, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 441, in service_get_all_by
    result = self.db.service_get_by_args(context, host, binary)
  File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 134, in service_get_by_args
    return IMPL.service_get_by_args(context, host, binary)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/api.py", line 147, in __getattr__
    self._load_backend()
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/api.py", line 142, in _load_backend
    backend_mod = importutils.import_module(backend_path)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/importutils.py", line 57, in import_module
    __import__(import_str)
  File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 57, in <module>
    from nova.openstack.common.db.sqlalchemy import utils as sqlalchemyutils
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/utils.py", line 22, in <module>
    from migrate.changeset import UniqueConstraint
  File "/usr/lib/python2.6/site-packages/migrate/__init__.py", line 31, in <module>
    from migrate.changeset import *
  File "/usr/lib/python2.6/site-packages/migrate/changeset/__init__.py", line 13, in <module>
    warnings.simplefilter('always', DeprecationWarning)
  File "/usr/lib64/python2.6/warnings.py", line 76, in simplefilter

Comment 6 Nikola Dipanov 2014-06-12 11:08:03 UTC
Seeing that there is a root cause identified - I assume we can patch this in RHOS for the 5.0 release and do the proper fix upstream and drop the patch once that's in?

Comment 7 Brent Eagles 2014-06-12 13:34:11 UTC
Nikola - I agree. We are moving to fix this at the "source" (i.e. sqlalchemy-migrate) instead of the previously posted workarounds. This should satisfy the concerns posted against the original approach. So to be clear, we'd be patching python-sqlalchemy and not nova or oslo, etc.

Comment 8 Brent Eagles 2014-06-12 14:17:02 UTC
Indicated wrong package in Comment 7 - should have been python-migrate.

Comment 9 Brent Eagles 2014-06-13 15:09:23 UTC
To verify:

Scenario A: existing openstack deployment

- Before updating packages, inspect /var/log/messages and check for reports similar to the ones indicated in this bug report (e.g. auto_delete exchanges are deprecated, etc). 
- Shutdown openstack services (openstack-service stop)
- Verify there are no openstack services running (ps ax | grep python, etc)
- Install updated python-migrate package. You can update all RPMs, but for the sake of direct verification it would be better to update only this package.
- Start openstack services (openstack-service start)
- Inspect or tail /var/log/messages. You should not see any new warnings about auto_delete exchanges.

Scenario B: new openstack deployment on fresh system
- Verify openstack services are running
- Inspect and/or tail /var/log/messages. You should not see deprecation warnings about auto_delete exchanges.

It is not clear that additional errata information is required as this was an internally reported issue and would not otherwise impact users unless they were dependent on DeprecationWarnings being "enabled-by-side-effect" in their apps.

Comment 14 errata-xmlrpc 2014-07-08 15:14:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0845.html


Note You need to log in before you can comment on or make changes to this bug.