Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1451415

Summary:	rhosp-director: Repeating errors: "ERROR oslo_messaging._drivers.amqpdriver" - fill up the file system.
Product:	Red Hat OpenStack	Reporter:	Alexander Chuzhoy <sasha>
Component:	python-kombu	Assignee:	Matthias Runge <mrunge>
Status:	CLOSED ERRATA	QA Contact:	Alexander Chuzhoy <sasha>
Severity:	high	Docs Contact:
Priority:	high
Version:	12.0 (Pike)	CC:	ahrechan, apevec, dbecker, emacchi, jeckersb, lhh, mburns, mcornea, michele, morazi, oblaut, ohochman, rhel-osp-director-maint, scohen, srevivo, tvignaud
Target Milestone:	Upstream M2	Keywords:	Triaged
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-kombu-4.0.2-5.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-13 21:27:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2017-05-16 15:16:35 UTC

rhosp-director: Repeating errors: "ERROR oslo_messaging._drivers.amqpdriver" - fill up the file system.

Environment:
instack-undercloud-7.0.0-0.20170503001109.el7ost.noarch


The file system was filled up in a matter of few hours:
[root@undercloud-0 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       100G  100G   20K 100% /
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G  4.0K  7.8G   1% /dev/shm
tmpfs           7.8G   33M  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/0

The majority of space was used by neutron/nova logs:
59G     neutron
25G     nova

/var/log/neutron
26G     dhcp-agent.log
9.9G    dhcp-agent.log-20170516
25G     openvswitch-agent.log


Looking inside dhcp-agent.log - see repeating errors:

2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver Traceback (most recent call last):
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 332, in poll
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver     self.conn.consume(timeout=current_timeout)
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in consume
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver     error_callback=_error_callback)
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 810, in ensure
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver     except kombu.exceptions.OperationalError as exc:
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver AttributeError: 'module' object has no attribute 'OperationalError'
2017-05-15 19:20:08.336 2382 ERROR oslo_messaging._drivers.amqpdriver 
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver [-] Failed to process incoming message, retrying...
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver Traceback (most recent call last):
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 332, in poll
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver     self.conn.consume(timeout=current_timeout)
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in consume
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver     error_callback=_error_callback)
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 810, in ensure
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver     except kombu.exceptions.OperationalError as exc:
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver AttributeError: 'module' object has no attribute 'OperationalError'
2017-05-15 19:20:08.338 2382 ERROR oslo_messaging._drivers.amqpdriver 
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver [-] Failed to process incoming message, retrying...
(most recent call last):
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 332, in poll
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver     self.conn.consume(timeout=current_timeout)
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1088, in consume
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver     error_callback=_error_callback)
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 810, in ensure
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver     except kombu.exceptions.OperationalError as exc:
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver AttributeError: 'module' object has no attribute 'OperationalError'
2017-05-15 19:20:08.339 2382 ERROR oslo_messaging._drivers.amqpdriver 
2017-05-15 19:20:08.341 2382 ERROR oslo_messaging._drivers.amqpdriver [-] Failed to process incoming message, retrying...
2017-05-15 19:20:08.341 2382 ERROR oslo_messaging._drivers.amqpdriver Traceback (most recent call last):


Same errors in openvswitch-agent.log

Comment 1 Artem Hrechanychenko 2017-05-16 15:29:52 UTC

I also got that issue.

Comment 2 John Eckersberg 2017-05-16 15:56:16 UTC

Looks like kombu is too old:

python-kombu-3.0.32-2.el7ost.noarch

yet from pike requirements in oslo.messaging:

kombu!=4.0.2,>=4.0.0 # BSD

Fixing component.

Comment 4 Alexander Chuzhoy 2017-06-16 14:46:20 UTC

Environment:
openstack-tripleo-heat-templates-7.0.0-0.20170512193554.el7ost.noarch
instack-undercloud-7.0.0-0.20170503001109.el7ost.noarch
openstack-puppet-modules-10.0.0-0.20170315222135.0333c73.el7.1.noarch

The issue doesn't reproduce.

Comment 6 Alexander Chuzhoy 2017-06-19 21:33:48 UTC

Verified:
Environment: python-kombu-4.0.2-5.el7ost.noarch

The reported issue doesn't reproduce. 


[stack@undercloud-0 ~]$ uptime
 17:33:27 up  4:20,  3 users,  load average: 0.11, 0.21, 0.23
[stack@undercloud-0 ~]$ sudo du -hs /var/log/neutron/
23M     /var/log/neutron/

Comment 10 errata-xmlrpc 2017-12-13 21:27:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462