Bug 1713197 - Failed to establish a new connection: [Errno 24] Too many open files on nova-compute
Summary: Failed to establish a new connection: [Errno 24] Too many open files on nova-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z13
: 10.0 (Newton)
Assignee: Martin Schuppert
QA Contact: Paras Babbar
URL:
Whiteboard:
: 1716160 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-23 06:34 UTC by Ravi Singh
Modified: 2019-10-22 03:53 UTC (History)
16 users (show)

Fixed In Version: openstack-tripleo-heat-templates-5.3.10-27.el7ost puppet-tripleo-5.6.8-28.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 09:40:35 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1651526 0 None None None 2019-05-26 13:42:07 UTC
Red Hat Knowledge Base (Solution) 4172751 0 Troubleshoot None "conf_read_file: error code 24" tracebacks occur when destroying too many VMs on the same compute node at the same time 2019-05-26 16:57:58 UTC
Red Hat Product Errata RHBA-2019:3112 0 None None None 2019-10-16 09:40:52 UTC

Comment 4 Martin Schuppert 2019-06-07 08:58:52 UTC 
Adding public description to this BZ

Description of problem:
"conf_read_file: error code 24" tracebacks occur when destroying too many VMs on the same compute node at the same time

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create ~ 100VMs on the same compute
2. Destroy them
3.

Actual results:
Fails with the aforementioned error message

Expected results:
Shouldn't fail

Additional info:
We do have manual workarounds but this isn't a good solution as this should be handled through the deployment tool.   This looks similar to [1] but also affects nova-compute as it appears it needs to open lots of connections to ceph ...  or is this a FD leak ?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1372589



2019-05-07 12:09:02.653 4113 INFO nova.virt.libvirt.driver [req-096c9254-744d-44ca-9c04-e71eca962d06 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance failed to shutdown in 60 seconds.
2019-05-07 12:09:03.490 4113 INFO nova.virt.libvirt.driver [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance destroyed successfully.
2019-05-07 12:09:17.289 4113 INFO nova.compute.manager [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] VM Stopped (Lifecycle Event)
2019-05-07 12:09:18.755 4113 INFO nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Terminating instance
2019-05-07 12:09:18.761 4113 INFO nova.virt.libvirt.driver [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance destroyed successfully.
2019-05-07 12:09:20.270 4113 INFO nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Took 1.31 seconds to deallocate network for instance.
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Setting instance vm_state to ERROR
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Traceback (most recent call last):
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2385, in do_terminate_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._delete_instance(context, instance, bdms, quotas)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2348, in _delete_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     quotas.rollback()
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2312, in _delete_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._shutdown_instance(context, instance, bdms)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2217, in _shutdown_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     requested_networks)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2204, in _shutdown_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     block_device_info)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 863, in destroy
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     destroy_disks, migrate_data)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in cleanup
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._cleanup_rbd(instance)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1037, in _cleanup_rbd
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     LibvirtDriver._get_rbd_driver().cleanup_volumes(filter_fn)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 361, in cleanup_volumes
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 105, in __init__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 134, in _connect_to_rados
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 525, in rados.Rados.__init__ (rados.c:5719)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 568, in rados.Rados.__setup (rados.c:6448)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 631, in rados.Rados.conf_read_file (rados.c:7384)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Error: error calling conf_read_file: error code 24
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]
Comment 5 Martin Schuppert 2019-06-07 08:59:39 UTC 
*** Bug 1716160 has been marked as a duplicate of this bug. ***
Comment 13 Lon Hohberger 2019-08-09 10:42:23 UTC 
According to our records, this should be resolved by puppet-tripleo-5.6.8-27.el7ost.  This build is available now.
Comment 25 errata-xmlrpc 2019-10-16 09:40:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3112
Note You need to log in before you can comment on or make changes to this bug.