Bug 1713197

Summary: Failed to establish a new connection: [Errno 24] Too many open files on nova-compute
Product: Red Hat OpenStack Reporter: Ravi Singh <ravsingh>
Component: openstack-tripleo-heat-templatesAssignee: Martin Schuppert <mschuppe>
Status: CLOSED ERRATA QA Contact: Paras Babbar <pbabbar>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: amodi, bbowen, dasmith, dhill, eglynn, jhakimra, jschluet, kchamart, lyarwood, mbooth, mburns, mschuppe, pbabbar, sbauza, sgordon, vromanso
Target Milestone: z13Keywords: Patch, Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-5.3.10-27.el7ost puppet-tripleo-5.6.8-28.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 09:40:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 4 Martin Schuppert 2019-06-07 08:58:52 UTC 
Adding public description to this BZ

Description of problem:
"conf_read_file: error code 24" tracebacks occur when destroying too many VMs on the same compute node at the same time

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create ~ 100VMs on the same compute
2. Destroy them
3.

Actual results:
Fails with the aforementioned error message

Expected results:
Shouldn't fail

Additional info:
We do have manual workarounds but this isn't a good solution as this should be handled through the deployment tool.   This looks similar to [1] but also affects nova-compute as it appears it needs to open lots of connections to ceph ...  or is this a FD leak ?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1372589



2019-05-07 12:09:02.653 4113 INFO nova.virt.libvirt.driver [req-096c9254-744d-44ca-9c04-e71eca962d06 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance failed to shutdown in 60 seconds.
2019-05-07 12:09:03.490 4113 INFO nova.virt.libvirt.driver [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance destroyed successfully.
2019-05-07 12:09:17.289 4113 INFO nova.compute.manager [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] VM Stopped (Lifecycle Event)
2019-05-07 12:09:18.755 4113 INFO nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Terminating instance
2019-05-07 12:09:18.761 4113 INFO nova.virt.libvirt.driver [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance destroyed successfully.
2019-05-07 12:09:20.270 4113 INFO nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Took 1.31 seconds to deallocate network for instance.
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Setting instance vm_state to ERROR
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Traceback (most recent call last):
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2385, in do_terminate_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._delete_instance(context, instance, bdms, quotas)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2348, in _delete_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     quotas.rollback()
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2312, in _delete_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._shutdown_instance(context, instance, bdms)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2217, in _shutdown_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     requested_networks)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2204, in _shutdown_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     block_device_info)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 863, in destroy
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     destroy_disks, migrate_data)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in cleanup
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._cleanup_rbd(instance)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1037, in _cleanup_rbd
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     LibvirtDriver._get_rbd_driver().cleanup_volumes(filter_fn)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 361, in cleanup_volumes
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 105, in __init__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 134, in _connect_to_rados
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 525, in rados.Rados.__init__ (rados.c:5719)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 568, in rados.Rados.__setup (rados.c:6448)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 631, in rados.Rados.conf_read_file (rados.c:7384)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Error: error calling conf_read_file: error code 24
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]
Comment 5 Martin Schuppert 2019-06-07 08:59:39 UTC 
*** Bug 1716160 has been marked as a duplicate of this bug. ***
Comment 13 Lon Hohberger 2019-08-09 10:42:23 UTC 
According to our records, this should be resolved by puppet-tripleo-5.6.8-27.el7ost.  This build is available now.
Comment 25 errata-xmlrpc 2019-10-16 09:40:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3112