Bug 1716160

Summary:	"conf_read_file: error code 24" tracebacks occur when destroying too many VMs on the same compute node at the same time
Product:	Red Hat OpenStack	Reporter:	David Hill <dhill>
Component:	openstack-tripleo-heat-templates	Assignee:	Martin Schuppert <mschuppe>
Status:	CLOSED DUPLICATE	QA Contact:	Sasha Smolyak <ssmolyak>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	10.0 (Newton)	CC:	mbooth, mburns
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-07 08:59:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description David Hill 2019-06-02 13:13:59 UTC

Description of problem:
"conf_read_file: error code 24" tracebacks occur when destroying too many VMs on the same compute node at the same time

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Create ~ 100VMs on the same compute
2. Destroy them
3.

Actual results:
Fails with the aforementioned error message

Expected results:
Shouldn't fail

Additional info:
We do have manual workarounds but this isn't a good solution as this should be handled through the deployment tool.   This looks similar to [1] but also affects nova-compute as it appears it needs to open lots of connections to ceph ...  or is this a FD leak ?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1372589

Comment 1 David Hill 2019-06-02 13:16:21 UTC

This is the same as [1] but this bug is public instead... we see error messages similar to this:

019-05-07 12:09:02.653 4113 INFO nova.virt.libvirt.driver [req-096c9254-744d-44ca-9c04-e71eca962d06 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance failed to shutdown in 60 seconds.
2019-05-07 12:09:03.490 4113 INFO nova.virt.libvirt.driver [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance destroyed successfully.
2019-05-07 12:09:17.289 4113 INFO nova.compute.manager [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] VM Stopped (Lifecycle Event)
2019-05-07 12:09:18.755 4113 INFO nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Terminating instance
2019-05-07 12:09:18.761 4113 INFO nova.virt.libvirt.driver [-] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Instance destroyed successfully.
2019-05-07 12:09:20.270 4113 INFO nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Took 1.31 seconds to deallocate network for instance.
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [req-15ab85e1-b052-425a-aa8f-1ec611b9d19c 8c8b5d3da7a84db3958d47d8715172e9 1faa0efe1ea74d86a2954cfa5a080975 - - -] [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Setting instance vm_state to ERROR
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Traceback (most recent call last):
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2385, in do_terminate_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._delete_instance(context, instance, bdms, quotas)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2348, in _delete_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     quotas.rollback()
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2312, in _delete_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._shutdown_instance(context, instance, bdms)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2217, in _shutdown_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     requested_networks)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2204, in _shutdown_instance
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     block_device_info)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 863, in destroy
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     destroy_disks, migrate_data)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in cleanup
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     self._cleanup_rbd(instance)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1037, in _cleanup_rbd
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]     LibvirtDriver._get_rbd_driver().cleanup_volumes(filter_fn)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 361, in cleanup_volumes
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 105, in __init__
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 134, in _connect_to_rados
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 525, in rados.Rados.__init__ (rados.c:5719)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 568, in rados.Rados.__setup (rados.c:6448)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073]   File "rados.pyx", line 631, in rados.Rados.conf_read_file (rados.c:7384)
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] Error: error calling conf_read_file: error code 24
2019-05-07 12:09:20.290 4113 ERROR nova.compute.manager [instance: 8bdab8cd-427c-4f28-849c-0ea2c15b3073] 



[1] https://bugzilla.redhat.com/show_bug.cgi?id=1713197

Comment 2 Martin Schuppert 2019-06-07 08:59:39 UTC

I have added a public update in 1713197, will close this as a duplicate

*** This bug has been marked as a duplicate of bug 1713197 ***