Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2028233

Summary:	neutron-server can't start new thread
Product:	Red Hat OpenStack	Reporter:	Tommy Doucet <tdoucet>
Component:	openstack-tripleo-heat-templates	Assignee:	Brendan Shephard <bshephar>
Status:	CLOSED ERRATA	QA Contact:	Joe H. Rahme <jhakimra>
Severity:	high	Docs Contact:
Priority:	high
Version:	16.1 (Train)	CC:	bshephar, drosenfe, mburns, ramishra
Target Milestone:	zstream	Keywords:	Triaged
Target Release:	16.1 (Train on RHEL 8.2)
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-11.3.2-1.20211202153350.29a02c1.el8ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-24 11:02:18 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Tommy Doucet 2021-12-01 18:47:57 UTC

Description of problem:
Customer from [Case #03084974](https://gss--c.visualforce.com/apex/Case_View?id=5002K00000zdPXE&sfdc.override=1) hitting PIDsLimit of 4096 with neutron-server container. Similar to [BZ #1871885](https://bugzilla.redhat.com/show_bug.cgi?id=1871885), would it be possible to add a parameter to change this value here too? 

~~~
2021-11-11 21:05:10.076 169 DEBUG neutron.api.v2.base [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 81
0437b2d44f4936aac2ba22cea2f191 - default default] Request body: {'network': {'name': 'provider-data-7303', 'shared': False, 'provid
er:physical_network': 'phys_vxlan-cx', 'admin_state_up': True, 'provider:network_type': 'vxlan-evpn', 'provider:segmentation_id': '7303'}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719
2021-11-11 21:05:10.080 169 DEBUG neutron.db.quota.driver [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Resources port,network,subnet,security_group_rule,security_group,subnetpool have unlimited quota limit. It is not required to calculate headroom  make_reservation /usr/lib/python3.6/site-packages/neutron/db/quota/driver.py:221
2021-11-11 21:05:10.096 169 DEBUG neutron.pecan_wsgi.hooks.quota_enforcement [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Made reservation on behalf of 810437b2d44f4936aac2ba22cea2f191 for: {'network': 1} before /usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/quota_enforcement.py:55
2021-11-11 21:05:10.100 169 DEBUG neutron_lib.callbacks.manager [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Notify callbacks ['neutron.plugins.ml2.plugin.SecurityGroupDbMixin._ensure_default_security_group_handler-3201866'] for network, before_create _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2021-11-11 21:05:10.119 169 DEBUG neutron.plugins.ml2.drivers.helpers [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] vxlan-evpn segment {'physical_network': 'phys_vxlan-cx', 'vxlan_vni': 7303} create started allocate_fully_specified_segment /usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/helpers.py:108
2021-11-11 21:05:10.121 169 DEBUG neutron.plugins.ml2.drivers.helpers [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] vxlan-evpn segment {'physical_network': 'phys_vxlan-cx', 'vxlan_vni': 7303} create done allocate_fully_specified_segment /usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/helpers.py:112
2021-11-11 21:05:10.127 169 DEBUG neutron_lib.callbacks.manager [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Notify callbacks ['neutron.plugins.ml2.plugin.Ml2Plugin._handle_segment_change--9223372036851818362'] for segment, precommit_create _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2021-11-11 21:05:10.127 169 INFO neutron.db.segments_db [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Added segment aadd2a6f-a19e-4362-b2f1-4056e7a1022e of type vxlan-evpn for network b3a621ba-118e-4997-9740-60fb58215cde
2021-11-11 21:05:10.146 169 DEBUG neutron_lib.callbacks.manager [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Notify callbacks ['neutron.services.auto_allocate.db._ensure_external_network_default_value_callback-8755840522799'] for network, precommit_create _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2021-11-11 21:05:10.182 169 DEBUG neutron_lib.callbacks.manager [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Notify callbacks ['neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler.handle_event-1639370', 'neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api.DhcpAgentNotifyAPI._native_event_send_dhcp_notification-1893231'] for network, after_create _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Error during notification for neutron.plugins.ml2.ovo_rpc._ObjectChangeHandler.handle_event-1639370 network, after_create: RuntimeError: can't start new thread
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/ovo_rpc.py", line 95, in handle_event
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager     self.fts.append(self._worker_pool.submit(self.dispatch_events))
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.6/site-packages/futurist/_futures.py", line 189, in submit
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager     return self._gatherer.submit(fn, *args, **kwargs)
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.6/site-packages/futurist/_futures.py", line 95, in submit
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager     fut = self._submit_func(fn, *args, **kwargs)
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.6/site-packages/futurist/_futures.py", line 178, in _submit
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager     self._maybe_spin_up()
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3.6/site-packages/futurist/_futures.py", line 164, in _maybe_spin_up
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager     w.start()
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager   File "/usr/lib64/python3.6/threading.py", line 846, in start
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager     _start_new_thread(self._bootstrap, ())
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager RuntimeError: can't start new thread
2021-11-11 21:05:10.182 169 ERROR neutron_lib.callbacks.manager 
2021-11-11 21:05:10.183 169 DEBUG neutron.pecan_wsgi.hooks.policy_enforcement [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Attributes excluded by policy engine: ['vlan_transparent'] _exclude_attributes_by_policy /usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:256
2021-11-11 21:05:10.184 169 DEBUG neutron_lib.callbacks.manager [req-aa13de26-240e-4151-bbd8-8cc0f9e0beae 52408254be404b2e99519e2f5dc65f04 810437b2d44f4936aac2ba22cea2f191 - default default] Notify callbacks ['neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api.DhcpAgentNotifyAPI._send_dhcp_notification-1893246'] for network, before_response _notify_loop /usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py:193
~~~

Version-Release number of selected component (if applicable):
Red Hat OpenStack 16.1

How reproducible:
Limited, should not happen for most cases.

Actual results:
Unable to start new thread as PIDsLimit of 4096 is exceeded.

Expected results:
Being able to up PIDsLimit for neutron-server via tripleo-heat-templates.

Comment 1 Brendan Shephard 2021-12-01 21:24:57 UTC

Hi Tommy,

There is indeed an option for this in tripleo-heat-templates:
https://github.com/openstack/tripleo-heat-templates/commit/781beb757121c3b3e535af6d2240163e58a33ddd

And it has been backported back to Train as well:
https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/podman/podman-baremetal-ansible.yaml#L56-L60

Comment 2 Tommy Doucet 2021-12-01 21:44:33 UTC

Sorry about that and thank you for the info! I missed it. I gave the customer the solution.

Comment 3 Brendan Shephard 2021-12-01 21:47:23 UTC

No worries. It might not be in the customers version of tripleo-heat-templates, they might need to update.

I'll double check the packages for 16.1

Comment 4 Brendan Shephard 2021-12-01 22:03:09 UTC

Ah my bad, it's in 16.2 but not 16.1. I have added it in now, so it will be in an upcoming z-stream.

The patches they would need to address this prior to the z-stream are:
https://review.opendev.org/c/openstack/tripleo-ansible/+/775803
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/775767

Comment 10 David Rosenfeld 2022-02-04 18:06:40 UTC

Edited /usr/share/openstack-tripleo-heat-templates/deployment/podman/podman-baremetal-ansible.yaml to change the pids limit from its default value to 6000 and deployed overcloud.

After deployment this was seen on controller node:

more /etc/containers/containers.conf

[containers]
pids_limit = 6000
[engine]
events_logger = "journald"
runtime = "runc"

[heat-admin@controller-0 ~]$ sudo podman inspect neutron_api |grep PidsLimit
            "PidsLimit": 6000,

The default container pids limit may be modified.

Comment 17 errata-xmlrpc 2022-03-24 11:02:18 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.8 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0986