Bug 2132774

Summary:	On its default configuration mosquitto cannot accept more than 1014 clients
Product:	Red Hat Satellite	Reporter:	Pablo Mendez Hernandez <pmendezh>
Component:	Remote Execution	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	NEW ---	QA Contact:	Satellite QE Team <sat-qe-bz-list>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.12.0	CC:	aruzicka, ekohlvan
Target Milestone:	Unspecified	Keywords:	Performance, Triaged, WorkAround
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Pablo Mendez Hernandez 2022-10-06 15:53:36 UTC

Description of problem:

mosquitto cannot accept more than 1014 clients on its default configuration.

When trying to register more, it'll complain:

# journalctl -f -u mosquitto
. . .
Oct 06 15:17:12 F.Q.D.N mosquitto[47597]: 1665069432: Unable to accept new connection, system socket count has been exceeded. Try increasing "ulimit -n" or equivalent.


Version-Release number of selected component (if applicable):

Satellite 6.12.0 Snap 13


How reproducible:

Always


Steps to Reproduce:
1. Configure Satellite or capsule to use MQTT broker
2. Start registering systems against it
3. Wait until reaching 1015


Actual results:

Unable to accept new connection, system socket count has been exceeded. Try increasing "ulimit -n" or equivalent.


Expected results:

New client connected from $CLIENT_IP:$HIGH_PORT as $UUID (p2, c1, k30, u'$UUID').


Additional info:

I'm solving it with this:

cat /etc/systemd/system/mosquitto.service.d/limits.conf
[Service]
LimitNOFILE=10000

I've chosen 10000 because it's the "maximum number of clients" for a default Satellite installation according to the tuning guide.

Comment 3 Ewoud Kohl van Wijngaarden 2022-11-02 19:10:05 UTC

In https://github.com/theforeman/foreman-documentation/pull/1747 I was pointed to this issue. I've opened https://github.com/eclipse/mosquitto/issues/2674 for a long term solution.