Bug 2148170

Description of problem:

When a transient service is failing, it continues consuming a unit until "reset-failed" is issued.
This is problematic when reaching the maximum number of units (hardcoded to 128K in the sources):

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
 21 #define MANAGER_MAX_NAMES 131072 /* 128K */
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Indeed, once limit is reached, many problems appear, including:

1. socket units die when getting triggered by incoming traffic

    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
    systemd[1]: fakeauth.socket: Failed to listen on sockets: Argument list too long
    systemd[1]: fakeauth.socket: Failed with result 'resources'.
    systemd[1]: Failed to listen on fakeauth.socket.
    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

    This prevents service handling completely.

2. mount are not registered in systemd (but they are still working)

    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
    systemd[1]: Failed to set up mount unit: Argument list too long
    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

3. logins are not moved to expected user's cgroup

    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
    systemd-logind[905]: Failed to start session scope session-9.scope: Argument list too long
    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

4. admin cannot reboot the system using "reboot/shutdown" command

    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
    systemd-initctl[368371]: Failed to change runlevel: Argument list too long
    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------


Such limit can be "easily" reached when having a socket-triggered service which regularly dies.
This happens in real life with "authd.socket" whose service fails if the remote end vanished before "authd@.service" could read the addr/port of remote end.
Indeed, socket-triggered services of Stream type consume 2 units:
- one as "<service@instance>"
- one as "<service@instance-localaddr:localport-remoteaddr:remoteport>"

Hence, it's sufficient to have 64K failures in the past (which can be other a full year for example) to "take down" the system.

Additionally, failed services impact systemd's performance a lot: it appears that finding a slot in the "manager->units" hashmap takes more and more time, due to having these failed services consume buckets.
This can be easily seen when spawning transient failing services in loop: initially it's fast, then slows down and we see systemd taking up to 80% of a CPU (or more).

Version-Release number of selected component (if applicable):

systemd-239-68.el8.x86_64

How reproducible:

Always

Steps to Reproduce:

1. Create a dummy socket service listening on TCP stream that always fails

    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
    # cat /etc/systemd/system/fakeauth.socket 
    [Socket]
    ListenStream=113
    Accept=true
    
    # cat /etc/systemd/system/fakeauth@.service 
    [Service]
    ExecStart=/bin/false
    StandardInput=socket
    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

2. Start the service socket

    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
    # systemctl start fakeauth.socket
    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

3. Trigger the service in loop until 64K instances are failing

    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
    # i=0; while [ $i -le 65535 ]; do ncat --send-only localhost 113 </dev/null; let i++; sleep 0.1; done
    -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

    Note: due to slowness with time, the socket may drop incoming requests, hence more than 65535 recursion is actually required.

Actual results:

After having ~65535 units on the system, "Argument list too long" is seen and "fakeauth.socket" unit dies.

Nov 24 13:18:05 vm-fakeauth8 systemd[1]: fakeauth.socket: Failed to listen on sockets: Argument list too long
Nov 24 13:18:05 vm-fakeauth8 systemd[1]: fakeauth.socket: Failed with result 'resources'.
Nov 24 13:18:05 vm-fakeauth8 systemd[1]: Failed to listen on fakeauth.socket.

Expected results:

Socket doesn't die, reboot can be issue, ssh logins are not in "sshd.service" cgroup, etc.

Additional info:

"Argument list too long" is not a good errno at all. The errno should be handled by caller to explain what's going on more clearly.