Bug 1016548 - Systemd limits manual starts
Systemd limits manual starts
Status: CLOSED DUPLICATE of bug 821723
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: systemd-maint
qe-baseos-daemons
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-08 06:39 EDT by Nikolai Kondrashov
Modified: 2013-10-16 06:29 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-08 09:08:43 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Nikolai Kondrashov 2013-10-08 06:39:10 EDT
Description of problem:
Systemd applies StartLimitInterval and StartLimitBurst to services started "manually", e.g. from command line.

This interferes with scripting and automatic tests in particular, where many different service configurations need to be tried in a short time.

Version-Release number of selected component (if applicable):
systemd-sysv-207-2.el7.x86_64
systemd-libs-207-2.el7.x86_64
systemd-207-2.el7.x86_64

How reproducible:
Always

Steps to Reproduce:
Run the following commands as root:

    for i in `seq 1 6`; do systemctl restart named; done
    journalctl -xn | grep refusing

Actual results:

Output:
    Job for named.service failed. See 'systemctl status named.service' and 'journalctl -xn' for details.
    Oct 08 06:35:31 sgi-xe250-02.rhts.eng.bos.redhat.com systemd[1]: named.service start request repeated too quickly, refusing to start.

Service is not started.

Expected results:
No output. Service started.

Additional info:
This reproduces with the default systemd start limits. Seen with sssd service as well.
Comment 2 Michal Schmidt 2013-10-08 09:08:43 EDT
This has come up before. In bug 821723, for instance.
An old upstream mailing list posting:
http://lists.freedesktop.org/archives/systemd-devel/2012-September/006530.html

In short, "systemctl reset-failed named" can be used in the loop to reset the burst limit counter (documented in 'man systemd.service', section about StartLimitInterval=, StartLimitBurst=).

*** This bug has been marked as a duplicate of bug 821723 ***
Comment 3 Nikolai Kondrashov 2013-10-08 10:05:43 EDT
This is very unintuitive and I consider the above a workaround. I would like to have this fixed still. This is breaking "service" command and thus RHEL6 compatibility and makes our tests unnecessarily complicated.

I would say that this is logically possible to implement. I.e. limit restarts initiated by automatic restart on death, socket-triggered start, etc., but don't limit otherwise perfectly valid administrative restarts.
Comment 4 Lennart Poettering 2013-10-13 15:49:56 EDT
(In reply to Nikolai Kondrashov from comment #3)
> This is very unintuitive and I consider the above a workaround. I would like
> to have this fixed still. This is breaking "service" command and thus RHEL6
> compatibility and makes our tests unnecessarily complicated.
> 
> I would say that this is logically possible to implement. I.e. limit
> restarts initiated by automatic restart on death, socket-triggered start,
> etc., but don't limit otherwise perfectly valid administrative restarts.

We cannot distuingish that really. Also, your tests are *not* administrative restarts, they are scripts, which by your own admission should be rate limited.

We simply want to avoid busy loops, and to protect against that it doesn't matter what exactly is causing a restart.
Comment 5 Nikolai Kondrashov 2013-10-14 06:37:17 EDT
I'm sorry, you haven't provided any arguments, and I still consider it logically possible to distinguish restarts initiated with the "service" ("systemctl") command compared to other methods. However, I can understand if the current design of systemd doesn't allow implementing that easily.

Why administrative restarts cannot be done by a script? Would it be allowed to do them if it was reimplemented in C?

Sorry, I don't see where I admitted that my scripts should be limited on how often they restart a service. On the contrary, I need them to restart the service as fast as possible.

And no, I don't want to make my scripts change a service description file or execute a special command so they're allowed to restart a service as often as needed. Not in the least because these are incompatible with RHEL6 (and other distributions not using systemd), which I (may) still have to test.

I still insist that administrative restarts, i.e. run by an administrator, using administrative interface (i.e. "service" or "systemctl" commands) don't need to be protected against. Otherwise systemd is trying to protect administrators from themselves, thus limiting its own usability, akin to a hammer padded with feather cushions for "safety".
Comment 6 Lennart Poettering 2013-10-15 17:28:02 EDT
(In reply to Nikolai Kondrashov from comment #5)
> I'm sorry, you haven't provided any arguments, and I still consider it
> logically possible to distinguish restarts initiated with the "service"
> ("systemctl") command compared to other methods. However, I can understand
> if the current design of systemd doesn't allow implementing that easily.
> 
> Why administrative restarts cannot be done by a script? Would it be allowed
> to do them if it was reimplemented in C?

No, that's the point, scripts using systemctl, users running systemctl from the command line and C programs all use the same interface: the bus APIs, and to systemd they are all the same.

Also, how would you distuingish systemctl run by a script and run by the admin anyway?

To avoid busy loops we rate limit everything. And that's the right thing to do.

Note that you can reset the rate counter with "systemctl reset-failed" on the specific unit. If you really really think that busy loops are awesome you can invoke that before each "systemctl start", and not rate limits will every apply.

> Sorry, I don't see where I admitted that my scripts should be limited on how
> often they restart a service. On the contrary, I need them to restart the
> service as fast as possible.

That's a very bad idea. Note that even sysvinit employed rate limits for its services. 

> And no, I don't want to make my scripts change a service description file or
> execute a special command so they're allowed to restart a service as often
> as needed. Not in the least because these are incompatible with RHEL6 (and
> other distributions not using systemd), which I (may) still have to test.
> 
> I still insist that administrative restarts, i.e. run by an administrator,
> using administrative interface (i.e. "service" or "systemctl" commands)
> don't need to be protected against. Otherwise systemd is trying to protect
> administrators from themselves, thus limiting its own usability, akin to a
> hammer padded with feather cushions for "safety".

Well, your script is not the admin, and we cannot distuingish that and it's a bad idea anyway and there's a work-around using systemctl reset-failed. Sorry, but this is really not going to change. It's a safety and robustness feature, and it's going to stay.
Comment 7 Nikolai Kondrashov 2013-10-16 06:29:59 EDT
(In reply to Lennart Poettering from comment #6)
> (In reply to Nikolai Kondrashov from comment #5)
> > Why administrative restarts cannot be done by a script? Would it be allowed
> > to do them if it was reimplemented in C?
> 
> No, that's the point, scripts using systemctl, users running systemctl from
> the command line and C programs all use the same interface: the bus APIs,
> and to systemd they are all the same.
> 
> Also, how would you distuingish systemctl run by a script and run by the
> admin anyway?

Nohow, and that's not what I'm asking to do.
I'm asking for systemd not to limit how it is used.

The mechanisms for limiting re-spawning of a dying service or starting based
on socket connections have their merit, since it is done by systemd itself and
the triggers are potentially outside of administrator's control.

However, systemd limiting *outside* API (such as systemctl interface, or maybe
some D-Bus methods) is just assuming too much responsibility. There are valid
use cases where systemd might be driven by other software, it shouldn't be
limited to a slow human interface only. And it should be the driving
software's decision on how often a service can be restarted.

If the internal restart mechanisms use the same API (D-Bus methods?) as the
outside control is supposed to use, then make a separate API for the former
and make *that* limited, but please don't limit outside control.

> Note that you can reset the rate counter with "systemctl reset-failed" on
> the specific unit. If you really really think that busy loops are awesome
> you can invoke that before each "systemctl start", and not rate limits will
> every apply.

No, I don't think busy loops are "awesome" or "awful", they're just loops, and
"busy" is relative. When I need to try 100 different configurations of a
service in a test I would naturally want it to try them faster than 5 per 10
seconds, i.e. completing in *less* than 3.3 minutes, preferably as quickly as
possible. And, no, I don't want to call "systemctl reset-failed", not in the
least because it doesn't exist in "service" command interface, and on RHEL6
and other distros.

> > Sorry, I don't see where I admitted that my scripts should be limited on how
> > often they restart a service. On the contrary, I need them to restart the
> > service as fast as possible.
> 
> That's a very bad idea. Note that even sysvinit employed rate limits for its
> services. 

You probably mean services defined in /etc/inittab and (re-)spawned
automatically by "init". Limiting those is right. However, there was never any
limit on how often a service could be restarted using scripts from /etc/init.d
or the "service" command.

Note You need to log in before you can comment on or make changes to this bug.