The systemd and sm-client services maybe could be improved by adding a "StartLimitInterval=0" instruction. The reason of this modification is that the network-manager dispatcher script may restart the service at the same rate the user requests a network change, and this may easily hit the default start-hit-rate limit of systemd (5 bursts starts per 10 secs). When it happens, the service is silently _not_ restarted: Feb 15 18:00:43 bonobo.bellet.info systemd[1]: sendmail.service: Failed with result 'start-limit-hit'. As a side comment, I'm also wondering if the nm-dispatcher script for sendmail could not only restart sendmail when the hostname is modified ? This is what is important for sendmail I think (knowing its own hostname, to decide when delivery should be local or relayed).
(In reply to Fabrice Bellet from comment #0) > The systemd and sm-client services maybe could be improved by adding a > "StartLimitInterval=0" instruction. The reason of this modification is that > the network-manager dispatcher script may restart the service at the same > rate the user requests a network change, and this may easily hit the default > start-hit-rate limit of systemd (5 bursts starts per 10 secs). When it > happens, the service is silently _not_ restarted: > > Feb 15 18:00:43 bonobo.bellet.info systemd[1]: sendmail.service: Failed with > result 'start-limit-hit'. > Thanks for info. But didn't you mean "StartLimitIntervalSec"? > As a side comment, I'm also wondering if the nm-dispatcher script for > sendmail could not only restart sendmail when the hostname is modified ? > This is what is important for sendmail I think (knowing its own hostname, to > decide when delivery should be local or relayed). Last time when I looked on this problem, I wasn't able to find anything related in NM. Maybe worth NM RFE?
(In reply to Jaroslav Škarvada from comment #1) > Thanks for info. But didn't you mean "StartLimitIntervalSec"? Ah. that's interesting, because I didn't notice the option name mismatch. I copied this option from other service files (lvm2-pvscan@.service and kdump.service on my fedora 25 box). StartLimitIntervalSec=0 doesn't do the job. StartLimitBurst=0 seems the required option instead, if we want to stay with "documented" options :) > > > As a side comment, I'm also wondering if the nm-dispatcher script for > > sendmail could not only restart sendmail when the hostname is modified ? > > This is what is important for sendmail I think (knowing its own hostname, to > > decide when delivery should be local or relayed). > > Last time when I looked on this problem, I wasn't able to find anything > related in NM. Maybe worth NM RFE? For what I know of current networkmanager capabilities, dispatcher scripts can react on hostname change initiated by networkmanager itself (arg $2 == hostname). This event can be triggered automatically when receiving the IP address from a DHCP server (from address lookup), or manually when the user changes the hostname with "nmcli general hostname newhostname". This second possibility _also_ modifies the static hostname in /etc/hostname. There may also be subtleties, depending on whether /etc/hostname is empty, or whether hostname==localhost.localdomain at NetworkManager startup, but I'm not 100% sure about that.
(In reply to Fabrice Bellet from comment #2) > (In reply to Jaroslav Škarvada from comment #1) > > Thanks for info. But didn't you mean "StartLimitIntervalSec"? > > Ah. that's interesting, because I didn't notice the option name mismatch. I > copied this option from other service files (lvm2-pvscan@.service and > kdump.service on my fedora 25 box). > > StartLimitIntervalSec=0 doesn't do the job. StartLimitBurst=0 seems the > required option instead, if we want to stay with "documented" options :) > Has the "StartLimitInterval=0" worked for you? If yes I am going to sort it out with the systemd guys, because it doesn't seem to be documented. > > > > > As a side comment, I'm also wondering if the nm-dispatcher script for > > > sendmail could not only restart sendmail when the hostname is modified ? > > > This is what is important for sendmail I think (knowing its own hostname, to > > > decide when delivery should be local or relayed). > > > > Last time when I looked on this problem, I wasn't able to find anything > > related in NM. Maybe worth NM RFE? > > For what I know of current networkmanager capabilities, dispatcher scripts > can react on hostname change initiated by networkmanager itself (arg $2 == > hostname). This event can be triggered automatically when receiving the IP > address from a DHCP server (from address lookup), or manually when the user > changes the hostname with "nmcli general hostname newhostname". This second > possibility _also_ modifies the static hostname in /etc/hostname. > > There may also be subtleties, depending on whether /etc/hostname is empty, > or whether hostname==localhost.localdomain at NetworkManager startup, but > I'm not 100% sure about that. Thanks for info, but I am currently not 100% sure whether it's enough, because IIRC sendmail is doing reverse DNS check which could reveal inconsistency if the IP changes.
(In reply to Jaroslav Škarvada from comment #3) > Has the "StartLimitInterval=0" worked for you? If yes I am going to sort it > out with the systemd guys, because it doesn't seem to be documented. Yes, "StartLimitInterval=0" has the desired behaviour for me (both in sendmail.service and sm-client.service)
(In reply to Fabrice Bellet from comment #4) > (In reply to Jaroslav Škarvada from comment #3) > > Has the "StartLimitInterval=0" worked for you? If yes I am going to sort it > > out with the systemd guys, because it doesn't seem to be documented. > > Yes, "StartLimitInterval=0" has the desired behaviour for me (both in > sendmail.service and sm-client.service) Interesting, it seems StartLimitInterval is just obsoleted alias for StartLimitIntervalSec (in f25, i.e. systemd-231-13.fc25). Adding Michal to sort it out.
(In reply to Jaroslav Škarvada from comment #5) > (In reply to Fabrice Bellet from comment #4) > > (In reply to Jaroslav Škarvada from comment #3) > > > Has the "StartLimitInterval=0" worked for you? If yes I am going to sort it > > > out with the systemd guys, because it doesn't seem to be documented. > > > > Yes, "StartLimitInterval=0" has the desired behaviour for me (both in > > sendmail.service and sm-client.service) > > Interesting, it seems StartLimitInterval is just obsoleted alias for > StartLimitIntervalSec (in f25, i.e. systemd-231-13.fc25). Adding Michal to > sort it out. StartLimitInterval is the proper name of the option while StartLimitIntervalSec is name of the DBus property. In configuration files please use the former. For complete list of options (with pointers to additional docs) see "man 7 systemd.directives"
(In reply to Michal Sekletar from comment #6) > (In reply to Jaroslav Škarvada from comment #5) > > (In reply to Fabrice Bellet from comment #4) > > > (In reply to Jaroslav Škarvada from comment #3) > > > > Has the "StartLimitInterval=0" worked for you? If yes I am going to sort it > > > > out with the systemd guys, because it doesn't seem to be documented. > > > > > > Yes, "StartLimitInterval=0" has the desired behaviour for me (both in > > > sendmail.service and sm-client.service) > > > > Interesting, it seems StartLimitInterval is just obsoleted alias for > > StartLimitIntervalSec (in f25, i.e. systemd-231-13.fc25). Adding Michal to > > sort it out. > > StartLimitInterval is the proper name of the option while > StartLimitIntervalSec is name of the DBus property. In configuration files > please use the former. > > For complete list of options (with pointers to additional docs) see "man 7 > systemd.directives" Sorry I cannot find it in the "man 7 systemd.directives", is it systemd bug / non documented option?: $ man 7 systemd.directives | grep StartLimitInterval StartLimitIntervalSec= DefaultStartLimitIntervalSec= $ man systemd.unit | grep StartLimitInterval StartLimitIntervalSec=, StartLimitBurst= modified. Use StartLimitIntervalSec= to configure the checking interval (defaults to DefaultStartLimitIntervalSec= in Configure the action to take if the rate limit configured with StartLimitIntervalSec= and StartLimitBurst= is hit. Takes
(In reply to Jaroslav Škarvada from comment #7) > (In reply to Michal Sekletar from comment #6) > > (In reply to Jaroslav Škarvada from comment #5) > > > (In reply to Fabrice Bellet from comment #4) > > > > (In reply to Jaroslav Škarvada from comment #3) > > > > > Has the "StartLimitInterval=0" worked for you? If yes I am going to sort it > > > > > out with the systemd guys, because it doesn't seem to be documented. > > > > > > > > Yes, "StartLimitInterval=0" has the desired behaviour for me (both in > > > > sendmail.service and sm-client.service) > > > > > > Interesting, it seems StartLimitInterval is just obsoleted alias for > > > StartLimitIntervalSec (in f25, i.e. systemd-231-13.fc25). Adding Michal to > > > sort it out. > > > > StartLimitInterval is the proper name of the option while > > StartLimitIntervalSec is name of the DBus property. In configuration files > > please use the former. > > > > For complete list of options (with pointers to additional docs) see "man 7 > > systemd.directives" > > Sorry I cannot find it in the "man 7 systemd.directives", is it systemd bug > / non documented option?: > > $ man 7 systemd.directives | grep StartLimitInterval > StartLimitIntervalSec= > DefaultStartLimitIntervalSec= > > $ man systemd.unit | grep StartLimitInterval > StartLimitIntervalSec=, StartLimitBurst= > modified. Use StartLimitIntervalSec= to configure the checking > interval (defaults to DefaultStartLimitIntervalSec= in > Configure the action to take if the rate limit configured with > StartLimitIntervalSec= and StartLimitBurst= is hit. Takes You are right. Sorry for the confusion. My systemd version still have StartLimitInterval though. Here is upstream commit that renamed StartLimitInterval to StartLimitIntervalSec, https://github.com/systemd/systemd/commit/f0367da7d1a61ad698a55d17b5c28ddce0dc265a Note that old name still works. Hence I'd stick to old name if you don't want to have different unit file versions for different Fedora releases. But it is your call.
Thanks for info. I can see clearly in the code it's just rename, so I don't understand why StartLimitInterval=0 worked for reporter, but StartLimitIntervalSec=0 didn't. Michal what's the correct way to disable the rate limiting in the unit file (i.e. to allow unlimited number of service restarts even with multiple failures)? I am unable to find it in the documentation. According to the reporter in the comment 2, it seems that StartLimitInterval=0 works, StartLimitBurst=0 doesn't work, and StartLimitBurst=0 works, which seems quite strange for me (I am going to retest).
(In reply to Jaroslav Škarvada from comment #9) > Thanks for info. I can see clearly in the code it's just rename, so I don't > understand why StartLimitInterval=0 worked for reporter, but > StartLimitIntervalSec=0 didn't. > > Michal what's the correct way to disable the rate limiting in the unit file > (i.e. to allow unlimited number of service restarts even with multiple > failures)? I am unable to find it in the documentation. According to the > reporter in the comment 2, it seems that StartLimitInterval=0 works, > StartLimitBurst=0 doesn't work, and StartLimitBurst=0 works, which seems > quite strange for me (I am going to retest). Safest bet is to set both to zero. However, according to our docs setting just the interval should be sufficient, "Use StartLimitInterval= to configure the checking interval (defaults to DefaultStartLimitInterval= in manager configuration file, set to 0 to disable any kind of rate limiting)."
As I thought and proved by following simple experiment StartLimitIntervalSec works the same as StartLimitInterval. I have default systemd settings which means "by default, units which are started more than 5 times within 10 seconds are not permitted to start any more times until the 10 second interval ends". This means six quick restarts should do the trick. Reproducer: Default sendmail.service: # systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail Job for sendmail.service failed. See "systemctl status sendmail.service" and "journalctl -xe" for details. # echo $? 1 sendmail.service with StartLimitIntervalSec=0: # systemctl daemon-reload # systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail # echo $? 0 sendmail.service with StartLimitInterval=0 # systemctl daemon-reload # systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail # echo $? 0 sendmail.service with StartLimitBurst=0 # systemctl daemon-reload # systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail; systemctl restart sendmail # echo $? 0 I.e. all three variants works. I will probably go with StartLimitIntervalSec instead of StartLimitInterval which is marked as obsoleted and I am not going to backport this feature to older Fedoras.
That's odd, because I have a different behaviour when adding StartLimitInterval=0 or StartLimitIntervalSec=0. [root@bonobo ~]# rpm -q systemd systemd-231-12.fc25.x86_64 [root@bonobo ~]# rpm -V systemd S.5....T. c /etc/systemd/coredump.conf S.5....T. c /etc/systemd/logind.conf .......T. c /etc/systemd/system.conf [root@bonobo ~]# systemctl is-enabled sendmail enabled I modified /usr/lib/systemd/system/sendmail.service, adding StartLimitInterval=0 or StartLimitIntervalSec=0 at the end of section [Service]. Could the section be relevant in that case ?
(In reply to Fabrice Bellet from comment #12) > That's odd, because I have a different behaviour when adding > StartLimitInterval=0 or StartLimitIntervalSec=0. > > [root@bonobo ~]# rpm -q systemd > systemd-231-12.fc25.x86_64 > [root@bonobo ~]# rpm -V systemd > S.5....T. c /etc/systemd/coredump.conf > S.5....T. c /etc/systemd/logind.conf > .......T. c /etc/systemd/system.conf > [root@bonobo ~]# systemctl is-enabled sendmail > enabled > > > I modified /usr/lib/systemd/system/sendmail.service, adding > StartLimitInterval=0 or StartLimitIntervalSec=0 at the end of section > [Service]. Could the section be relevant in that case ? AFAIK (and according to the doc) it should be in the [unit] section, and my tests in comment 11 used it so, i.e.: # cat /usr/lib/systemd/system/sendmail.service [Unit] Description=Sendmail Mail Transport Agent After=syslog.target network.target Conflicts=postfix.service exim.service Wants=sm-client.service StartLimitIntervalSec=0 [Service] Type=forking PIDFile=/run/sendmail.pid Environment=SENDMAIL_OPTS=-q1h EnvironmentFile=-/etc/sysconfig/sendmail ExecStartPre=-/etc/mail/make ExecStartPre=-/etc/mail/make aliases ExecStart=/usr/sbin/sendmail -bd $SENDMAIL_OPTS $SENDMAIL_OPTARG [Install] WantedBy=multi-user.target Also=sm-client.service
Maybe StartLimitInterval somehow bubbles from the [Service] section, or maybe it's there allowed for backward compatibility, I didn't check this in the systemd code.
Yup, it seems so in the systemd code (load-fragment-gperf.gperf.m4): m4_dnl The following three only exist for compatibility, they moved into Unit, see above Service.StartLimitInterval, config_parse_sec, 0, offsetof(Unit, start_limit.interval) Service.StartLimitBurst, config_parse_unsigned, 0, offsetof(Unit, start_limit.burst) Service.StartLimitAction, config_parse_failure_action, 0, offsetof(Unit, start_limit_action) And there is no StartLimitIntervalSec.
Michal, I expected answer similar to comment 11 and comment 15, just nitpicking ;)
Fabrice could you confirm StartLimitIntervalSec worked for you in the [unit] section?
(In reply to Jaroslav Škarvada from comment #17) > Fabrice could you confirm StartLimitIntervalSec worked for you in the [unit] > section? Yes, that's right: StartLimitIntervalSec=0 works too, when inserted in the [unit] section.
Thanks, I am happy we sorted it out.
sendmail-8.15.2-8.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-b64ee22a45
Should sm-client.service be modified too ? I notice that the start-limit-hit failed state is now reported on sm-client.
sendmail-8.15.2-8.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-b64ee22a45
(In reply to Fabrice Bellet from comment #21) > Should sm-client.service be modified too ? I notice that the start-limit-hit > failed state is now reported on sm-client. Yes, you are right, it's also worth the fix.
(In reply to Jaroslav Škarvada from comment #16) > Michal, I expected answer similar to comment 11 and comment 15, just > nitpicking ;) Mehh, if I were to try and experiment with all configurations/options/setups I see in different bugs that I am handling then I wouldn't get any work done at all because I'd be just configuring systems all the time ;)
sendmail-8.15.2-8.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
This limit in restarting sendmail service is there for a reason I assume? Why does sendmail have to be patched to fix a buggy networkmanager? The fix should be in networkmanager, not in sendmail.
(In reply to joopbraak from comment #26) > This limit in restarting sendmail service is there for a reason I assume? > Why does sendmail have to be patched to fix a buggy networkmanager? The fix > should be in networkmanager, not in sendmail. It's workaround. The limit is by default on for all services, I guess it's there to prevent loops of never ending restarts which shouldn't be this case. And there are other services also utilizing this functionality. But feel free to propose better solution / patches. Unfortunately at the moment we have nothing better.
Like Fabrice Bellet said: "As a side comment, I'm also wondering if the nm-dispatcher script for sendmail could not only restart sendmail when the hostname is modified ? This is what is important for sendmail I think (knowing its own hostname, to decide when delivery should be local or relayed)." or just build in a delay in restarting the service? I'm no coder, but it doesn't seem that difficult to me.
(In reply to joopbraak from comment #28) > Like Fabrice Bellet said: > > "As a side comment, I'm also wondering if the nm-dispatcher script for > sendmail could not only restart sendmail when the hostname is modified ? > This is what is important for sendmail I think (knowing its own hostname, to > decide when delivery should be local or relayed)." > To be honest I don't know. Also see comment 2 there maybe more complication. The dispatcher script will also have to compare the new hostname with the current hostname and restart only if there is a change - and there may be races so we could miss restart. Nor counting that currently I don't know whether it's enough to just react on the hostname change. Please note I am not against such change, but there is no code at the moment. And even with the code, it would require testing. > or just build in a delay in restarting the service? > > I'm no coder, but it doesn't seem that difficult to me. This would need some layer in between to count the delay (i.e. the manager), locking, etc. Not trivial. I am currently not aware of such systemd functionality.