Bug 1116538 (postfix_network-online) - needs network-online.target instead of network.target
Summary: needs network-online.target instead of network.target
Keywords:
Status: CLOSED EOL
Alias: postfix_network-online
Product: Fedora
Classification: Fedora
Component: postfix
Version: 27
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jaroslav Škarvada
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1116539 postfix_network-online.target (view as bug list)
Depends On:
Blocks: network-online.target
TreeView+ depends on / blocked
 
Reported: 2014-07-06 01:24 UTC by David Benfell
Modified: 2023-06-01 06:31 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-30 21:33:09 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description David Benfell 2014-07-06 01:24:41 UTC
Description of problem: postfix starts before network initialization is complete


Version-Release number of selected component (if applicable): postfix-2.11.1-2.fc20.x86_64


How reproducible: For me, 100%


Steps to Reproduce:
1. reboot system
2.
3.

Actual results: postfix generates an error indicating its interface does not exist


Expected results: postfix starts


Additional info: This looks very much like bug #1096081 (which is for sshd). To solve this problem, I am currently using the following lines in my unit file:

Requires=network-online.target
After=syslog.target network-online.target

With these lines, postfix starts correctly. Please note, this bug also affects nsd and ejabberd.

Comment 1 Jaroslav Škarvada 2014-07-07 07:55:55 UTC
I don't like the network-online.target solution. This may add significant delays if booting without network and from the systemd docs [1] it also seems not to be recommended, citing:

> network-online.target .... It is strongly recommended not to pull in this
> target too liberally: for example network server software should generally not
> pull this in (since server software generally is happy to accept local
> connections even before any routable network interface is up), it's primary
> purpose is network client software that cannot operate without network.

If postfix cannot start (which is something I have never encountered, probably it's due to your customized configuration), it should be probably patched to use IP_FREEBIND. Could you provide more information about your configuration (e.g. the inet_interfaces line from the config) and the error mesasge displayed?

[1] http://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

Comment 2 Jaroslav Škarvada 2014-07-07 08:07:45 UTC
Also if your specific configuration requires network to be online, you can use the following:
# systemctl enable NetworkManager-wait-online.service

Comment 3 Jaroslav Škarvada 2014-07-07 08:08:21 UTC
*** Bug 1116539 has been marked as a duplicate of this bug. ***

Comment 4 David Benfell 2014-07-07 09:42:50 UTC
The systemctl status is as follows:

[root@munich]/home/benfell# systemctl status postfix
postfix.service - Postfix Mail Transport Agent
   Loaded: loaded (/etc/systemd/system/postfix.service; enabled)

Active: failed (Result: exit-code) since Wed 2014-07-02 05:00:31 PDT; 9h ago Process: 1196 ExecStart=/usr/sbin/postfix start (code=exited, status=1/FAILURE) Process: 1194 ExecStartPre=/usr/libexec/postfix/chroot-update (code=exited, status=0/SUCCESS) Process: 371 ExecStartPre=/usr/libexec/postfix/aliasesdb (code=exited, status=75)

and the configuration line is:

inet_interfaces = 127.0.0.1, [::1], 91.205.174.238, [2a02:c200:0:10::404:11]

(doesn't seem to me like an especially large number of interfaces.)

Comment 5 David Benfell 2014-07-07 09:44:15 UTC
I'm not using NetworkManager on this system. It is a dedicated server. postfix here runs for several domains.

Comment 6 Jaroslav Škarvada 2014-07-07 11:32:18 UTC
Thanks, could you also provide snip from the log (e.g. /var/log/maillog) saying more about the failure?

Comment 7 David Benfell 2014-07-07 20:00:57 UTC
This seems to be a relevant snippet covering a reboot:

Jun 30 04:50:35 munich spamd[27322]: spamd: server killed by SIGTERM, shutting down
Jun 30 04:50:36 munich spamd[27322]: spamd: cannot send SIGINT to child process [27339]: No such process
Jun 30 04:50:36 munich spamd[27322]: spamd: cannot send SIGINT to child process [27338]: No such process
Jun 30 04:50:36 munich dovecot: master: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill)
Jun 30 04:50:39 munich dovecot: config: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill)
Jun 30 04:50:39 munich dovecot: auth-worker(27494): Warning: Killed with signal 15 (by pid=1 uid=0 code=kill)
Jun 30 04:50:39 munich dovecot: auth: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill)
Jun 30 04:50:39 munich dovecot: log: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill)
Jun 30 04:50:39 munich dovecot: anvil: Warning: Killed with signal 15 (by pid=1 uid=0 code=kill)
Jun 30 04:50:43 munich authdaemond: stopping authdaemond children
Jun 30 04:53:20 munich authdaemond: modules="authpam", daemons=5
Jun 30 04:53:20 munich postfix/sendmail[450]: fatal: parameter inet_interfaces: no local interface found for 91.205.174.238
Jun 30 04:53:21 munich dovecot: master: Dovecot v2.2.13 starting up for imap, pop3, lmtp (core dumps disabled)
Jun 30 04:53:21 munich authdaemond: Installing libauthpam
Jun 30 04:53:22 munich authdaemond: Installation complete: authpam
Jun 30 04:53:22 munich postfix[610]: fatal: parameter inet_interfaces: no local interface found for 91.205.174.238
Jun 30 04:53:23 munich postfix/sendmail[732]: fatal: parameter inet_interfaces: no local interface found for 91.205.174.238
Jun 30 04:53:23 munich spamd[479]: spamd: server started on IO::Socket::IP [127.0.0.1]:783, IO::Socket::IP [::1]:783 (running version 3.4.0)
Jun 30 04:53:24 munich spamd[479]: spamd: server pid: 479
Jun 30 04:53:24 munich spamd[479]: spamd: server successfully spawned child process, pid 745
Jun 30 04:53:24 munich spamd[479]: spamd: server successfully spawned child process, pid 746
Jun 30 04:53:24 munich spamd[479]: prefork: child states: II
Jun 30 04:53:26 munich postfix/sendmail[1316]: fatal: parameter inet_interfaces: no local interface found for 91.205.174.238
Jun 30 04:54:14 munich dovecot: imap-login: Login: user=<benfell>, method=PLAIN, rip=::1, lip=::1, mpid=1788, secured, session=<4MNlUQz9LQAAAAAAAAAAAAAAAAAAAAAB>
Jun 30 04:54:15 munich dovecot: imap(benfell): Disconnected: Logged out in=2855 out=16240
Jun 30 04:55:12 munich dovecot: imap-login: Login: user=<benfell>, method=PLAIN, rip=::1, lip=::1, mpid=1819, secured, session=<CsjdVAz9WAAAAAAAAAAAAAAAAAAAAAAB>

Comment 8 Pavel Šimerda (pavlix) 2014-07-17 07:57:49 UTC
(In reply to Jaroslav Škarvada from comment #1)
> If postfix cannot start (which is something I have never encountered,
> probably it's due to your customized configuration), it should be probably
> patched to use IP_FREEBIND.

+1 for IP_FREEBIND

Comment 9 Balint Szigeti 2014-07-17 09:17:21 UTC
(In reply to Pavel Šimerda (pavlix) from comment #8)
> (In reply to Jaroslav Škarvada from comment #1)
> > If postfix cannot start (which is something I have never encountered,
> > probably it's due to your customized configuration), it should be probably
> > patched to use IP_FREEBIND.
> 
> +1 for IP_FREEBIND

?????
Do you suggest modify everything else, instead of systemd which cause the problem?


cite from http://man7.org/linux/man-pages/man7/ip.7.html
IP_FREEBIND (since Linux 2.4)
              If enabled, this boolean option allows binding to an IP
              address that is nonlocal or does not (yet) exist.  This
              permits listening on a socket, without requiring the
              underlying network interface or the specified dynamic IP
              address to be up at the time that the application is trying to
              bind to it.  This option is the per-socket equivalent of the
              ip_nonlocal_bind /proc interface described below

It is a good feature of Linux kernel but I think that is the worst approach to fix this issue.

Comment 10 Balint Szigeti 2014-07-17 09:20:11 UTC
systemd should start the network BEFORE the other services start which NEED network. the other way just stupidity.

Comment 11 Pavel Šimerda (pavlix) 2014-07-17 10:32:29 UTC
(In reply to Balint Szigeti from comment #9)
> (In reply to Pavel Šimerda (pavlix) from comment #8)
> > (In reply to Jaroslav Škarvada from comment #1)
> > > If postfix cannot start (which is something I have never encountered,
> > > probably it's due to your customized configuration), it should be probably
> > > patched to use IP_FREEBIND.
> > 
> > +1 for IP_FREEBIND
> 
> ?????
> Do you suggest modify everything else, instead of systemd which cause the
> problem?

I personally would be most happy if the kernel provided IP_FREEBIND sockets *by default* and therefore would not create the pointless dependency on configured addresses at all.

> cite from http://man7.org/linux/man-pages/man7/ip.7.html
> IP_FREEBIND (since Linux 2.4)
>               If enabled, this boolean option allows binding to an IP
>               address that is nonlocal or does not (yet) exist.  This
>               permits listening on a socket, without requiring the
>               underlying network interface or the specified dynamic IP
>               address to be up at the time that the application is trying to
>               bind to it.  This option is the per-socket equivalent of the
>               ip_nonlocal_bind /proc interface described below
> 
> It is a good feature of Linux kernel but I think that is the worst approach
> to fix this issue.

In my opinion, the best solutions would be to have freebind sockets by default (unless there are some bad side effects) and using explicitly IP_FREEBIND is IMO the second best solution plus it's supported by existing kernels.

(In reply to Balint Szigeti from comment #10)
> systemd should start the network BEFORE the other services start which NEED
> network. the other way just stupidity.

In this case systemd seems to behave exactly according to its documentation. It provides `network-online.target` as a means to try to wait for a working network before a service is started. This target simply waits for any network providing services to finish or timeout.

Don't ask me why `network-online.target` was defined instead of redifining `network.target`, I don't know more than what's in the manual page. But that's what we're working with.

Comment 12 Alex Regan 2014-07-28 02:46:20 UTC
For anyone experiencing this problem, what is the solution in the meantime? I'm also having a problem with postfix and dovecot.

Is the only temporary solution to start the services manually? Or will changing network.target to network-online.target for each service work for now?

Comment 13 Pavel Šimerda (pavlix) 2014-07-28 07:34:23 UTC
(In reply to Alex Regan from comment #12)
> For anyone experiencing this problem, what is the solution in the meantime?
> I'm also having a problem with postfix and dovecot.
> 
> Is the only temporary solution to start the services manually? Or will
> changing network.target to network-online.target for each service work for
> now?

You can fix the service files locally (see bug #1119787).

Comment 14 Timon 2015-01-23 06:34:27 UTC
steel persist in f21
postfix-2.11.3-1.fc21.x86_64
systemd-216-16.fc21.x86_64

Comment 15 Thomas Köller 2015-01-25 12:46:24 UTC
(In reply to Alex Regan from comment #12)
> For anyone experiencing this problem, what is the solution in the meantime?
> I'm also having a problem with postfix and dovecot.
> 
> Is the only temporary solution to start the services manually? Or will
> changing network.target to network-online.target for each service work for
> now?

I have the same problem. My workaround was to create an intermediate systemd target that orders itself after network-online.target, but before postfix.service.

I do think that postfix.service should really order itself after network-online.target, as other services of similar kind already do (dovecot, radicale).

Comment 16 Dominique Martinet 2015-04-08 07:05:11 UTC
For what it's worth, I have the same problem on rhel/centos7.

Basically, postfix will start depending on your configuration:
These two start:
inet_interfaces = all
inet_interfaces = localhost

These two won't:
inet_interfaces = $myhostname
inet_interfaces = $myhostname, localhost

All four are provided in comments in the default confiuration shipped in the RPM, so really are expected to work!

Comment 17 Fedora End Of Life 2015-05-29 12:18:23 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 18 Fedora End Of Life 2015-06-30 01:04:22 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 19 Mike A. Harris 2017-05-05 15:14:55 UTC
I can confirm this bug exists on fully up to date CentOS 7, and I experienced it on EL7 about 2 years ago also.  Good to see that nobody has fixed it yet though, makes system administration more fun as you get to re-engineer the OS yourself.

Who needs vendor updates! :)

Comment 20 David Hill 2017-10-18 05:14:32 UTC
I do have this issue too with Fedora rawhide ...

Comment 21 Michal Ambroz 2017-11-06 20:55:02 UTC
*** Bug 1510158 has been marked as a duplicate of this bug. ***

Comment 22 Michal Ambroz 2017-11-06 20:58:16 UTC
Hello,
I can confirm this bug exists in fully up-to-date Fedora 26 - and according the package sources affects also rawhide and Fedora 27.

Description of problem:
-----------------------
After reboot postfix fails to start with following message in the syslog:
Nov 06 19:17:22 testmachine.example.com postfix[1200]: fatal: parameter inet_interfaces: no local interface found for 127.0.0.2

- The postfix systemd start-script is depending to network.target (and not network-online.target as it probably should be). 
- Due to this it gets to race condition where network-manager is just started but not finished setting-up network interfaces. 
- At this precise point it tries to resolve its $myhostname as defined in /etc/postfix/main.cf or obtained from gethostname() by default and gots 127.0.0.2
- then it tries to identify a local device which has this IP address configured, but doen't find any (it would be wrong address anyway)
- so the start of the service after reboot fails
- then the network manager finishes the network configuration so all the resolving works fine from now on so it is hard to get idea where is this 127.0.0.2 coming from

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
tested with postfix-3.2.3-1.fc26, but the situation is the same in rawhide and fc27.


How reproducible:
-----------------
Race condition during start of network-manager probably dependent on the number of network cards.
On one of my machines it was failing in 100% cases (machine with cca 10 physical interfaces), but on some different machine I believe it could be 100% success (like machine with only 1 physical network card).


Steps to Reproduce:
-------------------
0. have many network cards
1. instal postfix (dnf -y install postfix)
2. enable it to start automatically (systemctl enable postfix)
3. reboot the machine
4. check the staus after reboot (systemctl status postfix)


Additional info:
----------------
I believe the dependency should be changed from network.target to network-online.target in /usr/lib/systemd/system/postfix.service:

[Unit]
Description=Postfix Mail Transport Agent
After=syslog.target network-online.target
Conflicts=sendmail.service exim.service
....

Even if you manualy define correct $myhostname in /etc/postfix/main/cf and its correct IP resolv in /etc/hosts, the postfix service still fails to start at reboot, because at that point it would know the correct IP address, but that would not be ready on some network interface. The error in this situation would be :
Nov 06 19:17:22 testmachine.example.com postfix[1200]: fatal: parameter inet_interfaces: no local interface found for 111.222.33.44


Best regards
Michal Ambroz

Comment 23 Michal Ambroz 2017-11-08 15:55:49 UTC
Affects the package during the Fedora 27 release cycle - changing to F27

Comment 24 Michal Ambroz 2017-11-09 02:01:02 UTC
Part of the problem is also that the unit files such as /usr/lib/systemd/system/postfix.service are usually not marked as configuration files so even if you change it manually from network.target to netork-online.target, it will revert back with another patching.

Comment 25 Jaroslav Škarvada 2017-11-10 13:52:40 UTC
(In reply to Michal Ambroz from comment #22)
> Hello,
> I can confirm this bug exists in fully up-to-date Fedora 26 - and according
> the package sources affects also rawhide and Fedora 27.
> 
> Description of problem:
> -----------------------
> After reboot postfix fails to start with following message in the syslog:
> Nov 06 19:17:22 testmachine.example.com postfix[1200]: fatal: parameter
> inet_interfaces: no local interface found for 127.0.0.2
> 
> - The postfix systemd start-script is depending to network.target (and not
> network-online.target as it probably should be).

From the NetworkManager documentation [1]:
...for example network server software should generally not pull this in (since server software generally is happy to accept local connections even before any routable network interface is up), it's primary purpose is network client software that cannot operate without network.

I am afraid that adding this dependency by default may introduce boot delays - in the default configuration we enable postfix only on the localhost interface. But if you need it for your configuration, you can run (according to [1]):

# systemctl enable NetworkManager-wait-online.service

Or:
# systemctl enable systemd-networkd-wait-online.service

But it may introduce significant boot delays.

> - Due to this it gets to race condition where network-manager is just
> started but not finished setting-up network interfaces. 
> - At this precise point it tries to resolve its $myhostname as defined in
> /etc/postfix/main.cf or obtained from gethostname() by default and gots
> 127.0.0.2
> - then it tries to identify a local device which has this IP address
> configured, but doen't find any (it would be wrong address anyway)

This seems to be systemd-resolved. To be honest I don't understand why it resolves your hostname to IP address 127.0.0.2 that doesn't exists (i.e. is not configured on any of the interfaces). I think without systemd-resolved you shouldn't encounter this error.

[1] https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

Comment 26 Jaroslav Škarvada 2017-11-10 13:54:50 UTC
(In reply to Michal Ambroz from comment #24)
> Part of the problem is also that the unit files such as
> /usr/lib/systemd/system/postfix.service are usually not marked as
> configuration files so even if you change it manually from network.target to
> netork-online.target, it will revert back with another patching.

I think this is correct - this is not user editable configuration and you shouldn't edit it. If you need to modify it, you should copy it to the /etc/systemd/system and make your modifications on the copy.

Comment 27 Ben Cotton 2018-11-27 15:45:26 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 28 Ben Cotton 2018-11-30 21:33:09 UTC
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 29 Kenny G 2020-10-16 12:30:35 UTC Comment hidden (spam)

Note You need to log in before you can comment on or make changes to this bug.