Have you contacted support? Are you sure the SELinux isn't causing AVC denials?

RHEL support is not exactly super helpful or efficient to contact them, pus this is clearly an issue on RHEL side.

And I'm absolutely sure SELinux is disabled (even if it was not, HAProxy is part of the RHEL system so you should have ensured it complies with SELinux - but, as I said, SELinux is disabled). Did not you read my ticket which explains that systemd starts the service BEFORE the network is comletely up?!

I just spotted on another RHEL-8 machine the same behaviour with sshd when ListenAddress is enabled in config (not 0.0.0.0, but a specific address on one of its interfaces). Seems there is a general issue with services that bind to an address. I'd suggest you look at this more closely.

(In reply to Curaden AG from comment #3)
> RHEL support is not exactly super helpful or efficient to contact them, pus
> this is clearly an issue on RHEL side.
> 
> And I'm absolutely sure SELinux is disabled (even if it was not, HAProxy is
> part of the RHEL system so you should have ensured it complies with SELinux
> - but, as I said, SELinux is disabled). Did not you read my ticket which
> explains that systemd starts the service BEFORE the network is comletely up?!

HAProxy does comply with SELinux policy. By default it only allows haproxy to bind to specific ports. That is why I asked -- to see if perhaps you were running into this common problem.

I read the bug report. There are options for binding to non-existent IP addresses. See the 'transparent' bind option in the official documentation.

> I just spotted on another RHEL-8 machine the same behaviour with sshd when
> ListenAddress is enabled in config (not 0.0.0.0, but a specific address on
> one of its interfaces). Seems there is a general issue with services that
> bind to an address. I'd suggest you look at this more closely.

We are looking at it, but adding restart capabilities to the systemd unit file is not widely appealing. We believe the correct solution is to change the systemd dependency from 'network.target' to 'network-online.target'.

I did some research on this and found some interesting information about differences between 'network.target' and 'network-online.target' dependencies. Aside from the obvious differences, there is good information here [1]. Note that it warns of moving to dependency on 'network-online.target' due to potentially slower boot times, etc. At the bottom, it suggests using IP_FREEBIND as a means to bind to addresses that are not yet configured. From the link:

"If you write a server: if you want to listen on other, explicitly configured addresses, consider using the IP_FREEBIND sockopt functionality of the Linux kernel. This allows your code to bind to an address even if it is not actually (yet or ever) configured locally. This also makes your code robust towards network configuration changes."

This is exactly what the 'transparent' option for 'bind' does. This I am inclined to say that we do not want to change to "After=network-online.target" as discussed.

What I find most strange is that, as reported in the original bug report, haproxy-1.8 from RHSCL on RHEL7 does *not* experience this problem yet haproxy-1.8 on RHEL8 does. I have not yet been able to pinpoint a reason for this behavior. Is it possible that RHEL7 was faster at getting the network configured, ie. before haproxy tried to bind, thus avoiding any problem? I'm still investigating.

In the meantime, could the reporter try using 'transparent' option and reporting back if this is a valid workaround? Much appreciated.

[1] https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

From what I see, "bind transparent" is deprecated in HAProxy in favour of the global "option transparent". However, it is always mentioned as a way of doing a transparent proxy and nothing is mentioned about the ability to bind to a non-existent IP address. 

I also don't see anything similar as an option for SSHd, which is equally affected by this issue.

In addition, personally I am quite happy to once extend the unit file for HAProxy with network-online.target - whether a server boot takes a second or two longer once a week is irrelevant to me (for servers we always use static IP addresses) - what matters is convenience and reliability; hence I have no incentive to experiment with "option transparent" and I'll stick with my current solution.

Closing this as NOTABUG for the following reasons:

1. The "option transparent" in haproxy.cfg will allow binding to specific IP address regardless if that address exists on the machine.

2. The sysctl booleans "net.ipv4.ip_nonlocal_bind" and "net.ipv6.ip_nonlocal_bind", when set to "1", will also allow binding to nonexistent addresses. Note that this is system-wide.

3. Based on what I've read in the systemd documentation and explained in comment #5, I don't think we want to depend on "network-online.target".

OK, your position understood.

I'm reopening this bug since I got another inquiry about this.

In the case of binding to a non-existent IP address we can use "option transparent" as a workaround, as stated above. But there is another situation where network-online is useful -- when using a resolver to resolve server names via DNS. If the network is not online, DNS queries will not work. Read more about haproxy and DNS resolution here [1].

I'm leaning towards changing the systemd service file to include:

After=network-online.target
Wants=network-online.target

The risk here is that it could cause a substantially delay boot time. Read more about that here [2]. But there are some other commonly used services that use this same model, so it is not unprecedented.

[1] http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.3
[2] https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (haproxy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4815