Bug 1401456

Hi Stefan,

This issue is probably the same as one described in this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1263853

It's fixed RHEL7.3, rsyslog-7.4.7-16.el7

It would be nice if you give us a response when you try with new version.

Hi,

Sorry for the delay, I just wanted to test this now. However, I see it works now, why still using the old rsyslog-7.4.7-12.el7. I tested on 2 servers of which I'm sure it didn't work before.
Now, in the meantime these servers did have some RHEL security patches installed (no rsyslog update though).
I can't easily revert back to the old situation (from early December) to test without those patches.

I updated to rsyslog-7.4.7-16.el7 now too, it keeps working.
So, I guess the security patches that were installed in the meantime also did have some impact on the problem, although I don't see any direct relation...?

For completeness, I add the list of updated packages below. But since it works, I consider this solved.

Regards,
Stefan


Packages updated in the meantime (with security fixes):
Jan 17 01:56:44 Updated: nss-util-3.21.3-1.1.el7_3.x86_64
Jan 17 01:56:44 Updated: nss-3.21.3-2.el7_3.x86_64
Jan 17 01:56:45 Updated: nss-sysinit-3.21.3-2.el7_3.x86_64
Jan 17 01:56:45 Updated: 2:vim-filesystem-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:47 Updated: 2:vim-common-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:48 Updated: policycoreutils-2.5-9.el7.x86_64
Jan 17 01:56:48 Updated: 2:vim-minimal-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:48 Updated: sudo-1.8.6p7-21.el7_3.x86_64
Jan 17 01:56:48 Updated: policycoreutils-python-2.5-9.el7.x86_64
Jan 17 01:56:49 Updated: 2:vim-enhanced-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:49 Updated: nss-tools-3.21.3-2.el7_3.x86_64
Jan 17 01:56:49 Updated: expat-2.1.0-10.el7_3.x86_64
Feb 21 01:32:57 Updated: 32:bind-license-9.9.4-38.el7_3.2.noarch
Feb 21 01:32:58 Updated: 32:bind-libs-9.9.4-38.el7_3.2.x86_64
Feb 21 01:32:59 Updated: ntpdate-4.2.6p5-25.el7_3.1.x86_64
Feb 21 01:33:01 Updated: kernel-tools-libs-3.10.0-514.6.1.el7.x86_64
Feb 21 01:33:05 Updated: kernel-tools-3.10.0-514.6.1.el7.x86_64
Feb 21 01:33:05 Updated: ntp-4.2.6p5-25.el7_3.1.x86_64
Feb 21 01:33:05 Updated: 32:bind-utils-9.9.4-38.el7_3.2.x86_64
Feb 21 01:33:05 Updated: 32:bind-libs-lite-9.9.4-38.el7_3.2.x86_64
Feb 21 01:33:06 Updated: libtiff-4.0.3-27.el7_3.x86_64
Feb 21 01:33:06 Updated: python-perf-3.10.0-514.6.1.el7.x86_64

So I'm closing this bugzilla, if problem appears again feel free to reopen it.

Reopening, because this problem is not resolved.

The solution implemented in bug 1263853 was to add this:

After=network.target

…to the /usr/lib/systemd/system/rsyslog.service unit file. But this is not adequate to ensure that rsyslog can resolve DNS names when it starts: ensuring that networking is up *and available* is the role of network-online.target. Without an explicit dependency on network-online.target, there is a race condition where the network may not be up when rsyslog starts, which breaks DNS resolution.

A stock RHEL7 system probably won't hit the race condition that often. But we have an updater service that runs very early in the boot process, and has these explicit dependencies:

Wants=network.target network-online.target rsyslog.service
After=network.target network-online.target rsyslog.service

This essentially causes systemd to bring up networking and start rsyslog simultaneously. And virtually every single time, rsyslog fails to resolve DNS names in its configuration files, because even for systems that use a static network configuration (instead of DHCP), rsyslog is reading its configuration files before networking is up.

The solution is to make rsyslog.service have explicit dependencies and ordering on both network.target and network-online.target:

Wants=network.target network-online.target
After=network.target network-online.target

This is justified/necessary for at least two reasons:

1. This makes RHEL7 systems consistent with RHEL6 behavior, where rsyslog always started after networking was up. If it was acceptable to have the *only* logging daemon on RHEL6 systems not start until after the network was up, then it certainly acceptable for RHEL7 to not start an *auxiliary* logging daemon until after the network is up.

2. Because RHEL7 systems already have a local logging service (journald), if the administrator has additionally enabled rsyslog, it is almost certainly because rsyslog is being used to forward local logs to a remote log server. Therefore, it is even *more* important on RHEL7 hosts that the network is fully up before rsyslog starts.

For now, we have worked around this problem by adding an /etc/systemd/system/rsyslog.service file on all of our hosts, with these contents:

.include /usr/lib/systemd/system/rsyslog.service
[Unit]
Wants=network.target network-online.target
After=network.target network-online.target

But we should not need to do this: these dependencies are necessary for proper rsyslog operation on RHEL7, and should be in /usr/lib/systemd/system/rsyslog.service by default.

(Cross-filed as support case 01899302.)

Created attachment 1320075 [details]
proposed patch

*** Bug 1498902 has been marked as a duplicate of this bug. ***

*** Bug 1507202 has been marked as a duplicate of this bug. ***

*** Bug 1507151 has been marked as a duplicate of this bug. ***

I just upgraded a syslog server to RHEL7 and have the same issue - trying to forward syslog to a remote hostname fails since rsyslog cannot resolve DNS when it starts up.  Any idea when the -13 RPM will be released?

*** Bug 1550191 has been marked as a duplicate of this bug. ***

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0856