Description of problem:
When the server boots, rsyslog can't resolve an FQDN used in the config to forwared messages to.
Version-Release number of selected component (if applicable):
rsyslog.x86_64 - 7.4.7-12.el7
Everytime on our RHEL7 servers.
Steps to Reproduce:
1. Add config file in rsyslog.d to forward messages to a remote server (TCP or UDP, doesn't matter). Use the remote server FQDN. For example: "*.info @remote.server.com"
2. Restart rsyslog service. Test if messages are forwarded. This should be OK.
3. Reboot the server. Test if messages are forwarded. This fails because during boot, when rsyslog config was read, it could not yet resolve the FQDN.
4. Restart rsyslog service. Forwarding will now work again, since it could resolve the FQDN.
FQDN used in rsyslog configuration can't be resolved during boot.
FQDN used in rsyslog configuration should be resolved during boot.
This will depend on the startup order of rsyslog in systemd environment (network not yet available).
I opened a ticket for RH support. They advised to use the IP address. This works indeed. But, this could not always be possible though (eg. FQDN pointing to redundant IP's). And having to use the IP is nowhere mentioned in man pages or rsyslog documentation as a requirement (use of fqdn is also shown in examples).
I think this worked before in RHEL7 (but not 100% sure - could depend on when the service started before).
This works in RHEL6, because not dependant on systemd.
Maybe the rsyslog unit should depend on network to solve this?
This issue is probably the same as one described in this bug:
It's fixed RHEL7.3, rsyslog-7.4.7-16.el7
It would be nice if you give us a response when you try with new version.
Sorry for the delay, I just wanted to test this now. However, I see it works now, why still using the old rsyslog-7.4.7-12.el7. I tested on 2 servers of which I'm sure it didn't work before.
Now, in the meantime these servers did have some RHEL security patches installed (no rsyslog update though).
I can't easily revert back to the old situation (from early December) to test without those patches.
I updated to rsyslog-7.4.7-16.el7 now too, it keeps working.
So, I guess the security patches that were installed in the meantime also did have some impact on the problem, although I don't see any direct relation...?
For completeness, I add the list of updated packages below. But since it works, I consider this solved.
Packages updated in the meantime (with security fixes):
Jan 17 01:56:44 Updated: nss-util-3.21.3-1.1.el7_3.x86_64
Jan 17 01:56:44 Updated: nss-3.21.3-2.el7_3.x86_64
Jan 17 01:56:45 Updated: nss-sysinit-3.21.3-2.el7_3.x86_64
Jan 17 01:56:45 Updated: 2:vim-filesystem-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:47 Updated: 2:vim-common-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:48 Updated: policycoreutils-2.5-9.el7.x86_64
Jan 17 01:56:48 Updated: 2:vim-minimal-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:48 Updated: sudo-1.8.6p7-21.el7_3.x86_64
Jan 17 01:56:48 Updated: policycoreutils-python-2.5-9.el7.x86_64
Jan 17 01:56:49 Updated: 2:vim-enhanced-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:49 Updated: nss-tools-3.21.3-2.el7_3.x86_64
Jan 17 01:56:49 Updated: expat-2.1.0-10.el7_3.x86_64
Feb 21 01:32:57 Updated: 32:bind-license-9.9.4-38.el7_3.2.noarch
Feb 21 01:32:58 Updated: 32:bind-libs-9.9.4-38.el7_3.2.x86_64
Feb 21 01:32:59 Updated: ntpdate-4.2.6p5-25.el7_3.1.x86_64
Feb 21 01:33:01 Updated: kernel-tools-libs-3.10.0-514.6.1.el7.x86_64
Feb 21 01:33:05 Updated: kernel-tools-3.10.0-514.6.1.el7.x86_64
Feb 21 01:33:05 Updated: ntp-4.2.6p5-25.el7_3.1.x86_64
Feb 21 01:33:05 Updated: 32:bind-utils-9.9.4-38.el7_3.2.x86_64
Feb 21 01:33:05 Updated: 32:bind-libs-lite-9.9.4-38.el7_3.2.x86_64
Feb 21 01:33:06 Updated: libtiff-4.0.3-27.el7_3.x86_64
Feb 21 01:33:06 Updated: python-perf-3.10.0-514.6.1.el7.x86_64
So I'm closing this bugzilla, if problem appears again feel free to reopen it.
Reopening, because this problem is not resolved.
The solution implemented in bug 1263853 was to add this:
…to the /usr/lib/systemd/system/rsyslog.service unit file. But this is not adequate to ensure that rsyslog can resolve DNS names when it starts: ensuring that networking is up *and available* is the role of network-online.target. Without an explicit dependency on network-online.target, there is a race condition where the network may not be up when rsyslog starts, which breaks DNS resolution.
A stock RHEL7 system probably won't hit the race condition that often. But we have an updater service that runs very early in the boot process, and has these explicit dependencies:
Wants=network.target network-online.target rsyslog.service
After=network.target network-online.target rsyslog.service
This essentially causes systemd to bring up networking and start rsyslog simultaneously. And virtually every single time, rsyslog fails to resolve DNS names in its configuration files, because even for systems that use a static network configuration (instead of DHCP), rsyslog is reading its configuration files before networking is up.
The solution is to make rsyslog.service have explicit dependencies and ordering on both network.target and network-online.target:
This is justified/necessary for at least two reasons:
1. This makes RHEL7 systems consistent with RHEL6 behavior, where rsyslog always started after networking was up. If it was acceptable to have the *only* logging daemon on RHEL6 systems not start until after the network was up, then it certainly acceptable for RHEL7 to not start an *auxiliary* logging daemon until after the network is up.
2. Because RHEL7 systems already have a local logging service (journald), if the administrator has additionally enabled rsyslog, it is almost certainly because rsyslog is being used to forward local logs to a remote log server. Therefore, it is even *more* important on RHEL7 hosts that the network is fully up before rsyslog starts.
For now, we have worked around this problem by adding an /etc/systemd/system/rsyslog.service file on all of our hosts, with these contents:
But we should not need to do this: these dependencies are necessary for proper rsyslog operation on RHEL7, and should be in /usr/lib/systemd/system/rsyslog.service by default.
(Cross-filed as support case 01899302.)
Created attachment 1320075 [details]
*** Bug 1498902 has been marked as a duplicate of this bug. ***
*** Bug 1507202 has been marked as a duplicate of this bug. ***
*** Bug 1507151 has been marked as a duplicate of this bug. ***
I just upgraded a syslog server to RHEL7 and have the same issue - trying to forward syslog to a remote hostname fails since rsyslog cannot resolve DNS when it starts up. Any idea when the -13 RPM will be released?
*** Bug 1550191 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.