Bug 1401456 - rsyslog starts before the network service during boot
Summary: rsyslog starts before the network service during boot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rsyslog
Version: 7.2
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: ---
Assignee: Radovan Sroka
QA Contact: Stefan Dordevic
URL:
Whiteboard:
: 1498902 1507151 1507202 1550191 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-12-05 10:34 UTC by Stefan Zwijsen
Modified: 2019-03-07 03:05 UTC (History)
15 users (show)

Fixed In Version: rsyslog-8.24.0-13.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 15:26:37 UTC


Attachments (Terms of Use)
proposed patch (437 bytes, patch)
2017-08-30 13:32 UTC, Radovan Sroka
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0856 None None None 2018-04-10 15:27:12 UTC
Red Hat Bugzilla 1379638 None CLOSED rsyslog does not start remote logging at boot 2019-03-15 12:16:21 UTC

Description Stefan Zwijsen 2016-12-05 10:34:50 UTC
Description of problem:
When the server boots, rsyslog can't resolve an FQDN used in the config to forwared messages to.

Version-Release number of selected component (if applicable):
Kernel: 3.10.0-327.36.1.el7.x86_64
rsyslog.x86_64 - 7.4.7-12.el7

How reproducible:
Everytime on our RHEL7 servers.

Steps to Reproduce:
1. Add config file in rsyslog.d to forward messages to a remote server (TCP or UDP, doesn't matter). Use the remote server FQDN. For example: "*.info        @remote.server.com"
2. Restart rsyslog service. Test if messages are forwarded. This should be OK.
3. Reboot the server. Test if messages are forwarded. This fails because during boot, when rsyslog config was read, it could not yet resolve the FQDN.
4. Restart rsyslog service. Forwarding will now work again, since it could resolve the FQDN.

Actual results:
FQDN used in rsyslog configuration can't be resolved during boot.

Expected results:
FQDN used in rsyslog configuration should be resolved during boot.

Additional info:
This will depend on the startup order of rsyslog in systemd environment (network not yet available).
I opened a ticket for RH support. They advised to use the IP address. This works indeed. But, this could not always be possible though (eg. FQDN pointing to redundant IP's). And having to use the IP is nowhere mentioned in man pages or rsyslog documentation as a requirement (use of fqdn is also shown in examples).
I think this worked before in RHEL7 (but not 100% sure - could depend on when the service started before).
This works in RHEL6, because not dependant on systemd.
Maybe the rsyslog unit should depend on network to solve this?

Comment 3 Stefan Dordevic 2017-01-06 09:50:57 UTC
Hi Stefan,

This issue is probably the same as one described in this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1263853

It's fixed RHEL7.3, rsyslog-7.4.7-16.el7

It would be nice if you give us a response when you try with new version.

Comment 4 Stefan Zwijsen 2017-02-23 09:48:12 UTC
Hi,

Sorry for the delay, I just wanted to test this now. However, I see it works now, why still using the old rsyslog-7.4.7-12.el7. I tested on 2 servers of which I'm sure it didn't work before.
Now, in the meantime these servers did have some RHEL security patches installed (no rsyslog update though).
I can't easily revert back to the old situation (from early December) to test without those patches.

I updated to rsyslog-7.4.7-16.el7 now too, it keeps working.
So, I guess the security patches that were installed in the meantime also did have some impact on the problem, although I don't see any direct relation...?

For completeness, I add the list of updated packages below. But since it works, I consider this solved.

Regards,
Stefan


Packages updated in the meantime (with security fixes):
Jan 17 01:56:44 Updated: nss-util-3.21.3-1.1.el7_3.x86_64
Jan 17 01:56:44 Updated: nss-3.21.3-2.el7_3.x86_64
Jan 17 01:56:45 Updated: nss-sysinit-3.21.3-2.el7_3.x86_64
Jan 17 01:56:45 Updated: 2:vim-filesystem-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:47 Updated: 2:vim-common-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:48 Updated: policycoreutils-2.5-9.el7.x86_64
Jan 17 01:56:48 Updated: 2:vim-minimal-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:48 Updated: sudo-1.8.6p7-21.el7_3.x86_64
Jan 17 01:56:48 Updated: policycoreutils-python-2.5-9.el7.x86_64
Jan 17 01:56:49 Updated: 2:vim-enhanced-7.4.160-1.el7_3.1.x86_64
Jan 17 01:56:49 Updated: nss-tools-3.21.3-2.el7_3.x86_64
Jan 17 01:56:49 Updated: expat-2.1.0-10.el7_3.x86_64
Feb 21 01:32:57 Updated: 32:bind-license-9.9.4-38.el7_3.2.noarch
Feb 21 01:32:58 Updated: 32:bind-libs-9.9.4-38.el7_3.2.x86_64
Feb 21 01:32:59 Updated: ntpdate-4.2.6p5-25.el7_3.1.x86_64
Feb 21 01:33:01 Updated: kernel-tools-libs-3.10.0-514.6.1.el7.x86_64
Feb 21 01:33:05 Updated: kernel-tools-3.10.0-514.6.1.el7.x86_64
Feb 21 01:33:05 Updated: ntp-4.2.6p5-25.el7_3.1.x86_64
Feb 21 01:33:05 Updated: 32:bind-utils-9.9.4-38.el7_3.2.x86_64
Feb 21 01:33:05 Updated: 32:bind-libs-lite-9.9.4-38.el7_3.2.x86_64
Feb 21 01:33:06 Updated: libtiff-4.0.3-27.el7_3.x86_64
Feb 21 01:33:06 Updated: python-perf-3.10.0-514.6.1.el7.x86_64

Comment 5 Radovan Sroka 2017-03-13 11:20:49 UTC
So I'm closing this bugzilla, if problem appears again feel free to reopen it.

Comment 6 James Ralston 2017-07-27 15:40:21 UTC
Reopening, because this problem is not resolved.

The solution implemented in bug 1263853 was to add this:

After=network.target

…to the /usr/lib/systemd/system/rsyslog.service unit file. But this is not adequate to ensure that rsyslog can resolve DNS names when it starts: ensuring that networking is up *and available* is the role of network-online.target. Without an explicit dependency on network-online.target, there is a race condition where the network may not be up when rsyslog starts, which breaks DNS resolution.

A stock RHEL7 system probably won't hit the race condition that often. But we have an updater service that runs very early in the boot process, and has these explicit dependencies:

Wants=network.target network-online.target rsyslog.service
After=network.target network-online.target rsyslog.service

This essentially causes systemd to bring up networking and start rsyslog simultaneously. And virtually every single time, rsyslog fails to resolve DNS names in its configuration files, because even for systems that use a static network configuration (instead of DHCP), rsyslog is reading its configuration files before networking is up.

The solution is to make rsyslog.service have explicit dependencies and ordering on both network.target and network-online.target:

Wants=network.target network-online.target
After=network.target network-online.target

This is justified/necessary for at least two reasons:

1. This makes RHEL7 systems consistent with RHEL6 behavior, where rsyslog always started after networking was up. If it was acceptable to have the *only* logging daemon on RHEL6 systems not start until after the network was up, then it certainly acceptable for RHEL7 to not start an *auxiliary* logging daemon until after the network is up.

2. Because RHEL7 systems already have a local logging service (journald), if the administrator has additionally enabled rsyslog, it is almost certainly because rsyslog is being used to forward local logs to a remote log server. Therefore, it is even *more* important on RHEL7 hosts that the network is fully up before rsyslog starts.

For now, we have worked around this problem by adding an /etc/systemd/system/rsyslog.service file on all of our hosts, with these contents:

.include /usr/lib/systemd/system/rsyslog.service
[Unit]
Wants=network.target network-online.target
After=network.target network-online.target

But we should not need to do this: these dependencies are necessary for proper rsyslog operation on RHEL7, and should be in /usr/lib/systemd/system/rsyslog.service by default.

Comment 7 James Ralston 2017-07-27 15:57:50 UTC
(Cross-filed as support case 01899302.)

Comment 9 Radovan Sroka 2017-08-30 13:32:33 UTC
Created attachment 1320075 [details]
proposed patch

Comment 10 Radovan Sroka 2017-10-09 10:18:19 UTC
*** Bug 1498902 has been marked as a duplicate of this bug. ***

Comment 13 Radovan Sroka 2017-10-30 09:31:50 UTC
*** Bug 1507202 has been marked as a duplicate of this bug. ***

Comment 14 Radovan Sroka 2017-10-30 09:32:42 UTC
*** Bug 1507151 has been marked as a duplicate of this bug. ***

Comment 17 Steve Snodgrass 2018-02-13 19:06:01 UTC
I just upgraded a syslog server to RHEL7 and have the same issue - trying to forward syslog to a remote hostname fails since rsyslog cannot resolve DNS when it starts up.  Any idea when the -13 RPM will be released?

Comment 19 Radovan Sroka 2018-03-01 08:24:49 UTC
*** Bug 1550191 has been marked as a duplicate of this bug. ***

Comment 21 errata-xmlrpc 2018-04-10 15:26:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0856


Note You need to log in before you can comment on or make changes to this bug.