RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1509997 - Cannot log anymore after rsyslog died
Summary: Cannot log anymore after rsyslog died
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rsyslog
Version: 7.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Radovan Sroka
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-06 14:27 UTC by Renaud Métrich
Modified: 2020-12-14 10:46 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-26 19:40:19 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Renaud Métrich 2017-11-06 14:27:13 UTC 
Description of problem:

While reproducing BZ https://bugzilla.redhat.com/show_bug.cgi?id=1509987 I found that after the above issue with rsyslog happened, one could not send any logs further to systemd-journald.

A strace of "logger" shows the following:

# strace logger '{ "key": "value" }'
execve("/usr/bin/logger", ["logger", "{ \"key\": \"value\" }"], [/* 26 vars */]) = 0
...
socket(AF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0

=> ECONNREFUSED


Version-Release number of selected component (if applicable):

systemd-219-42.el7_4.4.x86_64


How reproducible:

Always

Steps to Reproduce: use reproducer of BZ https://bugzilla.redhat.com/show_bug.cgi?id=1509987

1. Install rsyslog-mmjsonparse package

# yum -y install rsyslog-mmjsonparse

2. Configure mmjsonparse

# cat /etc/rsyslog.d/json.conf 
module(load="mmjsonparse") # Load mmjsonparse module for structured logs
action(type="mmjsonparse" cookie='')
if $parsesuccess == "OK" then {
   action(type="omfile" File="/tmp/output")
} else if $parsesuccess == "FAIL" then {
   action(type="omfile" File="/tmp/parsing_failure")
}

3. Restart rsyslog

# systemctl restart rsyslog

4. Rsyslog will die immediately, respin, then systemd will abort respinning

5. Send log using logger

# strace logger Test


Actual results:

connect(3, {sa_family=AF_LOCAL, sun_path="/dev/log"}, 110) = -1 ECONNREFUSED (Connection refused)


Expected results:

Log sent to journal


Additional info:

Restarting systemd (systemctl daemon-reexec) or restarting systemd-journald doesn't help.
I have to reboot the system to have logs work again (after fixing the rsyslog configuration issue).
Comment 2 Lukáš Nykrýn 2017-11-06 15:39:30 UTC 
Looks like rsyslog deletes the sockets and creates it again.


time->Mon Nov  6 10:37:33 2017
type=PROCTITLE msg=audit(1509982653.388:575): proctitle=2F7573722F7362696E2F727379736C6F6764002D6E
type=PATH msg=audit(1509982653.388:575): item=1 name="/dev/log" inode=33254 dev=00:05 mode=0140666 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:devlog_t:s0 objtype=DELETE
type=PATH msg=audit(1509982653.388:575): item=0 name="/dev/" inode=3 dev=00:05 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:device_t:s0 objtype=PARENT
type=CWD msg=audit(1509982653.388:575):  cwd="/"
type=SYSCALL msg=audit(1509982653.388:575): arch=c000003e syscall=87 success=yes exit=0 a0=7fc5b620ce7f a1=1 a2=0 a3=2 items=2 ppid=1 pid=10693 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rsyslogd" exe="/usr/sbin/rsyslogd" subj=system_u:system_r:syslogd_t:s0 key="what_touches_dev_log"
----
time->Mon Nov  6 10:37:33 2017
type=CONFIG_CHANGE msg=audit(1509982653.388:576): auid=4294967295 ses=4294967295 op=updated_rules path="/dev/log" key="what_touches_dev_log" list=4 res=1
----
time->Mon Nov  6 10:37:33 2017
type=PROCTITLE msg=audit(1509982653.388:577): proctitle=2F7573722F7362696E2F727379736C6F6764002D6E
type=PATH msg=audit(1509982653.388:577): item=1 name="/dev/log" inode=33327 dev=00:05 mode=0140711 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:devlog_t:s0 objtype=CREATE
type=PATH msg=audit(1509982653.388:577): item=0 name="/dev/" inode=3 dev=00:05 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:device_t:s0 objtype=PARENT
type=CWD msg=audit(1509982653.388:577):  cwd="/"
type=SOCKADDR msg=audit(1509982653.388:577): saddr=01002F6465762F6C6F67
type=SYSCALL msg=audit(1509982653.388:577): arch=c000003e syscall=49 success=yes exit=0 a0=3 a1=7ffebd8542e0 a2=a a3=2 items=2 ppid=1 pid=10693 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rsyslogd" exe="/usr/sbin/rsyslogd" subj=system_u:system_r:syslogd_t:s0 key="what_touches_dev_log"
Note You need to log in before you can comment on or make changes to this bug.