Bug 1388956

Description of problem:
In THEL 7.2 auditd stopped sending audit messages to central syslog and requires a restart before work again. 

Version-Release number of selected component (if applicable):

* Release 
Linux <NAME> 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

* Package 
rsyslog-7.4.7-12.el7.x86_64                                 Tue Dec 15 09:02:02 2015
rsyslog-relp-7.4.7-12.el7.x86_64                            Tue Dec 15 09:02:02 2015

How reproducible: EASILY

Steps to Reproduce:

1) - Enable the audisp syslog plugin
   
2)   - Reboot

3)   - Write audit messages with audit -m

4)   - Audit messages are not received on rsyslog


Actual results:
Service does not start manually and requires intervention to start

Expected results: 
For the service to boot normally


Additional info:

We can see in the file '10-central_syslog.conf' that the customer is sending messages to the remote server using the RELP protocol  (http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html)
This filters only messages from itself and not from the 'ansible' executable should be sent to the remote syslog server
If we look in the strace, we can see that auditd receives the test messages and writes it to the auditd log. Then rsyslog receives this message from the audispd daemon, notice the field msg='This is a test message...' 
And rsyslog calls the sendto() function to send the message across the network socket to be submitted over the network via RELP to the syslog server.


This is a reproducible issue on my RHEL7 system configured with the audisp syslog plugin enabled(and sending remote logs via the omrelp rsyslog module, but this is not relevant to the actual root issue). The problem is that when using the default unit file, auditd does not initialize the audisp plugins at boot time. It looks like a timing issue with systemd and auditd preventing audit from calling initializing the audispd plugins or some failure in this codepath, I suspect because this initialization code is run too early on in the boot process.

In the auditd main() function we attempt to try calling init_dispatcher() at startup

    /* Init the event handler thread */
    write_pid_file();
    if (init_event(&config)) {
        if (pidfile)
            unlink(pidfile);
        tell_parent(FAILURE);
        return 1;
    }

    if (init_dispatcher(&config)) {
        if (pidfile)
            unlink(pidfile);
        tell_parent(FAILURE);
        return 1;
    }


When auditd is started correctly, the audispd dispatcher is initialized by auditd which calls into the audispd code to initialize the plugins. This is evident when the service is restarted, we will see the below messages present in the logs:

    Sep 20 10:47:51 rhel7-sssd-ad.jstephen.local auditd[2002]: Started dispatcher: /sbin/audispd pid: 2006
    Sep 20 10:47:51 rhel7-sssd-ad.jstephen.local audispd: syslog plugin initialized
    Sep 20 10:47:51 rhel7-sssd-ad.jstephen.local audispd: audispd initialized with q_depth=150 and 2 active plugins
    Sep 20 10:47:51 rhel7-sssd-ad.jstephen.local audispd: node=rhel7-sssd-ad.jstephen.local type=DAEMON_START msg=audit(1474382871.980:792): auditd start, ver=2.4.1 format=raw kernel=3.10.0-327.28.3.el7.x86_64 auid=4294967295 pid=2002 res=success
    Sep 20 10:47:51 rhel7-sssd-ad.jstephen.local kernel: [ 1041.873306] type=1305 audit(1474382871.984:211): audit_enabled=1 old=1 auid=4294967295 ses=4294967295 res=1
    Sep 20 10:47:51 rhel7-sssd-ad.jstephen.local kernel: [ 1041.873314] type=1305 audit(1474382871.984:212): audit_pid=2002 old=0 auid=4294967295 ses=4294967295 res=1
    Sep 20 10:47:51 rhel7-sssd-ad.jstephen.local auditd[2002]: Init complete, auditd 2.4.1 listening for events (startup state enable)

At boot time we do not see these messages however. Restarting the auditd service calls this initialization code which works correctly to start the audispd syslog plugin.

We need to investigate further why at boot time auditd is not initializing audispd. That said, I tested a workaround on my system which delays the startup of auditd and causes the audispd initialization to succeed:

    # cat /etc/systemd/system/auditd.service 
    [Unit]
    Type=idle
    Description=Security Auditing Service
    DefaultDependencies=no
    After=local-fs.target systemd-tmpfiles-setup.service rsyslog.service
    Conflicts=shutdown.target
    Before=sysinit.target shutdown.target
    RefuseManualStop=yes
    ConditionKernelCommandLine=!audit=0

    [Service]
    ExecStart=/sbin/auditd -n
    ## To not use augenrules, copy this file to /etc/systemd/system/auditd.service
    ## and comment/delete the next line and uncomment the auditctl line.
    ## NOTE: augenrules expect any rules to be added to /etc/audit/rules.d/
    ExecStartPost=-/sbin/augenrules --load
    #ExecStartPost=-/sbin/auditctl -R /etc/audit/audit.rules
    ExecReload=/bin/kill -HUP $MAINPID

    [Install]
    WantedBy=multi-user.target