Description of problem: /var/log/audit.log is missing in recent builds. This makes it hard to debug selinux problems. It is around when the installer is launched.
<mburns> fabiand: i'm not seeing any libvirt logs... <fabiand> mburns, oh ... are you seeing audit.log? <mburns> no <fabiand> darn ... <mburns> hmm <fabiand> mburns, https://bugzilla.redhat.com/show_bug.cgi?id=953963 <mburns> actually, no logs at all <fabiand> mburns, I've seen this a while back ... maybe something with mounting changed .. <mburns> messages is empty <fabiand> mburns, messages can be due to journalctl <fabiand> mburns, does lsof | grep var/log give you something? <fabiand> mburns, It seems as if the daemons all start to write to the logfiles before we mount HostVG/Logging - so the logfiles are hidden when Logging is finally mounted <fabiand> mburns, if you comment out /var/log in fstab the logfiles appear <mburns> yes, i see ovirt.log getting written
There are two separate problems. First, the SELInux context is incorrect, which blocks logging from happening even if rsyslog is restarted. A fix for which is here: http://gerrit.ovirt.org/#/c/14662/ Secondly, there appears to be a problem with the way /var/log gets mounted, which is also present in 2.6.1. /var/log (appropriately) comes up early relatively early in the boot process, but the mount over from /var/lib/stateless/writable/var/log does not occur until after ovirt-post runs. Thus far, this is not correctable by RequiresMountFor=/var/log, running ovirt-early before rsyslog, issuing a "mount -a" from ovirt-early, etc. I'm currently checking the ordering/systemd debugging in 2.6.0 to find how mounting behaves when working in order to correct the second part of the problem.
Secondarily: http://gerrit.ovirt.org/#/c/14663/ There appears to be a race condition in service startup which delays visibility of /var/lib/stateless/writable/var/log. This is a temporary fix which makes sure that /var/log wins the race against rsyslog every time. I'll continue to investigate on Monday to find the root cause.
Ryan, Mike, I suppose we could solve this particular problem if we added the Logging entry to fstab at buildtime. But this is only a workaround to this particular problem. I wonder if it would help if we split out our custom persistence stuff and try to run it before the local-fs.target. Or if we avoid this and wait for the union mounts to land.
Fabian, We only need to get it mounted before auditd and rsyslog, both of which come up in multi-user. We shouldn't need to get into local-fs for this, but with the details of the problem somewhat clearer, I hope to have a better fix today.
Ryan, the fixes in gerrit look good. And yes, if it's only those two services that are affected then it's enough to adjust the order of execution.
The fix for this patch has been merged and is present in the released ovirt-node-3.0.0.