So are you saying if the host journald is from a version of systemd that ships with Fedora 24, a container built for centos7 or rhel 7 cannot read /var/log/journal mount from the Fedora 24 host?

So we need to build the fluentd image with the right version of systemd libraries it seems.  Or am I missing a subtly here?

(In reply to Peter Portante from comment #2)
> So are you saying if the host journald is from a version of systemd that
> ships with Fedora 24, a container built for centos7 or rhel 7 cannot read
> /var/log/journal mount from the Fedora 24 host?

Correct.

> So we need to build the fluentd image with the right version of systemd
> libraries it seems.  Or am I missing a subtly here?

Correct.  We need an EL7 fluentd image with EL7 systemd journal libraries that can read journal files created with Fedora 24 and later.  In order to do that, we need EL7 systemd journal libraries that can read journal files created with Fedora 24 and later.  Where can we get such libraries?

Let's hope journalctl is written in such a way that the vary latest version of the journal can read older versions of journald data.  If it can't, and we ship with the latest, we'll lock ourselves out entirely.

Can we build the container for fluentd local to the host on which it will be deployed so that we can pick up those compatible versions of the journalctl libraries?

RHEL7 version of sd-journal doesn't support LZ4 compressed journal files (now default on Fedora). We may need to backport this support to RHEL7. However I don't know how "backportable" those patches are.

I think that our systemd has the support for lz4, but we turn it off during compilation. And to be honest I am quite hesitant to turn it on. We had some issues with it in fedora and we will also break forward compatibility in rhel, since new journals would be automatically created with lz4 and older version of rhel, would not be able to read it.

(In reply to Lukáš Nykrýn from comment #6)
> I think that our systemd has the support for lz4, but we turn it off during
> compilation. And to be honest I am quite hesitant to turn it on. We had some
> issues with it in fedora and we will also break forward compatibility in
> rhel, since new journals would be automatically created with lz4 and older
> version of rhel, would not be able to read it.

That is a good point. Compression option in journald.conf is boolean option and compression algorithm is not configurable at runtime.

Isn't the problem here that journalctl can't *read* a compressed journal?  I don't think this is about turning on compression by default for what journald writes in RHEL 7.

(In reply to Peter Portante from comment #9)
> Isn't the problem here that journalctl can't *read* a compressed journal?  I
> don't think this is about turning on compression by default for what
> journald writes in RHEL 7.

Compression is turned on by default. If you compile journal with both lz4 and xz support then journalctl will be able to read both formats. However journald from such rpm package will start producing lz4 compressed journals that older RHEL7 releases can't read.

I think that one possible solution would be turning on lz4 across the board but I am not sure how viable this is (especially for RHEL-7.1 where we still have systemd-208).

I think better approach would be a small downstream patch, that would swap the order of xz and lz4 here:
https://github.com/lnykryn/systemd-rhel/blob/staging/src/journal/compress.h#L36

I guess that would work. However, I am not sure how I feel about such patch. We would then end up with different default compared to upstream.

(In reply to Peter Portante from comment #4)
> Let's hope journalctl is written in such a way that the vary latest version
> of the journal can read older versions of journald data.  If it can't, and
> we ship with the latest, we'll lock ourselves out entirely.
> 
> Can we build the container for fluentd local to the host on which it will be
> deployed so that we can pick up those compatible versions of the journalctl
> libraries?

I don't understand what you mean by "build the container for fluentd local to the host on which it will be deployed".  I don't understand what you mean by "pick up those compatible versions of the journalctl libraries".

I probably don't understand what I am talking about either! ;)

I was trying to communicate that on the host on which the fluentd pod will be deployed would be some version of systemd/journald that if we could link against that version at the time we deploy the fluentd pod, we would not have to link against the right version ahead of time.

Maybe to summarize:

The problem here is that rhel7 journal and all of its API are able to only process journal logs compressed with xz, but fedora uses lz4. So you are asking for support of lz4 in rhel7 :-)

Our problem is, that you can't choose compression algorithm, which journal will be using and if it knows lz4 it starts writing logs using that. So if we blindly turn on lz4 support in rhel7 it will mean that all new logs will be compressed with lz4 instead of xz and that might upset some customers.

But I just found out, that all of this does not matter much, since we don't build lz4 library for rhel7.

So if you really need to build your image on top of rhel7 you will have to start with a different bug for component "distribution" and ask for adding lz4 package.

The only way I think we could do this is to configure the host to use systemd-journal-remote to send to a fluentd http listener running in the container.

Feel free to close this bug if it isn't feasible for rhel7.

Can somebody explain why this is a problem with the default compression that is used when writing on RHEL 7 and not that this is about journalctl (and really the journald apis) being able to read lz4 journals on rhel 7 box?

This seems really simple: get a change in rhel 7 so that journals written using either xz or lz4 can be read by the journal handling library used by fluentd (and for that matter journalctl, rsyslog (via imjournal), etc.).

(In reply to Lukáš Nykrýn from comment #15)
> Maybe to summarize:
> 
> The problem here is that rhel7 journal and all of its API are able to only
> process journal logs compressed with xz, but fedora uses lz4. So you are
> asking for support of lz4 in rhel7 :-)

This seems like we have a good summary ... but ...

> Our problem is, that you can't choose compression algorithm, which journal
> will be using and if it knows lz4 it starts writing logs using that.

And here is where it seems we go off track.  Why are we talking about writing journals when we need to talk about reading them?

> So if we blindly turn on lz4 support in rhel7

Is the request to turn on lz4 support for writing journals in rhel7?  Seems like it is endow journalctl and its libraries with the ability to read lz4 journal files.

> it will mean that all new logs will
> be compressed with lz4 instead of xz and that might upset some customers.

Right, this is really bad, and nobody in this BZ seems to want that.  Well ... do they?

> But I just found out, that all of this does not matter much, since we don't
> build lz4 library for rhel7.

How hard would it be to build lz4 for RHEL 7?

> So if you really need to build your image on top of rhel7 you will have to
> start with a different bug for component "distribution" and ask for adding
> lz4 package.

Sounds like a path forward, as long as we get to the point where we talk about reading journals with lz4 and not writing them.  ;)

> > Our problem is, that you can't choose compression algorithm, which journal
> > will be using and if it knows lz4 it starts writing logs using that.
> 
> And here is where it seems we go off track.  Why are we talking about
> writing journals when we need to talk about reading them?

Again, current code does not support choosing between compression of algorithms. If you compile journal, so it is able to read lz4, it starts also writing logs with lz4. It is an implementation detail, that is not interesting for you, but we need to deal with it via some downstream patch.

(In reply to Lukáš Nykrýn from comment #19)
> If you compile journal, so it is able to read lz4, it starts
> also writing logs with lz4.

So what prevents us from filing an issue upstream against systemd, motivating the use case (which seems really obvious in a containerized environment), and working to get this patched upstream first?

I don't think there is any problem with the upstream, since there we want to push lz4 and upstream version is able to read the older journals.

Again only blocker here is the missing lz4 package in rhel.

fix merged to staging branch -> https://github.com/lnykryn/systemd-rhel/pull/169 -> post

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0711