Bug 2183546

Summary: Reenable LZ4 compression support in 9.2 journald
Product: Red Hat Enterprise Linux 9 Reporter: Scott Dodson <sdodson>
Component: systemdAssignee: Jan Macku <jamacku>
Status: CLOSED ERRATA QA Contact: Frantisek Sumsal <fsumsal>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.2CC: dtardon, eparis, jamacku, jwboyer, msekleta, systemd-maint-list, systemd-maint, walters, yuwatana
Target Milestone: rcKeywords: Patch, Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-252-15.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2184929 (view as bug list) Environment:
Last Closed: 2023-11-07 08:54:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2184929    

Description Scott Dodson 2023-03-31 15:07:15 UTC
Description of problem:
OCP Logging runs as a container and the latest currently supported version is based on RHEL 8 UBI and thus collects logs using `journalctl` from 8.6/8.7. systemd-journald in 9.2 cannot be configured to be compatible with journalctl from 8.6/8.7 without completely disabling compression.


Version-Release number of selected component (if applicable):
systemd-252


How reproducible:
100%


Steps to Reproduce:
1. Install RHEL 9.2
2. Run journalctl from within an 8.x container
3.

Actual results:
systemd-journald 252 cannot be configured to enable compatibility with 8.x journalctl without fully disabling compression which would significantly increase disk utilization and is undesirable.

Expected results:
systemd-journald 252 can be configured or defaults to LZ4 compression and can be configured such that 8.x journalctl can read the on disk format. We know that this will require the existing environment variables.

Environment="SYSTEMD_JOURNAL_COMPACT=0"
Environment="SYSTEMD_JOURNAL_KEYED_HASH=0"



Additional info:
We're requesting this as a transitional state, RHEL 9.3 and later can revert back to ZSTD being the only supported compression. By the time OCP uses 9.3 or later for RHCOS we expect all containers to have moved to UBI9.

Comment 3 Colin Walters 2023-03-31 21:12:09 UTC
I typed this up, it compiles, haven't tested it:

diff --git a/src/libsystemd/sd-journal/journal-file.c b/src/libsystemd/sd-journal/journal-file.c
index 8da4ca9cad..2fcc9157ae 100644
--- a/src/libsystemd/sd-journal/journal-file.c
+++ b/src/libsystemd/sd-journal/journal-file.c
@@ -336,6 +336,22 @@ static bool compact_mode_requested(void) {
         return true;
 }
 
+static Compression compression_requested(void) {
+        const char *alg;
+        Compression ret;
+
+        alg = getenv("SYSTEMD_JOURNAL_COMPRESS");
+        if (alg == NULL)
+                return DEFAULT_COMPRESSION;
+
+        ret = compression_from_string(alg);
+        if (ret == _COMPRESSION_INVALID) {
+                log_debug("Failed to parse SYSTEMD_JOURNAL_COMPRESS value, ignoring: %s", alg);
+                return DEFAULT_COMPRESSION;
+        }
+        return ret;
+}
+
 static int journal_file_init_header(
                 JournalFile *f,
                 JournalFileFlags file_flags,
@@ -1626,7 +1642,7 @@ static Compression maybe_compress_payload(JournalFile *f, uint8_t *dst, const ui
 
 #if HAVE_COMPRESSION
         if (JOURNAL_FILE_COMPRESS(f) && size >= f->compress_threshold_bytes) {
-                compression = compress_blob(src, size, dst, size - 1, rsize);
+                compression = compress_blob_explicit(f->compression, src, size, dst, size - 1, rsize);
                 if (compression > 0)
                         log_debug("Compressed data object %"PRIu64" -> %zu using %s",
                                   size, *rsize, compression_to_string(compression));
@@ -3785,6 +3801,7 @@ int journal_file_open(
                 .fd = fd,
                 .mode = mode,
                 .open_flags = open_flags,
+                .compression = compression_requested(),
                 .compress_threshold_bytes = compress_threshold_bytes == UINT64_MAX ?
                                             DEFAULT_COMPRESS_THRESHOLD :
                                             MAX(MIN_COMPRESS_THRESHOLD, compress_threshold_bytes),
diff --git a/src/libsystemd/sd-journal/journal-file.h b/src/libsystemd/sd-journal/journal-file.h
index 70d2276ced..a14f4ea51f 100644
--- a/src/libsystemd/sd-journal/journal-file.h
+++ b/src/libsystemd/sd-journal/journal-file.h
@@ -100,6 +100,7 @@ typedef struct JournalFile {
 
         unsigned last_seen_generation;
 
+        Compression compression;
         uint64_t compress_threshold_bytes;
 #if HAVE_COMPRESSION
         void *compress_buffer;

Comment 4 David Tardon 2023-04-03 08:39:37 UTC
(In reply to Colin Walters from comment #3)
> I typed this up, it compiles, haven't tested it:

This looks like the right way. And it should probably be done upstream, as it looks like an omission. I.e., we have env. vars to turn off other incompatible features, but none to force a particular compression algorithm.

(That said, I'm pretty sure the patch in its current form doesn't work. In particular, setting of .incompatible_flags in journal_file_init_header() needs to be updated too.)

Comment 9 Colin Walters 2023-04-03 13:29:00 UTC
> This looks like the right way. And it should probably be done upstream, as it looks like an omission. I.e., we have env. vars to turn off other incompatible features, but none to force a particular compression algorithm.

Right, agree.  Would someone mind taking this ball and doing a PR upstream?  I have to chase a different bug.

> (That said, I'm pretty sure the patch in its current form doesn't work. In particular, setting of .incompatible_flags in journal_file_init_header() needs to be updated too.)

Ah yes, indeed.

Comment 11 Yu Watanabe 2023-04-04 06:55:44 UTC
Upstream patch: https://github.com/systemd/systemd/pull/27126

Comment 19 Plumber Bot 2023-05-16 09:57:43 UTC
fix merged to github main branch -> https://github.com/redhat-plumbers/systemd-rhel9/pull/159

Comment 26 errata-xmlrpc 2023-11-07 08:54:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (systemd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6640