Bug 1685243

We have a partial solution in https://github.com/ViaQ/fluent-plugin-viaq_data_model/commit/8b5ef11cedec4c372b2cb082afc7f9cc08473654 and https://github.com/ViaQ/fluent-plugin-viaq_data_model/commit/d204d2fe732fc26201302c8fa466f20cc335e517

I would encourage you to read the README about undefined field handling and why we had to set MERGE_JSON_LOG=false by default, because of the serious problems which result.

The shared environments such as Online and Dedicated are difficult to handle.  Not only are those the environments where we see the most problems with MERGE_JSON_LOG=true because of the many shared applications logging conflicting fields, but also those are the most difficult to remediate.  Setting MERGE_JSON_LOG affects _all_ customers and _all_ applications.  What we really need is the ability to set MERGE_JSON_LOG on a per-customer _and_ per-application basis.  We also need the ability to configure the viaq undefined field handling on a per-customer _and_ per-application basis.  The problem is that we have no way to tell fluentd about customers and applications.

One possible solution is that the customer would be responsible for labeling or annotating their pods and namespaces with MERGE_JSON_LOG, and then with specific viaq filter settings for MERGE_JSON_LOG=true.  This would require quite a bit of work in the viaq filter and in the logging fluentd configuration, and would likely have a negative impact on performance.

Hi Rich,
I realize your proposed solution involves installing a fluentd plugin and configuring specific ViaQ filter settings, however, I'm not sure the SRE team will allow this as they generally discourage managing snowflake clusters.  

That being said, what other options exist for the customer aside from enabling the MERGE_JSON_LOG parameter and running into data and/or performance issues.  They are running on a dedicated cluster and are the only consumers of it.  All applications running on that cluster belong to the same customer.

(In reply to Daniel Del Ciancio from comment #4)
> Hi Rich,
> I realize your proposed solution involves installing a fluentd plugin and
> configuring specific ViaQ filter settings, however, I'm not sure the SRE
> team will allow this as they generally discourage managing snowflake
> clusters.  
> 
> That being said, what other options exist for the customer aside from
> enabling the MERGE_JSON_LOG parameter and running into data and/or
> performance issues.  They are running on a dedicated cluster and are the
> only consumers of it.  All applications running on that cluster belong to
> the same customer.

Then I think it should be safe to use MERGE_JSON_LOG=true in this case.  But the customer will need to closely monitor their fluentd logs to make sure they don't see any strange errors, which would be caused by the undefined fields created by MERGE_JSON_LOG=true.

(In reply to Rich Megginson from comment #5)
> (In reply to Daniel Del Ciancio from comment #4)
> > Hi Rich,
> > I realize your proposed solution involves installing a fluentd plugin and
> > configuring specific ViaQ filter settings, however, I'm not sure the SRE
> > team will allow this as they generally discourage managing snowflake
> > clusters.  
> > 
> > That being said, what other options exist for the customer aside from
> > enabling the MERGE_JSON_LOG parameter and running into data and/or
> > performance issues.  They are running on a dedicated cluster and are the
> > only consumers of it.  All applications running on that cluster belong to
> > the same customer.
> 
> Then I think it should be safe to use MERGE_JSON_LOG=true in this case.  But
> the customer will need to closely monitor their fluentd logs to make sure
> they don't see any strange errors, which would be caused by the undefined
> fields created by MERGE_JSON_LOG=true.

As you're aware, I've raised this to the OSD BU (and CC'ed you and Jeff on that email).  I suspect that if this parameter was to be re-enabled, then the SRE team could implement some monitoring script that could scan the fluentd logs for any "undefined field" errors.  There would also need to be some procedure in place to be able to deal with these errors should they arise.

I'll have the BU decide on the next steps.

Thanks!

https://github.com/openshift/origin-aggregated-logging/pull/1554

merged upstream https://github.com/openshift/origin-aggregated-logging/commit/3ba5aead07f9d451385bec38561e634863fd914c

The fix is implemented in rubygem-fluent-plugin-viaq_data_model-0.0.18-1.el7 - please verify that the fluentd image has this version (or later).

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0636