Bug 1917847

@lukas can you advise the workaround here.  I presume its possible to either update the config or modify/add a template to expand the field limit with the full knowledge about the impact this will have on ES

Hi,

the "workaround" is to increase the soft limit on number of allowed fields per index. Either via manual command updating the settings against specific index (or all applicable indices at the time of the command execution) or/and by adding index template to setup filed limit to a new value for new indices.

And all this has been already pointed out in several resources above.
I can add quick link to similar external resource like: https://stackoverflow.com/questions/55372330/what-does-limit-of-total-fields-1000-in-index-has-been-exceeded-means-in

We can elaborate more specific instructions if needed (aka RHKB).
That was for the immediate workaround.

But, the point is that increasing the limit itself may not be a good long term solution.
We need to learn more about the data being indexed into that index. If I understood correctly the audit log level was configured to the highest granularity possible (see comment #1) and we need to understand what this means for the data - how the data model is impacted by this. The chance is that it may be better to exclude some portion of the data from indexing if there is no value in the data in this context.

More specifically, should we record all logs and all log fields when "AllRequestBodies" profile is used? See https://docs.openshift.com/container-platform/4.6/security/audit-log-policy-config.html

- If yes, we need to find what value the field limit needs to be set to (right now I do not know what the new field limit should be, i.e. is 2000 enough?)
- If not, then we need to decide which parts of the data we can ignore (i.e. which fields do not have to be indexed)

In both cases we need to update our index templates.

Lukáš

Hello,

thank you for taking care of this issue. 

We tried to always increae "total_fields_limit", but we needed to do this for each new index which was kind of annoying because it was a rather manual process and we have many indexes.

E.g. a max limit of 2000 was not enough at some point.

As for the data, from our point of view, we need all of it since auditing is very important for us.


Kind regards,

David

David,

I believe you should be able to push custom index template into ES to override just the setting for the total field limit.
See https://www.elastic.co/guide/en/elasticsearch/reference/6.8/mapping.html#mapping-limit-settings
See https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html

(You will want to make the index template to match only specific indices, like the one matching "audit-*" index name template, ie. do not make index template to match all indices)

Let me know if you need more details we can turn this into documentation update.

Lukáš

I'm tracking a doc task related to this engineering bz in https://issues.redhat.com/browse/RHDEVDOCS-2888

I have closed https://issues.redhat.com/browse/RHDEVDOCS-2888. Please notify me if and when this issue requires changes to the OpenShift documentation.

I think the default here is too low - a vanilla cluster will quite fast reach this w/o any workload, if you add a couple of operators like knative, gitops (argo) etc

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days