Bug 1918876
| Summary: | Kibana message: "Payload content length greater than maximum allowed" | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Oscar Casal Sanchez <ocasalsa> |
| Component: | Logging | Assignee: | ewolinet |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | medium | Docs Contact: | Rolfe Dlugy-Hegwer <rdlugyhe> |
| Priority: | medium | ||
| Version: | 4.5 | CC: | aguadarr, aos-bugs, ewolinet |
| Target Milestone: | --- | Flags: | ocasalsa:
needinfo-
|
| Target Release: | 4.7.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | logging-exploration | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
* Previously, the code that uses environment variables to replace values in the Kibana configuration file did not consider commented lines. This prevented users from overriding the default value of server.maxPayloadBytes. The current release fixes this issue by uncommenting the default value of server.maxPayloadByteswithin. Now, users can override the value by using environment variables, as documented. (link:https://bugzilla.redhat.com/show_bug.cgi?id=1918876[*BZ#1918876*])
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-24 11:22:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1925626 | ||
The shortterm work around to this issue is to go unmanaged and add the needed env var to the Kibana deployment Hello Jeff,
I was testing it on a cluster like this:
1. Moved the CLO to Unmanaged
~~~
$ oc edit clusterlogging instance
...
spec
(...)
managementState: Unmanaged
~~~
2. Moved Kibana to Unmanaged
~~~
$ oc edit kibana Kibana
...
spec
(...)
managementState: Unmanaged
~~~
3. Edited kibana deployment:
~~~
$ oc edit deployment.apps/kibana
(...)
- env:
- name: SERVER_MAXPAYLOADBYTES
value: "2097154"
~~~
4. Checking if this value was taken by Kibana, it appears this value in the configuration file, but, it's not taking efect since the line is starting by '#'
~~~
$ oc rsh <kibana pod>
$ grep -i maxpayload config/*
config/kibana.yml:#server.maxPayloadBytes: 2097154
config/kibana_new.yml:#server.maxPayloadBytes: 1048576
~~~
Then, it seems that it's not possible to configure it in Unmanaged status since a different issue is happening with the replacement of the variable here [1] where the code is replacing the value of the variable, but not deleting the '#' symbol that it's by default in the line in the kibana.yaml
sed -i "s,${kibana_var}:.*,${kibana_var}:\ ${value}," "${conf_file}"
Do you want to open a new bug for this second issue or would it managed in this one?
[1] https://github.com/openshift/origin-aggregated-logging/blob/570778764d3b64a1ca73f8a3b2bd67ff0a65688a/kibana/utils#L75
Verified on elasticsearch-operator.5.0.0-56. The SERVER_MAXPAYLOADBYTES can be changed as Env. Hello, The PR in this Bugzilla is for allowing to move to Unmanaged and being able to modify the SERVER_MAXPAYLOADBYTES value. This is great, but should a different Bugzilla be created for the definitive fix where it's not needed to move to "Unmanaged"? Best regards, Oscar Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Errata Advisory for Openshift Logging 5.0.0), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0652 Version 5.2.2-21 still happen The workaround mentioned doesn't work OCP 4.7.15 @ocasalsa |
[Description of problem] When using Kibana, it's possible to read the message: "Payload content length greater than maximum allowed" [Version-Release number of selected component (if applicable):] OCP 4.5.7 [How reproducible:] The message error is similar to the redacted in this KCS [1] where it has a workaround for OCP 3.11 modifying the Kibana DC doing: ~~~ $ oc edit dc/logging-kibana ... - env: - name: SERVER_MAXPAYLOADBYTES value: "2097154" ... ~~~ But, in OCP 4.x, it's not possible to modify this value being affected the user by it. [Actual results] The Kibana error "Payload content length greater than maximum allowed" is showed. [Expected results] No error and it's possible to work with Kibana [Additional info] It seems that Kibana has by default a server.maxPayloadBytes defined by default. When this limit is reached, it throws the error showed above. Usually, it's easier to workaround this issue if you can modify the kibana config, but in OCP 4.x, it's not possible since the operator won't allow this change in a "Managed" status. [1] https://access.redhat.com/solutions/4848311