Bug 1899905

Summary: elasticsearch-rollover and elasticsearch-delete pods are in error states with 'ValueError: No JSON object could be decoded'
Product: OpenShift Container Platform Reporter: Yogiraj Karajagi <ykarajag>
Component: LoggingAssignee: Hui Kang <hkang>
Status: CLOSED ERRATA QA Contact: Qiaoling Tang <qitang>
Severity: medium Docs Contact: Rolfe Dlugy-Hegwer <rdlugyhe>
Priority: unspecified    
Version: 4.5CC: alchan, andcosta, anli, aos-bugs, apjagtap, asheth, cruhm, florian.hauser, hkang, mnoguera, naoto30, naygupta, nnosenzo, rabdulra, rdlugyhe, rsandu, tmicheli
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: logging-exploration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
* Previously, elasticsearch-rollover and elasticsearch-delete pods remained in error states with `Invalid JSON:` or 'ValueError: No JSON object could be decoded'. This exception was raised because there was no exception handler for invalid JSON input. The current release fixes this issue by providing a handler for invalid JSON input. As a result, it outputs an error message instead of an exception traceback, and the elasticsearch-rollover and elasticsearch-delete jobs should not remain that error state. (link:https://bugzilla.redhat.com/show_bug.cgi?id=1899905[*BZ#1899905*])
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 11:22:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1901756    

Description Yogiraj Karajagi 2020-11-20 11:04:03 UTC 
Description of problem:
elasticsearch-rollover and elasticsearch-delete pods are in error states with 'ValueError: No JSON object could be decoded'
 
~~~
$ oc get pods  | grep Error
elasticsearch-delete-app-1604990700-m5tzq       0/1     Error       0          6d7h
elasticsearch-delete-audit-1605496500-69cc2     0/1     Error       0          11h
elasticsearch-delete-infra-1605519000-lzngc     0/1     Error       0          4h45m
elasticsearch-rollover-app-1605261600-hk5nq     0/1     Error       0          3d4h
elasticsearch-rollover-audit-1605097800-bj4vm   0/1     Error       0          5d1h
elasticsearch-rollover-infra-1605372300-877hc   0/1     Error       0          45h
 
$ oc logs jobs/elasticsearch-delete-app-1604990700
 
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
 
 
$ oc logs jobs/elasticsearch-delete-infra-1605519000
 
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
~~~
 
 
Version-Release number of selected component (if applicable):
4.5
 
How reproducible:
NA
 
Actual results:
elasticsearch-rollover and elasticsearch-delete pods failing with 'ValueError: No JSON object could be decoded'
 
Expected results:
elasticsearch-rollover and elasticsearch-delete jobs should complete successfully.
 
Additional info:
 
The cluster is recently upgraded to 4.5.16. Sometimes jobs run successfully.
 
~~~
$ oc -n openshift-logging get jobs
NAME                                      COMPLETIONS   DURATION   AGE
curator-1605497400                        1/1           4s         10h
elasticsearch-delete-app-1604990700       0/1           6d7h       6d7h
elasticsearch-delete-app-1605535200       1/1           8s         13m
elasticsearch-delete-audit-1605496500     0/1           10h        10h
elasticsearch-delete-audit-1605535200     1/1           8s         13m
elasticsearch-delete-infra-1605519000     0/1           4h43m      4h43m
elasticsearch-delete-infra-1605535200     1/1           3s         13m
elasticsearch-rollover-app-1605261600     0/1           3d4h       3d4h
elasticsearch-rollover-app-1605535200     1/1           8s         13m
elasticsearch-rollover-audit-1605097800   0/1           5d1h       5d1h
elasticsearch-rollover-audit-1605535200   1/1           8s         13m
elasticsearch-rollover-infra-1605372300   0/1           45h        45h
elasticsearch-rollover-infra-1605535200   1/1           3s         13m
~~~
Comment 1 Florian Hauser 2020-11-22 08:31:19 UTC 
Same issue here - started with 4.5.16 and still occurs on 4.5.19.
Comment 3 Qiaoling Tang 2020-12-10 00:57:31 UTC 
Tested with elasticsearch-operator.4.7.0-202012080225.p0, the error message is changed to `Invalid JSON: ''`.
Comment 8 Hui Kang 2021-01-14 14:58:12 UTC 
The fix has been cherry picked to ocp 4.6
Comment 10 Hui Kang 2021-01-26 15:11:52 UTC 
@cam, I will cherry pick to 4.5 as well.
Comment 19 errata-xmlrpc 2021-02-24 11:22:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Errata Advisory for Openshift Logging 5.0.0), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0652