Bug 1112285 (CVE-2014-4616)

Summary:	CVE-2014-4616 python: missing boundary check in JSON module
Product:	[Other] Security Response	Reporter:	Vasyl Kaigorodov <vkaigoro>
Component:	vulnerability	Assignee:	Red Hat Product Security <security-response-team>
Status:	CLOSED ERRATA	QA Contact:
Severity:	medium	Docs Contact:
Priority:	medium
Version:	unspecified	CC:	a.badger, abaron, adev88, amcnabb, aortega, apevec, ayoung, bkabrda, bkearney, carnil, chrisw, cpelland, cperry, dallan, derks, dmalcolm, extras-orphan, fdc, fschwarz, gkotton, gmollett, ivazqueznet, jberan, jeffrey.ness, jonathansteffan, jorton, jrusnack, katello-bugs, katzj, kylev, lewk, lhh, lpeer, markmc, mmaslano, mmccune, ncoghlan, python-maint, rbryant, rhos-maint, sclewis, srevivo, tjay, tomspur, tradej
Target Milestone:	---	Keywords:	Security
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	python 2.7.7, python 3.3.6, python 3.4.1	Doc Type:	Bug Fix
Doc Text:	A flaw was found in the way the json module handled negative index argument passed to certain functions (such as raw_decode()). An attacker able to control index value passed to one of the affected functions could possibly use this flaw to disclose portions of the application memory.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-11-03 21:09:24 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1112293, 1112294, 1112903, 1115517, 1115518, 1115519, 1187779, 1206574, 1222534
Bug Blocks:	1112298, 1112724, 1210268

Description Vasyl Kaigorodov 2014-06-23 13:47:14 UTC

It was reported [1] that Python built-in _json module have a flaw (insufficient bounds checking), which allows a local user to read current process' arbitrary memory.

Quoting the upstream bug report:
...
The sole prerequisites of this attack are that the attacker is able to
control or influence the two parameters of the default scanstring
function: the string to be decoded and the index.

The bug is caused by allowing the user to supply a negative index
value. The index value is then used directly as an index to an array
in the C code; internally the address of the array and its index are
added to each other in order to yield the address of the value that is
desired. However, by supplying a negative index value and adding this
to the address of the array, the processor's register value wraps
around and the calculated value will point to a position in memory
which isn't within the bounds of the supplied string, causing the
function to access other parts of the process memory.
...

Upstream bug report with additional technical details:
[1] http://bugs.python.org/issue21529

Upstream commits (taken from upstream tracker):

2.7: http://hg.python.org/cpython/rev/50c07ed1743d
3.1: http://hg.python.org/cpython/rev/a8facac493ef
3.2: http://hg.python.org/cpython/rev/8130b8c06606
3.3: http://hg.python.org/cpython/rev/4f15bd1ab28f
3.4: http://hg.python.org/cpython/rev/7b95540ced5c
3.5: http://hg.python.org/cpython/rev/3a414c709f1f

Comment 1 Vasyl Kaigorodov 2014-06-23 14:02:53 UTC

Created python tracking bugs for this issue:

Affects: fedora-all [bug 1112293]

Comment 2 Vasyl Kaigorodov 2014-06-23 14:02:57 UTC

Created python3 tracking bugs for this issue:

Affects: fedora-all [bug 1112294]

Comment 3 Tomas Hoger 2014-06-23 14:28:16 UTC

HackerOne report:
https://hackerone.com/reports/12297

Comment 4 Vincent Danen 2014-06-24 13:37:43 UTC

This was assigned CVE-2014-4616: http://openwall.com/lists/oss-security/2014/06/24/7

Comment 5 Tomas Hoger 2014-06-24 14:38:51 UTC

The original report suggests that this is a high severity issue because of "ubiquity of JSON parsing in Python applications".  However, practical impact seems rather limited.  To trigger the issue, negative value is passed as idx argument of raw_decode().  However, this argument is not documented in the Python standard library documentation, which currently says:

  raw_decode(s)

    Decode a JSON document from s (a str or unicode beginning with a JSON
    document) and return a 2-tuple of the Python representation and the index
    in s where the document ended.

    This can be used to decode a JSON document from a string that may have
    extraneous data at the end.

  https://docs.python.org/2/library/json.html#json.JSONDecoder.raw_decode
  https://docs.python.org/3/library/json.html#json.JSONDecoder.raw_decode

Therefore, the idx argument for the function is rarely used.  I've not been able to identify any application that could pass arbitrary negative value as idx.  The only use of the idx argument uses the following pattern from decode(), which is used to strip leading spaces from the input string:

  WHITESPACE = re.compile(r'\s*', FLAGS)
  _w=WHITESPACE.match
  raw_decode(s, idx=_w(s, 0).end())

This does not allow passing arbitrary negative index value to raw_decode().

It should be noted that the json module that is part of Python standard library is based on simplejson, currently using version 2.0.9.  The json module was added to the Python standard library in version 2.6, therefore python packages in Red Hat Enterprise Linux 5 (using python 2.4.3) and older are not affected by this issue.  The python packages in Red Hat Enterprise Linux 6 (using python 2.6.6) use json module based on simplejson 1.9.  The code affected by this issue was only introduced later, and added to Python standard library as part of this rebase from 1.9 to 2.0.9:

http://hg.python.org/cpython/rev/52199
http://bugs.python.org/issue4136

Therefore python packages in Red Hat Enterprise Linux 6 are not affected.  There are also python-simplejson packages shipped in Red Hat Enterprise Linux 5 and 6 (version 2.0.9), which are affected by this issue.

Recent upstream versions of simplejson (such as the version used in Red Hat Software Collections 1) also mitigate this issue by not allowing arbitrary negative index in raw_decode().  Starting with version 2.6.1 (based on upstream git repository tags), raw_decode() also performs stripping of leading spaces, which deals with negative index as a side effect:

https://github.com/simplejson/simplejson/commit/0fb0aea

However, it is possible to reproduce the issue by calling (undocumented) scan_once() function directly.  This problem was reported to simplejson upstream via:

https://github.com/simplejson/simplejson/issues/98

For the sake of completeness, API documentation of current upstream simplejson versions document idx argument for the raw_decode() function:

  raw_decode(s[, idx=0])

    Decode a JSON document from s (a str or unicode beginning with a JSON
    document) starting from the index idx and return a 2-tuple of the Python
    representation and the index in s where the document ended.

    This can be used to decode a JSON document from a string that may have
    extraneous data at the end, or to decode a string that has a series of
    JSON objects.

    JSONDecodeError will be raised if the given JSON document is not valid.

  http://simplejson.readthedocs.org/en/latest/#simplejson.JSONDecoder.raw_decode

Comment 7 Tomas Hoger 2014-06-24 19:21:08 UTC

(In reply to Tomas Hoger from comment #5)
> However, it is possible to reproduce the issue by calling (undocumented)
> scan_once() function directly.  This problem was reported to simplejson
> upstream via:
> 
> https://github.com/simplejson/simplejson/issues/98

simplejson fixed upstream in version 3.5.3 via the following commit:

https://github.com/simplejson/simplejson/commit/b7486b8

Comment 8 Luke Macken 2014-06-24 23:05:44 UTC

(In reply to Tomas Hoger from comment #7)
> (In reply to Tomas Hoger from comment #5)
> > However, it is possible to reproduce the issue by calling (undocumented)
> > scan_once() function directly.  This problem was reported to simplejson
> > upstream via:
> > 
> > https://github.com/simplejson/simplejson/issues/98
> 
> simplejson fixed upstream in version 3.5.3 via the following commit:
> 
> https://github.com/simplejson/simplejson/commit/b7486b8

I just built python-simplejson-3.5.3 and queued it up for testing. https://admin.fedoraproject.org/updates/python-simplejson-3.5.3-1.fc20

Comment 11 Vincent Danen 2014-06-27 14:17:47 UTC

This has been fixed in Fedora for python-simplejson:

https://lists.fedoraproject.org/pipermail/package-announce/2014-June/134761.html

Comment 12 Fedora Update System 2014-06-30 10:31:42 UTC

python-2.7.5-13.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 14 Tomas Hoger 2014-07-02 13:45:28 UTC

Statement:

This issue affects the versions of python as shipped with Red Hat Enterprise Linux 7, the versions of python-simplejson as shipped with Red Hat Enterprise Linux 5 and 6, and the versions of python33-python and python33-python-simplejson as shipped with Red Hat Software Collections. Red Hat Product Security has rated this issue as having Moderate security impact. Future updates may address this issue. For additional information, refer to the Issue Severity Classification: https://access.redhat.com/security/updates/classification/.

Comment 15 Tomas Hoger 2014-07-02 13:59:37 UTC

Created python26-simplejson tracking bugs for this issue:

Affects: epel-5 [bug 1115518]

Comment 16 Tomas Hoger 2014-07-02 13:59:42 UTC

Created python-simplejson tracking bugs for this issue:

Affects: fedora-all [bug 1115517]
Affects: epel-7 [bug 1115519]

Comment 17 Fedora Update System 2014-07-17 04:28:11 UTC

python3-3.3.2-9.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 18 Fedora Update System 2014-07-17 04:29:28 UTC

python-2.7.5-13.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2014-11-09 15:45:14 UTC

python3-3.3.2-18.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2014-11-10 06:36:22 UTC

python3-3.4.1-16.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2014-11-13 18:14:22 UTC

python3-3.3.2-10.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 25 errata-xmlrpc 2015-06-04 08:29:54 UTC

This issue has been addressed in the following products:

  Red Hat Software Collections for Red Hat Enterprise Linux 7
  Red Hat Software Collections for Red Hat Enterprise Linux 6
  Red Hat Software Collections for Red Hat Enterprise Linux 6.5 EUS
  Red Hat Software Collections for Red Hat Enterprise Linux 6.6 EUS

Via RHSA-2015:1064 https://rhn.redhat.com/errata/RHSA-2015-1064.html

Comment 26 errata-xmlrpc 2015-11-19 12:42:43 UTC

This issue has been addressed in the following products:

  Red Hat Enterprise Linux 7

Via RHSA-2015:2101 https://rhn.redhat.com/errata/RHSA-2015-2101.html