Bug 1417838 - Posibility to turn off certification verification in python 2.7.13
Summary: Posibility to turn off certification verification in python 2.7.13
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Software Collections
Classification: Red Hat
Component: python
Version: python27
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 2.4
Assignee: Tomas Orsava
QA Contact: Branislav Náter
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-31 09:03 UTC by Branislav Náter
Modified: 2017-04-26 10:21 UTC (History)
3 users (show)

Fixed In Version: python27-python-2.7.13-3.el7 python27-python-2.7.13-3.el6
Doc Type: Release Note
Doc Text:
SSL/TLS certificate verification for the HTTP clients in the Python standard library have been enabled by default and new configuration options were added New per-application and per-process configuration options for SSL/TLS certificate verification have been added for the HTTP clients in the Python standard library. The options are described in the 493 Python Enhancement Proposal (https://www.python.org/dev/peps/pep-0493/). In addition, the default global setting was changed to be to verify certificates. Customers that opted out using the file-based configuration will not be affected. For details, see https://access.redhat.com/articles/2039753.
Clone Of:
Environment:
Last Closed: 2017-04-26 10:21:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:1162 0 normal SHIPPED_LIVE Moderate: python27 security, bug fix, and enhancement update 2017-04-26 14:18:34 UTC

Description Branislav Náter 2017-01-31 09:03:14 UTC
Description of problem:
It is possible to turn off hostname and certificate verification in previous version of rhscl python27 (python27-python-2.7.8-18.el6), see BZ#1311044.
It's not possible to do this in rebased python27 (python27-python-2.7.13-1.el6) and certification verification is turned on always. I believe customer should have this option even in new python.

Version-Release number of selected component (if applicable):
python27-python-2.7.13-1.el6

How reproducible:
always by running /CoreOS/python/Regression/bz1219108-python-backport-PEP-476-with-global-toggle-to test.

Actual results:
:: [  BEGIN   ] :: Set certificate and hostname verification to "disable" :: actually running 'sed -i 's/verify=.*/verify=disable/' /opt/rh/python27/root/etc/python/cert-verification.cfg'
:: [   PASS   ] :: Set certificate and hostname verification to "disable" (Expected 0, got 0)
:: [  BEGIN   ] :: HTTPS - no certificate verification, different hostname :: actually running 'python test.py localhost'
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    c1.request("GET", "/")
<snip>
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
:: [   FAIL   ] :: HTTPS - no certificate verification, different hostname (Expected 0, got 1)

Expected results:
:: [  BEGIN   ] :: Set certificate and hostname verification to "disable" :: actually running 'sed -i 's/verify=.*/verify=disable/' /opt/rh/python27/root/etc/python/cert-verification.cfg'
:: [   PASS   ] :: Set certificate and hostname verification to "disable" (Expected 0, got 0)
:: [  BEGIN   ] :: HTTPS - no certificate verification, different hostname :: actually running 'python test.py localhost'
(200, 'ok')
200
:: [   PASS   ] :: HTTPS - no certificate verification, different hostname (Expected 0, got 0)


Additional info:
Full beaker log: http://beaker-archive.app.eng.bos.redhat.com/beaker-logs/2017/01/16861/1686176/3451813/50490761/TESTOUT.log

If we decide to backport this functionality, we should probably turn on this feature by default as we are going to do for rhel-7.4 (see BZ#1219110).

Comment 1 Branislav Náter 2017-02-06 16:28:16 UTC
Tomas, can you please review this bug? Is it ok to have no option to turn off certification verification in rhscl python27? If no, are we going to set "platform_default" to ENABLED in python27?

Comment 2 Tomas Hoger 2017-02-06 22:21:47 UTC
(In reply to Branislav Náter from comment #1)
> Tomas, can you please review this bug? Is it ok to have no option to turn
> off certification verification in rhscl python27?

Speaking strictly from the Product Security point of view, having verification enabled by default and not providing a configuration file to change that default is ok, as it fixes the problem with missing certificate verification.

However, from the product point of view, this python27 rebase enabling verification by default can break user deployments and hence supporting configuration method that is supported in current python27 packages sounds reasonable.  It should also be noted that PEP 493 indicates that python versions 2.7.9 or later "SHOULD NOT" provide this configuration file feature.

It should also be noted that python 2.7.13 implements two other features specified in PEP 493, that (IIRC) are not implemented in the latest released RHSCL python27 packages:

- "Feature: environment based configuration", with PYTHONHTTPSVERIFY environment variable used to define verification default for given python/script run.

- "Feature: Configuration API", an easy way to set verification default from inside python scripts.

We have these supported in the latest RHEL-7 python, so besides PEP 493 (http://legacy.python.org/dev/peps/pep-0493/) itself, we have some info about these features in the following kbase:

https://access.redhat.com/articles/2039753

So technically, there is a method to disable verification without needing to modify scripts - the PYTHONHTTPSVERIFY environment variable.  It's still a change from the current config file based method.

> If no, are we going to set "platform_default" to ENABLED in python27?

If we decide to support cert-verification.cfg in our 2.7.13, we really should ensure that platform_default is enabled verification.

Comment 3 Tomas Orsava 2017-02-07 14:48:35 UTC
(In reply to Tomas Hoger from comment #2)
> (In reply to Branislav Náter from comment #1)
> 
> However, from the product point of view, this python27 rebase enabling
> verification by default can break user deployments and hence supporting
> configuration method that is supported in current python27 packages sounds
> reasonable.  It should also be noted that PEP 493 indicates that python
> versions 2.7.9 or later "SHOULD NOT" provide this configuration file feature.

The PEP states that python versions 2.7.9 and later should not use this method (which has verification disabled by default) because users will have a reasonable expectation that verification of HTTPS certificates will be enabled by default.

Therefore if we continue to support this method but switch the default to enable verification, all parties should be satisfied.


> We have these supported in the latest RHEL-7 python, so besides PEP 493
> (http://legacy.python.org/dev/peps/pep-0493/) itself, we have some info
> about these features in the following kbase:

There is a section specifically dealing with "Red Hat Software Collections Resolution" [0] that informs customers that the file based method is being used with the default set to disabled for now, and that they should expect this to change to enabled in the future.

I think this is a binding promise we should really avoid breaking, especially since the alternative methods to enable/disable verification were not present in the latest released version of python27.


Summary: I believe we should continue to support the file based approach and turn the default to enable verification (which is happening in RHEL-7.4 as well [1]).


[0] https://access.redhat.com/articles/2039753#red-hat-software-collections-resolution-5
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1219110

Comment 4 Tomas Hoger 2017-02-08 12:52:18 UTC
(In reply to Tomas Orsava from comment #3)
> The PEP states that python versions 2.7.9 and later should not use this
> method (which has verification disabled by default) because users will have
> a reasonable expectation that verification of HTTPS certificates will be
> enabled by default.

My understanding of the PEP wording is that it discourages the use of the config file regardless of what default it configured.

> There is a section specifically dealing with "Red Hat Software Collections
> Resolution" [0] that informs customers that the file based method is being
> used with the default set to disabled for now, and that they should expect
> this to change to enabled in the future.

The intention was to have a way to have a 3 state option, allowing users to explicitly enable or disable verification, or use whatever the system default is.  The default setting in the file should be platform_default.  This was meant to allow users to change it to disabled if they know they're applications fail with verification enabled.  It's not really needed while platform_default is disabled, but allows making configuration "future proof" - if platform_default is ever changed to enabled, users can prepare for that in advance and ensure it stays disabled for them.

The usage that was really considered was something like RHEL-7, where rebase to 2.7.9 or newer was not ever expected.  However, I do not see the mechanism unacceptable for 2.7.9+ if it's for product that previously provided older version, and the default in 2.7.9+ is enabled verification.

Comment 5 Tomas Orsava 2017-02-08 14:57:41 UTC
(In reply to Tomas Hoger from comment #4)
> (In reply to Tomas Orsava from comment #3)
> > The PEP states that python versions 2.7.9 and later should not use this
> > method (which has verification disabled by default) because users will have
> > a reasonable expectation that verification of HTTPS certificates will be
> > enabled by default.
> 
> My understanding of the PEP wording is that it discourages the use of the
> config file regardless of what default it configured.


The relevant paragraph in the PEP does not count with the possibility where the file based approach is used with the platform_default set to enabled. Here's the text:

"However, this approach SHOULD NOT be used for any Python installation that advertises itself as providing Python 2.7.9 or later, as most Python users will have the reasonable expectation that all such environments will verify HTTPS certificates by default."

I.e. if the platform_default is set to enabled then there won't be a problem as the Python will behave as expected (unless the setting is purposefully changed).


> > There is a section specifically dealing with "Red Hat Software Collections
> > Resolution" [0] that informs customers that the file based method is being
> > used with the default set to disabled for now, and that they should expect
> > this to change to enabled in the future.
> 
> The intention was to have a way to have a 3 state option, allowing users to
> explicitly enable or disable verification, or use whatever the system
> default is.  The default setting in the file should be platform_default. 
> This was meant to allow users to change it to disabled if they know they're
> applications fail with verification enabled.  It's not really needed while
> platform_default is disabled, but allows making configuration "future proof"
> - if platform_default is ever changed to enabled, users can prepare for that
> in advance and ensure it stays disabled for them.


I agree, that is precisely why I want to re-add the file based approach.


> The usage that was really considered was something like RHEL-7, where rebase
> to 2.7.9 or newer was not ever expected.  However, I do not see the
> mechanism unacceptable for 2.7.9+ if it's for product that previously
> provided older version, and the default in 2.7.9+ is enabled verification.


Thank you for the review! As we're in agreement, I'm going to go ahead and implement it.

Comment 7 Tomas Hoger 2017-02-22 13:08:30 UTC
Regarding the doc text, I will be turning the 2.4 erratum to RHSA to say that CVE-2014-9365 is now fixed as we're changing the default to enabled verification.

The erratum should not document that you can disable verification via cert-verification.cfg, as that's not a new feature.

We can document that there now is support for PYTHONHTTPSVERIFY and Configuration API from PEP 493, even though that's not what this bug is really about.  If we're mentioning that, consider re-using text that was used in RHEL 7.3 release notes:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.3_Release_Notes/new_features_compiler_and_tools.html

New configuration options for SSL/TLS certificate verification for the HTTP clients in the Python standard library

New per-application and per-process configuration options for SSL/TLS certificate verification have been added for the HTTP clients in the Python standard library. The options are described in the 493 Python Enhancement Proposal (https://www.python.org/dev/peps/pep-0493/). The default global setting continues to be to not verify certificates. For details, see https://access.redhat.com/articles/2039753. (BZ#1315758)

Of course, the linked kbase https://access.redhat.com/articles/2039753 is also going to be updated.

Comment 8 Tomas Orsava 2017-02-22 15:14:58 UTC
(In reply to Tomas Hoger from comment #7)

I reformulated it using the text from RHEL 7.3 release notes.

Should I take some action in updating the kbase article, or is that under your purview?

Comment 9 Tomas Hoger 2017-02-22 15:50:05 UTC
I can update the kbase, but I will not be doing that until much closer to GA.

Comment 10 Branislav Náter 2017-03-15 12:31:04 UTC
Verified on python27-python-2.7.13-3.el6 and python27-python-2.7.13-3.el7 packages.

It's possible to set hostname and certificate verification, and it is turn on by default.

TCMS test case runs:
https://tcms.engineering.redhat.com/run/305838/#caserun_15308637
https://tcms.engineering.redhat.com/run/305839/#caserun_15308807
https://tcms.engineering.redhat.com/run/305850/#caserun_15309973

Comment 12 errata-xmlrpc 2017-04-26 10:21:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:1162


Note You need to log in before you can comment on or make changes to this bug.