Hide Forgot
DESCRIPTION OF PROBLEM: Using cryptography.fernet.Fernet in RHEL 7.3.1 via mod_wsgi doesn't work the first few times after httpd is started. When it fails, it errors out in /usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py with: > InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=151441516L, lib=9, func=109, reason=108, reason_text='error:0906D06C:PEM routines:PEM_read_bio:no start line'), _OpenSSLErrorWithText(code=151441516L, lib=9, func=109, reason=108, reason_text='error:0906D06C:PEM routines:PEM_read_bio:no start line')]) After 5 or so failures, it starts working and then seems to not fail any more until the next httpd restart. VERSION-RELEASE NUMBER OF SELECTED COMPONENT (IF APPLICABLE): python2-cryptography-1.3.1-3.el7.x86_64 httpd-2.4.6-45.el7.x86_64 openssl-1.0.1e-60.el7.x86_64 (Also updated all packages to RHEL 7.3.1, Dec 7) HOW REPRODUCIBLE: 100%, right after starting/restarting httpd STEPS TO REPRODUCE: 1. yum install -y python-cffi mod_wsgi httpd python2-cryptography 2. yum update 3. reboot 4. echo WSGIScriptAlias /test_fernet /var/www/cryptotest.wsgi >/etc/httpd/conf.d/test_fernet.conf 5. cat >/var/www/cryptotest.wsgi <<\EOF def test_fernet(input='sample input text'): import logging logging.basicConfig(format="test_fernet: %(message)s", level='DEBUG') logging.debug(">>> from cryptography.fernet import Fernet") from cryptography.fernet import Fernet logging.debug(">>> Fernet.generate_key()") key = Fernet.generate_key() logging.info("KEY: '{}'".format(key)) f = Fernet(key) logging.info("INPUT: '{}'".format(input)) e = f.encrypt(bytes(input)) logging.info("ENC: '{}'".format(e)) d = f.decrypt(bytes(e)) logging.info("DEC: '{}'".format(d)) return "COMPLETED\n" def application(environ, start_response): output = test_fernet() response_headers = [('Content-type', 'text/plain'), ('Content-Length', str(len(output)))] start_response('200 OK', response_headers) return [output] if __name__ == '__main__': print(test_fernet()) EOF 6. systemctl restart httpd 7. for i in {1..30}; do curl -s --head localhost/test_fernet | head -1; done ACTUAL RESULTS: [root@r73 ~]# for i in {1..30}; do curl -s --head localhost/test_fernet | head -1; done HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 500 Internal Server Error HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK From httpd's error_log: test_fernet: >>> from cryptography.fernet import Fernet test_fernet: >>> Fernet.generate_key() test_fernet: KEY: 'D_0AykLrKC5wTwcJ6gBRJQqU_rvB9VSU1XWF-RFBHpk=' [client ::1:44590] mod_wsgi (pid=23141): Exception occurred processing WSGI script '/var/www/cryptotest.wsgi'. [client ::1:44590] Traceback (most recent call last): [client ::1:44590] File "/var/www/cryptotest.wsgi", line 18, in application [client ::1:44590] output = test_fernet() [client ::1:44590] File "/var/www/cryptotest.wsgi", line 9, in test_fernet [client ::1:44590] f = Fernet(key) [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/fernet.py", line 32, in __init__ [client ::1:44590] backend = default_backend() [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 40, in default_backend [client ::1:44590] _default_backend = MultiBackend(_available_backends()) [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 27, in _available_backends [client ::1:44590] "cryptography.backends" [client ::1:44590] File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load [client ::1:44590] entry = __import__(self.module_name, globals(),globals(), ['__name__']) [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> [client ::1:44590] from cryptography.hazmat.backends.openssl.backend import backend [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 46, in <module> [client ::1:44590] from cryptography.hazmat.bindings.openssl import binding [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 225, in <module> [client ::1:44590] Binding.init_static_locks() [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 181, in init_static_locks [client ::1:44590] cls._ensure_ffi_initialized() [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 176, in _ensure_ffi_initialized [client ::1:44590] cls._register_osrandom_engine() [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 141, in _register_osrandom_engine [client ::1:44590] _openssl_assert(cls.lib, cls.lib.ERR_peek_error() == 0) [client ::1:44590] File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 63, in _openssl_assert [client ::1:44590] errors_with_text [client ::1:44590] InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=151441516L, lib=9, func=109, reason=108, reason_text='error:0906D06C:PEM routines:PEM_read_bio:no start line'), _OpenSSLErrorWithText(code=151441516L, lib=9, func=109, reason=108, reason_text='error:0906D06C:PEM routines:PEM_read_bio:no start line')]) EXPECTED RESULTS: Should work immediately, i.e., thirty 200 OK status returns. ADDITIONAL INFO: Note that the same test in Fedora25 meets success all-around. [root@f25 ~]# rpm -q python2-cryptography openssl httpd python2-cryptography-1.5.3-3.fc25.x86_64 openssl-1.0.2j-1.fc25.x86_64 httpd-2.4.23-4.fc25.x86_64 [root@f25 ~]# systemctl restart httpd; for i in {1..10}; do curl -s --head localhost/test_fernet | head -1; done HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK [root@f25 ~]# tail -6 /var/log/httpd/error_log | grep -o ".pid.*" [pid 8546] test_fernet: >>> from cryptography.fernet import Fernet [pid 8546] test_fernet: >>> Fernet.generate_key() [pid 8546] test_fernet: KEY: 'Li8hOE73GFt8sGbtSCHSJ7tr7v6MCDc9DC7CFXNmJfc=' [pid 8546] test_fernet: INPUT: 'sample input text' [pid 8546] test_fernet: ENC: 'gAAAAABYR6bq3Rn1Dp1Tv2xLNeXnT8kWgyEtYbWc-R-HZjS-S57fHx3T0xDbubX_QW__-v1ya7GRkIylxTo5efFNpojJ_25EXEq8RKOrq8Ait42USPq2R-4=' [pid 8546] test_fernet: DEC: 'sample input text'
This kind of bug can occur when Apache / mod_ssl does not clean up OpenSSL's error stack properly. Other developers have reported similar issues with mod_wsgi applications.
https://github.com/pyca/cryptography/pull/3278 should address the bug.
(In reply to Christian Heimes from comment #2) > https://github.com/pyca/cryptography/pull/3278 should address the bug. Fascinating ... That made me question why this wasn't an issue in Fedora 25 and I realized: in addition to the reproducer-steps, in RHEL7 I also installed mod_ssl. Removing that, I can no longer reproduce in RHEL 7.3.1. [root@r73 ~]# rpm -q httpd python2-cryptography mod_ssl openssl httpd-2.4.6-45.el7.x86_64 python2-cryptography-1.3.1-3.el7.x86_64 package mod_ssl is not installed openssl-1.0.1e-60.el7.x86_64 [root@r73 ~]# systemctl restart httpd [root@r73 ~]# for i in {1..10}; do curl -s --head localhost/test_fernet | head -1; done HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK And if I install mod_ssl in F25, the issue surfaces. [root@f25 ~]# rpm -q httpd python2-cryptography mod_ssl openssl httpd-2.4.23-4.fc25.x86_64 python2-cryptography-1.5.3-3.fc25.x86_64 mod_ssl-2.4.23-4.fc25.x86_64 openssl-1.0.2j-1.fc25.x86_64 [root@f25 ~]# systemctl restart httpd [root@f25 ~]# for i in {1..15}; do curl -s --head localhost/test_fernet | head -1; done HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 500 Internal Server Error HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK HTTP/1.1 200 OK AWESOME. So it looks like the upstream PR to make python-cryptography pragmatically handle these kinds of errors hasn't even been merged yet. Any reason to keep this BZ open? I'm thinking it makes most sense to clone this to mod_ssl package. Concur?
The upstream PR will be merged soon. Paul wants me and Alex to review the PR before it gets merged. The fix will be in cryptography 1.7. Yes, it makes sense to clone the issue to mod_ssl and fix the root cause of the bug in mod_ssl.
Let's just move this bug to httpd. If they are handling errors (which is clearly the case) then they need to clean up the error stack after handling the errors.
The test cases you are showing don't show a mod_ssl configuration, or mod_ssl being exercised directly, so it's a bit hard for me to comment. curl -s --head localhost/test_fernet ... is plan HTTP. Having faced similar OpenSSL problems in mod_ssl itself, we took to doing: ERR_clear_error() some_SSL_function() in places where either the SSL function behaviour is dependent on the error stack being empty on entry, or doesn't clear it internally, and we need to check it afterwards. Note that Perl, PHP, ... stuff can also all get linked into the httpd process and change the error stack state. The above approach is the only one we've found reliable.
(In reply to Joe Orton from comment #8) > The test cases you are showing don't show a mod_ssl configuration, or > mod_ssl being exercised directly, so it's a bit hard for me to comment. > > curl -s --head localhost/test_fernet > > ... is plan HTTP. I hear you Joe, but this is reproduceable with a completely default ssl.conf and you don't need to query over https to hit it. Simply install python-cffi mod_wsgi httpd python2-cryptography mod_ssl and then follow the steps.
OK, fair enough. I'm not going to make a guarantee that mod_ssl will leave the error stack empty in all cases, we've never written it like that upstream, and every other OpenSSL consumer which has been linked into httpd (i.e. all of them) has managed to deal with this issue. I can either WONTFIX this bug or you can take it back and fix the issue in python-cryptography.
(In reply to Joe Orton from comment #10) > I'm not going to make a guarantee that mod_ssl will leave the error stack > empty in all cases, we've never written it like that upstream, and every > other OpenSSL consumer which has been linked into httpd (i.e. all of them) > has managed to deal with this issue. I can either WONTFIX this bug or you > can take it back and fix the issue in python-cryptography. Thanks Joe. Understood. @Nathaniel McCallum: See above and also note that upstream python-cryptography has taken note of Joe's response and one of the maintainers reluctantly merged a pull request (https://github.com/pyca/cryptography/pull/3278) to clear the error queue. I leave it to you to decide how to move forward with this BZ -- e.g., to change component back to python-cryptography or just close in favor of the RHEL7.4 rebase request (bz1398666).
python-cryptography is thorough and validates that OpenSSL is in a good state, before cryptography starts to use OpenSSL. After all OpenSSL is a security critical crypto and TLS library. I agree with upstream's approach validate-then-use. https://github.com/pyca/cryptography/pull/3278 is a workaround for an issue that upstream considers a bug in mod_ssl. Libraries should remove an error from the error stack before they hand over OpenSSL to another part of the application. The PR is new and not part of 1.7.2. It should be in 1.8.0 but that's not released yet. In bz1398666 I might go for 1.8.0 to fix another issue any way.
Reassigning to python-cryptography, fix needs to be done there.
Verified on: python2-cryptography-1.7.2-2.el7.x86_64 httpd-2.4.6-79.el7.x86_64 openssl-1.0.2k-11.el7.x86_64 python-cffi-1.6.0-5.el7.x86_64 mod_wsgi-3.4-12.el7_0.x86_64 mod_ssl-2.4.6-79.el7.x86_64
Created attachment 1367347 [details] verification_steps
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0720