Description of problem: Since 3 Dec 2021, all duplicity operations against Backblaze B2 are failing with: ``` B2ConnectionError: Connection error: HTTPSConnectionPool(host='api.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v2/b2_authorize_account (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) ``` Version-Release number of selected component (if applicable): duplicity-0.7.19-1.el7.x86_64 How reproducible: Every time Steps to Reproduce: Attempt any operation against a b2 bucket, e.g. `duplicity collection-status b2://xxx@xxx/xxx Actual results: ``` Traceback (innermost last): File "/usr/bin/duplicity", line 1581, in <module> with_tempdir(main) File "/usr/bin/duplicity", line 1567, in with_tempdir fn() File "/usr/bin/duplicity", line 1406, in main action = commandline.ProcessCommandLine(sys.argv[1:]) File "/usr/lib64/python2.7/site-packages/duplicity/commandline.py", line 1132, in ProcessCommandLine globals.backend = backend.get_backend(args[0]) File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 223, in get_backend obj = get_backend_object(url_string) File "/usr/lib64/python2.7/site-packages/duplicity/backend.py", line 209, in get_backend_object return factory(pu) File "/usr/lib64/python2.7/site-packages/duplicity/backends/b2backend.py", line 92, in __init__ self.service.authorize_account('production', account_id, account_key) File "/usr/lib/python2.7/site-packages/logfury/v0_1/trace_call.py", line 84, in wrapper return function(*wrapee_args, **wrapee_kwargs) File "/usr/lib/python2.7/site-packages/b2sdk/api.py", line 160, in authorize_account response = self.raw_api.authorize_account(realm_url, application_key_id, application_key) File "/usr/lib/python2.7/site-packages/b2sdk/raw_api.py", line 188, in authorize_account return self._post_json(realm_url, 'b2_authorize_account', auth) File "/usr/lib/python2.7/site-packages/b2sdk/raw_api.py", line 184, in _post_json return self.b2_http.post_json_return_json(url, headers, params) File "/usr/lib/python2.7/site-packages/b2sdk/b2http.py", line 327, in post_json_return_json return self.post_content_return_json(url, headers, data, try_count, params) File "/usr/lib/python2.7/site-packages/b2sdk/b2http.py", line 297, in post_content_return_json response = _translate_and_retry(do_post, try_count, post_params) File "/usr/lib/python2.7/site-packages/b2sdk/b2http.py", line 127, in _translate_and_retry return _translate_errors(fcn, post_params) File "/usr/lib/python2.7/site-packages/b2sdk/b2http.py", line 84, in _translate_errors raise B2ConnectionError(str(e0)) B2ConnectionError: Connection error: HTTPSConnectionPool(host='api.backblazeb2.com', port=443): Max retries exceeded with url: /b2api/v2/b2_authorize_account (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)) ``` Expected results: No error. Additional info: I wondered if it was an updated certificate (the server's Let's Encrypt cert was updated on 30 Nov), but the previous backup (2 Dec) worked fine. I also wondered if it was an effect of the September expiry of the Let's Encrypt X3 cross-signing CA, but I have the updated ca-certificates package (2021.2.50-72) already, so I don't think it's that. I've tried with `--ssl-no-check-certificate` but as expected it made no difference - the man page says this only works with webdav and lftp backends. Trying to get extra debug output (`-vd`) did not produce anything of relevance. I also wonder if it's something that's either fixed in a newer version of duplicity (0.8.21 is available for el8), or newer "b2"/"b2sdk" Python modules (but I tried `pip install --upgrade b2` and the latest version doesn't work with Python 2). I use duplicity on Fedora 35 and that has no issues.
Incidentally, `openssl s_client -connect api.backblaze.com:443 -status <<<''` shows me the cert and chain are valid ("Verify return code: 0 (ok)").
I can confirm the same behavior on my system. The main difference I can find is that a number of updates were installed via yum between the last working backup and the first failed backup: Transaction ID : 506 Begin time : Thu Dec 2 14:45:15 2021 Begin rpmdb : 969:029fbbb7229149a83706511de5b175c35338fa8a End time : 14:47:34 2021 (139 seconds) End rpmdb : 969:710292ceb44a36a236a684d94caea772e65fa9a9 User : Scott Bishop <drkbish> Return-Code : Success Command Line : update Transaction performed with: Updated rpm-4.11.3-46.el7_9.x86_64 @updates Updated subscription-manager-1.24.48-1.el7.centos.x86_64 @updates Installed yum-3.4.3-168.el7.centos.noarch @base Installed yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda Installed yum-plugin-fastestmirror-1.1.31-54.el7_8.noarch @updates Packages Altered: Updated bash-4.2.46-34.el7.x86_64 @base Update 4.2.46-35.el7_9.x86_64 @updates Updated bind-32:9.11.4-26.P2.el7_9.7.x86_64 @updates Update 32:9.11.4-26.P2.el7_9.8.x86_64 @updates Updated bind-export-libs-32:9.11.4-26.P2.el7_9.7.x86_64 @updates Update 32:9.11.4-26.P2.el7_9.8.x86_64 @updates Updated bind-libs-32:9.11.4-26.P2.el7_9.7.x86_64 @updates Update 32:9.11.4-26.P2.el7_9.8.x86_64 @updates Updated bind-libs-lite-32:9.11.4-26.P2.el7_9.7.x86_64 @updates Update 32:9.11.4-26.P2.el7_9.8.x86_64 @updates Updated bind-license-32:9.11.4-26.P2.el7_9.7.noarch @updates Update 32:9.11.4-26.P2.el7_9.8.noarch @updates Updated bind-utils-32:9.11.4-26.P2.el7_9.7.x86_64 @updates Update 32:9.11.4-26.P2.el7_9.8.x86_64 @updates Updated kbd-1.15.5-15.el7.x86_64 @base Update 1.15.5-16.el7_9.x86_64 @updates Updated kbd-legacy-1.15.5-15.el7.noarch @base Update 1.15.5-16.el7_9.noarch @updates Updated kbd-misc-1.15.5-15.el7.noarch @base Update 1.15.5-16.el7_9.noarch @updates Erase kernel-3.10.0-1160.31.1.el7.x86_64 @updates Install kernel-3.10.0-1160.49.1.el7.x86_64 @updates Erase kernel-devel-3.10.0-1160.31.1.el7.x86_64 @updates Install kernel-devel-3.10.0-1160.49.1.el7.x86_64 @updates Updated kernel-headers-3.10.0-1160.45.1.el7.x86_64 @updates Update 3.10.0-1160.49.1.el7.x86_64 @updates Updated kernel-tools-3.10.0-1160.45.1.el7.x86_64 @updates Update 3.10.0-1160.49.1.el7.x86_64 @updates Updated kernel-tools-libs-3.10.0-1160.45.1.el7.x86_64 @updates Update 3.10.0-1160.49.1.el7.x86_64 @updates Updated openssh-7.4p1-21.el7.x86_64 @base Update 7.4p1-22.el7_9.x86_64 @updates Updated openssh-clients-7.4p1-21.el7.x86_64 @base Update 7.4p1-22.el7_9.x86_64 @updates Updated openssh-server-7.4p1-21.el7.x86_64 @base Update 7.4p1-22.el7_9.x86_64 @updates Updated perf-3.10.0-1160.45.1.el7.x86_64 @updates Update 3.10.0-1160.49.1.el7.x86_64 @updates Updated python-perf-3.10.0-1160.45.1.el7.x86_64 @updates Update 3.10.0-1160.49.1.el7.x86_64 @updates Updated python-syspurpose-1.24.48-1.el7.centos.x86_64 @updates Update 1.24.50-1.el7.centos.x86_64 @updates Updated rpm-4.11.3-46.el7_9.x86_64 @updates Update 4.11.3-48.el7_9.x86_64 @updates Updated rpm-build-4.11.3-46.el7_9.x86_64 @updates Update 4.11.3-48.el7_9.x86_64 @updates Updated rpm-build-libs-4.11.3-46.el7_9.x86_64 @updates Update 4.11.3-48.el7_9.x86_64 @updates Updated rpm-libs-4.11.3-46.el7_9.x86_64 @updates Update 4.11.3-48.el7_9.x86_64 @updates Updated rpm-python-4.11.3-46.el7_9.x86_64 @updates Update 4.11.3-48.el7_9.x86_64 @updates Updated subscription-manager-1.24.48-1.el7.centos.x86_64 @updates Update 1.24.50-1.el7.centos.x86_64 @updates Updated subscription-manager-rhsm-1.24.48-1.el7.centos.x86_64 @updates Update 1.24.50-1.el7.centos.x86_64 @updates Updated subscription-manager-rhsm-certificates-1.24.48-1.el7.centos.x86_64 @updates Update 1.24.50-1.el7.centos.x86_64 @updates Scriptlet output: 1 grubby fatal error: unable to find a suitable template history info
I have a very similar set of updates (not the subscription-manager* or python-syspurpose ones). I don't think any of them relate to SSL cert validation though - unless anyone knows better?
Via `strace` I worked out that duplicity was actually loading its CA certs from /usr/lib/python2.7/site-packages/certifi/cacert.pem, not the bundle installed by the ca-certificates package, and that this file contains the expired "DST Root CA X3" cert known to cause problems with validating Let's Encrypt certificates (with some versions of openssl at least). [root@kwartz ~]# rpm -qf /usr/lib/python2.7/site-packages/certifi/cacert.pem file /usr/lib/python2.7/site-packages/certifi/cacert.pem is not owned by any package [root@kwartz ~]# pip list | grep certifi certifi (2019.9.11) `pip install --upgrade certifi` downloaded version 2021.10.8 but that didn't fix it - it's still included. However, manually editing /usr/lib/python2.7/site-packages/certifi/cacert.pem to remove the certificate labelled "DST Root CA X3" solved the problem, and my backups are running now. I've logged https://github.com/certifi/python-certifi/issues/182 to remove it.
Would this work? mv /usr/lib/python2.7/site-packages/certifi/cacert.pem /usr/lib/python2.7/site-packages/certifi/cacert.pem.bad && ln -s /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /usr/lib/python2.7/site-packages/certifi/cacert.pem Then by rights python should just reference the certs from the ca-certificates bundle instead of it's own messed up certs?