Created attachment 1950273 [details] broken log Description of problem: After upgrading to Fedora 38 my openvpn tunnel started failing. I am using a ECDSA certificate baked by a yubikey. The relevant configuration is (openvpn.conf): pkcs11-id "pkcs11:model=PKCS%2315%20emulated;token=xxxx;manufacturer=piv_II;serial=xxxx;id=%01" pkcs11-providers /usr/lib64/pkcs11/opensc-pkcs11.so management 127.0.0.1 8888 management-query-passwords Version-Release number of selected component (if applicable): openvpn-2.6.0-2.fc38 I am attaching a debug log with a working attempt from fedora 37 and a broken one from fedora 38. Please note that the certificates are not the same but they are both baked by yubikeys. In the log you can see that the broken configuration does "less" (compare with meld or another visual diff tool) and seems to return with an rv=1 indicating an error (?). Please let me know which other details I can provide.
Created attachment 1950274 [details] working log
Correction: The above log doesn't actually show the error. Nothing in the client log really shows an error, I think I missinterpreted rv=1. The server logs show this: TLS Error: TLS handshake failed | TLS Error: TLS object -> incoming plaintext read error | TLS\_ERROR: BIO read tls\_read\_plaintext error | OpenSSL: error:1417B07B:SSL routines:tls\_process\_cert\_verify:bad signature | OpenSSL: error:0D07803A:asn1 encoding routines:asn1\_item\_embed\_d2i:nested asn1 error | OpenSSL: error:0D0680A8:asn1 encoding routines:asn1\_check\_tlen:wrong tag | So it seems like OpenVPN now generates something weirdly broken :(
Proposed as a Freeze Exception for 38-beta by Fedora user apollo13 using the blocker tracking app because: Users that require a yubikey for their OpenVPN privatekey require pkcs11/openvpn to be in a functional state. Similar for older Fedora version: https://pagure.io/fedora-qa/blocker-review/issue/770
I think the OP wanted to propose this as a Final blocker, correcting
The configuration working on Fedora 37, is that running OpenVPN 2.5 or OpenVPN 2.6? If it is running OpenVPN 2.5, can you please test with OpenVPN 2.6 using this Fedora Copr repository? https://copr.fedorainfracloud.org/coprs/dsommers/openvpn-release-2.6/ This is needed to clarify if this is an issue in the PKCS#11 library stack or OpenVPN 2.6.
Fedora 37 is running OpenVPN 2.5. I did manually compile OpenVPN 2.5 on Fedora 38 and it works again. Do you need me to retry on Fedora 37 with 2.6 also or is it enough to know the Fedora 38 with Openvpn 2.5 works?
It would be good if you could run a 2.6 test on Fedora 37 too. I suspect it will fail, but with that confirmation we have a better grip on the failures. Also the same log outputs would be good for comparison. I will send this information to the upstream project once we have all details.
I can try Fedora 37 tomorrow. I managed to get it running with Fedora 38 and OpenVPN by using OpenVPN's external signing mechanism and manually implementing the required signature. What I noticed on the way when using the internal provider: xkey_provider: In xkey_sign_dispatch: xkey_provider: external sign op returned ret = 1 siglen = 64 When I use my own external signature I get don't get 64 but 70-72. I get to 70-72 by using https://python-pkcs11.readthedocs.io/en/latest/api.html?highlight=get_objects#pkcs11.util.ec.encode_ecdsa_signature which encodes the raw 64 byte signature into DER encoded signature. Now I wonder if the xkey_provider should be doing this somewhere -- but at this point I am just guessing, this is a bit out of my league.
Details for the previous message (xkey_provider logs): Working: xkey_provider: In ec_keymgmt_name: entry xkey_provider: In query_operation: op = 12 xkey_provider: In query_operation: op = 10 xkey_provider: In signature_newctx: entry xkey_provider: In signature_digest_sign_init: mdname = <SHA256> xkey_provider: In signature_set_ctx_params: entry xkey_provider: In signature_freectx: entry xkey_provider: In keydata_free: entry xkey_provider: In keymgmt_get_params: entry xkey_provider: In ec_keymgmt_name: entry xkey_provider: In signature_newctx: entry xkey_provider: In signature_digest_sign_init: mdname = <SHA2-256> xkey_provider: In signature_set_ctx_params: entry xkey_provider: In signature_digest_sign: entry xkey_provider: In signature_digest_sign: entry In xkey_management_sign with keytype = EC, op = DigestSign xkey_management_sign: computing digest In xkey_digest xkey management_sign: requesting sig with algorithm <ECDSA> xkey_provider: In xkey_sign_dispatch: xkey_provider: external sign op returned ret = 1 siglen = 70 xkey_provider: In signature_freectx: entry xkey_provider: In keydata_free: entry Broken: xkey_provider: In ec_keymgmt_name: entry xkey_provider: In query_operation: op = 12 xkey_provider: In query_operation: op = 10 xkey_provider: In signature_newctx: entry xkey_provider: In signature_digest_sign_init: mdname = <SHA256> xkey_provider: In signature_set_ctx_params: entry xkey_provider: In signature_freectx: entry xkey_provider: In keydata_free: entry xkey_provider: In keymgmt_get_params: entry xkey_provider: In ec_keymgmt_name: entry xkey_provider: In signature_newctx: entry xkey_provider: In signature_digest_sign_init: mdname = <SHA2-256> xkey_provider: In signature_set_ctx_params: entry xkey_provider: In signature_digest_sign: entry xkey_provider: In signature_digest_sign: entry xkey_pkcs11h_sign: computing digest In xkey_digest xkey_pkcs11h_sign: signing with EC key xkey_provider: In xkey_sign_dispatch: xkey_provider: external sign op returned ret = 1 siglen = 64 xkey_provider: In signature_freectx: entry xkey_provider: In keydata_free: entry If you compare working with broken you see that the returned signature length is quite a bit different. I wonder if the new xkey_provider should be DER encoding (?) the results in xkey_pkcs11h_sign (my working copy uses xkey_management_sign to get the proper signature)
I can confirm that this problem also occurs on Fedora 37 with OpenVPN 2.6.0. The client logs are basically irrelevant, they don't really show any error. Only the server complains that the signature is wrong and that it cannot parse the ASN.1 structure.
Thanks a lot! I've had a brief discussion with upstream, and there is already a patch being reviewed which seems relevant: https://patchwork.openvpn.net/project/openvpn2/patch/20230311052403.1154030-1-selva.nair@gmail.com/
I have a couple of Fedora Koji builds brewing now with the patch from the mailing list included. Can you give those a spin, Florian? Fedora 38: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680872 x86_64 packages: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680943 Fedora 37: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680878 x86_64 packages: https://koji.fedoraproject.org/koji/taskinfo?taskID=98680991
Jupp, the Fedora 38 package works for me. Didn't test the 37 variant.
Great! Thanks for testing! I'll report back.
FEDORA-2023-dde2bc874f has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-dde2bc874f
The same fix has been pushed to the Fedora Copr repository for the OpenVPN 2.6 releases. The upstream patch is included and will most likely be included in the next OpenVPN 2.6.2 release.
FEDORA-2023-dde2bc874f has been pushed to the Fedora 38 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-dde2bc874f See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2023-82ca19484b has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-82ca19484b
FEDORA-2023-82ca19484b has been pushed to the Fedora 39 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2023-dde2bc874f has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.