Bug 1219544

Did you try to escape ":" as "\:"?

Ah yes, if I build against GnuTLS instead of NSS *and* escape the colon, it works.

Is there a reason to continue building with NSS, or can we switch? NSS seems quite resistant to getting fixed (for anything, ever, but including this in particular).

Your answer is ambiguous.

Does curl-7.40.0-3.fc22 work in your scenario if you escape the colon?

If not, does the behavior differ from what is stated in comment #0?

Apologies. No, curl-7.40.0-3.fc22 does not work:

]$ curl -E 'pkcs11\:token=Gnome2%20Key%20Storage;id=%e6%34%78%b5%fd%86%77%ba%b9%86%95%c3%19%02%74%d1%f5%33%9a%97' -I -v https://auth.startssl.com
* Rebuilt URL to: https://auth.startssl.com/
*   Trying 192.116.242.27...
* Connected to auth.startssl.com (192.116.242.27) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS: client certificate not found: pkcs11:token=Gnome2%20Key%20Storage;id=%e6%34%78%b5%fd%86%77%ba%b9%86%95%c3%19%02%74%d1%f5%33%9a%97
* NSS error -12227 (SSL_ERROR_HANDSHAKE_FAILURE_ALERT)
* SSL peer was unable to negotiate an acceptable set of security parameters.
* Closing connection 0
curl: (35) NSS: client certificate not found: pkcs11:token=Gnome2%20Key%20Storage;id=%e6%34%78%b5%fd%86%77%ba%b9%86%95%c3%19%02%74%d1%f5%33%9a%97



If I build curl for myself, against GnuTLS instead of NSS, *then* it works.

Is the token listed by certutil?

I guess there needs to be a PKCS#11 NSS module installed to provide the token for applications that use NSS to load client certificates.

(In reply to Kamil Dudka from comment #5)
> Is the token listed by certutil?

No. The token was installed by the Fedora packaging which creates /usr/share/p11-kit/modules/opensc.module — that is supposed to be enough.

> I guess there needs to be a PKCS#11 NSS module installed to provide the
> token for applications that use NSS to load client certificates.

Yes, NSS needs to load p11-kit-proxy.so (or individually load the modules in the p11-kit configuration) by default. That is bug 1173577.

That doesn't solve the need to accept RFC7512 URIs as specifiers for the certificates therein, though.

Thanks for the info!  Do I understand it correctly that this bug will go away once bug #1173577 is fixed?

libcurl does use "sql:/etc/pki/nssdb" as the database if NSS is able to load it.

Only part of this bug will go away if bug #1173577 is fixed. But even if you work around that by manually adding a PKCS#11 token to /etc/pki/nssdb/pkcs11.txt I don't think there's any way to get curl to let you *use* a certificate from it. Let alone using the RFC7512 URI.

As noted in https://bugzilla.redhat.com/show_bug.cgi?id=1173577#c1 it's not even clear if it makes sense to manually load p11-kit-proxy.so (or add it to /etc/pki/nssdb/pkcs11.txt) or whether that will end up conflicting with a more generic solution in NSS.

(In reply to David Woodhouse from comment #8)
> Only part of this bug will go away if bug #1173577 is fixed. But even if you
> work around that by manually adding a PKCS#11 token to
> /etc/pki/nssdb/pkcs11.txt I don't think there's any way to get curl to let
> you *use* a certificate from it. Let alone using the RFC7512 URI.

So we need to make NSS accept RFC7512 URIs as certificate nicknames.  libcurl overrides the SelectClientCert() hook only for certificates loaded from files (detected by slash occurring in the name).  If the certificate is specified by nickname, libcurl just passes the callback over to NSS_GetClientAuthData().

> As noted in https://bugzilla.redhat.com/show_bug.cgi?id=1173577#c1 it's not
> even clear if it makes sense to manually load p11-kit-proxy.so (or add it to
> /etc/pki/nssdb/pkcs11.txt) or whether that will end up conflicting with a
> more generic solution in NSS.

How can we write a patch for libcurl then? :-)

It sounds like the NSS bug should be looked at first...

(In reply to Kamil Dudka from comment #9)
> How can we write a patch for libcurl then? :-)

One option is just to build curl against GnuTLS instead of NSS.

That would fix one bug and (re)open several others at the same time.

Please have a look at the comparison of SSL libraries supported by curl:

http://curl.haxx.se/docs/ssl-compared.html

I fail to read from that table how GnuTLS is strictly better than NSS for us.

I'm not sure how up-to-date that matrix is; I'll have a check. Is there anything in particular that concerns you?

I believe the Version row tells us time of check for each of the columns.  If you are aware of any feature that has been introduced/deprecated since then, feel free to post updates to the upstream mailing list.

In my opinion, a strong reason to switch the TLS backend would be if the set of GnuTLS features was a superset of the set of NSS features.  Otherwise it is just a matter of preference.  Some users need FIPS-140 certification, others may need particular cipher-suites not implemented by the other backend.

Well, the table needs rows for 'Uses system-configured PKCS#11 tokens' and 'Allows certificates to be specified with RFC7512 URIs', which are the missing features discussed in this bug. With a tick for GnuTLS (and for OpenSSL+engine_pkcs11), and a cross for NSS.

NPN can be removed, I think, since it is now deprecated in favour of ALPN.

GnuTLS *can* use "certificate/key database", since that is basically talking about PKCS#11. It can even use the NSS database if you ask nicely. I'm fairly sure it has EC support now too.

GnuTLS has also passed FIPS-140 certification, according to http://gnutls.org/manual/html_node/Certification.html

I'll look at submitting an update.

http://curl.haxx.se/docs/ssl-compared.html has now been updated.

I think the only interesting thing NSS offers that GnuTLS doesn't is *automatic* CRL handling. I haven't checked whether curl does actually use GnuTLS's CEL handling manually. If that's an issue I can look into it...

(In reply to David Woodhouse from comment #15)
> http://curl.haxx.se/docs/ssl-compared.html has now been updated.

Thanks for the update!  We should also set the "OpenSSL-like API" field to "no" for NSS because nss_compat_ossl is now deprecated:

https://fedoraproject.org/wiki/Nss_compat_ossl

However why the "Integrates with system token database" field of NSS is set to "no"?

I believe that libcurl linked against NSS loads certificates from /etc/pki/nssdb and ~/.pki/nssdb by default.  Does not that classify as "system token database"?

No. That's NSS-specific. The platform "token database" is the one you get from p11-kit.

If you add p11-kit-proxy.so to the database in /etc/pki/nssdb then that might suffice as a workaround for those apps which happen to use it (which isn't many), but it might conflict with a *real* solution in NSS later...

(In reply to David Woodhouse from comment #17)
> No. That's NSS-specific. The platform "token database" is the one you get
> from p11-kit.

I am new to p11-kit.  The project page says "Provides a *standard* configuration setup for installing PKCS#11 modules in such a way that they're discoverable." but it is not clear to me what is meant by *standard* here.  Is there any RFC defining it?

PKCS#11 modules provided by NSS are also discoverable IMO.

In the context of Fedora, the 'standard' is set by https://fedoraproject.org/wiki/Packaging:SSLCertificateHandling

As for sql:/etc/pki/nssdb being 'discoverable'... well, there isn't even any coherent best practice for NSS applications to discover whether it's installed and usable or whether they should use sql:$HOME/.pki/nssdb instead... and a whole bunch of common NSS apps (Firefox, Thunderbird) don't even manage to use the *latter*!

(In reply to David Woodhouse from comment #19)
> In the context of Fedora, the 'standard' is set by
> https://fedoraproject.org/wiki/Packaging:SSLCertificateHandling
> 
> As for sql:/etc/pki/nssdb being 'discoverable'... well, there isn't even any
> coherent best practice for NSS applications to discover whether it's
> installed and usable or whether they should use sql:$HOME/.pki/nssdb
> instead... and a whole bunch of common NSS apps (Firefox, Thunderbird) don't
> even manage to use the *latter*!

If you point NSS at /etc/pki/nssdb, it will load both it and $HOME/.pki/nssdb - user's private keys wouldn't work otherwise.

(In reply to Hubert Kario from comment #20)
> 
> If you point NSS at /etc/pki/nssdb, it will load both it and
> $HOME/.pki/nssdb - user's private keys wouldn't work otherwise.

I think it has to be sql:/etc/pki/nssdb, since /etc/pki/nssdb means the legacy DB. And it's also only true if the contents of /etc/pki/nssdb/pkcs11.txt are going to cause it to load libnsssysinit.sh. Which is untrue on most Linux distrubutions AFAIK and it's not even universally true on Fedora, which is why we have the 'setup-nsssysinit.sh' script to query its status and set it up.

Evolution has horrid code to actually open /etc/pki/nssdb/pkcs11.txt and check for 'library=libnsssysinit.so', which is what setup-nsssysinit.sh. And then open sql:/etc/pki/nssdb or sql:$HOME/.pki/nssdb according to the result. I am not aware of *any* user application other than Evolution which does this. Chrome doesn't (it ignores the one in /etc/). Firefox doesn't (it doesn't use *either* of the sane databases).

If we want to fix the NSS problem with p11-kit by using the Shared System Database then (aside from the fact that we're discussing this in the wrong bug and we should be doing it in bug 1173577) we probably need to:

 - Kill setup-nsssysinit.sh and make it *unconditionally* functional.
 - Ensure NSS apps on Fedora (ideally including Firefox etc.) are using it.
 - Add p11-kit-proxy.so to the default /etc/pki/nssdb/pkcs11.txt

And for that last one, we first need to establish what the 'real' plan is for NSS to fix this properly, and ensure that it doesn't conflict with adding p11-kit-proxy that way. See also the thread at http://www.mail-archive.com/dev-tech-crypto@lists.mozilla.org/msg12230.html which was unfortunately hijacked by some bizarre denialism about p11-kit's status and what appears to be wilful misunderstanding of the intent ("OMG YOU ARE STUPID AND IGNORANT YOU WANT TO MAKE IT LOAD THE MODULES EVEN WHEN I USE NSS_NoDB_Init()"... "Er, no that would be silly"... <crickets>). Not a great advert for trying to work with NSS and get things improved.

In the meantime, for the purpose of this *curl* bug, it does seem like just building against GnuTLS would be the better option. Then things will Just Work™.

Unless libcurl is asked to load a client certificate from file (which is not the case in comment #4), it just passes the SelectClientCert() hook over to NSS.  libcurl already initializes NSS with "sql:/etc/pki/nssdb" as the database.  I am not sure what else could be improved in the curl package.  I am moving this bug to the nss component so that nss maintainers can have a look at this.

(In reply to Kamil Dudka from comment #22)
>  I am not sure what else could be improved in the curl package.  I
> am moving this bug to the nss component so that nss maintainers can have a
> look at this.

We already had bugs against NSS. For the failure to load the correct PKCS#11 tokens by default, there was bug 1173577 and the corresponding upstream bug https://bugzilla.mozilla.org/show_bug.cgi?id=1161219

For the failure to accept RFC7512 PKCS#11 URIs, I've now filed Fedora bug 1227376 referencing the upstream https://bugzilla.mozilla.org/show_bug.cgi?id=1162897

This bug was against the Fedora curl package, for a reason — I don't really understand why you say you're not sure how it could be improved. As I mentioned, it could be improved by simply building it against GnuTLS, and then this bug would instantly be fixed.

I do understand your concern that such a change might introduce other problems. I've updated the feature matrix in the curl documentation to help address that concern, so if there are any *specific* issues you'd still like to look at please let me know.

GnuTLS upstream is extremely responsive so if you have any problems I'm sure we can get them fixed. NSS upstream does not show any signs of even engaging in productive discussion of fixing *its* problems — as you can see from the archive link I gave in comment 21, when I finally managed to comprehensively address the pointless FUD, they just reverted to silence.

(In reply to Kamil Dudka from comment #22)
> Unless libcurl is asked to load a client certificate from file (which is not
> the case in comment #4), it just passes the SelectClientCert() hook over to
> NSS.  libcurl already initializes NSS with "sql:/etc/pki/nssdb" as the
> database.  I am not sure what else could be improved in the curl package.  I
> am moving this bug to the nss component so that nss maintainers can have a
> look at this.

I don't like to play ping-pong, but after a number of months of NSS predictably still not being fixed, I'm shifting this back to curl (and updating it to Fedora 23).

Until we can actually get progress on NSS, *please* switch to building against any of the other crypto libraries we ship in Fedora, which *do* support building packages that conform to Fedora's packaging guidelines.

I believe I updated an old feature matrix w.r.t. GnuTLS features. Is there any specific technical reason why we shouldn't just switch curl to build against GnuTLS instead of NSS, and finally fix this bug?

Elio, have you had any time to look at this?

There is no reply on the upstream bug since May:

https://bugzilla.mozilla.org/show_bug.cgi?id=1162897

> https://bugzilla.mozilla.org/show_bug.cgi?id=1162897

That one actually looks ideal for a GSoC project. Just today I got email from a student who may be interested in doing it, in fact (which is why I was looking again). Ideally we *would* want a mentor on the NSS side though; I'm not a good person to mentor an NSS project. Could we make that happen?


(Note that for curl (and indeed anything using NSS in Fedora which wants to comply with the packaging guidelines) we *also* need to fix https://bugzilla.mozilla.org/show_bug.cgi?id=1161219 — but one thing at a time is fine.)

(In reply to David Woodhouse from comment #26)
> That one actually looks ideal for a GSoC project. Just today I got email
> from a student who may be interested in doing it, in fact (which is why I
> was looking again). Ideally we *would* want a mentor on the NSS side though;
> I'm not a good person to mentor an NSS project. Could we make that happen?

Sounds great!  Bob offered to co-mentor such a project.  I am switching the bug back to nss...

The NSS patches are there, this is easily fixable for F25. If this isn't fixed in F25 (code freeze tomorrow, right?) then can I switch it back to curl please?

If we miss code freeze for f25, it can still be fixed in f26.  I fail to see how switching the bug back to curl would speed up delivery of the patches...

It can fix the bug for curl, because we can rebuild curl against GnuTLS *today* and fix the bug that was actually filed above. Just like we could have rebuilt it against GnuTLS in May of last year, when this bug was originally filed.

I'm OK with "let's fix it in NSS", followed by actually applying the patches.

I'm less OK with "let's fix it in NSS"... followed by not actually doing so.

There is very little justification for this fix not making F25.

(In reply to David Woodhouse from comment #30)
> It can fix the bug for curl, because we can rebuild curl against GnuTLS
> *today* and fix the bug that was actually filed above. Just like we could
> have rebuilt it against GnuTLS in May of last year, when this bug was
> originally filed.

It is too late to rebuild curl against GnuTLS in f25.  It would be way more disruptive than just applying those patches on NSS.

> I'm OK with "let's fix it in NSS", followed by actually applying the patches.
> 
> I'm less OK with "let's fix it in NSS"... followed by not actually doing so.
> 
> There is very little justification for this fix not making F25.

This bug is assigned to you instead of NSS maintainers (apparently my mistake).  I am resetting the assignee now to the default value...

(In reply to Kamil Dudka from comment #31)
> It is too late to rebuild curl against GnuTLS in f25.  It would be way more
> disruptive than just applying those patches on NSS.

I agree. Let's do that then.

With the NSS fixes from https://bugzil.la/1296263 and https://bugzil.la/1162897 and with our system crypto policy amended as discussed in Fedora bug 1173577 this should all work nicely.

It seems this bug is waiting for a feature enhancement to be loaded into upstream NSS, we should avoid carrying downstream feature patches to NSS.

This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This will be fixed when we rebase NSS to 3.31 in Fedora, right?

If yes, please mark as duplicate of bug 1460051.

No. We've fixed https://bugzilla.mozilla.org/show_bug.cgi?id=1162897 but not https://bugzilla.mozilla.org/show_bug.cgi?id=248722 

That is, if a cert/key is available in our database then we can reference it by the PKCS#11 URI... but the proper tokens *aren't* in our database by default because we're not using p11-kit-proxy.so as we should.

I had a proof-of-concept patch which attempted to resolve that by piggy-backing on the mechanism we use to load the system crypto policy — it was basically a two-line if (!modDB) SECMOD_LoadModule(p11-kit-proxy) in the right place, IIRC. But Bob wasn't happy with it...

This is the discussion of the bit that remains:

https://bugzilla.redhat.com/show_bug.cgi?id=1173577
https://bugzilla.mozilla.org/show_bug.cgi?id=1296263

Although didn't curl get rebuilt against OpenSSL in bug 1445153 thus resolving this original bug (before it got reassigned to nss) anyway?

This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Hi Kamil,
 I'm resetting that bug to curl, as it is moving to openssl and that should work out of the box (so if that's hopefully the case, the issue should be closed).

IIRC one needs to use curl --engine to enable PKCS#11 with OpenSSL but I have never tried it myself.  David, could you please retest this with f27+ curl?

0. Create and example client certificate and private key
1. cat >config
directories.tokendir = db
objectstore.backend = file

2. export SOFTHSM2_CONF=config
3. mkdir db
4. softhsm2-util --init-token --slot 0 --label test --so-pin 1234 --pin 1234
5. p11tool --provider /usr/lib64/pkcs11/libsofthsm2.so --write --load-certificate cert.pem  --label test --login
6. p11tool --provider /usr/lib64/pkcs11/libsofthsm2.so --write --load-privkey privkey.pem  --label test --login

# List items
7. p11tool --list-all "pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;token=test" --login


8. If you use these URIs with curl -E and --key, you should be able to connect with a server which requires certificates. (e.g, gnutls-serv --x509certfile cert.pem --x509keyfile key.pem --cafile ca.pem --require-client-cert)

After some testing:
$ curl https://localhost:5556 -E 'pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=68bb9e47276a0d77;token=test;id=%a4%d2%6e%2d%cf%6e%36%86%bb%10%1b%2d%de%ef%30%ba%a9%74%de%44;object=test;type=cert' --key 'pkcs11:model=SoftHSM%20v2;manufacturer=SoftHSM%20project;serial=68bb9e47276a0d77;token=test;id=%af%55%6d%b4%a8%a9%b1%d6%81%9c%24%cc%de%7b%9d%b9%80%cb%24%f0;object=test;type=private'
curl: (58) could not load PEM client certificate, OpenSSL error error:02001002:system library:fopen:No such file or directory, (no key found, wrong pass phrase, or wrong file format?)


Adding the --engine pkcs11 doesn't change that error message. So it seems that curl doesn't pass the key/cert parameters to the lower level back-end, nor does seem to consider the presence of a 'pkcs11:' URL, a reason to load the engine.

I fixed cURL to automatically load the engine when given a pkcs11: URI. That was a long time ago; I'm fairly sure we should have that by now.

You're probably tripping up on cURL's password handling for key files. It uses a colon for that, so interprets your argument as a filename of "pkcs11" and a password of "model=SoftHSM...".

Escape the colon... is it with a backslash? I don't call. Although I actually thought I'd fixed that too, with a nasty special case based on the assumption that nobody will ever call their key file literally just "pkcs11". And if they do they can refer to it as ./pkcs11 if the really must ;)

(In reply to David Woodhouse from comment #43)
> Escape the colon... is it with a backslash?

I think so.  See comment #1.

Oh, I never did fix this in cURL's OpenSSL support. There were... other issues amd it got ratholed.

https://github.com/curl/curl/issues/974

You were supposed to switch to GnuTLS :)

When I run:
$ curl --engine pkcs11 -E "pkcs11:token=test;object=test;type=cert" --cert-type ENG --key "pkcs11:token=test;object=test;type=private" --key-type ENG --cacert ca.crt https://localhost:5556

It works for me.

(In reply to Anderson Sasaki from comment #46)
> When I run:
> $ curl --engine pkcs11 -E "pkcs11:token=test;object=test;type=cert"
> --cert-type ENG --key "pkcs11:token=test;object=test;type=private"
> --key-type ENG --cacert ca.crt https://localhost:5556
> 
> It works for me.

The issue is that you needed to specify "--engine pkcs11", "--cert-type ENG", "--key-type ENG" in addition to the PKCS#11 URI.

The request is for transparent support for URIs such as:
"curl -E 'pkcs11:token=Gnome2%20Key%20Storage;id=%e6%34%78%b5%fd%86%77%ba%b9%86%95%c3%19%02%74%d1%f5%33%9a%97' -I -v https://auth.startssl.com", similarly to how wget handles that.

This bug appears to have been reported against 'rawhide' during the Fedora 28 development cycle.
Changing version to '28'.

This upstream pull request seems related:

https://github.com/curl/curl/pull/2333

It is already merged upstream:
https://github.com/curl/curl/pull/2333

Thank you for working on this upstream!

Downstream commits:
https://src.fedoraproject.org/cgit/rpms/curl.git/commit/?id=178b0fc8 (f29)
https://src.fedoraproject.org/cgit/rpms/curl.git/commit/?id=ab86f699 (f28)
https://src.fedoraproject.org/cgit/rpms/curl.git/commit/?id=85cdb484 (f27)

curl-7.55.1-13.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d204581751

curl-7.59.0-6.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d342cf1a7f

curl-7.55.1-13.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d204581751

curl-7.59.0-6.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d342cf1a7f

This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

curl-7.59.0-6.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

curl-7.55.1-13.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

*** Bug 1594847 has been marked as a duplicate of this bug. ***

I know that this is closed, but I'll make a comment here to follow up because it's taken so long. 

David Woodhouse asked about passing using openonnect and curl.  I don't us openconnect.  I only have need to use curl. 

Using the form or the longer version of the url provided by pckcs11 tool gives the following answer.

curl -E 'pkcs11:manufacturer=piv_II;id=%01' $URL
curl: (58) SSL Engine 'pkcs11' not found

I also get the following when using the environment variables: 
fatal: unable to access '<url>': could not load PEM client certificate, OpenSSL error error:02001002:system library:fopen:No such file or directory, (no key found, wrong pass phrase, or wrong file format?)

do you have engine_pkcs11 package installed?

There is no such package.  Did you mean openssl-pkcs11?

Yes, openssl-pkcs11 is a rename of the old engine_pkcs11 package.