Bug 1577572

Summary: openconnect doesn't work with key on rutoken
Product: [Fedora] Fedora Reporter: Denis Kostousov <denis.kostousov>
Component: openconnectAssignee: David Woodhouse <dwmw2>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 28CC: denis.kostousov, dwmw2, nmavrogi
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openconnect-8.01-1.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-11 04:34:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
openconnect on Fedora27
none
openconnect on Fedora28
none
p11tool --list-all pkcs11:model=PKCS%2315 fedora 28
none
p11tool --list-all pkcs11:model=PKCS%2315 fedora 27
none
openconnect on Fedora28 gnutls from Fedora27
none
openconnect patch to force login on tokens which don't ask for it
none
p11tool --list-tokens fedora 28 gnutls was downgraded
none
patched openconnect, actual gnutls
none
patched openconnect, downgraded gnutls
none
p11tool --list-tokens fedora 28 gnutls from f28
none
openconnect.f28.gnutls-f27.txt
none
patched openconnect, actual gnutls
none
patched openconnect, actual gnutls, full cert url
none
Work around incompatible GnuTLS ABI change. none

Description Denis Kostousov 2018-05-13 04:53:45 UTC
Description of problem:
I have a rsa private key and x509 cert on rutoken. Usually I connect to remote VPN with a command:
sudo openconnect -c "pkcs11:id=newkeyset" --script /etc/openconnect/vpnc-script -b https://vpn.company.com/+webvpn+/index.html

On Fedora 27 (with last updates) the command dialog looks like this:
---
POST https://vpn.company.com/+webvpn+/index.html
Connected to x.x.x.x:443
PIN required for User PIN (Rutoken ECP)
Enter PIN:
Using client certificate 'kostousov-ds-ra'
SSL negotiation with vpn.company.com
Connected to HTTPS on vpn.company.com
Got HTTP response: HTTP/1.1 301 Moved Permanently
GET https://vpn.company.com/+webvpn+/index.html
Connected to x.x.x.x:443
SSL negotiation with vpn.company.com
Connected to HTTPS on vpn.company.com
Please enter your username and password.
Password:
POST https://vpn.company.com/+webvpn+/index.html
SSL negotiation with vpn.company.com
Connected to HTTPS on vpn.company.com
Got CONNECT response: HTTP/1.1 200 OK
CSTP connected. DPD 30, Keepalive 20
Connected as 10.100.1.10, using SSL
Established DTLS connection (using GnuTLS). Ciphersuite (DTLS0.9)-(DHE-RSA-4294967237)-(AES-256-CBC)-(SHA1). 
---

After upgrade to Fedora 28 I can't connect to VPN. I tried some variants with different key id:

1.
% sudo openconnect -c "pkcs11:id=newkeyset" --script /etc/openconnect/vpnc-script https://vpn.company.com/+webvpn+/index.html         
POST https://vpn.company.com/+webvpn+/index.html
Connected to x.x.x.x:443
Error importing PKCS#11 URL pkcs11:model=PKCS%231;manufacturer=Aktiv%20Co;serial=0000000031296E3;token=User%20PIN%20%28Rutoken%20ECP;id=%6e%65%77%6b%65%79%73%65%74;type=private: The requested PKCS #11 object is not available
Loading certificate failed. Aborting.
Failed to open HTTPS connection to vpn.company.com
Failed to obtain WebVPN cookie

2.
% sudo openconnect -c "pkcs11:id=6e65776b6579736574" --script /etc/openconnect/vpnc-script https://vpn.company.com/+webvpn+/index.html
POST https://vpn.company.com/+webvpn+/index.html
Connected to x.x.x.x:443
PIN required for System Trust
Enter PIN:
Error loading certificate from PKCS#11: PKCS #11 user error
Loading certificate failed. Aborting.
Failed to open HTTPS connection to vpn.company.com
Failed to obtain WebVPN cookie

3.
% sudo openconnect -c "pkcs11:manufacturer=Aktiv%20Co." --script /etc/openconnect/vpnc-script https://vpn.company.com/+webvpn+/index.html
POST https://vpn.company.com/+webvpn+/index.html
Connected to x.x.x.x:443
PIN required for User PIN (Rutoken ECP)
Enter PIN:
Error loading certificate from PKCS#11: The requested data were not available.
Loading certificate failed. Aborting.
Failed to open HTTPS connection to vpn.company.com
Failed to obtain WebVPN cookie

Some diagnostik info:

% p11tool --list-all pkcs11:model=PKCS%2315
Object 0:
        URL: pkcs11:model=PKCS%2315;manufacturer=Aktiv%20Co.;serial=0000000031296E36;token=User%20PIN%20%28Rutoken%20ECP%29;id=%6e%65%77%6b%65%79%73%65%74;object=kostousov-ds-ra;type=public
        Type: Public key (RSA-2048)
        Label: kostousov-ds-ra
        Flags: CKA_WRAP/UNWRAP; 
        ID: 6e:65:77:6b:65:79:73:65:74

Object 1:
        URL: pkcs11:model=PKCS%2315;manufacturer=Aktiv%20Co.;serial=0000000031296E36;token=User%20PIN%20%28Rutoken%20ECP%29;id=%6e%65%77%6b%65%79%73%65%74;object=current;type=cert
        Type: X.509 Certificate (RSA-2048)
        Label: current
        ID: 6e:65:77:6b:65:79:73:65:74

% rpm -qa p11\* pcsc\* opensc\* gnutls\* openconnect\*
gnutls-3.6.2-1.fc28.x86_64
p11-kit-trust-0.23.10-1.fc28.x86_64
pcsc-lite-libs-1.8.23-2.fc28.x86_64
p11-kit-0.23.10-1.fc28.x86_64
opensc-0.17.0-10.fc28.x86_64
gnutls-utils-3.6.2-1.fc28.x86_64
gnutls-dane-3.6.2-1.fc28.x86_64
pcsc-lite-1.8.23-2.fc28.x86_64
openconnect-7.08-5.fc28.x86_64
pcsc-lite-ccid-1.4.29-1.fc28.x86_64

Comment 1 David Woodhouse 2018-05-13 15:02:10 UTC
Hm... my first inclination is to blame Nikos...

Comment 2 Nikos Mavrogiannopoulos 2018-05-14 07:57:16 UTC
Could you please try running the three commands above on F27 and F28 with GNUTLS_DEBUG_LEVEL=6 and attach the 6 log files to see the difference? Unfortunately we don't test rutokens as they cannot be exported from Russia. Do other tokens like yubikey or softhsm work in your setup?

Comment 3 Denis Kostousov 2018-05-14 08:48:27 UTC
Created attachment 1436073 [details]
openconnect on Fedora27

Thanks! I didn't know hot is it to enable log for this command.

Comment 4 Denis Kostousov 2018-05-14 14:31:53 UTC
Created attachment 1436219 [details]
openconnect on Fedora28

The log contains several openconnect invocations

Comment 5 Nikos Mavrogiannopoulos 2018-05-14 19:31:10 UTC
Does the 'p11tool --list-all pkcs11:model=PKCS%2315' command listed above print the same output in F27 and F28?

Comment 6 Denis Kostousov 2018-05-15 03:28:21 UTC
Created attachment 1436638 [details]
p11tool --list-all pkcs11:model=PKCS%2315 fedora 28

Comment 7 Denis Kostousov 2018-05-15 03:28:54 UTC
Created attachment 1436640 [details]
p11tool --list-all pkcs11:model=PKCS%2315 fedora 27

Comment 8 Nikos Mavrogiannopoulos 2018-05-15 07:05:59 UTC
One more request could you run "p11tool --list-tokens" on F28?
What has changed is that gnutls will try to login only when the "Requires login" flag is present. For example:

Token 2:
	URL: pkcs11:model=PKCS%2315%20emulated;manufacturer=www.CardContact.de;serial=DENK0100424;token=UserPIN%20%28SmartCard-HSM%29
	Label: UserPIN (SmartCard-HSM)
	Type: Hardware token
	Flags: RNG, Requires login
	Manufacturer: www.CardContact.de
	Model: PKCS#15 emulated
	Serial: DENK0100424
	Module: opensc-pkcs11.so

Comment 9 Denis Kostousov 2018-05-15 07:41:55 UTC
I can try to downgrade gnutls

Comment 10 Nikos Mavrogiannopoulos 2018-05-15 10:03:18 UTC
Please send the output, so we know whether this is the problem.

Comment 11 Denis Kostousov 2018-05-16 16:33:24 UTC
Created attachment 1437472 [details]
openconnect on Fedora28 gnutls from Fedora27

Comment 12 Nikos Mavrogiannopoulos 2018-05-17 07:21:23 UTC
Denis I wouldn't expect bringing the f27 gnutls in f28 to work. Please sent the output of:
"p11tool --list-tokens"

I'll attach a patch on openconnect which I believe addresses the issue.

Comment 13 Nikos Mavrogiannopoulos 2018-05-17 07:22:25 UTC
Created attachment 1437774 [details]
openconnect patch to force login on tokens which don't ask for it

Comment 14 Nikos Mavrogiannopoulos 2018-05-17 07:32:01 UTC
You can test whether the patch works at:
https://koji.fedoraproject.org/koji/taskinfo?taskID=27007949

Comment 15 Denis Kostousov 2018-05-17 17:30:20 UTC
Created attachment 1438050 [details]
p11tool --list-tokens fedora 28 gnutls was downgraded

Comment 16 Denis Kostousov 2018-05-17 17:40:05 UTC
Created attachment 1438051 [details]
patched openconnect, actual gnutls

Comment 17 Denis Kostousov 2018-05-17 17:40:51 UTC
Created attachment 1438052 [details]
patched openconnect, downgraded gnutls

Comment 18 Nikos Mavrogiannopoulos 2018-05-18 09:29:53 UTC
Can you attach the "p11tool --list-tokens" output from the normal gnutls in fedora 28?

btw. I see that openconnect with the patch now asks you for a PIN, which did not happen previously, is that correct?

btw. It seems the URI you are providing doesn't correspond to any object that is listed by 'p11tool --list-all pkcs11:model=PKCS%2315'. Is it clear that you need to specify an object that you see in that list?

Comment 19 Nikos Mavrogiannopoulos 2018-05-18 09:31:20 UTC
Note that in fedora28 also opensc changed, and for some cards it updated the information that it provides. That means that the PKCS#11 URI may have changed, in your case it seems to be happening.

Comment 20 Denis Kostousov 2018-05-19 20:02:56 UTC
Created attachment 1438989 [details]
p11tool --list-tokens fedora 28 gnutls from f28

Comment 21 Denis Kostousov 2018-05-19 20:07:12 UTC
Created attachment 1438990 [details]
openconnect.f28.gnutls-f27.txt

URL from list-all was used

Comment 22 Nikos Mavrogiannopoulos 2018-05-21 06:52:49 UTC
It gets quite confusing. Please stick to F28 with gnutls for F28. 

If that does not work, try the build that I submit it. Does it work?

Comment 23 Denis Kostousov 2018-05-26 16:55:19 UTC
Created attachment 1442008 [details]
patched openconnect, actual gnutls

Comment 24 Nikos Mavrogiannopoulos 2018-05-28 06:24:35 UTC
Thank you.

In the openconnect command line you specify a token, not an object in the token (e.g., certificate or key):
"pkcs11:token=User%20PIN%20%28Rutoken%20ECP%29"

It is also different than the token that you use in the --list-all command shown at the first comment above. Did you change the token? In openconnect try to specify the output you see in p11tool "--list-all --login" for the token you are interested.

Try the following options if possible:
$ openconnect -c "URL of certificate" 

$ openconnect -c "URL of certificate"  -k "URL of private key"

The "URL of certificate" and key, should match the key and cert URLs you see in
$ p11tool --list-all --login "pkcs11:token=User%20PIN%20%28Rutoken%20ECP%29"

Comment 25 Denis Kostousov 2018-05-28 15:42:28 UTC
Created attachment 1443406 [details]
patched openconnect, actual gnutls, full cert url

Thanks!

This command works

Comment 26 Denis Kostousov 2018-05-28 15:48:48 UTC
After some experiments I found that the patched openconnect works with this short url "pkcs11:model=PKCS%2315;id=%6e%65%77%6b%65%79%73%65%74"

But it fails after removing "model=PKCS%2315" from the url with strange error:
---
Error importing PKCS#11 URL pkcs11:model=PKCS%231;manufacturer=Aktiv%20Co;serial=0000000031296E3;token=User%20PIN%20%28Rutoken%20ECP;id=%6e%65%77%6b%65%79%73%65%74;object=current;type=private: The requested PKCS #11 object is not available
---

Strangeness lies in the fact that I deleted the "PKCS%2315" from the url and in the error only "PKCS%231" without "5".

Comment 27 Nikos Mavrogiannopoulos 2018-05-29 08:02:32 UTC
That because "PKCS%231" is actually a bad hexadecimal number. So I take it that the original openconnect code doesn't work without the patch and the _right_ URI, right?

In that case, David could you bring that patch in openconnect?

Comment 28 David Woodhouse 2018-05-29 10:31:51 UTC
So... GnuTLS no longer automatically logs in when the 'requires login' flag is not present. But... OpenConnect should effectively revert that same change and *should* always force a login? What is the logic here, which makes it the right thing for GnuTLS in the general case, but not for OpenConnect?

I believe it's possible to have private keys which don't require login — I'm fairly sure I can set something up with p11-remote and OpenSC so that key can be used by (e.g.) Apache on the local box, but *without* a password.

OpenConnect can run as an unprivileged user, and I can imagine people setting up unattended VPN that way too. Will OpenConnect be doing the right thing with your patch, in that case?

Should we implement this as a fallback where we try without login first? 

I have other questions too... like *why* isn't the 'requires login' flag set, if a login is required? Is this a bug in OpenSC that we're working around?

Comment 29 Denis Kostousov 2018-05-29 11:11:36 UTC
Hmm... I downgraded openconnect to openconnect-7.08-5.fc28.x86_64 And all works fine on this version with URL pkcs11:model=PKCS%2315;id=%6e%65%77%6b%65%79%73%65%74
If I use URL pkcs11:id=%6e%65%77%6b%65%79%73%65%74 then the openconnect fails with error
---
Error importing PKCS#11 URL pkcs11:model=PKCS%231;manufacturer=Aktiv%20Co;serial=0000000031296E3;token=User%20PIN%20%28Rutoken%20ECP;id=%6e%65%77%6b%65%79%73%65%74;type=private: The requested PKCS #11 object is not available
---

One can see difference between "model=PKCS%2315" in the good URL and "model=PKCS%231" it the error message. I think, this is the bug.

Comment 30 David Woodhouse 2018-05-29 12:03:33 UTC
Created attachment 1445372 [details]
Work around incompatible GnuTLS ABI change.

When OpenConnect is given just an id and the token isn't specified, it looks for the cert in all available tokens.

It then looks for the key in the *same* token in which it found the cert... by querying the label/manufacturer/model/serialNumber fields of the cert, and creating a URI for the key with those.

However... GnuTLS 3.6.0 made an incompatible ABI change and stopped including the NUL terminator on the end of those fields in the output_size it returns. Thus we are dropping the last byte from *all* of those, as you can see in the URI if you look hard. It's less obvious when it's dropping a close parenthesis which is %-encoded anyway, or the dot after "Aktiv Co.", or the final '6' in the serial number.

Try this patch, and a long hard stare at the GnuTLS maintainer for breaking API and ABI...

Comment 31 Nikos Mavrogiannopoulos 2018-05-29 14:34:31 UTC
Ouch, which function do you refer to? That we can revert.

Comment 32 David Woodhouse 2018-05-29 20:47:05 UTC
(In reply to Nikos Mavrogiannopoulos from comment #31)
> Ouch, which function do you refer to? That we can revert.

(summarising the IRC conversation for posterity)

https://www.gnutls.org/reference/gnutls-pkcs11.html#gnutls-pkcs11-obj-get-info
indicates that gnutls_pkcs11_obj_get_info() no longer includes the NUL terminator in the returned size, which is what led to those four fields being truncated by one byte after updating to GnuTLS 3.6.

That was an ABI change which shouldn't have been made without an soname bump. I've worked around it with the above patch, which is now committed as
http://git.infradead.org/users/dwmw2/openconnect.git/commitdiff/9362354f48f

I was dubious about reverting it (on the basis that changing ABI *twice* isn't necessarily better than changing it once).... but on reflection, maybe it is the best idea after all. After all, there have been no interim *major* releases of GnuTLS, so we can fix it in 3.6.3 and have a reasonable expectation that distros *should* upgrade and not stay on 3.6.[012]. And I can update my workaround to cope.

Comment 33 Fedora Update System 2019-01-07 07:43:01 UTC
openconnect-8.01-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-267b29539a

Comment 34 Fedora Update System 2019-01-08 02:04:54 UTC
openconnect-8.01-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-267b29539a

Comment 35 Fedora Update System 2019-01-11 04:34:00 UTC
openconnect-8.01-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.