Bug 1446795

Summary: OpenVPN does not execute (--tls-verify) script in /etc/openvpn/server
Product: [Fedora] Fedora EPEL Reporter: Levente Farkas <lfarkas>
Component: openvpnAssignee: David Sommerseth <dazo>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: epel7CC: dazo, huzaifas, mauricio.teixeira, steve
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-12 21:15:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Levente Farkas 2017-04-28 22:32:49 UTC
while the old fashioned systemd file still working the new files are not working if you try to use crl and tls-verify.
while exactly the same config files working in the /etc/openvpn directory if i move everything into /etc/openvpn/server then clients no longer be able to connect since i've got this errors:
--------------------------------------
80.99.47.225:1075 WARNING: Failed to stat CRL file, not (re)loading CRL.
80.99.47.225:1075 WARNING: Failed running command (--tls-verify script): could not execute external program
80.99.47.225:1075 OpenSSL: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
80.99.47.225:1075 TLS_ERROR: BIO read tls_read_plaintext error
80.99.47.225:1075 TLS Error: TLS object -> incoming plaintext read error
80.99.47.225:1075 TLS Error: TLS handshake failed
-----------------------------------------
of course it's not ssl or cert error since if i move the files back it's working again. but i'm not able to find out the reason why.

Comment 1 David Sommerseth 2017-04-28 23:27:24 UTC
Oh, interesting!  It might be the ownership is too strict on /etc/openvpn/server.  Try doing: # chown openvpn /etc/openvpn/server

I see that the default is root:root, which normally is fine.  Except when you place scripts and CRL files there and use --user/--group in addition.

That said, I'm not too fond of having scripts in the same directory as the configuration files.  And I'd like to see SELinux be improved for OpenVPN as well too, which would also make that harder to achieve.  Perhaps we should consider to ship a new directory in addition, /etc/openvpn/scripts where those can be placed.

Comment 2 Levente Farkas 2017-04-29 11:51:59 UTC
i didn't mention but i already try many things. the whole directoty owned by openvpn, change permission, etc. but none of them working. anyway it's clean that if i move everything to /etc/openvpn with the same owner permission etc then everything is working while in th server subdirectorz it's not working. so it's clear that it's not permsission and ownnership etc.

Comment 3 David Sommerseth 2017-05-11 20:37:19 UTC
I have setup a brand new installation, using the EPEL-7 build.  This works out-of-the box with both the new and old unit files.

Please share your OpenVPN configuration file, so we can have a deeper look into what is going on.  And also an overview of the related OpenVPN files in /etc/openvpn ... which files are located in which directory?

Comment 4 Levente Farkas 2017-05-11 21:36:00 UTC
proto udp
dev-type tun
dev vpn-udp

tls-server

ca              ca.crt
tls-auth        ta.key 0
cert            vpn.example.com.crt
key             vpn.example.com.key
dh              dh2048.pem
crl-verify      crl.pem
tls-verify      "./verify-cn ccd-udp"
script-security 2
remote-cert-tls client
#tls-version-min 1.2
auth            SHA256
cipher          AES-256-CBC
ncp-ciphers     AES-256-GCM:AES-128-GCM:AES-256-CBC
tls-cipher      TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA

topology subnet
client-to-client
server 172.30.40.0 255.255.255.0
ifconfig-pool-persist ipp-udp.txt
verb 1
#status  /var/log/openvpn-status-udp.log
log     /var/log/openvpn-server-udp.log

user openvpn
group openvpn

comp-lzo no
persist-tun
persist-key
persist-local-ip
keepalive 10 120

client-config-dir ccd-udp

push "comp-lzo no"
push "persist-tun"
push "persist-key"

Comment 5 Levente Farkas 2017-05-11 21:37:43 UTC
use a simple shell script which return true as verify-cn and fill all cert with a valid value.
the server starts and after the first client try to connect it gives the above error.

Comment 6 David Sommerseth 2017-05-11 22:22:45 UTC
My test script:
-----------------------------------------------
# cat ovpn-test-script.sh 
#!/bin/bash

printenv > /var/log/test.log
echo "---------" >>  /var/log/test.log
echo "ARGS: $*" >>  /var/log/test.log
exit 0
# touch /var/log/test.log
# chown openvpn: /var/log/test.log
# chmod /etc/openvpn/server/ovpn-test-script.sh
# chmod o+x /etc/openvpn/server
-----------------------------------------------

This works like a charm on my setup.  But do notice the last 'chmod' line.  You use --user/--group openvpn.  By default only root have access to /etc/openvpn/{client,server}.  You may also get around this by using setfacl or changing the group-owner of the that directory.

I anyhow recommend you to rather save your VPN scripts outside of /etc ... it is not the most ideal place to save scripts in the same directory as you have configuration files and keying material - especially not when considering that remote users is triggering the execution of these scripts.

Comment 7 David Sommerseth 2017-05-11 22:25:06 UTC
By the way ... this is not related to the new unit files at all.  I could trigger the exact same issue with lacking permissions by running as root:

   # /usr/sbin/openvpn --config /etc/openvpn/server/testserver.conf

Comment 8 Levente Farkas 2017-05-12 09:36:50 UTC
ok. you've got right!
but in this case it's a packaging bug since with the old unit file the same config working. imho the server directory should have to be 755 mode and not 750!
so IMHO it's still a bug since it's not just the scripts but the crl file reading is also cause an error. and i hope you agree that it's have to be in that directory!

Comment 9 Levente Farkas 2017-05-12 09:37:25 UTC
otherwise the directory /etc/openvpn also can be 750 but it's also 755.

Comment 10 David Sommerseth 2017-05-12 21:15:19 UTC
We still ship the *old* unit file, nothing changes there.  It should work out-of-the-box as before.  The *new* unit files does not in anyway depend on the old way of doing things.  These new unit files introduces a newer and more stricter regime, to protect sensitive information.

/etc/openvpn will *not* change mode from 0755, because that would break existing setups.  That is something we always try to avoid doing as far as it is possible to avoid.

In regards to the CRL causing an error, I do hope you are aware of the remarks done here: https://github.com/OpenVPN/openvpn/blob/v2.4.1/Changes.rst#deprecated-features

"the crl file reading is also cause an error" doesn't provide any information at all, it's like saying "the car doesn't move forward".  I suspect it is a similar issue to your scripts.  The user OpenVPN is running as (--user/--group) needs to be able to read the CRL file.  Whether you place that file in /etc/openvpn, /etc/openvpn/server, /etc/pki/tls or some other directory is really not that relevant - it is completely decided by your configuration file; and you need to ensure the privileges on the files are set accordingly to what you have configured OpenVPN to expect.

Since your tunnel work with the *old* regime and not the *new* regime, this issues with privileges is hardly a packaging bug.


On the more general side of things ...

The *new* /etc/openvpn/client and /etc/openvpn/server enforces an improved security to protect keying material which often is seen together with the OpenVPN configuration files.  In fact, private keys and --tls-auth/--tls-crypt keys can also be _embedded_ into the configuration files.  Since these directories are _new_, no existing setups should break.

When you migrate your configuration from the old to the new model, you need to take care of all the needed details to ensure things works.  It is not expected that all setups will work instantly by just moving the files.  Since there are more ways to configure OpenVPN than we two together can manage to imagine (OpenVPN consists of over 240 options), this is also why we do not do any type of automatic migration.  Migrations needs to be done manually and be supervised by someone who can look at the machine directly and do the needed adaptations.

Comment 11 Levente Farkas 2017-05-12 21:45:24 UTC
all other files should have to be readable. ca, cert, key ....
and as all docs examples place these files into the same dir as the conf files most people will do it in the same way. it has nothing to do with the new regime.
so imho /etc/openvpn/server either should have to owned by openvpn or has to be 755 mode.

Comment 12 David Sommerseth 2017-05-15 16:16:10 UTC
(In reply to Levente Farkas from comment #11)
> all other files should have to be readable. ca, cert, key ....
> and as all docs examples place these files into the same dir as the conf
> files most people will do it in the same way. it has nothing to do with the
> new regime.
> so imho /etc/openvpn/server either should have to owned by openvpn or has to
> be 755 mode.

No.  This is grave misunderstanding of how OpenVPN should be configured.

You have two options:

a)  use --persist-key.  This makes OpenVPN to load the all the keying material before it drops privileges and it will cache them in memory during the process' lifetime.  It will never need to re-read those files again and the files can be owned by root if you wish.

b) use inline files.  --ca/--cert/--key/--dh/--pkcs12/--tls-auth/--tls-crypt all support <ca>...</ca>, etc in the configuration files.  For PKCS#12 files, they must be BASE64 encoded.  The configuration file can be owned by root only and will not be re-read after it have dropped privileges.

Of course, you have the option of not using --user/--group too, but that is not helping much on the security side.  And if using --user/--group you should anyway add --persist-tun, otherwise OpenVPN will fail in those cases requiring re-init of the tun/tap adapter.

Yes, there are other challenges with --crl and all the various script hooks.  Those are most commonly not security sensitive (such as private keys), so they _may_ reside in /etc/openvpn.  But it is by far better to have the scripts in a bin/ or libexec/ directory somewhere and perhaps even the CRL file in /var/lib/openvpn ... which is where application data more belongs.  A while ago we started a discussion of the file directories needed for OpenVPN and the recommended security settings for them, unfortunately that did not complete - but I will ensure this gets brought up again.

We really try to help people increasing the security of their setups, and not having a "status quo because that works best for me".  Which is why the new folders are setup stricter than then base /etc/openvpn.  This ensures the less sensitive files can be stored in /etc/openvpn while the new security defaults are enforced in /etc/openvpn/{client,server}.


If there are bugs related to how OpenVPN itself behaves in these scenarios, then that needs to be handled upstream.  But for packaging OpenVPN we really want to have an increased security level on basic installs.


You may be of a different opinion, that's fine.  But this should also go both ways.

Comment 13 Levente Farkas 2017-05-15 18:56:11 UTC
(In reply to David Sommerseth from comment #12)
> You have two options:
> 
> a)  use --persist-key.  This makes OpenVPN to load the all the keying
> material before it drops privileges and it will cache them in memory during
> the process' lifetime.  It will never need to re-read those files again and
> the files can be owned by root if you wish.

i've persist-key but it's still not able to read crl!
 
> b) use inline files.  --ca/--cert/--key/--dh/--pkcs12/--tls-auth/--tls-crypt
> all support <ca>...</ca>, etc in the configuration files.  For PKCS#12
> files, they must be BASE64 encoded.  The configuration file can be owned by
> root only and will not be re-read after it have dropped privileges.

it's not a real life situation to put crl inline. since under normal circumstances you'd have to regenerate it regularly.

so none of these 2 options works.  
 
> Yes, there are other challenges with --crl and all the various script hooks.
> Those are most commonly not security sensitive (such as private keys), so
> they _may_ reside in /etc/openvpn.  But it is by far better to have the
> scripts in a bin/ or libexec/ directory somewhere and perhaps even the CRL
> file in /var/lib/openvpn ... which is where application data more belongs. 

it's not a really fhs compatible way. form
https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

/var/lib
State information. Persistent data modified by programs as they run, e.g., databases, packaging system metadata, etc.

imho crl is not an openvpn generated file:-)

Comment 14 David Sommerseth 2017-05-15 21:23:41 UTC
(In reply to Levente Farkas from comment #13)
> (In reply to David Sommerseth from comment #12)
> > You have two options:
> > 
> > a)  use --persist-key.  This makes OpenVPN to load the all the keying
> > material before it drops privileges and it will cache them in memory during
> > the process' lifetime.  It will never need to re-read those files again and
> > the files can be owned by root if you wish.
> 
> i've persist-key but it's still not able to read crl!

CRL is not a key, and it is read upon each client connect to fetch the latest updates.

> > b) use inline files.  --ca/--cert/--key/--dh/--pkcs12/--tls-auth/--tls-crypt
> > all support <ca>...</ca>, etc in the configuration files.  For PKCS#12
> > files, they must be BASE64 encoded.  The configuration file can be owned by
> > root only and will not be re-read after it have dropped privileges.
> 
> it's not a real life situation to put crl inline. since under normal
> circumstances you'd have to regenerate it regularly.

Did I list --crl above?  No, because that is not really much of a useful feature.

> > Yes, there are other challenges with --crl and all the various script hooks.
> > Those are most commonly not security sensitive (such as private keys), so
> > they _may_ reside in /etc/openvpn.  But it is by far better to have the
> > scripts in a bin/ or libexec/ directory somewhere and perhaps even the CRL
> > file in /var/lib/openvpn ... which is where application data more belongs. 
> 
> it's not a really fhs compatible way. form
> https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
> 
> /var/lib
> State information. Persistent data modified by programs as they run, e.g.,
> databases, packaging system metadata, etc.

CRLs are essentially a signed list of revoked certificates, which can also be considered a database of revoked certificates.  But if you want to be so strict, you also don't use /var/lib/libvirt/images or what about docker images?

> imho crl is not an openvpn generated file:-)

No, OpenPVN does not generate the CRL files. But you also have such a strict interpreation of the FHS that there is no real location for CRL files - I can even argue that the CRL is not a configuration file.  But this really leads nowhere and will just end up on word-splitting.  The quotation you bring in from FHS also says "[...], etc".

And, despite all this ... there is *nothing* restricting you to put the CRL in /etc/openvpn if you wish so.  The CRL is by default considered to be a public file, so it does not require the same protection as private or shared keys - nor other more sensitive which configuration files may contain (like LDAP or database passwords).  Putting it in /etc/openvpn does not require you to change the ACLs in any ways.  Plus CRL files can be shared across server and client configurations, as long as the CA issuing the CRL and the certificates for both server and client configurations is the same.

You also seem to ignore the fact that when using script hooks or plug-ins, they run with the same privileges as OpenVPN - so they can very much get access to the private keys/passwords unless those files are strictly protected.  I would even argue that configuration files and private keys/passwords is not something they should really need access to - the information they need should be passed to them via the arguments provided in the OpenVPN configuration.

You might argue that there should be a private/ directory for private keys and passwords, but when considering that the OpenVPN configuration may very well contain that information inlined, you can't really put the configuration file easily inside a private/ directory - and to handle that in a good way through systemd unit files will be just even worse ... so we don't use the private/ name.  We use client/ and server/ for that - as that makes the overall directory layout simpler and less confusing.