Bug 1760253

Summary: --ping-restart does not work
Product: [Fedora] Fedora Reporter: Sami Farin <hvtaifwkbgefbaei>
Component: NetworkManager-openvpnAssignee: Lubomir Rintel <lkundrak>
Status: ASSIGNED --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 38CC: bgalvani, bojan, choeger, code, david.00.fernandez, dazo, dcbw, fgiudici, huzaifas, klember, lkundrak, steve, thaller
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-25 17:33:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sami Farin 2019-10-10 09:08:41 UTC
Description of problem:
openvpn started with --ping-restart 30 --persist-key --persist-tun does not properly retry on connection timeout.

Version-Release number of selected component (if applicable):
2.4.7

How reproducible:
100%

Steps to Reproduce:
1. connect
2. wait for/cause some network trouble
3.

Actual results:
Oct 10 02:42:17 nm-openvpn[252391]: [x] Inactivity timeout (--ping-restart), restarting
Oct 10 02:42:17 nm-openvpn[252391]: SIGUSR1[soft,ping-restart] received, process restarting
Oct 10 02:42:22 nm-openvpn[252391]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Oct 10 02:42:22 nm-openvpn[252391]: NOTE: setsockopt TCP_NODELAY=1 failed
Oct 10 02:42:22 nm-openvpn[252391]: UDP link local: (not bound)
Oct 10 02:42:29 nm-openvpn[252391]: GDG6: NLMSG_ERROR: error No route to host
Oct 10 02:42:29 nm-openvpn[252391]: ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
Oct 10 02:42:29 NetworkManager[6022]: <warn>  [1570668149.6681] vpn-connection[0x560fca5b0710,232064d1-973c-430c-a290-384a8ff5247b,"x",8:(tun0)]: VPN plugin: failed: connect-failed (1)
Oct 10 02:42:29 nm-openvpn[252391]: Exiting due to fatal error


Expected results:
retry! I used --ping-retry .

Additional info:

Comment 1 David Sommerseth 2019-10-10 09:14:50 UTC
When you get the first initial connection running (before --ping-restart kicks in), can you please provide the output if `ps faxuw | grep openvpn` ?

The nm-openvpn plugin has its own configuration parser which is regenerated as command line options which is used when the openvpn process is started.  This is just to verify how the configuration is being interpreted by NetworkManager.

Comment 2 David Sommerseth 2019-10-10 09:15:44 UTC
oh, you need 'ps faxuww | grep openvpn' .... two 'w', to get the complete command line.

Comment 3 Sami Farin 2019-10-10 09:33:17 UTC
nm-open+  276272  0.3  0.0  10980  7684 ?        S    10:41   0:00 /usr/sbin/openvpn --remote x 1301 udp --remote-random --tun-ipv6 --comp-lzo yes --ping 60 --ping-restart 30 --nobind --dev tun --cipher AES-256-GCM --auth SHA384 --auth-nocache --remote-cert-tls server --reneg-sec 0 --verb 1 --syslog nm-openvpn --script-security 2 --up /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 276269 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_13 --tun -- --up-restart --persist-key --persist-tun --management /var/run/NetworkManager/nm-openvpn-232064d1-973c-430c-a290-384a8ff5247b unix --management-client-user root --management-client-group root --management-query-passwords --auth-retry interact --route-noexec --ifconfig-noexec --client --auth-user-pass --ca /etc/x --user nm-openvpn --group nm-openvpn

Comment 4 David Sommerseth 2019-10-10 10:13:33 UTC
Thank you!  This will be a bit of fumbling in the dark, unless we can get the openvpn process to be more verbose (--verb 4 would be ideal now, but I never managed to figure out how to tweak this with NM).

There are a few things here which might cause an issue, but at this point it's far from a theory.

* "GDG6: NLMSG_ERROR: error No route to host" ... This indicates an error retrieving default IPv6 gateway - which can be due to privilege issues or that the IPv6 config is suddenly not available.
* "ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)" ... This indicates privilege issues, not being able to modify the tun device setup

Even though --persist-tun is used, there are situations where the tun device needs to be recreated too - like if the configuration pushed from the server modifies the VPN IP address of the client or passes a different routing table.

Can you try to run this configuration as root using the openvpn command line directly and see how that behaves?  What is unclear here is whether it is a misbehaviour in the OpenVPN code base in some corner cases with --ping-restart, or if it is the privilege separation model used by NM causing OpenVPN to lack enough privileges to do what it need to do to recover.
e-noexec --ifconfig-noexec --client --auth-user-pass --ca /etc/x --user nm-openvpn --group nm-openvpn

Comment 5 Sami Farin 2019-10-10 10:45:30 UTC
I had omitted some debug lines from this report:

Oct 10 02:42:28 nm-openvpn[252391]: Preserving previous TUN/TAP instance: tun0
Oct 10 02:42:28 nm-openvpn[252391]: /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 252388 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_10 --tun -- tun0 1500 1553 10.15.0.9 255.255.0.0 restart
Oct 10 02:42:28 dbus-broker[6021]: A security policy denied :1.1048 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetConfig to :1.1040.
Oct 10 02:42:28 dbus-broker[6021]: A security policy denied :1.1048 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetIp4Config to :1.1040.
Oct 10 02:42:28 nm-openvpn[252391]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Oct 10 02:42:28 dbus-broker[6021]: A security policy denied :1.1048 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetIp6Config to :1.1040.
Oct 10 02:42:28 NetworkManager[6022]: <info>  [1570668148.6601] device (tun0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
Oct 10 02:42:28 systemd[1]: Starting Network Manager Script Dispatcher Service...
Oct 10 02:42:28 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Oct 10 02:42:28 systemd[1]: Started Network Manager Script Dispatcher Service.
Oct 10 02:42:28 nm-dispatcher[259223]: ok

So it got some new configs...  and NetworkManager didn't restart openvpn.  openvpn was running as user nm-openvpn, and it didn't have CAP_NET_ADMIN capability?

Comment 6 David Sommerseth 2019-10-10 11:01:20 UTC
The whole interaction between OpenVPN and NM is fairly tricky and nasty.  OpenVPN 2.x was designed without any network management in mind (18 years ago or so).  But it provides a few different interfaces (scripts, plug-ins and TCP based management interface), which NM tries to take advantage of.  The challenge here is that both NM and OpenVPN wants to manage the tun interface and related configurations.

When I said increasing the debug level, I really meant the --verb argument which NM-openvpn sets when starting the OpenVPN process.  This gives an insight to what OpenVPN is doing and why things are failing.  The lines tagged with 'dbus-broker' in your comment #5 is the NM-opepnvpn plugin log.

To the privilege issues.  OpenVPN is designed to be started as root, to create and prepare the tun interface and then it drops privileges once everything is set up.  Now, things gets more complicated with NM-openvpn, as it instructs OpenVPN to "don't configure routing or tun interface" (
--route-noexec --ifconfig-noexec).  And then all this info is rather passed to NM-openvpn via the /usr/libexec/nm-openvpn-service-openvpn-helper executable, which then does all the network configuration.  But even with this setup, OpenVPN is the one needing to create and destroy the tun interface - it just "outsources" the IP address and routing setup to go via NM-openvpn.  

Which leads back to try running OpenVPN from the command line directly, without using NM at all.  Just to see if you still have an issue.  So run OpenVPN as root like this:    openvpn --config $CONFIG_FILE --verb 4   .... and see how this runs when --ping-restart kicks in.

If the manual OpenVPN run above works fine, then it definitely is a bug in NM-openvpn and not something OpenVPN can really fix.  If you still have --ping-restart issues, then with --verb 4 it should be enough information to better understand what happens and to see what needs to be improved.

Comment 7 Sami Farin 2019-10-10 13:58:52 UTC
I ran with --verb 4, then  just rebooted the router and waited.

Oct 10 15:15:17 openvpn[300901]: PUSH: Received control message: 'PUSH_REPLY,dhcp-option ...
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: compression parms modified
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: --socket-flags option modified
Oct 10 15:15:17 openvpn[300901]: NOTE: setsockopt TCP_NODELAY=1 failed
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: --ifconfig/up options modified
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: route options modified
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: route-related options modified
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: peer-id set
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: adjusting link_mtu to 1625
Oct 10 15:15:17 openvpn[300901]: OPTIONS IMPORT: data channel crypto options modified
Oct 10 15:15:17 openvpn[300901]: Data Channel MTU parms [ L:1553 D:1450 EF:53 EB:406 ET:0 EL:3 ]
Oct 10 15:15:17 openvpn[300901]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Oct 10 15:15:17 openvpn[300901]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Oct 10 15:15:17 openvpn[300901]: Preserving previous TUN/TAP instance: tun0
Oct 10 15:15:17 openvpn[300901]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Oct 10 15:15:17 openvpn[300901]: /sbin/ip route del 186.1.2.3/32
Oct 10 15:15:17 openvpn[300901]: ERROR: Linux route delete command failed: external program exited with error status: 2
...and so on...
Oct 10 15:15:17 openvpn[300901]: Closing TUN/TAP interface
Oct 10 15:15:17 openvpn[300901]: /sbin/ip addr del dev tun0 10.15.0.4/16
Oct 10 15:15:17 openvpn[300901]: Linux ip addr del failed: external program exited with error status: 2
...
Oct 10 15:15:18 openvpn[300901]: ROUTE_GATEWAY 192.168.178.1/255.255.255.0 IFACE=eno1 HWADDR=aa:ee:48:a9:09:e0
Oct 10 15:15:18 openvpn[300901]: GDG6: remote_host_ipv6=n/a
Oct 10 15:15:18 openvpn[300901]: GDG6: NLMSG_ERROR: error No route to host
Oct 10 15:15:18 openvpn[300901]: ROUTE6: default_gateway=UNDEF
Oct 10 15:15:18 openvpn[300901]: ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
Oct 10 15:15:18 openvpn[300901]: Exiting due to fatal error

Comment 8 David Sommerseth 2019-10-10 15:12:16 UTC
Alright, this makes a lot more sense now.  The NOTE line in your log (copied below) is the cause if this issue.  This is a known issue with many OpenVPN configurations.

   Oct 10 15:15:17 openvpn[300901]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.

This means something changed with the VPN IP address and/or routing, which required a tear-down and re-establishing of the tun device.

One way to avoid this is to run OpenVPN as root, then it will have the needed privileges.  But I also wonder if starting OpenVPN via systemd (using the openvpn-client@$CONFIGNAME.service approach) will also allow this to function, as this model grants the CAP_NET_ADMIN capability to the OpenVPN process, regardless of the root-downgrade.  And if the systemd fails now, it should work with the coming OpenVPN 2.5 release (no ETA yet), which integrates with the Linux kernel Netlink API; thus not requiring to run external programs to configure the tun device.

But you can also consider to try the OpenVPN 3 based Linux client ... https://community.openvpn.net/openvpn/wiki/OpenVPN3Linux  (This is my project).  This should also not have these issues, as it tries to rethink how OpenVPN should work on a modern Linux distribution, including a far better privilege separation model.

I'm sorry I don't have any better solutions integrating with NM currently.

Comment 9 Ben Cotton 2020-04-30 20:59:09 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 10 Ben Cotton 2020-05-26 17:25:37 UTC
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 11 David Fernandez 2020-10-29 12:41:50 UTC
This also happens in Fedora 33...

> $ journalctl --follow
> nm-openvpn[9273]: [OpenVPN] Inactivity timeout (--ping-restart), restarting
> nm-openvpn[9273]: SIGUSR1[soft,ping-restart] received, process restarting
> nm-openvpn[9273]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
> nm-openvpn[9273]: TCP/UDP: Preserving recently used remote address: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[9273]: UDP link local: (not bound)
> nm-openvpn[9273]: UDP link remote: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[9273]: [OpenVPN] Inactivity timeout (--ping-restart), restarting
> nm-openvpn[9273]: SIGUSR1[soft,ping-restart] received, process restarting
> nm-openvpn[9273]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
> nm-openvpn[9273]: TCP/UDP: Preserving recently used remote address: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[9273]: UDP link local: (not bound)
> nm-openvpn[9273]: UDP link remote: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[9273]: [OpenVPN] Peer Connection Initiated with [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[9273]: Preserving previous TUN/TAP instance: tun0
> nm-openvpn[9273]: /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 9265 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_29 --tun -- tun0 1500 1552 XXX.XXX.XXX.XXX 255.255.255.0 restart
> dbus-broker[1457]: A security policy denied :1.1366 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetConfig to :1.1299.
> dbus-broker[1457]: A security policy denied :1.1366 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetIp4Config to :1.1299.
> dbus-broker[1457]: A security policy denied :1.1366 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetIp6Config to :1.1299.
> nm-openvpn[9273]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
> NetworkManager[1584]: <info>  [1603971369.8287] device (tun0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
> gnome-shell[2487]: Removing a network device that was not added
> systemd[1]: Starting Network Manager Script Dispatcher Service...
> systemd[1]: Started Network Manager Script Dispatcher Service.
> audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> nm-openvpn[9273]: ERROR: Cannot ioctl TUNSETIFF tun0: Operation not permitted (errno=1)
> nm-openvpn[9273]: Exiting due to fatal error
> NetworkManager[1584]: <warn>  [1603971370.8361] vpn-connection[0x5628fb4fa350,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",17:(tun0)]: VPN plugin: failed: connect-failed (1)
> NetworkManager[1584]: <info>  [1603971370.8361] vpn-connection[0x5628fb4fa350,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",17:(tun0)]: VPN plugin: state changed: stopping (5)
> NetworkManager[1584]: <info>  [1603971370.8361] vpn-connection[0x5628fb4fa350,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",17:(tun0)]: VPN plugin: state changed: stopped (6)
> NetworkManager[1584]: <info>  [1603971370.8392] vpn-connection[0x5628fb4fa350,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",0]: VPN service disappeared
> systemd[1]: NetworkManager-dispatcher.service: Succeeded.
> audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

From David Sommerseth <dazo>
> One way to avoid this is to run OpenVPN as root, then it will have the needed
> privileges.  But I also wonder if starting OpenVPN via systemd (using the
> openvpn-client@$CONFIGNAME.service approach) will also allow this to function,
> as this model grants the CAP_NET_ADMIN capability to the OpenVPN process,
> regardless of the root-downgrade.  And if the systemd fails now, it should work
> with the coming OpenVPN 2.5 release (no ETA yet), which integrates with the
> Linux kernel Netlink API; thus not requiring to run external programs to
> configure the tun device.

Given that in the Fedora Desktop, OpenVPN runs out of NetworkManager, is there a
way to make NetworkManager call it with the right privileges or the correct form?
e.g. Modifying some NM script for this or all connections as a workaround while
an official fix appears.

Comment 12 David Sommerseth 2020-10-29 14:26:50 UTC
OpenVPN 2.5 was released yesterday.  For Fedora 33 and older, this is available from here: https://copr.fedorainfracloud.org/coprs/dsommers/openvpn-release/

What could be a solution, which would require some changes to NetworkManager-openvpn, is to ensure the openvpn process runs with CAP_NETADMIN privileges. Then it should be possible for OpenVPN 2.5 to run without root and still be able to reconfigure the network, as OpenVPN 2.5 uses a netlink API directly against the kernel instead of executing binaries.

I'm switching this over to the NetworkManager-openvpn component to get their attention.

Comment 13 David Fernandez 2020-10-30 11:10:30 UTC
Tried with openvpn-2.5.0 from your repo copr.fedorainfracloud.org/dsommers/openvpn-release, but it all appears the same:

> # yum copr enable dsommers/openvpn-release
> Enabling a Copr repository. Please note that this repository is not part
> of the main distribution, and quality may vary.
> 
> The Fedora Project does not exercise any power over the contents of
> this repository beyond the rules outlined in the Copr FAQ at
> <https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,
> and packages are not held to any quality or security level.
> 
> Please do not file bug reports about these packages in Fedora
> Bugzilla. In case of problems, contact the owner of this repository.
> 
> Do you really want to enable copr.fedorainfracloud.org/dsommers/openvpn-release? [y/N]: y
> Repository successfully enabled.
> 
> # yum update openvpn
> Last metadata expiration check: 0:00:43 ago on Fri 30 Oct 2020 10:40:19 GMT.
> Dependencies resolved.
> ========================================================================================
>  Package
>       Arch   Version      Repository                                               Size
> ========================================================================================
> Upgrading:
>  openvpn
>       x86_64 2.5.0-1.fc33 copr:copr.fedorainfracloud.org:dsommers:openvpn-release 637 k
> 
> Transaction Summary
> ========================================================================================
> Upgrade  1 Package
> 
> Total download size: 637 k
> Is this ok [y/N]: y
> Downloading Packages:
> openvpn-2.5.0-1.fc33.x86_64.rpm                         396 kB/s | 637 kB     00:01    
> ----------------------------------------------------------------------------------------
> Total                                                   395 kB/s | 637 kB     00:01     
> warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:dsommers:openvpn-release-f5cc63c490988984/packages/openvpn-2.5.0-1.fc33.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 17454df8: NOKEY
> Copr repo for openvpn-release owned by dsommers          14 kB/s | 1.0 kB     00:00    
> Importing GPG key 0x17454DF8:
>  Userid     : "dsommers_openvpn-release (None) <dsommers#openvpn-release.org>"
>  Fingerprint: 6E8D A604 0BB3 E856 FD0A E9D5 54C5 1981 1745 4DF8
>  From       : https://download.copr.fedorainfracloud.org/results/dsommers/openvpn-release/pubkey.gpg
> Is this ok [y/N]: y
> Key imported successfully
> Running transaction check
> Transaction check succeeded.
> Running transaction test
> Transaction test succeeded.
> Running transaction
>   Preparing        :                                                                1/1 
>   Running scriptlet: openvpn-2.5.0-1.fc33.x86_64                                    1/1 
>   Running scriptlet: openvpn-2.5.0-1.fc33.x86_64                                    1/2 
>   Upgrading        : openvpn-2.5.0-1.fc33.x86_64                                    1/2 
>   Running scriptlet: openvpn-2.5.0-1.fc33.x86_64                                    1/2 
>   Running scriptlet: openvpn-2.4.9-2.fc33.x86_64                                    2/2 
>   Cleanup          : openvpn-2.4.9-2.fc33.x86_64                                    2/2 
>   Running scriptlet: openvpn-2.4.9-2.fc33.x86_64                                    2/2 
>   Verifying        : openvpn-2.5.0-1.fc33.x86_64                                    1/2 
>   Verifying        : openvpn-2.4.9-2.fc33.x86_64                                    2/2 
> 
> Upgraded:
>   openvpn-2.5.0-1.fc33.x86_64                                                           
> 
> Complete!
> 
> # journalctl --follow | grep --color=none -Ee '[Oo]penvpn|[Nn]etwork'
> ...
> nm-openvpn[107443]: [OpenVPN] Inactivity timeout (--ping-restart), restarting
> nm-openvpn[107443]: SIGUSR1[soft,ping-restart] received, process restarting
> nm-openvpn[107443]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
> nm-openvpn[107443]: TCP/UDP: Preserving recently used remote address: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: UDP link local: (not bound)
> nm-openvpn[107443]: UDP link remote: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: [UNDEF] Inactivity timeout (--ping-restart), restarting
> nm-openvpn[107443]: SIGUSR1[soft,ping-restart] received, process restarting
> nm-openvpn[107443]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
> nm-openvpn[107443]: TCP/UDP: Preserving recently used remote address: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: UDP link local: (not bound)
> nm-openvpn[107443]: UDP link remote: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: [OpenVPN] Peer Connection Initiated with [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: [OpenVPN] Inactivity timeout (--ping-restart), restarting
> nm-openvpn[107443]: SIGUSR1[soft,ping-restart] received, process restarting
> nm-openvpn[107443]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
> nm-openvpn[107443]: TCP/UDP: Preserving recently used remote address: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: UDP link local: (not bound)
> nm-openvpn[107443]: UDP link remote: [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: [OpenVPN] Peer Connection Initiated with [AF_INET]xxx.xxx.xxx.xxx:yyyy
> nm-openvpn[107443]: Preserving previous TUN/TAP instance: tun0
> nm-openvpn[107443]: /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 107435 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_77 --tun -- tun0 1500 1552 XXX.XXX.XXX.XXX 255.255.255.0 restart
> dbus-broker[1457]: A security policy denied :1.7949 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetConfig to :1.7890.
> dbus-broker[1457]: A security policy denied :1.7949 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetIp4Config to :1.7890.
> dbus-broker[1457]: A security policy denied :1.7949 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetIp6Config to :1.7890.
> nm-openvpn[107443]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
> nm-openvpn[107443]: net_addr_v4_del: XXX.XXX.XXX.XXX dev tun0
> nm-openvpn[107443]: sitnl_send: rtnl: generic error (-1): Operation not permitted
> nm-openvpn[107443]: Linux can't del IP from iface tun0
> systemd-networkd[107409]: tun0: Link DOWN
> systemd-networkd[107409]: tun0: Lost carrier
> NetworkManager[1584]: <info>  [1604055348.7811] device (tun0): state change: activated -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
> gnome-shell[50230]: Removing a network device that was not added
> systemd[1]: Starting Network Manager Script Dispatcher Service...
> systemd[1]: Started Network Manager Script Dispatcher Service.
> audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> nm-openvpn[107443]: ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
> nm-openvpn[107443]: Exiting due to fatal error
> NetworkManager[1584]: <warn>  [1604055349.7874] vpn-connection[0x5628fb4fa560,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",39:(tun0)]: VPN plugin: failed: connect-failed (1)
> NetworkManager[1584]: <info>  [1604055349.7876] vpn-connection[0x5628fb4fa560,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",39:(tun0)]: VPN plugin: state changed: stopping (5)
> NetworkManager[1584]: <info>  [1604055349.7876] vpn-connection[0x5628fb4fa560,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",39:(tun0)]: VPN plugin: state changed: stopped (6)
> NetworkManager[1584]: <info>  [1604055349.7946] vpn-connection[0x5628fb4fa560,a146c70d-74a8-4ae6-b13c-47e6e3c2b5b4,"MYvpn",0]: VPN service disappeared
> systemd[1]: NetworkManager-dispatcher.service: Succeeded.
> audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

My kernel is currently:

> # uname -r
> 5.8.16-300.fc33.x86_64

I guess openvpn-2/5/0 is not enough then?

Any ideas?

Regards

Comment 14 David Sommerseth 2020-11-05 17:43:44 UTC
I did say: "[...], which would require some changes to NetworkManager-openvpn".

Yes, NetworkManager-openvpn needs to ensure the openvpn process it spawns runs with CAP_NET_ADMIN privileges.

Comment 15 Fedora Program Management 2021-04-29 16:59:51 UTC
This message is a reminder that Fedora 32 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '32'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 32 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 16 Ben Cotton 2021-05-25 17:33:40 UTC
Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 17 Bojan Smojver 2023-07-21 03:00:51 UTC
Still happens in F38, it seems.

Comment 18 Bojan Smojver 2023-07-21 03:11:23 UTC
The openvpn process and nm-openvpn are running with CAP_NET_ADMIN already, BTW. But, routes are not set correctly with --ping-restart.

Comment 19 Bojan Smojver 2023-07-24 03:26:58 UTC
Error messages related to this failure on F38:

Jul 24 09:33:16 fedora nm-openvpn[1624]: /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 1563 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_3 --tun -- tun0 1500 0 10.0.4.58 255.25>
Jul 24 09:33:16 fedora dbus-broker[579]: A security policy denied :1.81 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetConfig to :1.76.
Jul 24 09:33:16 fedora dbus-broker[579]: A security policy denied :1.81 to send method call /org/freedesktop/NetworkManager/VPN/Plugin:org.freedesktop.NetworkManager.VPN.Plugin.SetIp4Config to :1.76.

Comment 20 Bojan Smojver 2023-07-24 23:10:34 UTC
Or in more detail for some of the errors:
-----
method call time=1690238101.742098 sender=:1.97 -> destination=:1.91 serial=6 path=/org/freedesktop/NetworkManager/VPN/Plugin; interface=org.freedesktop.NetworkManager.VPN.Plugin; member=SetConfig
   array [
      dict entry(
         string "gateway"
         variant             uint32 3826925876
      )
      dict entry(
         string "can-persist"
         variant             boolean true
      )
      dict entry(
         string "tundev"
         variant             string "tun0"
      )
      dict entry(
         string "mtu"
         variant             uint32 1500
      )
      dict entry(
         string "has-ip4"
         variant             boolean true
      )
   ]
error time=1690238101.742199 sender=org.freedesktop.DBus -> destination=:1.97 error_name=org.freedesktop.DBus.Error.AccessDenied reply_serial=6
   string "Sender is not authorized to send message"
method call time=1690238101.742208 sender=:1.97 -> destination=:1.91 serial=7 path=/org/freedesktop/NetworkManager/VPN/Plugin; interface=org.freedesktop.NetworkManager.VPN.Plugin; member=SetIp4Config
   array [
      dict entry(
         string "address"
         variant             uint32 1023672330
      )
      dict entry(
         string "prefix"
         variant             uint32 29
      )
      dict entry(
         string "prefix"
         variant             uint32 29
      )
      dict entry(
         string "preserve-routes"
         variant             boolean true
      )
      dict entry(
         string "dns"
         variant             array [
               uint32 2299199498
               uint32 3322019850
            ]
      )
      dict entry(
         string "domains"
         variant             array [
               string "<sanitised>"
            ]
      )
   ]
error time=1690238101.742344 sender=org.freedesktop.DBus -> destination=:1.97 error_name=org.freedesktop.DBus.Error.AccessDenied reply_serial=7
   string "Sender is not authorized to send message"
-----

I do not think these are related to SELinux. They appear to be the result of polkit policy.

Comment 21 Bojan Smojver 2023-07-25 03:53:18 UTC
Actually, it's D-Bus policy that is preventing messages from reaching the destination. A brute force policy in /etc/dbus-1/system.d/org.freedesktop.NetworkManager.openvpn.conf that allows this to work:
-----
<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
    <policy context="default">
        <deny own_prefix="org.freedesktop.NetworkManager.openvpn"/>

        <deny send_destination_prefix="org.freedesktop.NetworkManager.openvpn"/>

        <allow send_destination_prefix="org.freedesktop.NetworkManager.openvpn"
               send_interface="*"/>
    </policy>

    <limit name="max_replies_per_connection">1024</limit>
    <limit name="max_match_rules_per_connection">2048</limit>
</busconfig>
-----

This was cannibalised from from system-wide NetworkManager D-Bus policy and likely contains errors, too many relaxed rules etc. Anyhow, just an example of a workaround.

Comment 22 Bojan Smojver 2023-07-25 05:32:04 UTC
Or slightly less broad version:
-----
<!DOCTYPE busconfig PUBLIC
 "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
 "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
<busconfig>
    <policy context="default">
        <deny own_prefix="org.freedesktop.NetworkManager.openvpn"/>

        <deny send_destination_prefix="org.freedesktop.NetworkManager.openvpn"/>

        <!-- Basic D-Bus API stuff -->
        <allow send_destination_prefix="org.freedesktop.NetworkManager.openvpn"
               send_interface="org.freedesktop.DBus.Introspectable"/>
        <allow send_destination_prefix="org.freedesktop.NetworkManager.openvpn"
               send_interface="org.freedesktop.DBus.Properties"/>
        <allow send_destination_prefix="org.freedesktop.NetworkManager.openvpn"
               send_interface="org.freedesktop.DBus.ObjectManager"/>

        <allow send_destination_prefix="org.freedesktop.NetworkManager.openvpn"
               send_interface="org.freedesktop.NetworkManager.VPN.Plugin"/>
    </policy>

    <limit name="max_replies_per_connection">1024</limit>
    <limit name="max_match_rules_per_connection">2048</limit>
</busconfig>
-----