Bug 1932473 - Cannot establish VPN any more after updates: Permission denied
Summary: Cannot establish VPN any more after updates: Permission denied
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: NetworkManager-strongswan
Version: 36
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Petr Menšík
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1945885 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-24 16:56 UTC by Jan Vlug
Modified: 2023-05-25 19:31 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-25 19:31:21 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github strongswan strongswan issues 739 0 None open NM Gnome applet sends path to charon, which might be unreachable to system daemon 2021-11-08 21:35:14 UTC

Description Jan Vlug 2021-02-24 16:56:31 UTC
Description of problem:
After that I restarted my system after a dnf update, I cannot establish a VPN any more. NetworkManager was one of the updated packages IIRC.

Version-Release number of selected component (if applicable):
NetworkManager.x86_64                              1:1.26.6-1.fc33
NetworkManager-strongswan.x86_64                   1.5.0-2.fc33
NetworkManager-strongswan-gnome.x86_64             1.5.0-2.fc33

How reproducible:
Always

Steps to Reproduce:
1. Reuse a previously working VPN Connection set up with GNOME Settings.
2. Try to switch the VPN via the GNOME Settings Network GUI on.

Actual results:
The VPN connection is not establised.

This is in the log:

Feb 24 17:41:21 nyx charon-nm[6481]: 05[LIB]   opening '/home/jan/.cert/jan.vlug.blah.key.pem' failed: Permission denied
Feb 24 17:41:21 nyx charon-nm[6481]: 05[LIB] building CRED_PRIVATE_KEY - ANY failed, tried 7 builders
Feb 24 17:41:21 nyx NetworkManager[1650]: <info>  [1614184881.7942] vpn-connection[0x5555a9128500,246bceb2-ddb7-4627-9e61-804e8b0dfff4,"NSF",0]: VPN connection: (ConnectInteractive) reply received
Feb 24 17:41:21 nyx charon-nm[6481]: 05[CFG] received initiate for NetworkManager connection NSF
Feb 24 17:41:21 nyx charon-nm[6481]: 05[LIB]   opening '/home/jan/.cert/VpnServerRoot.cer' failed: Permission denied
Feb 24 17:41:21 nyx charon-nm[6481]: 05[LIB] building CRED_CERTIFICATE - X509 failed, tried 6 builders
Feb 24 17:41:21 nyx NetworkManager[1650]: <warn>  [1614184881.7954] vpn-connection[0x5555a9128500,246bceb2-ddb7-4627-9e61-804e8b0dfff4,"NSF",0]: VPN connection: failed to connect: 'Loading gateway certificate failed.'
Feb 24 17:41:21 nyx NetworkManager[1650]: <info>  [1614184881.7981] vpn-connection[0x5555a9128500,246bceb2-ddb7-4627-9e61-804e8b0dfff4,"NSF",0]: VPN plugin: state changed: stopped (6)


Expected results:
A correctly established VPN


Additional info:
I have two systems. This issue appeared first on one of the systems, that had been updated and rebooted. Then I tried on the other system, that also had been updated, but not been restarted yet. With tracer, I noticed that NetworkManager needed a restart, amongst many others. I rebooted the system, and now this system gives also the messages above. Taking the uptime of the system in consideration, the breaking change has probably happened between 15 days ago and now.

The VPN type is: IPsec/IKEv2 (strongswan).

After reboot, I was asked for the password when I tried to enable the VPN. I had to cancel to find the password (the password prompt blocks further access to the system). I found the password and added it the next time. But the VPN still cannot be established.

Comment 1 Mikhail Zabaluev 2021-02-25 08:48:50 UTC
It looks like newly applied selinux policy prevents charon-nm from accessing the certificates in your home directory that are probably configured with the NetworkManager UI plugin.

There is an architectural issue at the root of it whereby NetworkManager-strongswan UI passes user-provided file paths to the daemon expecting it to be able to access them. There is no quick fix to this; the unfortunate workaround is to relax the security policy.

Comment 2 Jan Vlug 2021-02-25 09:34:54 UTC
Thanks Mikhail. That is what I also initially thought, but I have some doubts now because of the following reasons:
1. I do not get an SELinux Alert when trying to enable the VPN.
2. when I set SELinux to permissive by doing:
   # setenforce Permissive
   # setenforce 0
   I still get the same Permission denied messages in the log.

Am I setting SELinux in the correct way to permissive? Normally, I use the SETroubleshout application to modify SELinux, if needed, but because there is no alert, I cannot use this tool.

Could you suggest a location where I could put the files (temporarily as a test) where you expect that they would be accessible?

Unfortunately, I have not found a work-around yet, I would happily test any suggestions.

Comment 3 Jan Vlug 2021-02-25 12:19:52 UTC
After some experimentation, I found this work around, where SELinux can still be enabled (maybe I had to change some policies as suggested by SETroubleshoot):
I copied the following files to /tmp/ and gave everyone read permissions:
1. The Server Certificate
2. The Client Certificate file
3. The Client Private key

So, now I have these files:
$ ll /tmp/jan.vlug.blah.key.pem /tmp/jan.vlug.blah.pem /tmp/VpnServerRoot.cer
-r--r--r--. 1 jan jan 1911 Feb 25 10:35 /tmp/jan.vlug.blah.key.pem
-r--r--r--. 1 jan jan 2691 Feb 25 10:35 /tmp/jan.vlug.blah.pem
-r--r--r--. 1 jan jan  947 Feb 25 10:35 /tmp/VpnServerRoot.cer

Comment 4 Mirek Svoboda 2021-03-02 09:31:22 UTC
This issue happened to me too.
The issue is related to the regular Linux file permissions.
Before, it was sufficient that the files had access rights set to 400 for my user.
Now, access by root user is required too.

I have worked around the issue by adding ACL, using the following commands:

$ cd /home/myuser/.local/share/networkmanagement/certificates
$ setfacl -m g:root:r-- ./*

No other changes are necessary, SElinux is enforcing.

Comment 5 Mikhail Zabaluev 2021-03-02 09:54:06 UTC
I see. I'm getting lost in all the various ways the Fedora kernel can use to control access to files.

(In reply to Jan Vlug from comment #2)
> Could you suggest a location where I could put the files (temporarily as a
> test) where you expect that they would be accessible?

The default location since 5.9.0-2 is
/etc/strongswan/ipsec.d/cacerts/

See the discussion in bug #1886759.

Comment 6 Mirek Svoboda 2021-03-03 00:45:42 UTC
> The default location since 5.9.0-2 is
> /etc/strongswan/ipsec.d/cacerts/

In my view, it is not the best practice to store per-user settings in /etc. /etc is typically used for system-wide settings, not for user-specific settings. User-specific settings should be stored at ~/.local or ~/.config
Another potential issue I see with /etc approach is that as far as I understand /etc is rolled back during a rollback to an older image in Silverblue, what could be undesirable and surprising to end users.

Comment 7 Fedora Admin user for bugzilla script actions 2021-03-18 00:05:19 UTC
This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.

Comment 8 Mirek Svoboda 2021-03-18 08:19:49 UTC
Hi Petr Menšík,

In my understanding you are the new maintainer of this package.
Let me wish you every success in this role!

Is there any information I could provide with regard to this bug?

Comment 9 svelosol 2021-03-18 17:04:42 UTC
Same problem with Fedora 33, before updating it was working fine:

- Linux redhat 5.10.22-200.fc33.x86_64
- NetworkManager.x86_64 1:1.26.6-1.fc33

My certificates files are placed in ~/.cert and have the following permissions:

$ ls -alZ ~/.cert/strongswan-vpn-*.pem
-rw-r--r--+ 1 myuser myuser unconfined_u:object_r:home_cert_t:s0 2857 nov  3 20:44 /home/myuser/.cert/strongswan-vpn-ca.pem
-rw-r--r--+ 1 myuser myuser unconfined_u:object_r:home_cert_t:s0 2654 jul 24  2020 /home/myuser/.cert/strongswan-vpn-myuser-certificate.pem
-rw-r-----+ 1 myuser myuser unconfined_u:object_r:home_cert_t:s0 1679 jul 24  2020 /home/myuser/.cert/strongswan-vpn-myuser-key.pem

But when I try to connect I get this error:

mar 18 17:58:39 redhat charon-nm[187395]: 05[CFG] received initiate for NetworkManager connection VPN Lab
mar 18 17:58:39 redhat charon-nm[187395]: 05[LIB]   opening '/home/myuser/.cert/strongswan-vpn-ca.pem' failed: Permission denied
mar 18 17:58:39 redhat charon-nm[187395]: 05[LIB] building CRED_CERTIFICATE - X509 failed, tried 6 builders
mar 18 17:58:39 redhat NetworkManager[176002]: <warn>  [1616086719.6282] vpn-connection[0x556df201e2d0,14bf82db-c470-4c99-8ee2-eee010ec2ea8,"VPN Lab",0]: VPN connection: failed to connect: 'Loading gateway certificate failed.'
mar 18 17:58:39 redhat NetworkManager[176002]: <info>  [1616086719.6294] vpn-connection[0x556df201e2d0,14bf82db-c470-4c99-8ee2-eee010ec2ea8,"VPN Lab",0]: VPN plugin: state changed: stopped (6)

I have also added the ACL for root as stated in https://bugzilla.redhat.com/show_bug.cgi?id=1932473#c4 but it doesn't work either.

I have SELinux enabled but there isn't any avc:

$ ausearch -m avc --start recent
<no matches>

Comment 10 Peter Beurle 2021-04-03 01:10:30 UTC
You need to put the certificate in, as root user;

/etc/strongswan/ipsec.d/cacerts

Make sure in the GUI for the VPN settings the certificate under identity says (none). It doesn't seem you can clear this from the file picker and I had to remove the old path to the certificate in the VPN connection details under etc/NetworkManager/system-connections/

Comment 11 Ivan Larionov 2021-04-23 09:34:50 UTC
(In reply to Peter Beurle from comment #10)
> You need to put the certificate in, as root user;
> 
> /etc/strongswan/ipsec.d/cacerts
> 
> Make sure in the GUI for the VPN settings the certificate under identity
> says (none). It doesn't seem you can clear this from the file picker and I
> had to remove the old path to the certificate in the VPN connection details
> under etc/NetworkManager/system-connections/

I had had the same problem and this workaround worked. Thanks.

Comment 12 Timo Eissler 2021-06-08 09:07:30 UTC
I have the same issue, but the workaround doesn't work for me. Any news on this topic in permanently fixing this?

Comment 13 Yevgeny Popovych 2021-07-06 18:45:27 UTC
Same issue here Fedora 34, Silverblue.
This is not SELinux related in this case (no AVC denials).

NetworkManager and charon-nm both run as root and root can access the file (tested).
I've also entered all charon-nm namespaces (nsenter --all) and still could access the file.

Need help.

Comment 14 Yevgeny Popovych 2021-07-07 18:43:36 UTC
I was finally able to find a solution..

Although the permissions on the file itself were OK,
permissions on a path to file were restrictive (in my case `/var/home/ievgenp/.cert/gw-cachain.pem`)..

This is what I had:

# ls -la /var/home/
total 4
drwxr-xr-x.  1 root    root      14 Apr 13  2020 .
drwxr-xr-x. 25 root    root    4096 Jun 15 15:30 ..
drwx--x---+  1 ievgenp ievgenp 1700 Jul  7 20:29 ievgenp
# getfacl /var/home/ievgenp/
getfacl: Removing leading '/' from absolute path names
# file: var/home/ievgenp/
# owner: ievgenp
# group: ievgenp
user::rwx
user:qemu:--x
group::---
mask::--x
other::---

I've added list directory permission to group root with `setfacl -m 'g:root:--x' /var/home/ievgenp` and the VPN now works fine.

What I do not understand though is that I could not reproduce it manually as root (explanation is welcome!).

Comment 15 Yevgeny Popovych 2021-07-07 20:38:56 UTC
> What I do not understand though is that I could not reproduce it manually as root (explanation is welcome!).

Answering my question - this is most likely related to capabilities (since my root shell has all of them but not charon-nm, specifically CAP_DAC_OVERRIDE and CAP_DAC_READ_SEARCH).

The change (which brought us here) is most likely related to the implementation/fix of bug 1911572
via https://src.fedoraproject.org/rpms/strongswan/c/8f80a71a0186483c087f77348b6b1cf020f2b351
that most likely has changed charon-nm behavior (and the timeframe matches).

So basically one must ensure that charon-nm user (which is root at the moment) is able to access your files.
You can test by dropping relevant capabilites:
    sudo capsh --drop=cap_dac_override,cap_dac_read_search -- -c 'cat /var/home/ievgenp/.cert/gw-cachain.pem'

Comment 16 Petr Menšík 2021-07-12 19:22:00 UTC
Hi, yes, I also think those problems have arisen, because CAP_DAC_OVERRIDE and CAP_DAC_READ_SEARCH capabilities are now not granted even to root process such as strongswan.service. More precise, it needs just CAP_DAC_READ_SEARCH I think. Because it does not have permission granted to read any file like common root user can, it cannot enter /home directory of user and their subdirectories.

It should work to put copy of your certificates to /etc/strongswan, but you have to have sudo access on your machine to solve it. It is by default accessible only to root even to read its contents. Which unfortunately does not help with easy GUI selection of certificate or key file.

This might help:

sudo chmod o+rx /etc/strongswan # allow common user to enter strongswan directory
sudo mkdir /etc/strongswan/usercert.d
for FILE in $USER.pem; # use any
do
  sudo touch /etc/strongswan/usercert.d/$FILE # prepare well-set empty file with desired permissions
  sudo chown $USER:root /etc/strongwan/usercert.d/$FILE
  sudo chmod 0640 /etc/strongwan/usercert.d/$FILE
  sudo cp $USER.pem /etc/strongwan/usercert.d/$FILE # write actual secret to strongswan directory
done

Now you should be able to add any key from that usercert directory to UI in gnome. It should not matter whether it is key or certificate, a new copy would be readable only by you as an user and root - therefore charon daemon.

Another possibility to use cert in your home, might be safer:

mkdir $HOME/.certs
sudo chgrp root $HOME $HOME/.certs
sudo chmod g+x $HOME
chmod 0750 $HOME/.certs

Now put your cert copies into ~/.certs and use those path. If your $HOME is not /home/$USER, make sure path above your $HOME lets root user inside - at least g+x with root group or o+x.

I am not sure what is correct solution, those are more workarounds. I am not sure how should safe certificate and key exchange look like to stay safe for both user and strongswan daemon. I think the best way would be using some kind of socket to push secrets from user-only home to strongswan only store for use by charon. I am not sure how it is solved by other VPN backends, they should have similar problem with key data supplied by ordinary user.

Comment 17 Ben Cotton 2021-11-04 13:50:55 UTC
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 18 Ben Cotton 2021-11-04 14:20:22 UTC
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 19 Ben Cotton 2021-11-04 15:18:05 UTC
This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 20 Petr Menšík 2021-11-08 21:37:47 UTC
*** Bug 1945885 has been marked as a duplicate of this bug. ***

Comment 21 Petr Menšík 2021-11-08 21:44:06 UTC
The issue is still there, but I am not sure how it should be solved. I do not think charon should be able to read all user's files, but that would be the easiest fix available.

Better would be some refactoring in charon, so user would be able to send certificate data over d-bus (or some other pipe?) to charon, which would not require access to home directory. I hope upstream can propose some solution, this is somehow hard for me to fix.

Interesting link to Ask Fedora from duplicate bug:
https://ask.fedoraproject.org/t/cannot-connect-to-ikev2-vpn-on-fedora-33-no-trusted-rsa-public-key-found/

Comment 22 Petr Menšík 2021-11-09 11:43:46 UTC
There is special directory in home, which has different context assigned: ~/.cert

I think that should be always used to store certificated by the user. Haven't found any hints to it in the plugin itself. I think it should be recommended location, not sure how to do that.

It seems cap_dac_read_search is needed by /usr/libexec/strongswan/charon-nm to read that files, but is not granted it.

/usr/libexec/nm-libreswan-service has different SELinux context, could that be the difference?

$ ls -lZ /usr/libexec/nm-libreswan-service* /usr/libexec/strongswan/charon-nm
-rwxr-xr-x. 1 root root system_u:object_r:ipsec_mgmt_exec_t:s0 79672 Jan 25  2021 /usr/libexec/nm-libreswan-service
-rwxr-xr-x. 1 root root system_u:object_r:bin_t:s0             24704 Jan 25  2021 /usr/libexec/nm-libreswan-service-helper
-rwxr-xr-x. 1 root root system_u:object_r:ipsec_exec_t:s0      45656 Oct 21 05:44 /usr/libexec/strongswan/charon-nm

Comment 23 Yevgeny Popovych 2021-11-09 15:50:29 UTC
Hi Petr,

My understanding is that it is not about `cap_dac_read_search` - even if the capability is granted, charon doesn't want it and will drop it at startup (this was enabled relatively recently), and this is by design.
This is also not about selinux since people have verified that it does not resolve the issue but fixing permissions helps (see `man path_resolution`).
See my previous comments for more information (https://bugzilla.redhat.com/show_bug.cgi?id=1932473#c14, https://bugzilla.redhat.com/show_bug.cgi?id=1932473#c15).

With current implementation files should be normally (without CAP_DAC*) accessible to the daemon, and I believe that `/.cert` exists to handle selinux (and it works); which is not ideal since users have to put their certificates there manually.
Like you said, ideally NetworkManager should be the one that opens and reads the file (via portal?) and passes it to charon-nm.

Also, do not forget that in case of NetworkManager integration - charon-nm is executed in the context of (and by) Networkmanager service.

Comment 24 Ben Cotton 2022-02-08 21:31:04 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

Comment 25 vangel.attila@gmail.com 2022-03-07 17:30:17 UTC
Got the same issue on Fedora 35.

https://bugzilla.redhat.com/show_bug.cgi?id=1932473#c16 worked, thanks!

I did this part of that comment:

mkdir $HOME/.certs
sudo chgrp root $HOME $HOME/.certs
sudo chmod g+x $HOME
chmod 0750 $HOME/.certs

Now put your cert copies into ~/.certs and use those path.

Comment 26 Franz H. 2023-04-12 08:06:45 UTC
still valid for Fedora 37

Comment 27 Ben Cotton 2023-04-25 18:22:28 UTC
This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 36 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 28 Ludek Smid 2023-05-25 19:31:21 UTC
Fedora Linux 36 entered end-of-life (EOL) status on 2023-05-16.

Fedora Linux 36 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.