RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1936842 - mariadb:10.5/mariadb: Review permissions of auth_pam_tool and auth_pam_tool_dir
Summary: mariadb:10.5/mariadb: Review permissions of auth_pam_tool and auth_pam_tool_dir
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: mariadb
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Michal Schorm
QA Contact: Lukáš Zachar
Lenka Špačková
URL:
Whiteboard:
Depends On:
Blocks: 1937841 1940436
TreeView+ depends on / blocked
 
Reported: 2021-03-09 10:44 UTC by Tomas Hoger
Modified: 2021-05-18 15:55 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
.`MariaDB 10.5` provides the PAM plug-in version 2.0 `MariaDB 10.5` adds a new version of the Pluggable Authentication Modules (PAM) plug-in. The PAM plug-in version 2.0 performs PAM authentication using a separate `setuid root` helper binary, which enables `MariaDB` to utilize additional PAM modules. In `MariaDB 10.5`, the Pluggable Authentication Modules (PAM) plug-in and its related files have been moved to a new package, `mariadb-pam`. This package contains both PAM plug-in versions: version 2.0 is the default, and version 1.0 is available as the `auth_pam_v1` shared object library. Note that the `mariadb-pam` package is not installed by default with the `MariaDB` server. To make the PAM authentication plug-in available in `MariaDB 10.5`, install the `mariadb-pam` package manually. See also known issue xref:BZ-1942330[].
Clone Of:
: 1937841 1940436 (view as bug list)
Environment:
Last Closed: 2021-05-18 15:55:23 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Tomas Hoger 2021-03-09 10:44:54 UTC
Description of problem:

MariaDB 10.5 introduces new setuid root application - auth_pam_tool.  Please review the following permission-related issues related to this new tool.

* The tool is installed in the /usr/lib*/mariadb/plugin/auth_pam_tool_dir/ directory.  This directory has ownership/permission of mysql:mysql 755.  We should not be installing any binaries form rpm packages into directories from where they can be removed by non-root users, in this case the mysql user.

Relevant .spec file entry is:

# Change from root:root to mysql:mysql, so it can be accessed by the server
%attr(0755,mysql,mysql) %dir %{_libdir}/%{pkg_name}/plugin/auth_pam_tool_dir

Note that the comment does not seem to make sense - directory owned by root:root and 755 would be accessible to the server running as mysql (or any other) user.

I suspect this may be coming from upstream rpms.  I noticed that those have the directory packaged as root:root 700, and chowned to mysql via post install script.  The resulting permission should be mysql:root 700.

If the intention is to restrict access to mysql user only, the directory permission should be root:mysql 750.  No write access for mysql user or group.  And this point brings us the the following...

* Review if the suid tool auth_pam_tool needs to be accessible to all local users.  Its current permissions are root:root 4755.  If it's only meant to be run by the mysql server, we should consider making it only accessible to the mysql group.  Either via file permissions - root:mysql 4750 - or via parent directory permissions - root:mysql 750, as discussed in the previous paragraph.

I believe upstream rpms already have such restriction, but I've not investigated closely.

Comment 2 Michal Schorm 2021-03-10 12:10:00 UTC
I started the research.

Comment 3 Michal Schorm 2021-03-10 22:09:24 UTC
(In reply to Tomas Hoger from comment #0)

> We should not be installing any binaries form rpm packages into directories
> from where they can be removed by non-root users, in this case the mysql user.

Understood.

I haven't found any necessity for write access for non-root users on the '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory.

---

The client side does NOT use the 'auth_pam_tool.so' plugin or the 'auth_pam_tool' binary.
Clients use the 'dialog.so' auth plugin, which includes the necessary logic to connect to the DB server account using PAM on the client side.

Therefore no one other than the *user running the DB server* needs access to the 'auth_pam_tool' binary.

However while uncommon, this user can be any system user.
You need to configure everything to a location accessible by the system user, the 'mysql_install_db' tool will fail on few things, but it works.
Example follows:

  $ mysql_install_db --datadir=/home/testusr/DB --user=testusr

  $ mysqld_safe --binlog_format=ROW --default-storage-engine=innodb --bind-address=0.0.0.0 --datadir=/home/testusr/DB --socket=/home/testusr/DB/mysql.sock --log-error=/home/testusr/DB/mariadb.log --pid-file=/home/testusr/DB/mariadb.pid --port=3307

In such case, if the '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory is not accessible by the user running the DB server, he won't be able to use the server PAM auth module.
Error log message when user tries to acces the server via PAM auth module while the server can't access the '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory:
  [ERROR] mariadbd: PAM: Cannot execute /usr/lib64/mariadb/plugin/auth_pam_tool_dir/auth_pam_tool (errno: 13 "Permission denied")

---

There are two possible outcomes:

1)
  '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory should be owned by root:root with 755 mode.
  That means anyone can access the SUID-to-root binary and execute it.
  However it allows any user on the system to run a DB server and use the PAM mariadb auth plugin.

2)
  '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory should be owned by root:mysql with 750 mode.
  That means only (root and) mysql user will be able to access the directory and thus use the PAM mariadb auth plugin.

  This approach might help if there would be security concerns about anyone executing the SUID-to-root binary.


---

I agree the current ownership/permission of mysql:mysql 755 of the '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory is wrong.

Personally I'd go with the first approach, but I can't imagine all the security consequences.

---

The code of the plugin can be found here:
https://github.com/MariaDB/server/tree/10.5/plugin/auth_pam

Comment 7 Tomas Hoger 2021-03-11 13:55:17 UTC
(In reply to Michal Schorm from comment #3)
> Therefore no one other than the *user running the DB server* needs access to
> the 'auth_pam_tool' binary.

Right, that understood.

> However while uncommon, this user can be any system user.
> You need to configure everything to a location accessible by the system
> user, the 'mysql_install_db' tool will fail on few things, but it works.
> Example follows:
> 
>   $ mysql_install_db --datadir=/home/testusr/DB --user=testusr

Out of curiosity, is chown/chmod of the auth_pam_tool_dir directory one of those failing things?  I see the install script does that, unless it's run with --rpm option, which is described as "For internal use".  So if you run that DB install command as root, it will actually make the directory and the tool inaccessible to the mysql user.  At least it does not attempt to chown/chmod the auth_pam_tool itself when it's already owned by root.

> In such case, if the '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory
> is not accessible by the user running the DB server, he won't be able to use
> the server PAM auth module.

This is a trade-off.  We can either make it accessible to mysql user, and in that case, we reduce the impact of any security issue found in the tool and make it only exploitable to the mysql system/service user rather than any local user.  Leaving it accessible to all users means larger exposure in case of security flaws found in the tool.  It's not uncommon to restrict setuid helpers to a specific user or group if it's not expected to be required by all users.  So the question is if we expect it common enough usage to run mariadb server under different users than mysql.  Also note that even for those use cases, only those that attempt to use PAM authentication would be affected.

Did you have an opportunity to check upstream RPMs?  As noted in comment 0, those seems to set auth_pam_tool_dir directory permissions in a way that it's only accessible to mysql and not all users.  If they do not see a need to be more open, maybe we should not go beyond what they do.

> There are two possible outcomes:
> 
> 1)
>   '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory should be owned by
> root:root with 755 mode.
>   That means anyone can access the SUID-to-root binary and execute it.
>   However it allows any user on the system to run a DB server and use the
> PAM mariadb auth plugin.
> 
> 2)
>   '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory should be owned by
> root:mysql with 750 mode.
>   That means only (root and) mysql user will be able to access the directory
> and thus use the PAM mariadb auth plugin.
> 
>   This approach might help if there would be security concerns about anyone
> executing the SUID-to-root binary.
>
> ---
> 
> I agree the current ownership/permission of mysql:mysql 755 of the
> '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory is wrong.
> 
> Personally I'd go with the first approach, but I can't imagine all the
> security consequences.

My preference is to start with 2) - this is the first time we're adding this tool, so we have an opportunity to set a safe baseline.  We can review later if there's customer demand for more open permissions.  It's usually easier move to less strict permissions that more strict.

> The code of the plugin can be found here:
> https://github.com/MariaDB/server/tree/10.5/plugin/auth_pam

I looked at the tool already.  While I do not have any specific attack, my concern is that there may be exposures created by the use of specific pam modules / configurations.

Comment 10 Michal Schorm 2021-03-11 17:05:05 UTC
(In reply to Tomas Hoger from comment #7)
> My preference is to start with 2) - this is the first time we're adding this
> tool, so we have an opportunity to set a safe baseline.  We can review later
> if there's customer demand for more open permissions.  It's usually easier
> move to less strict permissions that more strict.

That is the same opinion as of Honza Horak & Petr Kubat.

Let's go that way, then.

---

That means:

* Changing the ownership of the '/usr/lib*/mariadb/plugin/auth_pam_tool_dir/' directory from 'mysql:mysql' to 'root:mysql'
  AND
  Changing the directory mode/permissions from 755 to 750

---

As Honza Horak pointed out, as we don't want all users to execute the binary,
we should ALSO change the binary permissions from 'root:root' 4755 to 'root:mysql' 4750.

Do you agree, Tomas ?

---

Then there is this piece of code, in the upstream 'mysql_install_db.sh' script:

|    if test -n "$user"
|    then
|      if test -z "$srcdir" -a "$in_rpm" -eq 0
|      then
|        chown 0 "$pamtooldir/auth_pam_tool_dir/auth_pam_tool" && \
|        chmod 04755 "$pamtooldir/auth_pam_tool_dir/auth_pam_tool"
|        if test $? -ne 0
|        then
|            echo "Couldn't set an owner to '$pamtooldir/auth_pam_tool_dir/auth_pam_tool'."
|            echo "It must be root, the PAM authentication plugin doesn't work otherwise.."
|            echo
|        fi
|        chown $user "$pamtooldir/auth_pam_tool_dir" && \
|        chmod 0700 "$pamtooldir/auth_pam_tool_dir"
|        if test $? -ne 0
|        then
|            echo "Cannot change ownership of the '$pamtooldir/auth_pam_tool_dir' directory"
|            echo "to the '$user' user. Check that you have the necessary permissions and try again."
|            echo
|        fi
|      fi
|      args="$args --user=$user"
|    fi

Many of the pieces of code in that script rely on chown from the root user, since the upstream implementation of the systemd service run as root,
rather than running as mysql:mysql as our downstream implementation.

We actually patch this piece of code in following way:
https://src.osci.redhat.com/rpms/mariadb/blob/stream-10.5-rhel-8.4.0/f/mariadb-auth_pam_tool_dir.patch

I may update the patch so it will try to change owner:group and permissions to the state we newly want,
or I can kick this piece of code from the script totally. (which might be better in this downstream case IMO)

Comment 11 Honza Horak 2021-03-11 18:01:22 UTC
(In reply to Michal Schorm from comment #10)
> |      if test -z "$srcdir" -a "$in_rpm" -eq 0

The script mysql-prepare-db-dir.sh runs @bindir@/mysql_install_db with --rpm option, so this whole piece of code is not executed when running as part of the systemd service.

Comment 12 Tomas Hoger 2021-03-11 20:44:36 UTC
(In reply to Michal Schorm from comment #10)
> As Honza Horak pointed out, as we don't want all users to execute the binary,
> we should ALSO change the binary permissions from 'root:root' 4755 to
> 'root:mysql' 4750.
> 
> Do you agree, Tomas ?

Sound ok ot me.  As already noted in comment 0, we can restrict access to the binary via binary permissions or parent directory permissions.  Probably don't need both.  All these should be valid options:

1) auth_pam_tool_dir root:root 755
   auth_pam_tool root:mysql 4750

2) auth_pam_tool_dir root:mysql 750
   auth_pam_tool root:root 4755

3) auth_pam_tool_dir root:mysql 750
   auth_pam_tool root:mysql 4750

I do not think I have any strong preference.  Would 1) be the most expected for an RPM package?

I realize that I probably wasn't clear enough in comment 7 - my preference is to prevent users other than mysql from running the binary, but I'm not trying to force any particular way to implement the restriction.

Comment 13 Michal Schorm 2021-03-12 13:51:38 UTC
Test 1:
From the application POV, it is irrelevant, whether it cannot access the directory or the binary.
In both cases the same error is printed to the error log:
  [ERROR] mysqld: PAM: Cannot execute /usr/lib64/mariadb/plugin/auth_pam_tool_dir/auth_pam_tool (errno: 13 "Permission denied")

---

Upstream packaging issue reported upstream:
https://jira.mariadb.org/browse/MDEV-25126

The first reply is very important:

| The goal is to have a suid-root binary that only mysql user can execute.
| It is very intentional that other users should not be able to access it, otherwise they would be able to use it to brute-force passwords.
| You suggest to limit the access to mysql group, not to mysql user. I'm unsure of consequences,
| if it would potentially allow more users to run the tool, it would allow them to brute-force OS passwords.
| Who can have access to mysql group, some admin performing a backup?


What do you think now, Tomas ?

would
  direcotry:   mysql:root  500
or
  binary:      mysql:root 4500
be better choice ?

Comment 14 Tomas Hoger 2021-03-12 14:27:28 UTC
(In reply to Michal Schorm from comment #13)
> would
>   direcotry:   mysql:root  500

No write access is not going to help.  mysql as owner would be able to chmod it and add write access.

> or
>   binary:      mysql:root 4500

Did you really mean mysql:root rather than root:mysql?  The binary needs to be suid root, and hence must be root:<something>.

Comment 15 Michal Schorm 2021-03-12 15:14:37 UTC
So, security don't like that binary installed from RPM could be deleted by non-root.
I (as per Upstream comment 1) don't like execute permission for the group.

Would there be a possibility to solve it with SELinux, or a combination of SELinux and Extended file attributes (man xattr) ?

Comment 16 Tomas Hoger 2021-03-12 21:28:06 UTC
The concern around having non-root user remove and replace (or modify) binary installed from RPM (or really any in the $PATH) would allow the non-root user to compromise other users if they attempt to run that tampered binary.  The auth_pam_tool may seem less risky, because users are unlikely to want to run it directly, it's not in the $PATH, and the default config (assuming mysql:root 700 directory permission) would not allow that at all.  However, mysql user is able to open the directory permissions in that config, and hence other user trying to run mysqld under their own uid may have the auth_pam_tool executed by the server.

Restricting to the mysql group seems like a reasonable restriction.  Assuming that the mysql group is a dedicated group for the mysql user, it does not seem less safe than granting access to the user.  Using root:<somegroup> 4750 is not an uncommon way to restrict what local users can run a suid binary.

Comment 17 Michal Schorm 2021-03-16 11:58:56 UTC
I asked Zdenek <zpytela> if the SELinux could help.
It might help to restrict that only 'mysqld' would be allowed to execute binary with a context allowing reading /etc/shadow.
However that won't stop unconfined users.

Comment 18 Michal Schorm 2021-03-16 12:03:34 UTC
Restricting (In reply to Tomas Hoger from comment #14)
> (In reply to Michal Schorm from comment #13)
> > would
> >   direcotry:   mysql:root  500
> 
> No write access is not going to help.  mysql as owner would be able to chmod
> it and add write access.

Yeah, my bad.

However, it still lies an obstacle for the attacker. It might only be a security through obscurity though.

If the standard issue would be a priviledge escalation of the mysqld or some similar executable, it still doesn't necessarily mean the attacker would be able to call chmod, does it ?

Comment 19 Tomas Hoger 2021-03-16 13:41:14 UTC
The above question is not really relevant any more as we're going for root:root 755 directory.  Anyway, the risk from having a mysql-owned directory would not be related to mysql -> root privilege escalation via exploiting the auth_pam_tool suid binary, but rather from mysql user replacing the binary with a malicious binary and tricking or waiting for some other user to run it (and hence get compromised).  The mysql:root 500 permissions would not stop mysql from replacing the binary, and the directory ownership would make it possible to "open" the directory to other local users.  For this attack, I'm assuming that attacker can run arbitrary code as mysql.

Comment 21 Michal Schorm 2021-03-18 11:50:27 UTC
THE CURRENT STATUS:

I am applying two-stage approach.

---

1/ First stage - fix permissions on the files

In this stage, the permissions of the SUID-to-root binary and the directory it resides in are fixed.
The new permissions are:
  auth_pam_tool_dir   root:root    755
  auth_pam_tool       root:mysql  4750

The new permissions disallow non-root to tamper with the SUID-to-root binary.
The new permissions disallow all but root & mysql group to execute the SUID-to-root binary.
The new permissions allow anyone in the mysql group to execute the SUID-to-root binary.

Please note that anyone with the access to this binary can use it to brute-force the passwords to the users of the OS.
It would be ideal to further restrict the access from 'mysql group' to 'mysql user' only, however we haven't found ideal solution for that.
Even though by default the 'mysql group' contains only one user - mysql - it is not uncommon in the real deploy scenarios, that there are more users added to the 'mysql group' which then serves as 'mysql DB' admins or a way to backup the DB files.


This fix was pushed, built and tested in Fedora:
  https://src.fedoraproject.org/rpms/mariadb/c/51dee9887b9398bd3a9ca4806b4a923f518f00a8?branch=rawhide

---

2/ Second stage - move the PAM plugin to the separate subpackage

In this stage, we move the PAM plugin to a standalone subpackage.
This way, the PAM plugin won't be installed by default anymore, minimising the security risks to only those users, that would install this subpackage.

I believe this is important and correct way to minimise the security concern.
It won't be possible to implement it later on. Anyone who would install the MariaDB 10.5 server would automatically get the PAM plugin and we won't have a sane way to transition to this state.

The commit is prepared for review in Fedora:
  https://src.fedoraproject.org/rpms/mariadb/pull-request/26#request_diff

---

I will push the first step to the RHEL immediatelly.

However when the Exception+ was granted to me, this was expected to be a trivial, ideally one-line, fix.
While I see the second step as vital, I will wait with it until a I get a confirmation, the Exception+ is granted to the second step too.

Comment 39 errata-xmlrpc 2021-05-18 15:55:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new module: mariadb:10.5), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:1893


Note You need to log in before you can comment on or make changes to this bug.