RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2003556 - mariadb-server in centos9 stream is not shipping wsrep_sst_rsync_tunnel
Summary: mariadb-server in centos9 stream is not shipping wsrep_sst_rsync_tunnel
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: mariadb
Version: CentOS Stream
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Michal Schorm
QA Contact: Jakub Heger
URL:
Whiteboard:
Depends On:
Blocks: 2003658
TreeView+ depends on / blocked
 
Reported: 2021-09-13 08:38 UTC by Michele Baldessari
Modified: 2023-09-15 01:36 UTC (History)
8 users (show)

Fixed In Version: mariadb-10.5.12-3.el9
Doc Type: Enhancement
Doc Text:
Feature: wsrep_sst_rsync_tunnel script added Reason: requested by OpenStack team. See comment 9 for justification. Result: Feature added
Clone Of:
: 2003658 (view as bug list)
Environment:
Last Closed: 2022-05-17 12:42:41 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-96938 0 None None None 2021-09-13 08:39:20 UTC
Red Hat Product Errata RHBA-2022:2381 0 None None None 2022-05-17 12:42:59 UTC

Description Michele Baldessari 2021-09-13 08:38:53 UTC
Description of problem:
[root@controller-0 mysql]# podman run -it --net=host --rm --name=foo undercloud-0.ctlplane.home.arpa:8787/tripleomastercentos9/openstack-mariadb:a8b72470998eaa3e1039457bf1eb53f2 sh -c 'rpm -q mariadb; which -a wsrep_sst_rsync_tunnel' 
mariadb-10.5.12-2.el9.x86_64
which: no wsrep_sst_rsync_tunnel in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

This will then error out or OSP centos9 tls-e deployments like the following:
2021-09-09  9:08:53 0 [Note] WSREP: Running: 'wsrep_sst_rsync_tunnel --role 'joiner' --address 'controller-0.internalapi.home.arpa' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '1386' --mysqld-args --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mariadb/plugin --wsrep_on=ON --wsrep_provider=/usr/lib64/galera/libgalera_smm.so --wsrep-cluster-address=gcomm://controller-0.internalapi.home.arpa,controller-1.internalapi.home.arpa,controller-2.internalapi.home.arpa --log-error=/var/log/mysql/mysqld.log --open-files-limit=16384 --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'
sh: line 1: wsrep_sst_rsync_tunnel: command not found

Version-Release number of selected component (if applicable):
mariadb-10.5.12-2.el9.x86_64

Can we get this back please? I chatted with Damien and upstream does not yet have feature parity and we still need this script.

Comment 1 Michal Schorm 2021-09-13 10:56:57 UTC
This enhancement has been lost because of the (Fedora) upstream to (CentOS Stream 9) downstream code sync at the early stage of CentOS Stream 9 development.

The command:
 # git log --all -- wsrep_sst_rsync_tunnel
on top of Fedora repository:
  https://src.fedoraproject.org/rpms/mariadb
does not suggest this enhancement has ever been upstreamed.

I don't see any reason against including this script again.
I don't recall any negative feedback on it and I know the OSP rely on it.

So yes, I would say you can have it back.

Comment 8 Damien Ciabrini 2021-09-22 09:06:00 UTC
We just tested the scratch build from comment #7, the script is included as expected and using it works as expected again for our OpenStack use case.

Note that this script will eventually be replaced by improvements in wsrep_sst_rsync that have already landed in mariadb upstream. However for the time being, it seems we can't configure the certificate in such a way that allow us to rotate them easily, unlike wsrep_sst_rsync_tunnel. I am going to report that upstream as an RFE.

Comment 9 Damien Ciabrini 2021-09-24 13:25:47 UTC
A full rationale for that why we need that bz in RHEL9 below.

After further analysis, we don't need an RFE upstream, but the feature we need is only available starting mariadb 10.6. SST encryption in 10.5 is not practical and I doubt it's used at all in the field. In mariadb 10.5, TLS support in wsrep_sst_rsync is implemented via
stunnel connections:

  . The donor configures its stunnel process with its certificate and key,

  . The joiner configures its stunnel process to validate the TLS certificate it receives from the donor.

The joiner can only validate the donor's certificate if and only if it can find a file in capath (/etc/pki/tls/certs) whose name is the hash of the certificate. That is to say, the generated stunnel configuration always enforce the donor's certificate to be present in the joiner's capath directory.

This requirement to have all certs available to every galera hosts is not practical for our OpenStack environments, because each galera host is in charge of requesting its own certificate from a single IPA server and it has no means/channels to push its locally acquired certificate to other galera peers.

On the other hand, one could configure stunnel to only verify a CA chain, that it verify that the CA that issued the galera's certificate is trusted, rather than also forcing the mysql certificate itself to be trusted. This is way more practical because in this configuration, each galera  host can request a certificate renewal and doesn't need to propagate it. The only cert that has to be updated is the IPA's one, which is usually handled by another, orchestrated means.

The CA chain verification is the behaviour that we need in OpenStack, but it is only available starting mariadb 10.6. The SST scripts are largely different between 10.5 and 10.6, so a simple downstream backport is likely not achievable.

So until mariadb 10.6 is available in RHEL and consumable in OpenStack, our only practical option is to still rely on wsrep_sst_rsync_tunnel, which does what we want. This script will get naturally superseeded by the official upstream one eventually.

Comment 12 Michal Schorm 2021-10-11 13:20:34 UTC
The update has been merged to the CentOS Stream 9 repository:
https://gitlab.com/redhat/centos-stream/rpms/mariadb/-/commit/5de93c78f5de1146b70b8fedf5bc544a731e4391

Comment 21 errata-xmlrpc 2022-05-17 12:42:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: mariadb), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2381

Comment 22 Red Hat Bugzilla 2023-09-15 01:36:03 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days


Note You need to log in before you can comment on or make changes to this bug.