RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2032806 - Error replacing a replica with CentOS Stream 9
Summary: Error replacing a replica with CentOS Stream 9
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: ipa
Version: 8.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Trivino
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On: 1998016
Blocks: 2062402 2062403 2062404
TreeView+ depends on / blocked
 
Reported: 2021-12-15 09:15 UTC by Monkey Bizness
Modified: 2022-05-10 14:35 UTC (History)
11 users (show)

Fixed In Version: idm-DL1-8060020220203151553.92098735
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2062402 2062403 2062404 (view as bug list)
Environment:
Last Closed: 2022-05-10 14:09:17 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)
Installation logs (6.63 KB, text/plain)
2021-12-15 09:15 UTC, Monkey Bizness 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FREEIPA-7629 0 None None None 2021-12-15 09:17:58 UTC
Red Hat Issue Tracker RHELPLAN-105848 0 None None None 2021-12-15 09:18:02 UTC
Red Hat Product Errata RHEA-2022:1884 0 None None None 2022-05-10 14:09:52 UTC

Description Monkey Bizness 2021-12-15 09:15:26 UTC 
Created attachment 1846341 [details]
Installation logs

Description of problem:
I have currently a 2 node cluster running on CentOS Stream 8. In order to upgrade to CentOS 9, I have removed one of the replica from the
configuration, installed a fresh centos stream 9 and run ipa-replica-install.
It fails with this error (full log attached):
  [22/29]: Importing RA key
Error storing key "keys/ra/ipaCert": CalledProcessError(Command ['/usr/libexec/ipa/custodia/ipa-custodia-ra-agent', '--import', '-']
returned non-zero exit status 1: 'Traceback (most recent call last):\n  File "/usr/libexec/ipa/custodia/ipa-custodia-ra-agent", line 8, in
<module>\n    main(ra_agent_parser())\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 114, in main\n
common.main(parser, export_key, import_key)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/common.py", line 73, in
main\n    func(args, tmpdir, **kwargs)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 69, in
import_key\n    ipautil.run(cmd, umask=0o027)\n  File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in run\n    raise
CalledProcessError(\nipapython.ipautil.CalledProcessError: CalledProcessError(Command [\'/usr/bin/openssl\', \'pkcs12\', \'-in\',
\'/tmp/tmp7jrs5dqp/import.p12\', \'-clcerts\', \'-nokeys\', \'-out\', \'/var/lib/ipa/ra-agent.pem\', \'-password\',
\'file:/tmp/tmp7jrs5dqp/passwd\'] returned non-zero exit status 1: \'Error outputting keys and
certificates\\n80EB2D6B5D7F0000:error:0308010C:digital envelope
routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:346:Global default library context, Algorithm (RC2-40-CBC : 0),
Properties ()\\n\')\n')
  [error] FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Version-Release number of selected component (if applicable):
Original ipa : 4.9.6-6 on Centos Stream 8
New ipa : The one in Centos Stream 9

How reproducible:
Always


Steps to Reproduce:
1.Perform the migration procedure as described in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/installing_identity_management/migrate-7-to-8_migrating (addapted for centos stream 8 -> 9 
2. Remove existing replica from config
3. Destroy existing replica
4. Install new vm with centos stream 9
5. sudo ipa-replica-install --no-ntp --mkhomedir --setup-ca --setup-dns --no-dnssec-validation --forwarder 1.1.1.1 --forwarder 8.8.8.8 --principal admin --admin-password xxxxxxxxxxxx

Actual results:
See attached logs

Expected results:
Successful with the new ipa-replica joined in the ipa cluster

Additional info:
Comment 1 Alexander Bokovoy 2021-12-15 09:44:23 UTC 
The issue looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1998016. Can you please detail what pki packages were installed? This should be fixed with pki-ca-11.0.1-3.el9
Comment 2 Alexander Bokovoy 2021-12-15 09:54:33 UTC 
So my thinking is that there are two parts here:

 - pki-core fix in bug 1998016 is required but not enough. It only fixes the first step -- when RA Agent file is generated, it is no longer encrypted with a legacy cipher

 - IPA needs to drop encryption of existing RA Agent file generated with older PKI version _before_ submitting it to replica via Custodia

So we need to fix it on our side as well and have to do it on the 'older' master side, hence RHEL 8.
Comment 3 Monkey Bizness 2021-12-15 10:51:14 UTC 
(In reply to Alexander Bokovoy from comment #1)
> The issue looks similar to
> https://bugzilla.redhat.com/show_bug.cgi?id=1998016. Can you please detail
> what pki packages were installed? This should be fixed with
> pki-ca-11.0.1-3.el9

My version of pki-ca is 10.11.2-2.module_el8.5.0+945+a81e57da on the CS8 node.
I have destroyed the CS9 node where I tried that. I don't know the version at the time.
I can try again and get the versions in CS9 if needed
Comment 4 Florence Blanc-Renaud 2022-01-24 17:10:37 UTC 
I can confirm the issue is reproducible with RHEL 8.5 server + RHEL 9.0 replica:

[server]# rpm -qa ipa-server pki-ca
ipa-server-4.9.6-6.module+el8.5.0+12660+88e16a2c.x86_64
pki-ca-10.11.2-2.module+el8.5.0+12735+8eb38ccc.noarch


[replica]# rpm -qa ipa-server pki-ca
pki-ca-11.0.1-3.el9.noarch
ipa-server-4.9.8-1.el9.x86_64

meaning we need to fix this issue for RHEL 8->9 migration.
Comment 5 Christian Heimes 2022-01-26 13:51:12 UTC 
The problem is not the encryption of the ra-agent.key file. The Custodia client on the new RHEL 9 replica talks to the Custodia server on the RHEL 8.5 source server. Custodia's export layer on the source server creates an encrypted PKCS#12 file that contains the cert and the key. The code uses OpenSSL's default encryption scheme for PKCS#12, which is no longer supported on RHEL 9. To fix the problem you have to modify the export_key() function at https://github.com/freeipa/freeipa/blob/master/ipaserver/secrets/handlers/pemfile.py#L29 to use a stronger encryption algorithm.

The args:

    -keypbe AES-256-CBC -certpbe AES-256-CBC -macalg sha384

should do the trick to use stronger PBEv2 instead of old, insecure PBEv1. It's a trivial fix, but you have to backport the fix to supported RHEL 8 versions...

The weak encryption is not a security problem. The communication is protected by GSSAPI and HTTPS. We only encrypt because FIPS does not let us export key material without encryption.
Comment 6 Christian Heimes 2022-01-26 14:00:40 UTC 
PS: There is also a layer of JOSE authentication and encryption in the Custodia protocol. The PKCS#12 encryption is just there for FIPS compliance. The server sends the PKCS#12 blob and plain password inside the JOSE encrypted, JOSE authenticated, HTTPS encrypted, and GSSAPI authenticated response.
Comment 7 Trivino 2022-01-26 18:23:25 UTC 
(In reply to Christian Heimes from comment #5)
> The problem is not the encryption of the ra-agent.key file. The Custodia
> client on the new RHEL 9 replica talks to the Custodia server on the RHEL
> 8.5 source server. Custodia's export layer on the source server creates an
> encrypted PKCS#12 file that contains the cert and the key. The code uses
> OpenSSL's default encryption scheme for PKCS#12, which is no longer
> supported on RHEL 9. To fix the problem you have to modify the export_key()
> function at
> https://github.com/freeipa/freeipa/blob/master/ipaserver/secrets/handlers/
> pemfile.py#L29 to use a stronger encryption algorithm.
> 
> The args:
> 
>     -keypbe AES-256-CBC -certpbe AES-256-CBC -macalg sha384
> 
> should do the trick to use stronger PBEv2 instead of old, insecure PBEv1.
> It's a trivial fix, but you have to backport the fix to supported RHEL 8
> versions...
> 
> The weak encryption is not a security problem. The communication is
> protected by GSSAPI and HTTPS. We only encrypt because FIPS does not let us
> export key material without encryption.

thanks Christian, I can confirm that the new args fix the issue. I tested with RHEL 8.5 server + RHEL 9.0 replica:

[root@master /]# rpm -qa ipa-server pki-ca
ipa-server-4.9.6-10.module+el8.5.0+13587+92118e57.x86_64
pki-ca-10.11.2-3.module+el8.5.0+13650+763a7202.noarch


[root@replica cloud-user]# rpm -qa ipa-server pki-ca
pki-ca-11.0.3-1.el9.noarch
ipa-server-4.9.8-1.el9.x86_64
Comment 8 Trivino 2022-01-28 09:44:52 UTC 
Upstream PR: https://github.com/freeipa/freeipa/pull/6155

moving to POST.
Comment 9 Florence Blanc-Renaud 2022-01-31 09:09:35 UTC 
Fixed upstream
master:
https://pagure.io/freeipa/c/fd7f4a7411eb639d72dbe1f7749f1932fdcc9e62
Comment 10 Florence Blanc-Renaud 2022-02-01 07:58:11 UTC 
Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/653a7fe02880c168755984133ee143567cc7bb4e
Comment 11 Monkey Bizness 2022-02-01 08:07:45 UTC 
Thank you for the quick fix.
May I ask if this new version will be pushed to Centos Stream 8?
Comment 12 Trivino 2022-02-01 10:58:55 UTC 
(In reply to Monkey Bizness from comment #11)
> Thank you for the quick fix.
> May I ask if this new version will be pushed to Centos Stream 8?

The fix will be released in rhel8 first, then at some point it will land in c8s as well.
Comment 13 Theodoros Apazoglou 2022-02-01 11:12:52 UTC 
@ssidhaye pls qa_ack if you have agreed about it with Francisco/Rafael
@rjeffman pls dev_ack (Francisco is missing perms for now)
Comment 20 Michal Polovka 2022-03-02 17:10:13 UTC 
Verified using automation with these metadata http://idm-artifacts.usersys.redhat.com/mpolovka/trigger/4063/test-suite/metadata.orig.yaml.

All tests passed, as seen in the http://idm-artifacts.usersys.redhat.com/mpolovka/trigger//4063/test-suite//restraint.01/index.html

Therefore marking as verified.
Comment 26 errata-xmlrpc 2022-05-10 14:09:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (idm:client and idm:DL1 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:1884
Note You need to log in before you can comment on or make changes to this bug.