Bug 2032806

Summary: Error replacing a replica with CentOS Stream 9
Product: Red Hat Enterprise Linux 8 Reporter: Monkey Bizness <monkey>
Component: ipaAssignee: Trivino <ftrivino>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.5CC: abokovoy, bstinson, cheimes, frenaud, jwboyer, mpolovka, rcritten, rjeffman, ssidhaye, sumenon, tscherf
Target Milestone: rcKeywords: Regression, Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: idm-DL1-8060020220203151553.92098735 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2062402 2062403 2062404 (view as bug list) Environment:
Last Closed: 2022-05-10 14:09:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1998016    
Bug Blocks: 2062402, 2062403, 2062404    
Attachments:
Description Flags
Installation logs none

Description Monkey Bizness 2021-12-15 09:15:26 UTC
Created attachment 1846341 [details]
Installation logs

Description of problem:
I have currently a 2 node cluster running on CentOS Stream 8. In order to upgrade to CentOS 9, I have removed one of the replica from the
configuration, installed a fresh centos stream 9 and run ipa-replica-install.
It fails with this error (full log attached):
  [22/29]: Importing RA key
Error storing key "keys/ra/ipaCert": CalledProcessError(Command ['/usr/libexec/ipa/custodia/ipa-custodia-ra-agent', '--import', '-']
returned non-zero exit status 1: 'Traceback (most recent call last):\n  File "/usr/libexec/ipa/custodia/ipa-custodia-ra-agent", line 8, in
<module>\n    main(ra_agent_parser())\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 114, in main\n
common.main(parser, export_key, import_key)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/common.py", line 73, in
main\n    func(args, tmpdir, **kwargs)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 69, in
import_key\n    ipautil.run(cmd, umask=0o027)\n  File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in run\n    raise
CalledProcessError(\nipapython.ipautil.CalledProcessError: CalledProcessError(Command [\'/usr/bin/openssl\', \'pkcs12\', \'-in\',
\'/tmp/tmp7jrs5dqp/import.p12\', \'-clcerts\', \'-nokeys\', \'-out\', \'/var/lib/ipa/ra-agent.pem\', \'-password\',
\'file:/tmp/tmp7jrs5dqp/passwd\'] returned non-zero exit status 1: \'Error outputting keys and
certificates\\n80EB2D6B5D7F0000:error:0308010C:digital envelope
routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:346:Global default library context, Algorithm (RC2-40-CBC : 0),
Properties ()\\n\')\n')
  [error] FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Version-Release number of selected component (if applicable):
Original ipa : 4.9.6-6 on Centos Stream 8
New ipa : The one in Centos Stream 9

How reproducible:
Always


Steps to Reproduce:
1.Perform the migration procedure as described in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/installing_identity_management/migrate-7-to-8_migrating (addapted for centos stream 8 -> 9 
2. Remove existing replica from config
3. Destroy existing replica
4. Install new vm with centos stream 9
5. sudo ipa-replica-install --no-ntp --mkhomedir --setup-ca --setup-dns --no-dnssec-validation --forwarder 1.1.1.1 --forwarder 8.8.8.8 --principal admin --admin-password xxxxxxxxxxxx

Actual results:
See attached logs

Expected results:
Successful with the new ipa-replica joined in the ipa cluster

Additional info:

Comment 1 Alexander Bokovoy 2021-12-15 09:44:23 UTC
The issue looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1998016. Can you please detail what pki packages were installed? This should be fixed with pki-ca-11.0.1-3.el9

Comment 2 Alexander Bokovoy 2021-12-15 09:54:33 UTC
So my thinking is that there are two parts here:

 - pki-core fix in bug 1998016 is required but not enough. It only fixes the first step -- when RA Agent file is generated, it is no longer encrypted with a legacy cipher

 - IPA needs to drop encryption of existing RA Agent file generated with older PKI version _before_ submitting it to replica via Custodia

So we need to fix it on our side as well and have to do it on the 'older' master side, hence RHEL 8.

Comment 3 Monkey Bizness 2021-12-15 10:51:14 UTC
(In reply to Alexander Bokovoy from comment #1)
> The issue looks similar to
> https://bugzilla.redhat.com/show_bug.cgi?id=1998016. Can you please detail
> what pki packages were installed? This should be fixed with
> pki-ca-11.0.1-3.el9

My version of pki-ca is 10.11.2-2.module_el8.5.0+945+a81e57da on the CS8 node.
I have destroyed the CS9 node where I tried that. I don't know the version at the time.
I can try again and get the versions in CS9 if needed

Comment 4 Florence Blanc-Renaud 2022-01-24 17:10:37 UTC
I can confirm the issue is reproducible with RHEL 8.5 server + RHEL 9.0 replica:

[server]# rpm -qa ipa-server pki-ca
ipa-server-4.9.6-6.module+el8.5.0+12660+88e16a2c.x86_64
pki-ca-10.11.2-2.module+el8.5.0+12735+8eb38ccc.noarch


[replica]# rpm -qa ipa-server pki-ca
pki-ca-11.0.1-3.el9.noarch
ipa-server-4.9.8-1.el9.x86_64

meaning we need to fix this issue for RHEL 8->9 migration.

Comment 5 Christian Heimes 2022-01-26 13:51:12 UTC
The problem is not the encryption of the ra-agent.key file. The Custodia client on the new RHEL 9 replica talks to the Custodia server on the RHEL 8.5 source server. Custodia's export layer on the source server creates an encrypted PKCS#12 file that contains the cert and the key. The code uses OpenSSL's default encryption scheme for PKCS#12, which is no longer supported on RHEL 9. To fix the problem you have to modify the export_key() function at https://github.com/freeipa/freeipa/blob/master/ipaserver/secrets/handlers/pemfile.py#L29 to use a stronger encryption algorithm.

The args:

    -keypbe AES-256-CBC -certpbe AES-256-CBC -macalg sha384

should do the trick to use stronger PBEv2 instead of old, insecure PBEv1. It's a trivial fix, but you have to backport the fix to supported RHEL 8 versions...

The weak encryption is not a security problem. The communication is protected by GSSAPI and HTTPS. We only encrypt because FIPS does not let us export key material without encryption.

Comment 6 Christian Heimes 2022-01-26 14:00:40 UTC
PS: There is also a layer of JOSE authentication and encryption in the Custodia protocol. The PKCS#12 encryption is just there for FIPS compliance. The server sends the PKCS#12 blob and plain password inside the JOSE encrypted, JOSE authenticated, HTTPS encrypted, and GSSAPI authenticated response.

Comment 7 Trivino 2022-01-26 18:23:25 UTC
(In reply to Christian Heimes from comment #5)
> The problem is not the encryption of the ra-agent.key file. The Custodia
> client on the new RHEL 9 replica talks to the Custodia server on the RHEL
> 8.5 source server. Custodia's export layer on the source server creates an
> encrypted PKCS#12 file that contains the cert and the key. The code uses
> OpenSSL's default encryption scheme for PKCS#12, which is no longer
> supported on RHEL 9. To fix the problem you have to modify the export_key()
> function at
> https://github.com/freeipa/freeipa/blob/master/ipaserver/secrets/handlers/
> pemfile.py#L29 to use a stronger encryption algorithm.
> 
> The args:
> 
>     -keypbe AES-256-CBC -certpbe AES-256-CBC -macalg sha384
> 
> should do the trick to use stronger PBEv2 instead of old, insecure PBEv1.
> It's a trivial fix, but you have to backport the fix to supported RHEL 8
> versions...
> 
> The weak encryption is not a security problem. The communication is
> protected by GSSAPI and HTTPS. We only encrypt because FIPS does not let us
> export key material without encryption.

thanks Christian, I can confirm that the new args fix the issue. I tested with RHEL 8.5 server + RHEL 9.0 replica:

[root@master /]# rpm -qa ipa-server pki-ca
ipa-server-4.9.6-10.module+el8.5.0+13587+92118e57.x86_64
pki-ca-10.11.2-3.module+el8.5.0+13650+763a7202.noarch


[root@replica cloud-user]# rpm -qa ipa-server pki-ca
pki-ca-11.0.3-1.el9.noarch
ipa-server-4.9.8-1.el9.x86_64

Comment 8 Trivino 2022-01-28 09:44:52 UTC
Upstream PR: https://github.com/freeipa/freeipa/pull/6155

moving to POST.

Comment 9 Florence Blanc-Renaud 2022-01-31 09:09:35 UTC
Fixed upstream
master:
https://pagure.io/freeipa/c/fd7f4a7411eb639d72dbe1f7749f1932fdcc9e62

Comment 10 Florence Blanc-Renaud 2022-02-01 07:58:11 UTC
Fixed upstream
ipa-4-9:
https://pagure.io/freeipa/c/653a7fe02880c168755984133ee143567cc7bb4e

Comment 11 Monkey Bizness 2022-02-01 08:07:45 UTC
Thank you for the quick fix.
May I ask if this new version will be pushed to Centos Stream 8?

Comment 12 Trivino 2022-02-01 10:58:55 UTC
(In reply to Monkey Bizness from comment #11)
> Thank you for the quick fix.
> May I ask if this new version will be pushed to Centos Stream 8?

The fix will be released in rhel8 first, then at some point it will land in c8s as well.

Comment 13 Theodoros Apazoglou 2022-02-01 11:12:52 UTC
@ssidhaye pls qa_ack if you have agreed about it with Francisco/Rafael
@rjeffman pls dev_ack (Francisco is missing perms for now)

Comment 20 Michal Polovka 2022-03-02 17:10:13 UTC
Verified using automation with these metadata http://idm-artifacts.usersys.redhat.com/mpolovka/trigger/4063/test-suite/metadata.orig.yaml.

All tests passed, as seen in the http://idm-artifacts.usersys.redhat.com/mpolovka/trigger//4063/test-suite//restraint.01/index.html

Therefore marking as verified.

Comment 26 errata-xmlrpc 2022-05-10 14:09:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (idm:client and idm:DL1 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:1884