(In reply to Eric Helms from comment #0)
> 5. ruby -e "require
> 'openssl';OpenSSL::PKey.read(File.read('/etc/pki/consumer/key.pem'))"

Where to obtain this key? Or could it be attached to this BZ?

It's the key that comes from registering a RHEL host with the portal:

subscription-manager register

BTW to me, this sounds as and cipher configuration issue than anything else.

Also, to my knowledge, there were some issues with older version of OpenSSL library available in RHEL 9.1. Would you mind to test with CentOS, where more recent version of OpenSSL is available already?

> Where to obtain this key? Or could it be attached to this BZ?

You can create your own: for RSA with 4096 bits: `openssl genrsa -out key.pem 4096`

Failure is the same. Whether 4096 bit RSA is enough for FIPS mode is different question.

But I think the problem is that rubygem-openssl has to be compiled with OPENSSL_FIPS defined [0]? Not sure what kind of bearing does that #ifdef have on behaviour, rubygem-openssl should be just a wrapper for system OpenSSL. I'd have to go through the code paths to find out why I am seeing that message at all.

openssl NVR: openssl-3.0.7-2.el9.x86_64

Using 1minutetip with `--fips` (hopefully that is the correct way) and compose image: 1MT-RHEL-9.2.0-20230209.13

[0] https://github.com/ruby/openssl/blob/79aa3309fa0f0e28e732250b8a914f09dc77e161/ext/openssl/ossl.c#L421

(In reply to Jarek Prokop from comment #4)
> > Where to obtain this key? Or could it be attached to this BZ?
> 
> You can create your own: for RSA with 4096 bits: `openssl genrsa -out
> key.pem 4096`
> 
> Failure is the same. Whether 4096 bit RSA is enough for FIPS mode is
> different question.
> 
> But I think the problem is that rubygem-openssl has to be compiled with
> OPENSSL_FIPS defined [0]?

Thank you for pointing this bit. I was not aware about it up until now.

1) It is unfortunate to see such code around. Was not aware of it. To my knowledge, OpenSSL has always available methods such as `FIPS_mode`, so we should certainly take a look and ideally drop these conditionals.
2) As far as I can see, it just defines some methods/constants, which should have no influence on anything.

You guys, thank you for your investigation! Checking the source code in the openssl, the macro `OPENSSL_FIPS` definition was deleted by the commit below. And there is no OPENSSL_FIPS on the current master branch.

https://github.com/openssl/openssl/commit/b53338cbf8822dd774f9e4057307f347d2b63ff0

```
diff --git a/Configure b/Configure
index 4404963aa7..f6d5a7cfd3 100755
--- a/Configure
+++ b/Configure
...
-    if ($config{fips}) {
-       push @{$config{openssl_other_defines}}, "OPENSSL_FIPS";
-    }
-
...
```

Ok, "fun" stuff debugging OpenSSL and surrounding libraries...
Firstly technical Ruby vs Python, why I think python works here and ruby doesn't.

The key I used throughout was a mentioned command `openssl genrsa -out key.pem 4096`.

Python:
From what I can tell, Python's cryptography module uses EVP_MD* function family of OpenSSL.
That works, so sanity check for FIPS ended up OK.

Ruby:

This one's a bit difficult. Looking at ltrace of rubygem-openssl
~~~
$ ltrace -f -l openssl.so ruby -ropenssl -e "OpenSSL::PKey.read(File.read('key.pem'))"
[pid 6636] openssl.so->ossl_to_der_if_possible(0x7f194fbf23a8, 0x7f194fba2d20, 0x7f194fa58d08, 0x7f194fb79b10) = 0x7f194fbf23a8
[pid 6636] openssl.so->ossl_obj2bio(0x7ffc290a77a0, 156, 1, 0xd6410c)    = 0x56525f45ef30
[pid 6636] openssl.so->ossl_clear_error(0x56525f7ef910, 0x56525f782570, 0x7f194f90cfb8, 0x56525f7d7290) = 0x7f194f7b07e0
[pid 6636] openssl.so->ossl_pkey_read_generic(0x56525f45ef30, 8, 88, 0 <unfinished ...>
[pid 6636] openssl.so->ossl_clear_error(0x56525f7ef910, 0x56525f782570, 0x7f194f90cfb8, 0x56525f7d7290) = 0x7f194f7b07e0
[pid 6636] <... ossl_pkey_read_generic resumed> )                        = 0x56525f822630
[pid 6636] openssl.so->ossl_obj2bio(0x7ffc290b19d0, 0x7f19530ff048, 0x7f194fbb7208, 0x7f194fb74390) = 0x56525f8f0920
[pid 6636] openssl.so->ossl_pem_passwd_value(8, 0, 0, 0x56525f83bed0)    = 8
[pid 6636] openssl.so->ossl_pkey_read_generic(0x56525f8f0920, 8, 0, 0x56525f83bed0 <unfinished ...>
[pid 6636] openssl.so->ossl_clear_error(0x56525f83bed0, 0x56525f8f69e0, 0x7f194f90cfb8, 0x56525f83b6d0) = 0x7f194f7b07e0
[pid 6636] <... ossl_pkey_read_generic resumed> )                        = 0
[pid 6636] openssl.so->ossl_raise(0x7f194fbb7050, 0x7f194fb91c35, 0x56573aaafef0, 7 <unfinished ...>
[pid 6636] openssl.so->ossl_make_error(0x7f194fbb7050, 0x7f194fa4d930, 136, 0x91d08) = 0x7f194fa4d908
-e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
	from -e:1:in `<main>'
[pid 6636] +++ exited (status 1) +++
~~~

It calls `ossl_pkey_read_generic`. I confirmed this by running it through GDB and setting a breakpoint on the function. Triggered it twice, once I received a valid EVP_PKEY pointer, that is for DH class initialization.

The second time I received NULL (in ltrace this corresponds to: `<... ossl_pkey_read_generic resumed> ) = 0`), that's the decoding of the actual file we passed as string.


I tried distilling it down to a C equivalent, learned an interesting point:
~~~
#include <openssl/decoder.h>
#include <openssl/core_dispatch.h>
#include <openssl/evp.h>

int main() {
        EVP_PKEY *pkey = NULL;
        void *ppass = NULL;
        BIO *in = NULL;
        // uncomment if using file instead of the BIO
        //FILE *fp = fopen("key.pem", "rb");

        in = BIO_new_file("key.pem", "rb");

        OSSL_DECODER_CTX *dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "DER", NULL, NULL, 0, NULL, NULL);
        if (dctx == NULL) {
                printf("warp core breach, this shouldn't happen\n");
                return 1;
        }

        printf("%d 'OSSL_KEYMGMT_SELECT_PRIVATE_KEY'\n", OSSL_KEYMGMT_SELECT_PRIVATE_KEY);
        printf("%d 'EVP_PKEY_KEY_PARAMETERS'\n", EVP_PKEY_KEY_PARAMETERS);
        printf("%d 'EVP_PKEY_PUBLIC_KEY'\n", EVP_PKEY_PUBLIC_KEY);
        printf("%d 'EVP_PKEY_KEYPAIR'\n", EVP_PKEY_KEYPAIR);


        OSSL_DECODER_CTX_set_input_type(dctx, "PEM");
        //Fixes for FIPS
        //OSSL_DECODER_CTX_set_selection(dctx, EVP_PKEY_KEYPAIR);

        // uncomment this line and comment out following 'if' line, if using the file method instead of BIO
        //if (OSSL_DECODER_from_fp(dctx, fp)) {
        if (OSSL_DECODER_from_bio(dctx, in)) {
                printf("should be ok %p\n", pkey);
        } else {
                printf("Error decoding\n");
                return 1;
        }


        return 0;
}
~~~

If I pass 0 to the "selection" argument of the function `OSSL_DECODER_CTX_new_for_pkey`, the EVP_PKEY pointer is NULL, but this runs OK on non-RHEL machine apparently (Fedora Rawhide NVR: openssl-3.0.7-4.fc38.x86_64).
OpenSSL breaks its teeth on RHEL in this code.

If I initialize the OSSL_DECODER and use selection argument with "EVP_PKEY_KEYPAIR", then it works. But if I set it after the struct initialization via the `OSSL_DECODER_CTX_set_selection` with EVP_PKEY_KEYPAIR, it does not work.

The generic read (<https://github.com/ruby/openssl/blob/v3.0.0/ext/openssl/ossl_pkey.c#L93>) uses selection 0.
Man page states: "selection can also be set to 0 to indicate that the code will auto detect the selection." What code is meant here? OpenSSL code, or the code using the function should do the detection? If the latter, selection setting not working properly is a bit of a hurdle in this.

If I uncomment the `OSSL_DECODER_CTX_set_selection` line from the C example and recompile, it cannot still be parsed then on FIPS or RHEL 9 for that matter, but if I call `OSSL_DECODER_CTX_new_for_pkey` again with that EVP_PKEY_KEYPAIR in selection argument instead of setting it later, it passes. I also verified through GDB that the value of selection is 135 which corresponds to `EVP_PKEY_KEYPAIR` by the time it gets to `OSSL_DECODER_from_bio` function if OSSL_DECODER_CTX_set_selection is used.

In short, there is something broken on RHEL and OpenSSL, or the way rubygem-openssl handles loading is completely broken and should be remade.

openssl NVR: openssl-3.0.7-2.el9.x86_64

@dbelyavs could you PTAL?

(In reply to Jarek Prokop from comment #7)
> or the way rubygem-openssl handles loading is completely broken and should be remade.

Assuming we are talking about Ruby 3.0, we could also miss some upstream bits, since the OpenSSL 3.x support was backported. Is there by a chance any difference with Ruby 3.1?

@ehelms have you had a chance to test with Ruby 3.1 module?

Current (since 9.0) OpenSSL version loads and activates FIPS and base providers when FIPS mode is detected if the library loads non-empty config file. Could you please check it?

If you use the default openssl context, could you please try the output of the following functions:

OSSL_PROVIDER_available(NULL, "default");
OSSL_PROVIDER_available(NULL, "base");
OSSL_PROVIDER_available(NULL, "fips");

> have you had a chance to test with Ruby 3.1 module?

That won't be much help.

I tested this with Ruby 3.1 and Ruby 3.0. Not even rubygem-openssl 3.1.0 has any success. OTOH, there is this line: https://github.com/ruby/openssl/blob/v3.1.0/ext/openssl/ossl_pkey.c#L128 that could work, assuming the "OSSL_DECODER_CTX_set_selection" work as I expected it to.

This also means that the fix will have to be duplicated to 3.0 and 3.1

> If you use the default openssl context, could you please try the output of the following functions:

Reused example from before and just mounted the following:

```
#include <openssl/provider.h>

// ...

printf("default: %d, base: %d, fips: %d\n",
			OSSL_PROVIDER_available(NULL, "default"),
			OSSL_PROVIDER_available(NULL, "base"),
			OSSL_PROVIDER_available(NULL, "fips")
	      );
// ...
```
FIPS RHEL 9: `default: 1, base: 1, fips: 1`

RHEL 9 without FIPS looks ok. I think I forgot to generate the key when using the newly prepared machine...
without FIPS: default: 1, base: 0, fips: 0

But it seems to have some side effects??
I wanted to query it from inside rubygem-openssl using this:
~~~
diff -Naur a/ossl_pkey.c b/ossl_pkey.c
--- a/ossl_pkey.c	2023-02-21 06:23:27.901287679 -0500
+++ b/ossl_pkey.c	2023-02-21 06:24:28.341287679 -0500
@@ -81,6 +81,7 @@

 #if OSSL_OPENSSL_PREREQ(3, 0, 0)
 # include <openssl/decoder.h>
+#include <openssl/provider.h>

 EVP_PKEY *
 ossl_pkey_read_generic(BIO *bio, VALUE pass)
@@ -90,6 +91,12 @@
     EVP_PKEY *pkey = NULL;
     int pos = 0, pos2;

+    printf("default: %d, base: %d, fips: %d\n",
+		    OSSL_PROVIDER_available(NULL, "default"),
+		    OSSL_PROVIDER_available(NULL, "base"),
+		    OSSL_PROVIDER_available(NULL, "fips")
+	  );
+
     dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "DER", NULL, NULL, 0, NULL, NULL);
     if (!dctx)
         goto out;
~~~

And that made rubygem-openssl work on FIPS! Not that C example I created, but that's ok for me.
~~~
$ RUBYOPT=-I$PWD ruby -ropenssl -e "p OpenSSL::PKey.read(File.read('key.pem'))"
default: 1, base: 1, fips: 1
default: 1, base: 1, fips: 1
#<OpenSSL::PKey::RSA:0x00007f78b4d9b500 oid=rsaEncryption>
~~~

To reproduce that command with the rubygem-openssl rebuild, this should suffice:
~~~
$ sudo dnf install -y openssl-devel ruby-devel gcc make
$ gem fetch openssl --version 3.0.0
$ gem unpack openssl-3.0.0.gem
$ cd openssl-3.0.0/ext/openssl
$ ruby extconf.rb
// Apply the patch from the comment
$ patch -p1 < patch
$ make
// Stay in the directory and run the RUBYOPT=-I$PWD ......
~~~

Could you please check that the OpenSSL configuration file is loaded in FIPS mode when it doesn't work for you? We heavily relate on presence of it.

> Could you please check that the OpenSSL configuration file is loaded in FIPS mode

No idea how to do that or what I am looking for exactly. Any way to do this from the code?

I usually use strace for this purpose

So, it opens some cnf files that are OpenSSL, there does not seem to be relation to whether it fails or not:

With patch:
~~~
$ RUBYOPT=-I$PWD strace ruby -ropenssl -e "p OpenSSL::PKey.read(File.read('key.pem'))" 2>&1 | grep cnf
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 5
newfstatat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", {st_mode=S_IFREG|0644, st_size=619, ...}, 0) = 0
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 6
~~~

Without patch:
~~~
$ strace ruby -ropenssl -e "p OpenSSL::PKey.read(File.read('key.pem'))" 2>&1 | grep cnf
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 5
newfstatat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", {st_mode=S_IFREG|0644, st_size=619, ...}, 0) = 0
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 6
~~~

> Could you please check that the OpenSSL configuration file is loaded in FIPS mode when it doesn't work for you? We heavily relate on presence of it.

I executed the 1minutetip by my machine. Maybe it is the same environment with Jarek's one. I was able to reproduce the reported issue.

```
$ 1minutetip --fips 1MT-RHEL-9.2.0-20230209.13
```

I found this command to check if the machine is run by the FIPS mode. The machine is running on the FIPS mode.

```
[root@ci-vm-10-0-137-30 ~]# update-crypto-policies --show
FIPS
```

I think the proper command is fips-mode-setup --check but it's not the issue.

Is this a problem for RHEL 9.0? I strongly doubt any changes in OpenSSL in 9.2 should affect this behavior.

How can I assist you?

> I think the proper command is fips-mode-setup --check but it's not the issue.

```
[root@ci-vm-10-0-138-69 ~]# which fips-mode-setup
/usr/bin/fips-mode-setup

[root@ci-vm-10-0-138-69 ~]# rpm -qf /usr/bin/fips-mode-setup
crypto-policies-scripts-20221215-1.git9a18988.el9.noarch

[root@ci-vm-10-0-138-69 ~]# fips-mode-setup --check
FIPS mode is enabled.
```

> Is this a problem for RHEL 9.0? I strongly doubt any changes in OpenSSL in 9.2 should affect this behavior.

I only tested and reproduced the issue for RHEL 9.2. In my understanding Jarek and I are using the same version of the RHEL 9 instance: 1MT-RHEL-9.2.0-20230209.13

```
[root@ci-vm-10-0-138-69 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.2 Beta (Plow)

[root@ci-vm-10-0-138-69 ~]# rpm -q openssl
openssl-3.0.7-2.el9.x86_64
```

> How can I assist you?

In my current approach, I am trying to create a Dockerfile with RHEL 9 (UBI 9) with fips mode. If I can reproduce the issue on the container, it's easy to report it to the Ruby OpenSSL binding (ruby/openssl) project. So, if you can tell me how to set up FIPS mode on RHEL 9 or UBI 9, it's helpful and it saves my time. :) I assume that Jarek may try to fix this issue by a different approach.

I'm not sure what happens in case of Docker. We don't have any regressions on our side before the latest CVEs emerged and the CVEs fixes shouldn't affect it.

> I am trying to create a Dockerfile with RHEL 9 (UBI 9) with fips mode

Interesting idea, from what I've heard this can be very tricky. So, good luck!

> Is this a problem for RHEL 9.0

I will check and report back.

> Interesting idea, from what I've heard this can be very tricky. So, good luck!

It's not simple. We may need to change the kernel parameter.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/security_hardening/index
> Chapter 2. Installing the system in FIPS mode
> ...
> Procedure
>    Add the fips=1 option to the kernel command line during the system installation.
>    During the software selection stage, do not install any third-party software. 

Q. Do you have a FIPS mode enabled machine to test?
https://github.com/ruby/openssl/issues/601

Continuing my C example: adding this line after variable declarations was enough to get it to work: `OSSL_PROVIDER_load(NULL, "fips");`

Using the latest rubygem-openssl on a FIPS enabled RHEL 9 machine was not enough.
A patch utilizing OSSL_PROVIDER_load fixed the situation:
```
diff --git a/ext/openssl/ossl_pkey.c b/ext/openssl/ossl_pkey.c
index 4762566..ea81caf 100644
--- a/ext/openssl/ossl_pkey.c
+++ b/ext/openssl/ossl_pkey.c
@@ -90,6 +90,8 @@ ossl_pkey_read_generic(BIO *bio, VALUE pass)
     EVP_PKEY *pkey = NULL;
     int pos = 0, pos2;

+    OSSL_PROVIDER_load(NULL, "fips");
+
     dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "DER", NULL, NULL, 0, NULL, NULL);
     if (!dctx)
         goto out;
```

Then running:
```
$ bundle exec ruby -e "require 'openssl'; p OpenSSL::VERSION; p OpenSSL::PKey.read(File.read('./key.pem'))"
"3.1.0"
#<OpenSSL::PKey::RSA:0x00007f294bb08f08 oid=rsaEncryption>
```
shows that the patch helped.

Thanks for your help, Dmitry. I will bring this up with upstream on Jun's ticket. Maybe we'll come up with some reasonable way to get this to work.

I don't think it's a proper solution. Looks like the initialization of OpenSSL in FIPS mode bypasses the "normal" way and so we don't have FIPS provider automatically loaded. As this is not the case for any of our applications, I think it's not an openssl downstream bug but the ruby-specific problem

It might not be a proper solution but it works. I do not propose it as a fix, it helps to illustrate that something is not properly loading the cryptography for something as "simple" as decoding a PEM key.

Hmm, weird, if I create an openssl conf (inspired by this page https://www.openssl.org/docs/man3.0/man7/fips_module.html)
that specifies FIPS provider then everything is OK:
~~~
$ cat openssl_conf.cnf
config_diagnostics = 1
openssl_conf = openssl_init

[openssl_init]
providers = provider_sect

[provider_sect]
fips = fips_sect
base = base_sect

[base_sect]
activate = 1
$ cat decoder.c
#include <openssl/decoder.h>
#include <openssl/core_dispatch.h>
#include <openssl/evp.h>

int main() {
        EVP_PKEY *pkey = NULL;
        void *ppass = NULL;
        BIO *in = NULL;

        in = BIO_new_file("key.pem", "rb");

        OSSL_DECODER_CTX *dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "DER", NULL, NULL, 0, NULL, NULL);
        if (dctx == NULL) {
                printf("warp core breach, this shouldn't happen\n");
                return 1;
        }

        OSSL_DECODER_CTX_set_input_type(dctx, "PEM");

        if (OSSL_DECODER_from_bio(dctx, in)) {
                printf("should be ok %p\n", pkey);
        } else {
                printf("Error decoding\n");
                return 1;
        }


        return 0;
}
$ gcc -lssl -lcrypto -g ./decoder.c
$ OPENSSL_CONF=$(pwd)/openssl_conf.cnf ./a.out
should be ok 0x8bc1d0
$ ./a.out
Error decoding
~~~
I assume the config does something comparable to `OSSL_PROVIDER_load(NULL, "fips");`

This brings me to a question, can we expect, on proper FIPS RHEL 9, that there is going to exist a default openssl.cnf that specifies "fips = fips_sect" in provider_sect that will be loaded unless overriden?

In normal circumstances the fips.so should be loaded automatically when the system is in FIPS mode (as it happens for OpenSSL commandline, openssh, etc) without any config changes.

I opened the upstream ticket.

OpenSSL::PKey.read: can't parse PKey on Linux with FIPS mode enabled
https://github.com/ruby/openssl/issues/603

Let me share the update from the last comment above.

* I sent a PR to fix the FIPS mode flag methods (basic features of the FIPS mode) in the case of OpenSSL 3, also adding the OpenSSL 3.0.8 FIPS mode CI case. This PR is not directly related to this Bugzilla's issue about reading PEM file. However, adding the FIPS mode case to the upstream CI is a great way to maintain the case. And it's also to create an environment for another PR to fix the read error to be merged when we are ready for that.
* Implement FIPS functions, adding OpenSSL FIPS mode case on CI.
  https://github.com/ruby/openssl/pull/608

I was able to reproduce this issue with the upstream Ruby compiled from the source (not RPM package), and with the upstream OpenSSL 3.0.8 with FIPS mode built from the source (not RPM package) on both Fedora and Ubuntu.

I tried to contact to the maintainer of the Ruby OpenSSL bindings, ruby/openssl in several ways to ask him to review the PR above. However, I haven't seen the response from the maintainer so far.

I opened the issue ticket to ask a question on OpenSSL project.
* How to fix OSSL_DECODER_from_bio returning 0 as failure on OpenSSL 3 FIPS mode?
  https://github.com/openssl/openssl/issues/20657

You guys, thank you for your help.

I've marked this up as OCP being dependent on this. We can live with it in 4.13 but we'd like to remove the systemd-journald compatibility configuration we had to put in place to make the journal compatible with UBI8 journalctl in 4.14. As such we'd like this to be fixed in 9.2.z as soon as possible.

> I've marked this up as OCP being dependent on this. We can live with it in 4.13 but we'd like to remove the systemd-journald compatibility configuration we had to put in place to make the journal compatible with UBI8 journalctl in 4.14. As such we'd like this to be fixed in 9.2.z as soon as possible.

Scott,
I created the patch to fix this specific issue below on the upstream. However, the problem is there are some other issues in the Ruby OpenSSL bindings in the FIPS mode. I am currently working to fix other issues to pass all the unit tests in the FIPS mode. It takes time. However if you want us to fix this specific issue by `ruby -e "require 'openssl';OpenSSL::PKey.read(File.read('/etc/pki/consumer/key.pem'))"`. I think we can apply the downstream patch to the RHEL 9. What do you think?
https://github.com/ruby/openssl/issues/603#issuecomment-1507152493

Or it's great if the openssl RPM package can fix this issue.

(In reply to Jun Aruga from comment #49)
> Or it's great if the openssl RPM package can fix this issue.

We don't have an upstream fix for it yet.

(In reply to Scott Dodson from comment #47)
> I've marked this up as OCP being dependent on this. We can live with it in
> 4.13 but we'd like to remove the systemd-journald compatibility
> configuration we had to put in place to make the journal compatible with
> UBI8 journalctl in 4.14. As such we'd like this to be fixed in 9.2.z as soon
> as possible.

systemd-journald? What is the relation of this issue with systemd-journald? I am not sure I understand.

(In reply to Vít Ondruch from comment #51)
> (In reply to Scott Dodson from comment #47)
> > I've marked this up as OCP being dependent on this. We can live with it in
> > 4.13 but we'd like to remove the systemd-journald compatibility
> > configuration we had to put in place to make the journal compatible with
> > UBI8 journalctl in 4.14. As such we'd like this to be fixed in 9.2.z as soon
> > as possible.
> 
> systemd-journald? What is the relation of this issue with systemd-journald?
> I am not sure I understand.

Sure, when building the fluentd log collector on UBI9 it exposes this problem. When fluentd is built on UBI8 stack this issue is not present. As we'd like to remove the compatibility configuration required to make RHEL / RHCOS 9 systemd-journald compatible with UBI8 we'd like to have this fixed.

I changed this ticket's component from ruby to openssl.

If you read the upstream ticket, the root cause was found in the openssl side, and they also have a reproducer written by C without Ruby.
https://github.com/openssl/openssl/issues/20657

Though we have a workaround patch to avoid this issue on the ruby side.
https://github.com/ruby/openssl/issues/603#issuecomment-1507152493

Scott,

Is this PR related to what you are doing? It seems that the workaround patch on the ruby/openssl side is applied in the PR.
https://github.com/ViaQ/logging-fluentd/pull/70


Note that I opened a ticket in the Ruby project to fix the current FIPS related issues in the ruby/openssl effectively, and to maintain it sustainably. As a fastest timeline I have a possibility to start to work as a co-maintainer for the ruby/opessnl from next month 10th May.

Being a co-maintainer of the ruby/openssl for the OpenSSL FIPS mode
https://bugs.ruby-lang.org/issues/19608

(In reply to Jun Aruga from comment #54)
> Scott,
> 
> Is this PR related to what you are doing? It seems that the workaround patch
> on the ruby/openssl side is applied in the PR.
> https://github.com/ViaQ/logging-fluentd/pull/70

I'm not personally involved, that'd be @jcantril and @aconway However, I'm motivating fixes for these bugs on their behalf.

@Scott I do not believe the journal issue is related to this issue other then they were both discovered when we moved to RHEL9

(In reply to Jeff Cantrill from comment #56)
> @Scott I do not believe the journal issue is related to this issue other
> then they were both discovered when we moved to RHEL9

@jcantril 

What I meant was that your efforts to move to UBI9, as I've been told, were stalled due to this Ruby problem affecting FIPS mode clusters. This will likely block efforts to move Logging 5.7 toward UBI9, correct?

(In reply to Scott Dodson from comment #57)

> 
> What I meant was that your efforts to move to UBI9, as I've been told, were
> stalled due to this Ruby problem affecting FIPS mode clusters. This will
> likely block efforts to move Logging 5.7 toward UBI9, correct?

That is correct that this is a blocker to move to RHEL 9 but we decided RHEL9 is a target for logging 5.8.  We will continue with RHEL8 for logging 5.7

Note that I merged the PR <https://github.com/ruby/openssl/pull/615> that is a workaround to fix this issue on the ruby side, to the Ruby OpenSSL project repository (ruby/openssl)'s master branch.

I don't think we are going to fix this issue on the OpenSSL side other way than via rebase.

I saw this is closed wontfix, but also that a change was put into ruby-openssl. Can you help me understand what the path forward is? Are you saying this issue will not be fixed in any release of RHEL 9?

We don't have an OpenSSL upstream fix for it and don't plan to implement the fix ourselves. So the fix on the ruby-openssl level is a proper one. Currently we have no plans to fix it in RHEL 9 series.

"So the fix on the ruby-openssl level is a proper one."

Will this fixed into a Ruby release in RHEL 9? Do I need to file a clone of this BZ into the Ruby component to do so? The ability to read private keys via Ruby on a FIPS enabled RHEL is a key component to Red Hat Satellite being able to run on RHEL 9.

(In reply to Eric Helms from comment #63)
> "So the fix on the ruby-openssl level is a proper one."
> 
> Will this fixed into a Ruby release in RHEL 9?

I'll defer to @jaruga to judge this, if this is technically feasible (probably yes).

> Do I need to file a clone of
> this BZ into the Ruby component to do so?

Please note that this was originally reported against Ruby, so feel free to reopen and change the component, or clone. As you find suitable. But maybe clone with just the right amount of context / details would be better at this stage.

> The ability to read private keys
> via Ruby on a FIPS enabled RHEL is a key component to Red Hat Satellite
> being able to run on RHEL 9.

There was a fix merged to ruby-openssl -- https://bugzilla.redhat.com/show_bug.cgi?id=2170105#c59
This was deemed not going to be fixed in Openssl itself.

Red Hat Satellite is working towards running on RHEL 9 but has a requirement to run on FIPS-enabled RHEL and this bug would prevent Satellite's ability to do so.

(In reply to Vít Ondruch from comment #64)
> (In reply to Eric Helms from comment #63)
> > "So the fix on the ruby-openssl level is a proper one."
> > 
> > Will this fixed into a Ruby release in RHEL 9?
> 
> I'll defer to @jaruga to judge this, if this is technically feasible
> (probably yes).

Yes, I am happy to fix this issue by applying the workaround patches in ruby side in this case, and I will. I believe this is technically feasible.

(In reply to Eric Helms from comment #65)
> There was a fix merged to ruby-openssl --
> https://bugzilla.redhat.com/show_bug.cgi?id=2170105#c59
> This was deemed not going to be fixed in Openssl itself.
> 
> Red Hat Satellite is working towards running on RHEL 9 but has a requirement
> to run on FIPS-enabled RHEL and this bug would prevent Satellite's ability
> to do so.

Eric, 
I just want to know which Ruby RPM version are you using to run Red Hat Satellite on RHEL 9? Ruby 3.1 or 3.0? Can you share the result of the `rpm -q ruby`?

> Eric, 
> I just want to know which Ruby RPM version are you using to run Red Hat
> Satellite on RHEL 9? Ruby 3.1 or 3.0? Can you share the result of the `rpm
> -q ruby`?

Is it only RHEL 9? How about RHEL 8? I would like to know all the possible RHEL streams (cases) that Red Hat Satelite is running in RHEL FIPS mode.

Satellite is not yet running on RHEL 9 -- we are working towards that and trying to stay ahead of issues. This issue only shows itself on RHEL 9 so far. And we intend to run Ruby 3.1 most likely and then follow up to Ruby 3.2 when it's available.

I sent the PR to fix this issue to Fedora rawhide.
https://src.fedoraproject.org/rpms/ruby/pull-request/160

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.

(In reply to Jarek Prokop from comment #7)
> Ok, "fun" stuff debugging OpenSSL and surrounding libraries...
> Firstly technical Ruby vs Python, why I think python works here and ruby
> doesn't.
> 
> The key I used throughout was a mentioned command `openssl genrsa -out
> key.pem 4096`.
> 
> Python:
> From what I can tell, Python's cryptography module uses EVP_MD* function
> family of OpenSSL.
> That works, so sanity check for FIPS ended up OK.
> 
> Ruby:
> 
> This one's a bit difficult. Looking at ltrace of rubygem-openssl
> ~~~
> $ ltrace -f -l openssl.so ruby -ropenssl -e
> "OpenSSL::PKey.read(File.read('key.pem'))"
> [pid 6636] openssl.so->ossl_to_der_if_possible(0x7f194fbf23a8,
> 0x7f194fba2d20, 0x7f194fa58d08, 0x7f194fb79b10) = 0x7f194fbf23a8
> [pid 6636] openssl.so->ossl_obj2bio(0x7ffc290a77a0, 156, 1, 0xd6410c)    =
> 0x56525f45ef30
> [pid 6636] openssl.so->ossl_clear_error(0x56525f7ef910, 0x56525f782570,
> 0x7f194f90cfb8, 0x56525f7d7290) = 0x7f194f7b07e0
> [pid 6636] openssl.so->ossl_pkey_read_generic(0x56525f45ef30, 8, 88, 0
> <unfinished ...>
> [pid 6636] openssl.so->ossl_clear_error(0x56525f7ef910, 0x56525f782570,
> 0x7f194f90cfb8, 0x56525f7d7290) = 0x7f194f7b07e0
> [pid 6636] <... ossl_pkey_read_generic resumed> )                        =
> 0x56525f822630
> [pid 6636] openssl.so->ossl_obj2bio(0x7ffc290b19d0, 0x7f19530ff048,
> 0x7f194fbb7208, 0x7f194fb74390) = 0x56525f8f0920
> [pid 6636] openssl.so->ossl_pem_passwd_value(8, 0, 0, 0x56525f83bed0)    = 8
> [pid 6636] openssl.so->ossl_pkey_read_generic(0x56525f8f0920, 8, 0,
> 0x56525f83bed0 <unfinished ...>
> [pid 6636] openssl.so->ossl_clear_error(0x56525f83bed0, 0x56525f8f69e0,
> 0x7f194f90cfb8, 0x56525f83b6d0) = 0x7f194f7b07e0
> [pid 6636] <... ossl_pkey_read_generic resumed> )                        = 0
> [pid 6636] openssl.so->ossl_raise(0x7f194fbb7050, 0x7f194fb91c35,
> 0x56573aaafef0, 7 <unfinished ...>
> [pid 6636] openssl.so->ossl_make_error(0x7f194fbb7050, 0x7f194fa4d930, 136,
> 0x91d08) = 0x7f194fa4d908
> -e:1:in `read': Could not parse PKey (OpenSSL::PKey::PKeyError)
> 	from -e:1:in `<main>'
> [pid 6636] +++ exited (status 1) +++
> ~~~
> 
> It calls `ossl_pkey_read_generic`. I confirmed this by running it through
> GDB and setting a breakpoint on the function. Triggered it twice, once I
> received a valid EVP_PKEY pointer, that is for DH class initialization.
> 
> The second time I received NULL (in ltrace this corresponds to: `<...
> ossl_pkey_read_generic resumed> ) = 0`), that's the decoding of the actual
> file we passed as string.
> 
> 
> I tried distilling it down to a C equivalent, learned an interesting point:
> ~~~
> #include <openssl/decoder.h>
> #include <openssl/core_dispatch.h>
> #include <openssl/evp.h>
> 
> int main() {
>         EVP_PKEY *pkey = NULL;
>         void *ppass = NULL;
>         BIO *in = NULL;
>         // uncomment if using file instead of the BIO
>         //FILE *fp = fopen("key.pem", "rb");
> 
>         in = BIO_new_file("key.pem", "rb");
> 
>         OSSL_DECODER_CTX *dctx = OSSL_DECODER_CTX_new_for_pkey(&pkey, "DER",
> NULL, NULL, 0, NULL, NULL);
>         if (dctx == NULL) {
>                 printf("warp core breach, this shouldn't happen\n");
>                 return 1;
>         }
> 
>         printf("%d 'OSSL_KEYMGMT_SELECT_PRIVATE_KEY'\n",
> OSSL_KEYMGMT_SELECT_PRIVATE_KEY);
>         printf("%d 'EVP_PKEY_KEY_PARAMETERS'\n", EVP_PKEY_KEY_PARAMETERS);
>         printf("%d 'EVP_PKEY_PUBLIC_KEY'\n", EVP_PKEY_PUBLIC_KEY);
>         printf("%d 'EVP_PKEY_KEYPAIR'\n", EVP_PKEY_KEYPAIR);
> 
> 
>         OSSL_DECODER_CTX_set_input_type(dctx, "PEM");
>         //Fixes for FIPS
>         //OSSL_DECODER_CTX_set_selection(dctx, EVP_PKEY_KEYPAIR);
> 
>         // uncomment this line and comment out following 'if' line, if using
> the file method instead of BIO
>         //if (OSSL_DECODER_from_fp(dctx, fp)) {
>         if (OSSL_DECODER_from_bio(dctx, in)) {
>                 printf("should be ok %p\n", pkey);
>         } else {
>                 printf("Error decoding\n");
>                 return 1;
>         }
> 
> 
>         return 0;
> }
> ~~~
> 
> If I pass 0 to the "selection" argument of the function
> `OSSL_DECODER_CTX_new_for_pkey`, the EVP_PKEY pointer is NULL, but this runs
> OK on non-RHEL machine apparently (Fedora Rawhide NVR:
> openssl-3.0.7-4.fc38.x86_64).
> OpenSSL breaks its teeth on RHEL in this code.
> 
> If I initialize the OSSL_DECODER and use selection argument with
> "EVP_PKEY_KEYPAIR", then it works. But if I set it after the struct
> initialization via the `OSSL_DECODER_CTX_set_selection` with
> EVP_PKEY_KEYPAIR, it does not work.
> 
> The generic read
> (<https://github.com/ruby/openssl/blob/v3.0.0/ext/openssl/ossl_pkey.c#L93>)
> uses selection 0.
> Man page states: "selection can also be set to 0 to indicate that the code
> will auto detect the selection." What code is meant here? OpenSSL code, or
> the code using the function should do the detection? If the latter,
> selection setting not working properly is a bit of a hurdle in this.
> 
> If I uncomment the `OSSL_DECODER_CTX_set_selection` line from the C example
> and recompile, it cannot still be parsed then on FIPS or RHEL 9 for that
> matter, but if I call `OSSL_DECODER_CTX_new_for_pkey` again with that
> EVP_PKEY_KEYPAIR in selection argument instead of setting it later, it
> passes. I also verified through GDB that the value of selection is 135 which
> corresponds to `EVP_PKEY_KEYPAIR` by the time it gets to
> `OSSL_DECODER_from_bio` function if OSSL_DECODER_CTX_set_selection is used.
> 
> In short, there is something broken on RHEL and OpenSSL, or the way
> rubygem-openssl handles loading is completely broken and should be remade.
> 
> openssl NVR: openssl-3.0.7-2.el9.x86_64

@Jarek, 
I found something that actually worked on my end. Although I am not sure if it involved Ruby or Python on the back end.

If I upload the same exact report to Red Hat Insights, it works! 

On a RHEL 9 test system, I executed the following:

[jordi_Rubalcaba@hostname ~]$ sudo insights-client --compliance
System uses SSG version 0.1.69
Saved tailoring file for xccdf_org.ssgproject.content_profile_stig_gui to /var/tmp/oscap_tailoring_file-xccdf_org.ssgproject.content_profile_stig_gui.lo2boapo.xml
Running scan for xccdf_org.ssgproject.content_profile_stig_gui... this may take a while
Uploading Insights data.
Successfully uploaded report for hostname.fqdn
[jordi_Rubalcaba@hostname ~]$

I hope this helps on your end...

Regards,
Jordi Rubalcaba