@Jun could you PTAL? Thanks in advance.

BTW this is the test scratch build of Ruby 3.4.1:

https://koji.fedoraproject.org/koji/taskinfo?taskID=127511258

So very simple code actually fails:

$ cat openssl-flip-test.c 
#include <stdlib.h>
#include <openssl/ssl.h>

int main(void) {
        setenv("OPENSSL_FORCE_FIPS_MODE", "1", 1);

        if (!OPENSSL_init_ssl(0, NULL))
                abort();
}
$ gcc -lssl -o openssl-flip-test{,.c} && ./openssl-flip-test 

With openssl-3.2.2-8.fc42 this succeeds, with -10.fc42 this abort() .

> @Jun could you PTAL? Thanks in advance.

OK. I think the above Mamoru's reproducing code helps to fix this issue. I will also take a look at the issue.

Seeing the following openssl koji builds on rawhide, there is no openssl-3.2.2-9.fc42 build.
https://koji.fedoraproject.org/koji/packageinfo?packageID=109

So, I tested the above simple reproducing code with the rpms/openssl <1b863c4f9a3f99ac06765016f3deec199af27e39> that is openssl-3.2.2-9.fc42 on Fedora rawhide mock environment.
https://src.fedoraproject.org/rpms/openssl/c/1b863c4f9a3f99ac06765016f3deec199af27e39?branch=rawhide

The scratch build is below.
https://koji.fedoraproject.org/koji/taskinfo?taskID=127517026

```
$ rpm -q openssl openssl-devel
openssl-3.2.2-9.fc42.x86_64
openssl-devel-3.2.2-9.fc42.x86_64

$ gcc -lssl -o openssl-flip-test openssl-flip-test.c

$ ./openssl-flip-test
Aborted (core dumped)
```

Below is my debug log with the openssl-3.2.2-9.fc42.x86_64. I am not sure why next step of the crypto/init.c:612 moves to the `abort()`.

```
$ rm openssl-flip-test
$ gcc -lssl -O0 -g3 -ggdb3 -gdwarf-5 -o openssl-flip-test openssl-flip-test.c

$ gdb openssl-flip-test
...
(gdb) f
#0  OPENSSL_init_crypto (opts=opts@entry=76, settings=<optimized out>) at crypto/init.c:612
612	            if (ret <= 0)
(gdb) l
607	                                   ossl_init_config);
608	                conf_settings = NULL;
609	                CRYPTO_THREAD_unlock(init_lock);
610	            }
611	
612	            if (ret <= 0)
613	                return 0;
614	        }
615	    }
616	
(gdb) p ret
$1 = -1
(gdb) bt
#0  OPENSSL_init_crypto (opts=opts@entry=76, settings=<optimized out>) at crypto/init.c:612
#1  0x00007ffff7ef8a8d in OPENSSL_init_ssl (opts=76, settings=<optimized out>) at ssl/ssl_init.c:115
#2  0x000000000040116d in main () at openssl-flip-test.c:7
(gdb) n
main () at openssl-flip-test.c:8
8	        abort();
```

So looks like this is really strange.

Even tested on F41 environment, testing my test program,
openssl-3.2.2-7.fc41 koji binary works:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2544726
https://kojipkgs.fedoraproject.org/packages/openssl/3.2.2/7.fc41/


but openssl-3.2.2-7.fc41 with rebuilt by koji scratch-build for f41 today:
https://koji.fedoraproject.org/koji/taskinfo?taskID=127534355
https://koji.fedoraproject.org/scratch/mtasaka/task_127534355/
this causes abort() for my test program.

So maybe this is due to some toolchain change??

Looks like debugedit change is related to this. (debugedit-5.0-17.fc41 -> 5.1-2.fc42).

openssl build "edits" .rodata1 section of fips.so , looks like this section changes
after /usr/bin/find-debuginfo is executed.

Confirmed that rebuilding openssl-3.2.2-10.fc42 with debugedit downgraded to debugedit-5.0-17.fc41 cures this issue.

ref: https://src.fedoraproject.org/rpms/openssl/blob/3e4eea28e54d0550f621d087a90b82d7a4b57883/f/openssl.spec#_398
LL398-406

Once asking debugedit maintainer for help.

gdb shows with koji openssl-3.2.2-10.fc42 OSSL_SELF_TEST_oncorrupt_byte() called in
verify_integrity() in providers/fips/self_test.c is returning an unexpected result:

(gdb) p ret
$50 = 0
(gdb) bt
#0  verify_integrity_rodata.constprop.0 (bio=bio@entry=0x441180, read_ex_cb=<optimized out>, 
    expected=expected@entry=0x7ffff77d80e0 <fips_hmac_container.lto_priv> "Z\357\314&\201\331\276\023\213\301j\017\016{^\362E3\v\002S\327_1\207\231S2\301\327\207*\001\033\003;,3", 
    expected_len=<optimized out>, libctx=<optimized out>, ev=ev@entry=0x422af0, event_type=0x7ffff77c9350 "Module_Integrity") at providers/fips/self_test.c:353
#1  0x00007ffff761f6a5 in SELF_TEST_post (st=st@entry=0x422698, on_demand_test=on_demand_test@entry=0) at providers/fips/self_test.c:540
#2  0x00007ffff7623655 in OSSL_provider_init_int (handle=0x419e90, in=<optimized out>, out=0x7fffffffdf98, provctx=0x7fffffffdfa0) at providers/fips/fipsprov.c:839
#3  0x00007ffff793761a in provider_init (prov=0x419e90) at crypto/provider_core.c:967
#4  provider_activate (prov=prov@entry=0x419e90, lock=0, lock@entry=1, upcalls=upcalls@entry=1) at crypto/provider_core.c:1176
#5  0x00007ffff7937b3f in ossl_provider_activate (prov=prov@entry=0x419e90, upcalls=upcalls@entry=1, aschild=aschild@entry=0) at crypto/provider_core.c:1313
#6  0x00007ffff7938e6f in provider_conf_activate (libctx=libctx@entry=0x7ffff7caec60 <default_context_int.lto_priv>, name=name@entry=0x7ffff7b983e8 "fips", 
    value=value@entry=0x7ffff7b9a725 "fips_sect", path=path@entry=0x0, soft=soft@entry=0, cnf=cnf@entry=0x419b80) at crypto/provider_conf.c:245
#7  0x00007ffff7939261 in provider_conf_load (libctx=libctx@entry=0x7ffff7caec60 <default_context_int.lto_priv>, name=<optimized out>, name@entry=0x7ffff7b983e8 "fips", 
    value=value@entry=0x7ffff7b9a725 "fips_sect", cnf=cnf@entry=0x419b80) at crypto/provider_conf.c:325
#8  0x00007ffff79399c7 in provider_conf_init (md=<optimized out>, cnf=<optimized out>) at crypto/provider_conf.c:395
#9  0x00007ffff7870105 in module_init (pmod=0x4194d0, name=0x41a840 "providers", value=0x41a890 "provider_sect", cnf=<optimized out>) at crypto/conf/conf_mod.c:425
#10 module_run (cnf=<optimized out>, name=0x41a840 "providers", value=<optimized out>, flags=<optimized out>) at crypto/conf/conf_mod.c:272
#11 CONF_modules_load (cnf=cnf@entry=0x418c70, appname=appname@entry=0x0, flags=32, flags@entry=50) at crypto/conf/conf_mod.c:165
#12 0x00007ffff78706a8 in CONF_modules_load_file_ex (libctx=<optimized out>, filename=0x0, appname=0x0, flags=50) at crypto/conf/conf_mod.c:214
#13 0x00007ffff7927da8 in ossl_config_int (settings=0x0) at crypto/conf/conf_sap.c:68
#14 ossl_config_int (settings=0x0) at crypto/conf/conf_sap.c:44
#15 ossl_init_config () at crypto/init.c:258
#16 ossl_init_config_ossl_ () at crypto/init.c:256
#17 0x00007ffff7d6628b in __pthread_once_slow.isra.0 () from /lib64/libc.so.6
#18 0x00007ffff7d662f9 in pthread_once.5 () from /lib64/libc.so.6
#19 0x00007ffff793493d in CRYPTO_THREAD_run_once (once=<optimized out>, init=<optimized out>) at crypto/threads_pthread.c:152
#20 0x00007ffff7928883 in OPENSSL_init_crypto (opts=opts@entry=76, settings=0x0) at crypto/init.c:601
#21 0x00007ffff7ef8a8d in OPENSSL_init_ssl (opts=76, settings=<optimized out>) at ssl/ssl_init.c:115
#22 0x000000000040116d in main () at openssl-flip-test.c:7
(gdb) p out
$51 = "\"\000\250\374\006,\034\317\343\b\200x\376=t\221\227\324X\006W\242\216\371$\217\275\301h\r\206C", '\000' <repeats 31 times>

(In reply to Mamoru TASAKA from comment #8)
> ref:
> https://src.fedoraproject.org/rpms/openssl/blob/
> 3e4eea28e54d0550f621d087a90b82d7a4b57883/f/openssl.spec#_398
> LL398-406
> 
> Once asking debugedit maintainer for help.

Here is a bit of history, should it help:

https://gitlab.com/redhat/centos-stream/rpms/openssl/-/commit/5c4e10ac266bbf0fc928716eb2be3f5641d1e46c
https://gitlab.com/redhat/centos-stream/rpms/openssl/-/commit/3ff0db7558173e618711f618c1e8c6f9db8a5bfb

(In reply to Mamoru TASAKA from comment #6)
> Looks like debugedit change is related to this. (debugedit-5.0-17.fc41 ->
> 5.1-2.fc42).
> 
> openssl build "edits" .rodata1 section of fips.so , looks like this section
> changes
> after /usr/bin/find-debuginfo is executed.

For example on the following openssl-3.2.2-10.fc42 x86_64 build's build.log, I can see the following part. Did you mean the part of the "openssl build "edits" .rodata1 section of fips.so" is the part " objcopy --update-section ...", right? How did you find the debugedit package was used in this part? I don't find how the debugedit package (or command) is used in this part.

https://koji.fedoraproject.org/koji/buildinfo?buildID=2608512
https://kojipkgs.fedoraproject.org//packages/openssl/3.2.2/10.fc42/data/logs/x86_64/build.log

```
...
+ OPENSSL_CONF=/dev/null
+ LD_LIBRARY_PATH=.
+ apps/openssl dgst -binary -sha256 -mac HMAC -macopt hexkey:f4556650ac31d35461610bac4ed81b1a181b2d8a43ea2854cbae22ca74560813
+ objcopy --update-section .rodata1=/builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.hmac /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.mac
+ mv /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.mac /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so
+ rm /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.hmac
Reading /builddir/build/BUILD/openssl-3.2.2-build/SPECPARTS/rpm-debuginfo.specpart
Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.tDmB8i
...
```

(In reply to Jun Aruga from comment #11)
> (In reply to Mamoru TASAKA from comment #6)
> > Looks like debugedit change is related to this. (debugedit-5.0-17.fc41 ->
> > 5.1-2.fc42).
> > 
> > openssl build "edits" .rodata1 section of fips.so , looks like this section
> > changes
> > after /usr/bin/find-debuginfo is executed.
> 
> For example on the following openssl-3.2.2-10.fc42 x86_64 build's build.log,
> I can see the following part. Did you mean the part of the "openssl build
> "edits" .rodata1 section of fips.so" is the part " objcopy --update-section
> ...", right? 

Yes.

> How did you find the debugedit package was used in this part? I
> don't find how the debugedit package (or command) is used in this part.

Actually not that part, however:

```
+ ln -s /etc/crypto-policies/back-ends/openssl_fips.config /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/etc/pki/tls/fips_local.cnf
+ /usr/bin/find-debuginfo -j6 --strict-build-id -m -i --build-id-seed 3.2.2-10.fc42 --unique-debug-suffix -3.2.2-10.fc42.x86_64 --unique-debug-src-base openssl-3.2.2-10.fc42.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/openssl-3.2.2-build/openssl-3.2.2
find-debuginfo: starting
Extracting debug info from 9 files
DWARF-compressing 9 files
sepdebugcrcfix: Updated 9 CRC32s, 0 CRC32s did match.
Creating .debug symlinks for symlinks to ELF files
Copying sources found by 'debugedit -l' to /usr/src/debug/openssl-3.2.2-10.fc42.x86_64
find-debuginfo: done
+ /usr/lib/rpm/check-buildroot
+ /usr/lib/rpm/redhat/brp-ldconfig
+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
+ /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip
+ /usr/lib/rpm/check-rpaths
+ /usr/lib/rpm/redhat/brp-mangle-shebangs
mangling shebang in /usr/bin/make-dummy-cert from /bin/sh to #!/usr/bin/sh
mangling shebang in /usr/bin/renew-dummy-cert from /bin/bash to #!/usr/bin/bash
+ /usr/lib/rpm/brp-remove-la-files
+ env /usr/lib/rpm/redhat/brp-python-bytecompile '' 1 0 -j6
+ /usr/lib/rpm/redhat/brp-python-hardlink
+ /usr/bin/add-determinism --brp -j6 /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT
Scanned 148 directories and 7568 files,
               processed 0 inodes,
               0 modified (0 replaced + 0 rewritten),
               0 unsupported format, 0 errors
+ OPENSSL_CONF=/dev/null
+ LD_LIBRARY_PATH=.
+ apps/openssl dgst -binary -sha256 -mac HMAC -macopt hexkey:f4556650ac31d35461610bac4ed81b1a181b2d8a43ea2854cbae22ca74560813
+ objcopy --update-section .rodata1=/builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.hmac /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.mac
+ mv /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.mac /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so
+ rm /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so.hmac
```

During above, "/usr/bin/find-debuginfo" edits /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so 
(through debugedit).
The above command perhaps does not edit .rodata1 section itself, but edits some debug related sections, which seems to cause
difference.

From the maintainer perspective it looks like a toolchain issue. I'm not sure it would be easy to add the test for this issue.

The failing test ensures that the fips.so file has not been modified after the `objcopy --update-section .rodata1` invocation in the build.

If this didn't happen with debugedit-5.0-17.fc41, but is now happening with debugedit-5.1-2.fc42, something changed in debugedit that touches this file after the objcopy invocation. Either we need to find a way to override this behavior from the specfile, or we need to find a way to influence the debugedit behavior so that it leaves the file alone.

*** Bug 2336087 has been marked as a duplicate of this bug. ***

It is likely too late to revert openssl, because the change already landed in F41. What about disabling the check for now until there is proper solution? I am going to disable the affected tests in Ruby for now (we have ~week for Ruby mass rebuild prior the Fedora mass rebuild), but not sure what is the impact of bug 2336087

I cannot really follow how/when/where this extra rodata1 section is added to the fips.so file.

There is 0033-FIPS-embed-hmac.patch which seems to add the verify_integrity_rodata check.
Then in %check it seems to add and then update the fips.so rodata1 section again.
First using openssl dgst and objcopy --update-section.
It also %define __spec_install_post with a similar update. It isn't clear to be when this runs.

Are there build logs for when this apprears to work and when this doesn't work?
It the only difference the debugedit version installed?

Created attachment 2065034 [details]
Build log

Hi Mark, 
the build log is attached

You can reach me directly if you need any assistance or ask questions here

> During above, "/usr/bin/find-debuginfo" edits
> /builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/
> fips.so 
> (through debugedit).
> The above command perhaps does not edit .rodata1 section itself, but edits
> some debug related sections, which seems to cause
> difference.

Mamoru,
Thanks for your explanation. I understood it. 

> gdb shows with koji openssl-3.2.2-10.fc42 OSSL_SELF_TEST_oncorrupt_byte() called in
verify_integrity() in providers/fips/self_test.c is returning an unexpected result:

> ```
> (gdb) bt
> #0  verify_integrity_rodata.constprop.0 (bio=bio@entry=0x441180, read_ex_cb=<optimized out>, 
>     expected=expected@entry=0x7ffff77d80e0 <fips_hmac_container.lto_priv> "Z\357\314&\201\331\276\023\213\301j\017\016{^\362E3\v\002S\327_1\207\231S2\301\327\207*\001\033\003;,3", 
>     expected_len=<optimized out>, libctx=<optimized out>, ev=ev@entry=0x422af0, event_type=0x7ffff77c9350 "Module_Integrity") at providers/fips/self_test.c:353
> ```

According to your above gdb trace log, I can see the `OSSL_SELF_TEST_oncorrupt_byte()` is called in `verify_integrity_rodata()` in the `providers/fips/self_test.c`.[1] The function `verify_integrity_rodata()` is added by the patch `0033-FIPS-embed-hmac.patch`.

[1] https://src.fedoraproject.org/rpms/openssl/blob/3e4eea28e54d0550f621d087a90b82d7a4b57883/f/0033-FIPS-embed-hmac.patch#_147

could you please try https://koji.fedoraproject.org/koji/taskinfo?taskID=127629174 ?

(In reply to Clemens Lang from comment #14)
> The failing test ensures that the fips.so file has not been modified after
> the `objcopy --update-section .rodata1` invocation in the build.

Can we add this in %check section as a test of this issue? The steps are like this.

1. Back up fips.so, such as `cp -p /path/to/fips.so /path/to/fips_bak.so`.
2. Run `objcopy --update-section .rodata1 ..` for `fips.so`.
3. Compare the `fips.so` from the `fips_bak.so` such as `diff /path/to/fips_bak.so /path/to/fips.so`. If there is no difference, abort as an error case.

(In reply to Dmitry Belyavskiy from comment #21)
> could you please try
> https://koji.fedoraproject.org/koji/taskinfo?taskID=127629174 ?

I have tried to update openssl-libs and just rerun the Ruby tests (without rebuild or what not) and they keeps failing. I hope I have not messed it somehow.

(In reply to Dmitry Belyavskiy from comment #21)
> could you please try
> https://koji.fedoraproject.org/koji/taskinfo?taskID=127629174 ?

I tried it too. Unfortunately, I still see the issue with the scratch build openssl-3.2.2-11.fc41.

```
$ cat /etc/fedora-release
Fedora release 41 (Forty One)

$ rpm -qa | grep openssl
openssl-libs-3.2.2-11.fc41.x86_64
openssl-3.2.2-11.fc41.x86_64
openssl-devel-3.2.2-11.fc41.x86_64

$ rpm -q debugedit
debugedit-5.1-2.fc41.x86_64

$ gcc -lssl -o openssl-flip-test openssl-flip-test.c

$ ./openssl-flip-test
Aborted (core dumped)
```

(In reply to Dmitry Belyavskiy from comment #19)
> the build log is attached
> 
> You can reach me directly if you need any assistance or ask questions here

That give me: Sorry, you are not authorized to access attachment #2065034 [details].

Was that log against a specific version of the openssl.spec file?
Against which versions of which packages was it run?

https://kojipkgs.fedoraproject.org//packages/openssl/3.2.2/10.fc41/data/logs/x86_64/build.log - this is the build log for 3.2.2-10 (problematic)
https://kojipkgs.fedoraproject.org//packages/openssl/3.2.2/9.fc41/data/logs/x86_64/build.log - this is the build log for 3.2.2-9 (known good)

(In reply to Dmitry Belyavskiy from comment #26)
> https://kojipkgs.fedoraproject.org//packages/openssl/3.2.2/10.fc41/data/logs/
> x86_64/build.log - this is the build log for 3.2.2-10 (problematic)
> https://kojipkgs.fedoraproject.org//packages/openssl/3.2.2/9.fc41/data/logs/
> x86_64/build.log - this is the build log for 3.2.2-9 (known good)

OK, so the difference is the 0140-prov_no-cache.patch and possibly a change in buildroot package versions?
But there is no testing for this, except some other package (ruby) failing some tests after a rebuild?

I don't see any immediate issue in the build.logs in both you update the extra rodata1 section before and after running debugedit find-debuginfo.sh

Exactly. The Patch140 is the only addition.
The construction designed for embedding MAC into the fips.so looks fishy to me because I had to copy the original macros but it has been working for ~2 years now.

OK, replicated with the test program from comment #2.

Against openssl-3.2.2-9.fc41.x86_64 this succeeds.
Against openssl-3.2.2-10.fc41.x86_64 it aborts.

(In reply to Vít Ondruch from comment #16)
> It is likely too late to revert openssl, because the change already landed
> in F41. What about disabling the check for now until there is proper
> solution?

This check is required by FIPS guidelines. I'm assuming users that run Fedora in FIPS mode *want* those checks to be enabled. I don't think we should disable them for that reason.


(In reply to Jun Aruga from comment #22)
> (In reply to Clemens Lang from comment #14)
> > The failing test ensures that the fips.so file has not been modified after
> > the `objcopy --update-section .rodata1` invocation in the build.
> 
> Can we add this in %check section as a test of this issue? The steps are
> like this.
> 
> 1. Back up fips.so, such as `cp -p /path/to/fips.so /path/to/fips_bak.so`.
> 2. Run `objcopy --update-section .rodata1 ..` for `fips.so`.
> 3. Compare the `fips.so` from the `fips_bak.so` such as `diff
> /path/to/fips_bak.so /path/to/fips.so`. If there is no difference, abort as
> an error case.

The issue here is that for this test to be useful, it would have to run *after* debug symbol stripping, i.e., when we're sure that the `fips.so` file is no longer being modified.

The openssl spec file does the `objcopy --update-section .rodata1` dance in two places because of this: Once in %check before executing the test suite, where debug symbols are still in the library, so that the tests pass with fips.so with debug symbols, and once after debug symbols have been separated out into a different file, so that the HMAC in the file is correct for the installed file without the debug symbols. It seems now something modifies the binary even after that step, which cases the checksum to not match.

We can't catch that in %check, because to my knowledge there is no phase that we could hook after all debug symbol stripping and packaging has finished.

(In reply to Clemens Lang from comment #30)
> (In reply to Vít Ondruch from comment #16)
> > It is likely too late to revert openssl, because the change already landed
> > in F41. What about disabling the check for now until there is proper
> > solution?
> 
> This check is required by FIPS guidelines. I'm assuming users that run
> Fedora in FIPS mode *want* those checks to be enabled. I don't think we
> should disable them for that reason.

I was proposing disable this *temporary*.

* The very same users probably also want to use their systems, which are likely broken for them ATM. Some of them will be tech savvy enough to revert to older version. * And I was forced to disable the FIPS test cases, so we we are in danger that the FIPS in Ruby will get broken in the future without us noticing.

> The issue here is that for this test to be useful, it would have to run
> *after* debug symbol stripping, i.e., when we're sure that the `fips.so`
> file is no longer being modified.
> 
> The openssl spec file does the `objcopy --update-section .rodata1` dance in
> two places because of this: Once in %check before executing the test suite,
> where debug symbols are still in the library, so that the tests pass with
> fips.so with debug symbols, and once after debug symbols have been separated
> out into a different file, so that the HMAC in the file is correct for the
> installed file without the debug symbols. It seems now something modifies
> the binary even after that step, which cases the checksum to not match.
> 
> We can't catch that in %check, because to my knowledge there is no phase
> that we could hook after all debug symbol stripping and packaging has
> finished.

Okay. How about adding a test for this issue into tests/openssl repository?
We can use the reproducing code mentioned at <https://bugzilla.redhat.com/show_bug.cgi?id=2335414#c2>.

I'm going to investigate the issue and to check whether we can do anything with it on our side or we need some RPM/toolchain people.

What I see now is that the checksum IS NOT modified after calculation (presuming there is nothing that doesn't go to the build log).

https://koji.fedoraproject.org/koji/taskinfo?taskID=127670110 is the scratch build for f41
https://koji.fedoraproject.org/koji/taskinfo?taskID=127670308 is the scratch build for rawhide

The build contains file /usr/lib64/ossl-modules/fips.sha256 which is a build-time sha256 sum _after_ embedding HMAC into the FIPS module.
Jan, could you please check that the fips.so file is not changed after the installation?

I found the workaround, the issue is caused by fedpkg behavior change. Stay tuned.

(In reply to Dmitry Belyavskiy from comment #36)
> What I see now is that the checksum IS NOT modified after calculation
> (presuming there is nothing that doesn't go to the build log).
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=127670110 is the scratch
> build for f41
> https://koji.fedoraproject.org/koji/taskinfo?taskID=127670308 is the scratch
> build for rawhide
> 
> The build contains file /usr/lib64/ossl-modules/fips.sha256 which is a
> build-time sha256 sum _after_ embedding HMAC into the FIPS module.
> Jan, could you please check that the fips.so file is not changed after the
> installation?

Sure. I checked the fips.so file's sha256 value is the same with the fips.sha256 file's value in both your above f41 and rawhide scratch builds. Below is the command log.

On f41:

```
$ rpm -qa | grep openssl
openssl-libs-3.2.2-11.fc41.x86_64

$ rpm -ql openssl-libs | grep 'fips\.'
/usr/lib64/ossl-modules/fips.sha256
/usr/lib64/ossl-modules/fips.so

$ cat /usr/lib64/ossl-modules/fips.sha256
SHA2-256(/builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so)= d9e0e4f4ff4d1dab44c41ba1241a46154475b4d8d02d804aebd29f34231235c1

$ sha256sum /usr/lib64/ossl-modules/fips.so
d9e0e4f4ff4d1dab44c41ba1241a46154475b4d8d02d804aebd29f34231235c1  /usr/lib64/ossl-modules/fips.so

$ cat /usr/lib64/ossl-modules/fips.sha256 | cut -d' ' -f 2 > a.txt

$ sha256sum /usr/lib64/ossl-modules/fips.so | cut -d' ' -f 1 > b.txt

$ diff a.txt b.txt

$ echo $?
0
```

On rawhide:

```
$ rpm -qa | grep openssl
openssl-libs-3.2.2-11.fc42.x86_64

$ rpm -ql openssl-libs | grep 'fips\.'
/usr/lib64/ossl-modules/fips.sha256
/usr/lib64/ossl-modules/fips.so

$ cat /usr/lib64/ossl-modules/fips.sha256
SHA2-256(/builddir/build/BUILD/openssl-3.2.2-build/BUILDROOT/usr/lib64/ossl-modules/fips.so)= ee24c317d1a6cf0cb72f04f4a6d538220a173ae403b7c969809a1eb89f99685a

$ sha256sum /usr/lib64/ossl-modules/fips.so
ee24c317d1a6cf0cb72f04f4a6d538220a173ae403b7c969809a1eb89f99685a  /usr/lib64/ossl-modules/fips.so

$ cat /usr/lib64/ossl-modules/fips.sha256 | cut -d' ' -f 2 > a.txt

$ sha256sum /usr/lib64/ossl-modules/fips.so | cut -d' ' -f 1 > b.txt

$ diff a.txt b.txt

$ echo $?
```

Sorry, I missed pasting the last line of `0`.

> $ echo $?
> ```

$ echo $?
0
```

FEDORA-2025-8c4c41cb73 (openssl-3.2.2-11.fc42) has been submitted as an update to Fedora 42.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-8c4c41cb73

FEDORA-2025-be403a5054 (openssl-3.2.2-11.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-be403a5054

Jun,  Xiaofeng could you please test the build https://bodhi.fedoraproject.org/updates/FEDORA-2025-be403a5054 and vote for it if it works?

FEDORA-2025-8c4c41cb73 (openssl-3.2.2-11.fc42) has been pushed to the Fedora 42 stable repository.
If problem still persists, please make note of it in this bug report.

(In reply to Dmitry Belyavskiy from comment #42)
> Jun,  Xiaofeng could you please test the build
> https://bodhi.fedoraproject.org/updates/FEDORA-2025-be403a5054 and vote for
> it if it works?

It helps it seems:

https://koji.fedoraproject.org/koji/taskinfo?taskID=127682326

FEDORA-2025-be403a5054 has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-be403a5054`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-be403a5054

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

(In reply to Dmitry Belyavskiy from comment #42)
> Jun,  Xiaofeng could you please test the build
> https://bodhi.fedoraproject.org/updates/FEDORA-2025-be403a5054 and vote for
> it if it works?

Sure. I tested the following f41 build on Bodhi and also rawhide build. I confirmed that both builds worked by testing with the above simple reproducing code and running the rpms/ruby's FIPS tests using OPENSSL_FORCE_FIPS_MODE.

* f41: openssl-3.2.2-11.fc41
  * https://bodhi.fedoraproject.org/updates/FEDORA-2025-be403a5054
  * https://src.fedoraproject.org/rpms/openssl/c/133078051ec375bb970dca1e25fe79c814e4724f?branch=f41
* rawhide: openssl-3.2.2-11.fc42
  * https://src.fedoraproject.org/rpms/openssl/c/10959c52d0bb56c9423ac574c08234321ca80a8d?branch=rawhide

Below is the log.

## On f41:

### With the reproducing code

```
$ rpm -qa | grep openssl
openssl-libs-3.2.2-11.fc41.x86_64
openssl-devel-3.2.2-11.fc41.x86_64

$ gcc -lssl -o openssl-flip-test openssl-flip-test.c

$ ./openssl-flip-test

$ echo $?
0
```

### With rpms/ruby's FIPS tests

I tested with f41 mock environment. First installed the openssl-libs and openssl-devel from Bodhi.

```
# rpm -qa | grep openssl
openssl-libs-3.2.2-11.fc41.x86_64
openssl-devel-3.2.2-11.fc41.x86_64
```

Then executed the following mock command for ruby SRPM on rpms/ruby f41 branch.

```
$ mock -r fedora-41-x86_64 -n *.rpm
...
+ make -C redhat-linux-build runruby 'TESTRUN_SCRIPT=   -I/builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/tool/lib --enable-gems   /builddir/build/SOURCES/test_openssl_fips.rb /builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6 --verbose'
make: Entering directory '/builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/redhat-linux-build'
RUBY_ON_BUG='gdb -x /builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/.gdbinit -p' ./miniruby -I/builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/lib -I. -I.ext/common  /builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/tool/runruby.rb --extout=.ext  -- --disable-gems  -I/builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/tool/lib --enable-gems   /builddir/build/SOURCES/test_openssl_fips.rb /builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6 --verbose-
INFO: Loading /builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/test/openssl/test_fips.rb.
INFO: Loading /builddir/build/BUILD/ruby-3.3.6-build/ruby-3.3.6/test/openssl/test_pkey.rb.
Run options:-
  --seed=65249
  --verbose
# Running tests:
[ 1/13] OpenSSL::TestPKey#test_s_generate_parameters = 0.00 s
[ 2/13] OpenSSL::TestPKey#test_hmac_sign_verify = 0.00 s
[ 3/13] OpenSSL::TestPKey#test_to_text = 0.00 s
[ 4/13] OpenSSL::TestPKey#test_x25519 = 0.00 s
[ 5/13] OpenSSL::TestPKey#test_ed25519 = 0.00 s
[ 6/13] OpenSSL::TestPKey#test_compare? = 0.00 s
[ 7/13] OpenSSL::TestPKey#test_ed25519_not_approved_on_fips = 0.00 s
[ 8/13] OpenSSL::TestPKey#test_s_generate_key = 0.00 s
[ 9/13] OpenSSL::TestPKey#test_generic_oid_inspect = 0.00 s
[10/13] OpenSSL::TestFIPS#test_fips_mode_get_with_fips_mode_set = 0.14 s
[11/13] OpenSSL::TestFIPS#test_fips_mode_get_is_false_on_fips_mode_disabled = 0.00 s
[12/13] OpenSSL::TestFIPS#test_fips_mode_is_reentrant = 0.12 s
[13/13] OpenSSL::TestFIPS#test_fips_mode_get_is_true_on_fips_mode_enabled = 0.13 s
Finished tests in 0.410902s, 31.6377 tests/s, 180.0917 assertions/s.
13 tests, 74 assertions, 0 failures, 0 errors, 2 skips
...
```

## On rawhide:

### With the reproducing code

```
$ rpm -qa | grep openssl
openssl-libs-3.2.2-11.fc42.x86_64
openssl-devel-3.2.2-11.fc42.x86_64

$ gcc -lssl -o openssl-flip-test openssl-flip-test.c

$ ./openssl-flip-test

$ echo $?
0
```

### With rpms/ruby's FIPS tests

I executed ruby's scratch build [1] for ruby-3.4.1-20.fc42 including the commit[2] enabling FIPS tests again. According to the x86_64 build.log,[3] the FIPS tests are passing.

```
+ make -C redhat-linux-build runruby 'TESTRUN_SCRIPT=   -I/builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/tool/lib --enable-gems   /builddir/build/SOURCES/test_openssl_fips.rb /builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1 --verbose'
make: Entering directory '/builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/redhat-linux-build'
RUBY_ON_BUG='gdb -x /builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/.gdbinit -p' ./miniruby -I/builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/lib -I. -I.ext/common  /builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/tool/runruby.rb --extout=.ext  -- --disable-gems  -I/builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/tool/lib --enable-gems   /builddir/build/SOURCES/test_openssl_fips.rb /builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1 --verbose 
INFO: Loading /builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/test/openssl/test_fips.rb.
INFO: Loading /builddir/build/BUILD/ruby-3.4.1-build/ruby-3.4.1/test/openssl/test_pkey.rb.
Run options: 
  --seed=15072
  --verbose
# Running tests:
[ 1/13] OpenSSL::TestPKey#test_s_generate_parameters = 0.00 s
[ 2/13] OpenSSL::TestPKey#test_hmac_sign_verify = 0.00 s
[ 3/13] OpenSSL::TestPKey#test_to_text = 0.00 s
[ 4/13] OpenSSL::TestPKey#test_s_generate_parameters_with_block = 0.00 s
[ 5/13] OpenSSL::TestPKey#test_ed25519 = 0.00 s
[ 6/13] OpenSSL::TestPKey#test_x25519 = 0.00 s
[ 7/13] OpenSSL::TestPKey#test_compare? = 0.01 s
[ 8/13] OpenSSL::TestPKey#test_generic_oid_inspect = 0.01 s
[ 9/13] OpenSSL::TestPKey#test_s_generate_key = 0.00 s
[10/13] OpenSSL::TestFIPS#test_fips_mode_get_with_fips_mode_set = 0.21 s
[11/13] OpenSSL::TestFIPS#test_fips_mode_get_is_false_on_fips_mode_disabled = 0.00 s
[12/13] OpenSSL::TestFIPS#test_fips_mode_is_reentrant = 0.20 s
[13/13] OpenSSL::TestFIPS#test_fips_mode_get_is_true_on_fips_mode_enabled = 0.21 s
Finished tests in 0.668422s, 19.4488 tests/s, 110.7085 assertions/s.
13 tests, 74 assertions, 0 failures, 0 errors, 1 skips
```

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=127708045
[2] https://src.fedoraproject.org/rpms/ruby/c/e3690ac430df05803f10132080d180e5ee43f1b3?branch=rawhide
[3] x86_64 build.log https://kojipkgs.fedoraproject.org//work/tasks/8120/127708120/build.log

FEDORA-2025-be403a5054 (openssl-3.2.2-11.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.