Description of problem: Apptainer is compiled without FIPS support, even though it was compiled with it before Version-Release number of selected component (if applicable): apptainer-1.4.5-2.el8.x86_64 How reproducible: Always Build Date & Hardware: see rpm -qi output below 0. Ensure that apptainer-1.4.5-2.el8.x86_64 is installed on the system 1. Run apptainer command, get openssl error 2. Disable FIPS, reboot 3. Re-run same command, get a successful result 4. Compile and install an installation of apptainer that supports FIPS, use that to run apptainer successfully Steps to Reproduce: 0. Ensure that apptainer-1.4.5-2.el8.x86_64 is installed on the system, and that FIPS is enabled. $ rpm -qa | grep apptainer apptainer-1.4.5-2.el8.x86_64 apptainer-suid-1.4.5-2.el8.x86_64 $ yum search apptainer Not root, Subscription Management repositories not updated <snip> ==================================================================================================== Name Exactly Matched: apptainer ==================================================================================================== apptainer.x86_64 : Application and environment virtualization formerly known as Singularity =================================================================================================== Name & Summary Matched: apptainer =================================================================================================== apptainer-suid.x86_64 : Setuid component of Apptainer $ which apptainer /usr/bin/apptainer $ rpm -ql apptainer| grep bin /usr/bin/apptainer /usr/bin/run-singularity /usr/bin/singularity /usr/libexec/apptainer/bin /usr/libexec/apptainer/bin/fuse-overlayfs /usr/libexec/apptainer/bin/fuse2fs /usr/libexec/apptainer/bin/gocryptfs /usr/libexec/apptainer/bin/mksquashfs /usr/libexec/apptainer/bin/squashfuse_ll /usr/libexec/apptainer/bin/starter /usr/libexec/apptainer/bin/unsquashfs $ rpm -qi apptainer Name : apptainer Version : 1.4.5 Release : 2.el8 Architecture: x86_64 Install Date: Mon 26 Jan 2026 12:03:04 PM PST Group : Unspecified Size : 149398038 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : RSA/SHA256, Thu 04 Dec 2025 12:11:53 PM PST, Key ID 21ea45ab2f86d6a1 Source RPM : apptainer-1.4.5-2.el8.src.rpm Build Date : Thu 04 Dec 2025 10:20:28 AM PST Build Host : buildvm-x86-28.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Application and environment virtualization formerly known as Singularity Description : Apptainer provides functionality to make portable containers that can be used across host environments. $ sudo fips-mode-setup --check FIPS mode is enabled. 1. Run apptainer command, get openssl error $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version 2. Disable FIPS, reboot $ sudo fips-mode-setup --disable Setting system policy to DEFAULT Note: System-wide crypto policies are applied on application start-up. It is recommended to restart the system for the change of policies to fully take place. FIPS mode will be disabled. Please reboot the system for the setting to take effect. $ sudo reboot 3. Re-run same command, get a successful result $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine:latest cat /etc/alpine-release INFO: Using cached SIF image INFO: squashfuse not found, will not be able to mount SIF or other squashfs files INFO: gocryptfs not found, will not be able to use gocryptfs INFO: Converting SIF file to temporary sandbox... 3.23.2 INFO: Cleaning up image... 4. Compile and install an installation of apptainer that supports FIPS, use that to run apptainer successfully make-apptainer.sh 711 B #!/bin/bash # Script to compile apptainer with FIPS support. Tested in nonprod. # Compiles apptainer compiled with FIPS support in /usr/local/apptainer. # Based on comment from jaggyalex at https://github.com/apptainer/apptainer/issues/2893 export GOVERSION=1.25.5 OS=linux ARCH=amd64 # change this as you need wget https://dl.google.com/go/go${GOVERSION}.${OS}-${ARCH}.tar.gz tar xzvf go${GOVERSION}.${OS}-${ARCH}.tar.gz ./go/bin/go tool nm ./go/bin/go | grep FIPS export PATH=$HOME/go/bin:$PATH export TMPDIR=$HOME/tmp mkdir $TMPDIR which go go --version go version git clone https://github.com/apptainer/apptainer.git cd apptainer git checkout v1.4.5 ./mconfig -p /usr/local/apptainer cd builddir/ make install-apptainer.sh 584 B #!/bin/bash # Script to install apptainer with FIPS support. Tested on nonprod. Prerequisite is to run make-apptainer.sh first. # Needs to be run as admin user, i.e. with sudo access. # Installs apptainer compiled with FIPS support in /usr/local/apptainer. # Based on comment from jaggyalex at https://github.com/apptainer/apptainer/issues/2893 cd $HOME/apptainer/builddir sudo make install sudo find /usr/local/apptainer -type f -exec chmod 0644 {} \; sudo find /usr/local/apptainer -type f -path '*/bin/*' -exec chmod u+x,g+x,o+x {} \; /usr/local/apptainer/bin/apptainer version Use that to confirm that the version compiled with FIPS support runs as expected: $ /usr/local/apptainer/bin/apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release INFO: Using cached SIF image INFO: squashfuse not found, will not be able to mount SIF or other squashfs files INFO: gocryptfs not found, will not be able to use gocryptfs INFO: Converting SIF file to temporary sandbox... 3.23.2 INFO: Cleaning up image... Actual results: $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version Expected results: $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine:latest cat /etc/alpine-release INFO: Using cached SIF image INFO: squashfuse not found, will not be able to mount SIF or other squashfs files INFO: gocryptfs not found, will not be able to use gocryptfs INFO: Converting SIF file to temporary sandbox... 3.23.2 INFO: Cleaning up image... Additional info: Please update the latest version of apptainer available on EPEL to be compiled with FIPS. Note that this problem started on 1/12/26, when we upgraded from apptainer 1.4.5-1. As this command was working fine before, I believe this previous version was compiled with FIPS support. While I can't prove this since the previous versions of the RPM are not available on EPEL, all the information here supports this conclusion. Given that this RPM was compiled with FIPS support before, I'd like it updated to include FIPS support again.
Are you sure that you previously were using an apptainer from EPEL, and not one that was locally compiled? The way that the EPEL build is done has not changed. I wonder if it is required to have FIPS mode enabled when it is compiled in order for it to be enabled by default. According to https://go.dev/blog/fips140, FIPS support can be enabled at run time with a fips140=on GODEBUG option. I think maybe Red Hat used to modify their golang version for FIPS support. Now apparently it is supported by default so maybe they don't do that anymore, but maybe it needs to be enabled somehow.
Are you sure that you previously were using an apptainer from EPEL, and not one that was locally compiled? ================================================================================================================= Yes, I'm sure of that. Not only do we not normally locally recompile code from source RPMs, but my colleague did just that on a test system. I compare the results of rpm -qi on the two, and get distinctly different results on the test system where it was recompiled locally versus where it wasn't, i.e. the RPM from EPEL is installed. The host where it wasn't recompiled was where we are seeing the problem. The diff of the two results is below; the rpm -qi output from the problem system is what I posted earlier in this ticket, if you want to reference that. I've removed the name of the internal host where my colleague had recompiled, but again, they are different. Not only are they different, but the build host specifically says fedoraproject.org, which is definitely not one of our systems, meaning that it was not compiled locally. This same build host is mentioned earlier in the rpm -qi output I provided earlier. $ diff rpm-qi_not_recompiled.txt rpm-qi_is_recompiled.txt 3c3 < Release : 2.el8 --- > Release : 3.el8 5c5 < Install Date: Mon 12 Jan 2026 06:34:50 PM PST --- > Install Date: Thu 05 Feb 2026 03:38:29 PM PST 7c7 < Size : 149398038 --- > Size : 149372666 9,12c9,12 < Signature : RSA/SHA256, Thu 04 Dec 2025 12:11:53 PM PST, Key ID 21ea45ab2f86d6a1 < Source RPM : apptainer-1.4.5-2.el8.src.rpm < Build Date : Thu 04 Dec 2025 10:20:28 AM PST < Build Host : buildvm-x86-28.rdu3.fedoraproject.org --- > Signature : (none) > Source RPM : apptainer-1.4.5-3.el8.src.rpm > Build Date : Thu 05 Feb 2026 03:35:38 PM PST > Build Host : hostname-of-internal-server-removed 14,15d13 < Packager : Fedora Project < Vendor : Fedora Project 17d14 < Bug URL : https://bugz.fedoraproject.org/apptainer The way that the EPEL build is done has not changed. I wonder if it is required to have FIPS mode enabled when it is compiled in order for it to be enabled by default. ================================================================================================================= Dave, were you able to establish a baseline and recreate the error on a RHEL 8.10 system with apptainer from EPEL installed? I'm sure that the build process has not changed, but I wonder if an underlying component, e.g. go, had changed. Either way, the end result I see is the same. We had this working before 1/12/26, and after we patched on 1/12/26, the same command broke with the SSL error, and it's fixed by disabling FIPS. As I can only temporarily disable FIPS for troubleshooting per corporate policy, disabling it is not an option. I think from what I provided earlier, it seems to me that it's because the new RPM doesn't have FIPS support, i.e. it's not compiled in. That said, I'd think the EPEL build hosts may not have FIPS enabled. So, to test what you're saying, I did the following. I expected to get the same OpenSSL error, i.e. "panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version", and I do. This would seem to point to that, if FIPS enabled on the system, the code compiled would support FIPS, and if disabled, it would not support it. 1. Disabled FIPS on my test system and rebooted (commands I provided earlier) 2. Deleted the locally compiled (i.e. not from the source RPM, but in /usr/local/apptainer, per the scripts earlier) 3. Ran the two shell scripts I posted earlier in this ticket to install 4. Re-enabled FIPS on my test system and rebooted 5. Ran the same apptainer command I provided earlier Again, I get the same OpenSSL error I got earlier. Given this, It seems it does matter if FIPS is enabled on the build host or not. According to https://go.dev/blog/fips140, FIPS support can be enabled at run time with a fips140=on GODEBUG option. ================================================================================================================= Given that the scripts I posted earlier in this ticket don't have fips140=on, yet the compiled result resulted in a successful run of the apptainer command, I'm thinking that isn't necessary. But I also have FIPS enabled. Sure, so to test this idea, I tried a slight variation on the steps above, adding in #3 for the fips140 option: 1. Disabled FIPS on my test system and rebooted (commands I provided earlier) 2. Deleted the locally compiled (i.e. not from the source RPM, but in /usr/local/apptainer, per the scripts earlier) 3. Modified the shell script to use the fips140=on GODEBUG option, i.e. added the following lines: # Enable FIPS 140-2 support in Go's crypto packages at runtime export GODEBUG=fips140=on 4. Ran the two shell scripts I posted earlier in this ticket to install 5. Re-enabled FIPS on my test system and rebooted 5. Ran the same apptainer command I provided earlier So, in summary: 1. If FIPS is enabled on the build host, the GODEBUG variable fips140 needs to be set to on. 2. If FIPS is not enabled on the host, the GODEBUG variable fips140 does not need to be set. 3. Run the two scripts I provided. Can we make sure this functionality is in the build process, whether it's changed or not? I think maybe Red Hat used to modify their golang version for FIPS support. Now apparently it is supported by default so maybe they don't do that anymore, but maybe it needs to be enabled somehow. ================================================================================================================= I'm not understanding what you're saying here. Yes, from what I can tell, it is enabled by support as far as I can tell, i.e. the original version of my script without fips140=on installs a version of apptainer that has FIPS support, i.e. where I can run my apptainer command with no problems. If it's supported by default, why would it have to be enabled somehow?
Created attachment 2128499 [details] Script to compile code for apptainer
Created attachment 2128500 [details] Script to install apptainer code that's been compiled
> Dave, were you able to establish a baseline and recreate the error on a RHEL 8.10 system with apptainer from EPEL installed? No, I don't have any FIPS-enabled systems, and I don't want to risk turning it on in any of my existing systems. I need to rely on you to do that testing. > So, in summary: > 1. If FIPS is enabled on the build host, the GODEBUG variable fips140 needs to be set to on. > 2. If FIPS is not enabled on the host, the GODEBUG variable fips140 does not need to be set. That's very confusing. It seems opposite to what I would expect in order to have FIPS enabled by default. I don't control the EPEL build host and it does not have FIPS enabled, so if having FIPS enabled at build time is required in order to produce a FIPS-capable binary, there's nothing I can do. >> I think maybe Red Hat used to modify their golang version for FIPS support. Now apparently it is supported by default so maybe they don't do that anymore, but maybe it needs to be enabled somehow. > I'm not understanding what you're saying here. Yes, from what I can tell, it is enabled by support as far as I can tell, i.e. the original version of my script without fips140=on installs a version of apptainer that has FIPS support, i.e. where I can run my apptainer command with no problems. If it's supported by default, why would it have to be enabled somehow? I'm thinking that FIPS support might now already be in the compiled apptainer in EPEL, but turned off and only enabled when GODEBUG=fips140=on is set at runtime. If I'm understanding what you wrote, I don't think you've tried yet what I think might be needed. Try using the existing EPEL install but doing "export GODEBUG=fips140=on" before running apptainer.
No, I don't have any FIPS-enabled systems, and I don't want to risk turning it on in any of my existing systems. I need to rely on you to do that testing. ============================================= OK, I hope I've provided enough information here from my testing. My apologies, I had the points in the summary reversed. The conclusion is as follows: ===> 1. If FIPS is enabled on the build host, the GODEBUG variable fips140 does not need to be set. ===> 2. If FIPS is not enabled on the host, the GODEBUG variable fips140 needs to be set to on. In other words, there are no changes for FIPS required from an OS perspective on the build host, as I think currently #1 applies. If I'm understanding your last message correctly, this is already the case, i.e. I think that's what you mean by "now already be in the compiled apptainer in EPEL, but turned off and only enabled when GODEBUG=fips140=on is set at runtime". In that case, I'm not sure why I'm seeing the OpenSSL error with FIPS enabled, and it goes away if I disable it. Try using the existing EPEL install but doing "export GODEBUG=fips140=on" before running apptainer. ================================================ Is this what you mean? I try it and get the same thing. I thought the GODEBUG variable only applies during compilation. $ date; export GODEBUG=fips140=on Mon Feb 9 08:21:20 PST 2026 $ date; apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release Mon Feb 9 08:22:02 PST 2026 INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version
Ok so the answers are pretty definitive now. It appears that if I set GODEBUG=fips140=on in the rpm %build step it should do what you want. Are we sure that it will cause no harm on systems that do not have FIPS enabled? I just wonder why it is not the default. I'm guessing that it used to be on by default in the RedHat-supplied golang but now it's instead in the standard golang but needs to be enabled.
Ok so the answers are pretty definitive now. It appears that if I set GODEBUG=fips140=on in the rpm %build step it should do what you want. =========================================================================== Super! Do you have any estimate on when EPEL can be updated? I do have a workaround now, but it would be nice to have an estimate on when it would be updated, so I don't have to keep bugging you :) Are we sure that it will cause no harm on systems that do not have FIPS enabled? =========================================================================== I really can't see why it would. This is just setting a value for compilation; a value that seems to have been enabled before 1/12/26. Based on that, I wouldn't think there would be any adverse impact. That said, I'm not the expert on the code. I just wonder why it is not the default. =========================================================================== My guess is it changed for the latest build, for some reason. I'm guessing that it used to be on by default in the RedHat-supplied golang but now it's instead in the standard golang but needs to be enabled. =========================================================================== 1. So any reason why it wouldn't on by default in the Redhat supplied golang? It would seem that it did indeed change, but any thoughts on why? 2. The way I'm understanding this, it need to be enabled by setting the GODEBUG variable when apptainer is compiled if FIPS on the build host is not enabled. But yes, since your environment doesn't have FIPS enabled, that would apply, making the statement true. Basically, I'm confirming what you're saying. Of course, I don't understand why all this is, as I'm not familiar with the build process, etc.
I could get it built pretty soon. For now I did a scratch el8 build at https://koji.fedoraproject.org/koji/taskinfo?taskID=142200219. Please test the rpm there with and without FIPS enabled. I'm guessing that the change happened because Redhat convinced golang to include the support code when enabled, and then Redhat wanted to go back to having a standard golang install instead of a modified one. I would think that Fedora/EPEL could set the environment variable to enable it by default on their build system. Maybe Redhat does that on their internal build system.
So that test rpm is just to establish a baseline, right? I get the expected behavior that it fails with FIPS enabled, and works if disabled. This is the same as the behavior with the current stable RPM. I install the package and run the command with FIPS enabled, get the expected failure: $ sudo rpm -Uvh *.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:apptainer-1.4.5-2.1.el8 ################################# [ 50%] 2:apptainer-suid-1.4.5-2.1.el8 ################################# [100%] $ sudo fips-mode-setup --check FIPS mode is enabled. $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version $ rpm -qi apptainer apptainer-suid Name : apptainer Version : 1.4.5 Release : 2.1.el8 Architecture: x86_64 Install Date: Wed 11 Feb 2026 12:19:15 PM PST Group : Unspecified Size : 149402438 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : (none) Source RPM : apptainer-1.4.5-2.1.el8.src.rpm Build Date : Wed 11 Feb 2026 11:34:58 AM PST Build Host : buildvm-x86-26.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Application and environment virtualization formerly known as Singularity Description : Apptainer provides functionality to make portable containers that can be used across host environments. Name : apptainer-suid Version : 1.4.5 Release : 2.1.el8 Architecture: x86_64 Install Date: Wed 11 Feb 2026 12:19:16 PM PST Group : Unspecified Size : 25093600 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : (none) Source RPM : apptainer-1.4.5-2.1.el8.src.rpm Build Date : Wed 11 Feb 2026 11:34:58 AM PST Build Host : buildvm-x86-26.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Setuid component of Apptainer Description : Provides the optional setuid-root portion of Apptainer. With FIPS disabled, it works: $ fips-mode-setup --check FIPS mode is disabled. $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release INFO: Using cached SIF image 3.23.3
No, I was expecting that to fix the problem. It does "export GODEBUG=fips140=on" before building apptainer. You can see it being set if you click on the build.log from that koji page. I don't know what else to try. The src rpm can be found if you click from that koji page to its parent and from there down to rebuildSRPM. It has the modified spec file in it. If you want to experiment with it you can do that with the rpmbuild command, and/or extract the spec file with "rpm2cpio apptainer-1.4.5-2.1.el8.src.rpm |cpio -icv '*.spec'".
Created attachment 2129146 [details] Script to download source RPM, recompile for FIPS support and create a new RPM.
Created attachment 2129147 [details] Install RPM that's been compiled with FIPS support
@dwd - I just uploaded the updated scripts I used to recompile the source RPMS and install. Perhaps you can compare with what you have to see where mine works, i.e. what's missing from the build process?
It looks like your make-apptainer.sh not only sets the GODEBUG flag inside the spec file but it also sets it in the environment of the rpmbuild command. I don't have that option. Can you figure out how to do it in the spec file alone? You also set it in your %build step before the mconfig. I'll try that but I'm not hopeful that it will make a difference. Oh, it will affect the gocryptfs bundled package, but I don't think you're using that. Do I need to set it also on the %install step? I don't think that's doing any compiling, it's just moving files around.
Created attachment 2129415 [details] Install RPM that's been compiled with FIPS support - updated to lock packages, ensure both packages installed
For your first question, I think you're referring to this line in make-apptainer.sh: # Enable FIPS 140-2 support in Go’s crypto packages export GODEBUG=fips140=on I commented that out, ran the script to make the rpm, and installed it; the test ran fine and I was able to execute the run command noted in this ticket. So I don't think that environment variable is needed. Yes, I don't think I'm using gocryptofs; this is what I see on the build host: $ rpm -qa | grep gocryptofs | wc -l 0 $ yum search gocryptofs Not root, Subscription Management repositories not updated Last metadata expiration check: 3 days, 5:28:34 ago on Tue 10 Feb 2026 11:07:23 AM PST. No matches found. As far as I know, I'm not doing anything in the %install step. Are you able to run the make-apptainer.sh script? Do you see any errors?
Try the rpm from https://koji.fedoraproject.org/koji/taskinfo?taskID=142306511. I moved the GODEBUG setting earlier in %build. I don't see the value to me running your script, because we need it to work when built on the koji system and I would need you to test the result anyway.
@dwd OK, thanks. That worked. See output below, the end of which shows apptainer running the most recent version, i.e. from the new RPM, as well as the expected version number from the apptainer container. How long do you think it will take to get the public EPEL repository updated? [user@hostname ~]$ ./install-apptainer.sh Searching for the built binary RPM... Found RPMS /home/user/rpmbuild/RPMS/x86_64/apptainer-1.4.5-2.2.el8.x86_64.rpm /home/user/rpmbuild/RPMS/x86_64/apptainer-suid-1.4.5-2.2.el8.x86_64.rpm Removing currently installed apptainer ... Updating Subscription Management repositories. Dependencies resolved. ========================================================================================================================================================================================================== Package Architecture Version Repository Size ========================================================================================================================================================================================================== Removing: apptainer x86_64 1.4.5-2_generic.el8 @@System 139 M apptainer-suid x86_64 1.4.5-2_generic.el8 @System 23 M Transaction Summary ========================================================================================================================================================================================================== Remove 2 Packages Freed space: 162 M Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : apptainer-suid-1.4.5-2_generic.el8.x86_64 1/2 Erasing : apptainer-1.4.5-2_generic.el8.x86_64 2/2 Running scriptlet: apptainer-1.4.5-2_generic.el8.x86_64 2/2 Verifying : apptainer-1.4.5-2_generic.el8.x86_64 1/2 Verifying : apptainer-suid-1.4.5-2_generic.el8.x86_64 2/2 Installed products updated. Removed: apptainer-1.4.5-2_generic.el8.x86_64 apptainer-suid-1.4.5-2_generic.el8.x86_64 Complete! Locking RPM with string apptainer-1.4.5-2.2.el8*) Found built RPM: /home/user/rpmbuild/RPMS/x86_64/apptainer-1.4.5-2.2.el8.x86_64.rpm Installing the built RPM (requires sudo)... Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:apptainer-1.4.5-2.2.el8 ################################# [100%] Locking RPM with string apptainer-1.4.5-2.2.el8* Updating Subscription Management repositories. No such command: versionlock. Please use /bin/yum --help It could be a YUM plugin command, try: "yum install 'dnf-command(versionlock)'" Locking RPM with string apptainer-suid-1.4.5-2.2.el8*) Found built RPM: /home/user/rpmbuild/RPMS/x86_64/apptainer-suid-1.4.5-2.2.el8.x86_64.rpm Installing the built RPM (requires sudo)... Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:apptainer-suid-1.4.5-2.2.el8 ################################# [100%] Locking RPM with string apptainer-suid-1.4.5-2.2.el8* Updating Subscription Management repositories. No such command: versionlock. Please use /bin/yum --help It could be a YUM plugin command, try: "yum install 'dnf-command(versionlock)'" Locked packages: Updating Subscription Management repositories. No such command: versionlock. Please use /bin/yum --help It could be a YUM plugin command, try: "yum install 'dnf-command(versionlock)'" Name : apptainer Version : 1.4.5 Release : 2.2.el8 Architecture: x86_64 Install Date: Mon 16 Feb 2026 08:39:25 PM PST Group : Unspecified Size : 149401792 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : (none) Source RPM : apptainer-1.4.5-2.2.el8.src.rpm Build Date : Sat 14 Feb 2026 06:36:06 AM PST Build Host : buildvm-x86-18.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Application and environment virtualization formerly known as Singularity Description : Apptainer provides functionality to make portable containers that can be used across host environments. RPM installation complete. You may verify FIPS support by checking the binary symbols (for example, with 'go tool nm'). 1.4.5-2.2.el8 INFO: Using cached SIF image 3.23.3
It should be in testing tomorrow.
FEDORA-EPEL-2026-aba9cbc84b (apptainer-1.4.5-3.el10_1) has been submitted as an update to Fedora EPEL 10.1. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-aba9cbc84b
FEDORA-EPEL-2026-a2dfe68ed3 (apptainer-1.4.5-3.el9) has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-a2dfe68ed3
FEDORA-EPEL-2026-82f07c2a59 (apptainer-1.4.5-3.el8) has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-82f07c2a59
FEDORA-2026-fd2a9c3031 (apptainer-1.4.5-6.fc44) has been submitted as an update to Fedora 44. https://bodhi.fedoraproject.org/updates/FEDORA-2026-fd2a9c3031
FEDORA-EPEL-2026-cac32685d7 (apptainer-1.4.5-3.el10_2) has been submitted as an update to Fedora EPEL 10.2. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-cac32685d7
FEDORA-2026-221d4884f3 has been pushed to the Fedora 42 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-221d4884f3` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-221d4884f3 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2026-a2dfe68ed3 has been pushed to the Fedora EPEL 9 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-a2dfe68ed3 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2026-644f1847b7 has been pushed to the Fedora 43 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-644f1847b7` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-644f1847b7 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2026-aba9cbc84b has been pushed to the Fedora EPEL 10.1 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-aba9cbc84b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2026-82f07c2a59 has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-82f07c2a59 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2026-fd2a9c3031 has been pushed to the Fedora 44 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2026-fd2a9c3031` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2026-fd2a9c3031 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2026-cac32685d7 has been pushed to the Fedora EPEL 10.2 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2026-cac32685d7 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Burvil, can you please test this out of epel-testing?
@dwd are you sure there is no host you or any colleague can enable FIPS on and test? I think it would be more efficient than going back and forth. Below is the output showing it didn't work, as I get the same OpenSSL error I got earlier after installing the newest package. $ sudo dnf --enablerepo=epel-testing install apptainer Updating Subscription Management repositories. Extra Packages for Enterprise Linux 8 - Testing - x86_64 851 kB/s | 1.8 MB 00:02 Dependencies resolved. ================================================================================================================================================== Package Architecture Version Repository Size ================================================================================================================================================== Installing: apptainer x86_64 1.4.5-3.el8 epel-testing 46 M Transaction Summary ================================================================================================================================================== Install 1 Package Total download size: 46 M Installed size: 142 M Is this ok [y/N]: y Downloading Packages: apptainer-1.4.5-3.el8.x86_64.rpm 12 MB/s | 46 MB 00:03 -------------------------------------------------------------------------------------------------------------------------------------------------- Total 9.5 MB/s | 46 MB 00:04 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : apptainer-1.4.5-3.el8.x86_64 1/1 Running scriptlet: apptainer-1.4.5-3.el8.x86_64 1/1 Verifying : apptainer-1.4.5-3.el8.x86_64 1/1 Installed products updated. Installed: apptainer-1.4.5-3.el8.x86_64 Complete! $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version goroutine 1 [running]: crypto/internal/backend.init.0() crypto/internal/backend/openssl.go:65 +0x2a5 $ apptainer version 1.4.5-3.el8 $ rpm -qi apptainer Name : apptainer Version : 1.4.5 Release : 3.el8 Architecture: x86_64 Install Date: Wed 18 Feb 2026 04:30:49 PM PST Group : Unspecified Size : 149410198 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : RSA/SHA256, Tue 17 Feb 2026 02:09:38 PM PST, Key ID 21ea45ab2f86d6a1 Source RPM : apptainer-1.4.5-3.el8.src.rpm Build Date : Tue 17 Feb 2026 12:25:51 PM PST Build Host : buildvm-x86-17.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Application and environment virtualization formerly known as Singularity Description : Apptainer provides functionality to make portable containers that can be used across host environments.
I built with the exact same spec file as the previous scratch build so I don't know what else to do. The build log at https://kojipkgs.fedoraproject.org//packages/apptainer/1.4.5/3.el8/data/logs/x86_64/build.log shows that GODEBUG=fips140=on was set. I could probably with effort make a new VM on which I could experiment with FIPS, but I'm actually more concerned about being efficient with my limited time than with the total elapsed time. So I'd really rather have you do the testing since you're the one who is most motivated. It looks like for 1.4.5-2.2 you tested with apptainer-suid and this time you didn't. Could that be the difference?
Dave, thanks. While I'm definitely motivated to help, I do have an acceptable workaround in my environment for now, where I've installed and versionlocked my version where I've recompiled the code in the source RPM. So I'm fine if you need more time to set things up, work with someone else, etc. If that's the case, I'd just like to know what you'd still need from me. I get the same thing when repeating, and installing apptainer-suid. $ sudo yum -y install apptainer apptainer-suid Updating Subscription Management repositories. Extra Packages for Enterprise Linux 8 - Testing - x86_64 20 kB/s | 19 kB 00:00 Dependencies resolved. ========================================================================================================================================================= Package Architecture Version Repository Size ========================================================================================================================================================= Installing: apptainer x86_64 1.4.5-3.el8 epel-testing 46 M apptainer-suid x86_64 1.4.5-3.el8 epel-testing 7.4 M Transaction Summary ========================================================================================================================================================= Install 2 Packages Total download size: 53 M Installed size: 166 M Downloading Packages: (1/2): apptainer-suid-1.4.5-3.el8.x86_64.rpm 16 MB/s | 7.4 MB 00:00 (2/2): apptainer-1.4.5-3.el8.x86_64.rpm 52 MB/s | 46 MB 00:00 --------------------------------------------------------------------------------------------------------------------------------------------------------- Total 29 MB/s | 53 MB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : apptainer-1.4.5-3.el8.x86_64 1/2 Running scriptlet: apptainer-1.4.5-3.el8.x86_64 1/2 Installing : apptainer-suid-1.4.5-3.el8.x86_64 2/2 Running scriptlet: apptainer-1.4.5-3.el8.x86_64 2/2 Running scriptlet: apptainer-suid-1.4.5-3.el8.x86_64 2/2 Verifying : apptainer-1.4.5-3.el8.x86_64 1/2 Verifying : apptainer-suid-1.4.5-3.el8.x86_64 2/2 Installed products updated. Installed: apptainer-1.4.5-3.el8.x86_64 apptainer-suid-1.4.5-3.el8.x86_64 Complete! $ apptainer run --bind /etc/pki/tls:/etc/ssl docker://alpine cat /etc/alpine-release INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version $ rpm -qi apptainer Name : apptainer Version : 1.4.5 Release : 3.el8 Architecture: x86_64 Install Date: Thu 19 Feb 2026 09:55:17 AM PST Group : Unspecified Size : 149410198 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : RSA/SHA256, Tue 17 Feb 2026 02:09:38 PM PST, Key ID 21ea45ab2f86d6a1 Source RPM : apptainer-1.4.5-3.el8.src.rpm Build Date : Tue 17 Feb 2026 12:25:51 PM PST Build Host : buildvm-x86-17.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Application and environment virtualization formerly known as Singularity Description : Apptainer provides functionality to make portable containers that can be used across host environments. $ rpm -qi apptainer-suid Name : apptainer-suid Version : 1.4.5 Release : 3.el8 Architecture: x86_64 Install Date: Thu 19 Feb 2026 09:55:17 AM PST Group : Unspecified Size : 25093824 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : RSA/SHA256, Tue 17 Feb 2026 02:09:38 PM PST, Key ID 21ea45ab2f86d6a1 Source RPM : apptainer-1.4.5-3.el8.src.rpm Build Date : Tue 17 Feb 2026 12:25:51 PM PST Build Host : buildvm-x86-17.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Setuid component of Apptainer Description : Provides the optional setuid-root portion of Apptainer.
And does version 1.4.5-2.2 (from https://koji.fedoraproject.org/koji/taskinfo?taskID=142306511) succeed with FIPS enabled even when only installing apptainer? If so I'm out of ideas because 1.4.5-2.2 and 1.4.5-3 should be identical. They are installed and run in slightly different ways. Maybe try downloading 1.4.5-3 from https://koji.fedoraproject.org/koji/buildinfo?buildID=2942710 and installing it with install-apptainer.sh just like you do with 1.4.5-2.2.
Created attachment 2130231 [details] installs rpms from local build directory, locks them, supports symlinks
Created attachment 2130232 [details] updated version of install-apptainer.sh
1. If I only install only apptainer and not apptainer-suid, I do get the same error as above. See output below. FIPS is enabled. 2. I also tried your other suggestion, downloading the rpms manually and installing them. Output is also below, same OpenSSL error. I am also uploading the latest copy of install-apptainer.sh. Since I do have a workaround and you're out of ideas, perhaps you can ask around your team if they have any ideas on this? I think we're both spinning our wheels on this, and I'm also running out of ideas, too. I also have other things on my plate I need to tend to. $ ./install-apptainer.sh Checking FIPS status... FIPS mode is enabled. Searching for the built binary RPM... Found RPMS /home/adm-26051/rpmbuild/RPMS/x86_64/apptainer-1.4.5-2.2.el8.x86_64.rpm Enabling versionlock .. [main] enabled = 1 locklist = /etc/dnf/plugins/versionlock.list Removing currently installed apptainer ... Updating Subscription Management repositories. Dependencies resolved. ========================================================================================================================================================= Package Architecture Version Repository Size ========================================================================================================================================================= Removing: apptainer x86_64 1.4.5-3.el8 @epel-testing 142 M apptainer-suid x86_64 1.4.5-3.el8 @epel-testing 24 M Transaction Summary ========================================================================================================================================================= Remove 2 Packages Freed space: 166 M Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : apptainer-suid-1.4.5-3.el8.x86_64 1/2 Erasing : apptainer-1.4.5-3.el8.x86_64 2/2 Running scriptlet: apptainer-1.4.5-3.el8.x86_64 2/2 Verifying : apptainer-1.4.5-3.el8.x86_64 1/2 Verifying : apptainer-suid-1.4.5-3.el8.x86_64 2/2 Installed products updated. Removed: apptainer-1.4.5-3.el8.x86_64 apptainer-suid-1.4.5-3.el8.x86_64 Complete! Found built RPM: /home/adm-26051/rpmbuild/RPMS/x86_64/apptainer-1.4.5-2.2.el8.x86_64.rpm Installing the built RPM (requires sudo)... Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:apptainer-1.4.5-2.2.el8 ################################# [100%] Locking RPM with string apptainer-1.4.5-2.2.el8* Updating Subscription Management repositories. Last metadata expiration check: 0:53:20 ago on Thu 19 Feb 2026 04:16:34 PM PST. Adding versionlock on: apptainer-0:1.4.5-2.2.el8.* Locked packages: Updating Subscription Management repositories. Last metadata expiration check: 0:53:25 ago on Thu 19 Feb 2026 04:16:34 PM PST. apptainer-0:1.4.5-2.2.el8.* Name : apptainer Version : 1.4.5 Release : 2.2.el8 Architecture: x86_64 Install Date: Thu 19 Feb 2026 05:09:52 PM PST Group : Unspecified Size : 149401792 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : (none) Source RPM : apptainer-1.4.5-2.2.el8.src.rpm Build Date : Sat 14 Feb 2026 06:36:06 AM PST Build Host : buildvm-x86-18.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Application and environment virtualization formerly known as Singularity Description : Apptainer provides functionality to make portable containers that can be used across host environments. RPM installation complete. You may verify FIPS support by checking the binary symbols (for example, with 'go tool nm'). 1.4.5-2.2.el8 INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version goroutine 1 [running]: crypto/internal/backend.init.0() crypto/internal/backend/openssl.go:65 +0x2a5 $ ./install-apptainer.sh Checking FIPS status... FIPS mode is enabled. Searching for the built binary RPM... Found RPMS /home/adm-26051/rpmbuild/RPMS/x86_64/apptainer-suid-1.4.5-3.el8.x86_64.rpm /home/adm-26051/rpmbuild/RPMS/x86_64/apptainer-1.4.5-3.el8.x86_64.rpm Enabling versionlock .. [main] enabled = 1 locklist = /etc/dnf/plugins/versionlock.list Removing currently installed apptainer ... Updating Subscription Management repositories. No match for argument: apptainer-suid Dependencies resolved. ========================================================================================================================================================= Package Architecture Version Repository Size ========================================================================================================================================================= Removing: apptainer x86_64 1.4.5-2.2.el8 @@System 142 M Transaction Summary ========================================================================================================================================================= Remove 1 Package Freed space: 142 M Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Erasing : apptainer-1.4.5-2.2.el8.x86_64 1/1 Running scriptlet: apptainer-1.4.5-2.2.el8.x86_64 1/1 Verifying : apptainer-1.4.5-2.2.el8.x86_64 1/1 Installed products updated. Removed: apptainer-1.4.5-2.2.el8.x86_64 Complete! Found built RPM: /home/adm-26051/rpmbuild/RPMS/x86_64/apptainer-suid-1.4.5-3.el8.x86_64.rpm Installing the built RPM (requires sudo)... error: Failed dependencies: apptainer = 1.4.5-3.el8 is needed by apptainer-suid-1.4.5-3.el8.x86_64 Locking RPM with string apptainer-suid-1.4.5-3.el8* Updating Subscription Management repositories. Last metadata expiration check: 0:09:08 ago on Thu 19 Feb 2026 05:16:37 PM PST. No package found for: apptainer-suid-1.4.5-3.el8* Found built RPM: /home/adm-26051/rpmbuild/RPMS/x86_64/apptainer-1.4.5-3.el8.x86_64.rpm Installing the built RPM (requires sudo)... Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:apptainer-1.4.5-3.el8 ################################# [100%] Locking RPM with string apptainer-1.4.5-3.el8* Updating Subscription Management repositories. Last metadata expiration check: 0:09:17 ago on Thu 19 Feb 2026 05:16:37 PM PST. Adding versionlock on: apptainer-0:1.4.5-3.el8.* Locked packages: Updating Subscription Management repositories. Last metadata expiration check: 0:09:22 ago on Thu 19 Feb 2026 05:16:37 PM PST. apptainer-0:1.4.5-3.el8.* Name : apptainer Version : 1.4.5 Release : 3.el8 Architecture: x86_64 Install Date: Thu 19 Feb 2026 05:25:52 PM PST Group : Unspecified Size : 149410198 License : LicenseRef-Callaway-BSD AND BSD-3-Clause-LBNL AND Apache-2.0 Signature : (none) Source RPM : apptainer-1.4.5-3.el8.src.rpm Build Date : Tue 17 Feb 2026 12:25:51 PM PST Build Host : buildvm-x86-17.rdu3.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://apptainer.org Bug URL : https://bugz.fedoraproject.org/apptainer Summary : Application and environment virtualization formerly known as Singularity Description : Apptainer provides functionality to make portable containers that can be used across host environments. RPM installation complete. You may verify FIPS support by checking the binary symbols (for example, with 'go tool nm'). 1.4.5-3.el8 INFO: Using cached SIF image panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version goroutine 1 [running]: crypto/internal/backend.init.0() crypto/internal/backend/openssl.go:65 +0x2a5
FEDORA-EPEL-2026-a2dfe68ed3 (apptainer-1.4.5-3.el9) has been pushed to the Fedora EPEL 9 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2026-aba9cbc84b (apptainer-1.4.5-3.el10_1) has been pushed to the Fedora EPEL 10.1 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2026-221d4884f3 (apptainer-1.4.5-3.fc42) has been pushed to the Fedora 42 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2026-82f07c2a59 (apptainer-1.4.5-3.el8) has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report.
@dwd it looks like the changes were pushed to production/stable. But I don't see any changes since my last post, i.e. as far as I can tell, the problem still exists there. Have you had a chance to ask around as to how this problem can be addressed in the stable EPEL repositories? Note that by my last post, I wasn't saying this was no longer an issue. I was just saying I had a workaround for now where I recompiled what's in the source RPM and locked to that version, but was hoping for a permanent long term solution. So, per the last message from the Fedora Update System that stated "If problem still persists, please make note of it in this bug report.", I am saying the problem still persists.
FEDORA-2026-644f1847b7 (apptainer-1.4.5-3.fc43) has been pushed to the Fedora 43 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2026-cac32685d7 (apptainer-1.4.5-3.el10_2) has been pushed to the Fedora EPEL 10.2 stable repository. If problem still persists, please make note of it in this bug report.
Yeah, I didn't do anything to stop the automated process of promoting the packages that were built, and that automatically closed the issue. Reopening.