*** Bug 1020292 has been marked as a duplicate of this bug. ***

SPEC file contains the necessary information to build on CentOS / RHEL 7 as well.

I was packaging a Dash client for Fedora for quite some time. Dash was originally a fork of Bitcoin, and now it still synchronises its codebase with newer versions of Bitcoin. Hence, Dash is not different than Bitcoin from packaging point of view, only some file names and TCP port numbers are different.
You can take a look at spec and other files I use to build Dash here: https://obs.infoserver.lv/package/show/cryptocurrency/dash-core
If you find any ideas from this useful, feel free to use them. :-)

My spec file was originally based on spec file that was in Bitcoin source's contrib subdirectory (it was removed from there since then), but significantly evolved.

It has provisions for building with depends system, making deterministic build closer to the one used for binary distribution by vendor (but not exactly the same: still using compiler and some system libraries from Fedora). In order to be suitable for building in isolated environment without network connectivity, all sources needed for building are added as "SourceN:" directives in autogenerated section of spec file, and "update-sources.sh" script has to be run to update these sources every time you change the version number. This feature is not suitable for official Fedora package because it essentially bundles various libraries, so it's disabled by default.

Also, functional tests in "%check" section are run in parallel, and temporary directory is placed in "/var/tmp" for them, because "/tmp" is sometimes tmpfs in isolated build environment. and some tests require 8 gigabytes of space in tmpdir.

Thanks I will look at it.

Selinux* things must be optional.
If user set "selinux=disabled" than he can remove most of *selinux* rpms from host.
But your packages require to reinstall them again.

Summary good job, I use this for last month ok (f30 x32, f32 x64).

 - 
Source0:    http://github.com/%{name}/%{name}/archive/v%{version}.tar.gz#/%{name}-%{version}.tar.gz

→

Source0:    http://github.com/%{name}/%{name}/archive/v%{version}/%{name}-%{version}.tar.gz

 - Why does the core subpackage obsoletes the main package? Also for a rename the Obsolete part must be fixed, to the release above the last release of the obsoleted package.

%package core
Summary:    Peer to Peer Cryptographic Currency
Obsoletes:  %{name} < %{version}-%{release}
Provides:   %{name} = %{version}-%{release}

 - GZipping the man pages is handled by rpm, do not do it yourself:

gzip %{buildroot}%{_mandir}/man1/$i.1

 - This is only needed in EPEL7:

%post core
/bin/touch --no-create %{_datadir}/icons/hicolor &>/dev/null || :

%postun core
if [ $1 -eq 0 ] ; then
    /bin/touch --no-create %{_datadir}/icons/hicolor &>/dev/null
    /usr/bin/gtk-update-icon-cache %{_datadir}/icons/hicolor &>/dev/null || :
fi

%posttrans core
/usr/bin/gtk-update-icon-cache %{_datadir}/icons/hicolor &>/dev/null || :

 Please put it behind a condition.

 - Do not package libtool archive. Static library are generally not packaged, and, if really needed, must have their own static subpackage:

%{_libdir}/libbitcoinconsensus.a
%{_libdir}/libbitcoinconsensus.la

 
 - In order to avoid unintentional soname bump, we recommend not globbing the major SONAME version from shared library. Be more specific instead:

%{_libdir}/libbitcoinconsensus.so.*

 - /var/lib/, {_localstatedir}/lib → use %{_sharedstatedir}

 - /var/, %{_var} → use %{_localstatedir}

 - /run → %{_rundir}

 - I know nothing about SELinux, I need to find another reviewer for this

Also create a logrotate file for the log: https://docs.fedoraproject.org/en-US/packaging-guidelines/#_logrotate_config_file

Thanks for the feedback, updating the package now.

(In reply to Robert-André Mauchin from comment #6)
> Source0:   
> http://github.com/%{name}/%{name}/archive/v%{version}/%{name}-%{version}.tar.
> gz

Updated.

>  - Why does the core subpackage obsoletes the main package? Also for a
> rename the Obsolete part must be fixed, to the release above the last
> release of the obsoleted package.

Long old gone thing, removed the Obsoletes.

>  - GZipping the man pages is handled by rpm, do not do it yourself:
> 
> gzip %{buildroot}%{_mandir}/man1/$i.1

Missed it. Actually man pages are now installed automatically, so I removed the section entirely.

>  - This is only needed in EPEL7:
> 
> %post core
> /bin/touch --no-create %{_datadir}/icons/hicolor &>/dev/null || :
> 
> %postun core
> if [ $1 -eq 0 ] ; then
>     /bin/touch --no-create %{_datadir}/icons/hicolor &>/dev/null
>     /usr/bin/gtk-update-icon-cache %{_datadir}/icons/hicolor &>/dev/null || :
> fi
> 
> %posttrans core
> /usr/bin/gtk-update-icon-cache %{_datadir}/icons/hicolor &>/dev/null || :
> 
>  Please put it behind a condition.

Missed it, fixed.

>  - Do not package libtool archive. Static library are generally not
> packaged, and, if really needed, must have their own static subpackage:
> 
> %{_libdir}/libbitcoinconsensus.a
> %{_libdir}/libbitcoinconsensus.la

Disabled static library building and removed the local archive.

>  - In order to avoid unintentional soname bump, we recommend not globbing
> the major SONAME version from shared library. Be more specific instead:
> 
> %{_libdir}/libbitcoinconsensus.so.*

Fixed.

>  - /var/lib/, {_localstatedir}/lib → use %{_sharedstatedir}
>  - /var/, %{_var} → use %{_localstatedir}
>  - /run → %{_rundir}

Missed, all fixed.

>  - I know nothing about SELinux, I need to find another reviewer for this

I will not separate the selinux policies from the main package. Many packages in the distribution require SELinux (ex. FreeIPA). It's not something that should be disabled, pretty much like the firewall.


Testing now the above changes with 0.20, will post the updated src.rpm after all functional tests have run.

(In reply to Robert-André Mauchin from comment #7)
> Also create a logrotate file for the log:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/
> #_logrotate_config_file

A couple of notes here:

- The log file after months of a running daemon is only 9.7 Mb, so I'm not sure there is really any benefit here.
- When enabling -debug on the daemon, the -shrinkdebugfile is automatically set to 1, so the log file is trimmed at startup.

So I see two options:

1. Add -shrinkdebugfile=1 to the systemd unit, so the log file is also trimmed when NOT starting in debug mode
2. Add -shrinkdebugfile=0 to the systemd unit so the log file is never trimmed even when debugging and then add the logrotate file.

I think solution 1 is better.

In the meanwhile, before changing anything for the logging:

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.20.0-2.fc32.src.rpm

* Tue Jun 30 2020 Simone Caronni <negativo17> - 0.20.0-2
- Update Source0 URL.
- Do not obsolete "bitcoin", just leave the provider for it.
- Let the build install the man pages.
- Make sure old post scriptlets run only on RHEL/CentOS 7.
- Do not install static library and archive.
- Be explicit with share object versions.
- Use macros for more directories.
- Use GCC 9 and not 7 to build on RHEL/CentOS 7.

 - Could you follow the rules specified at https://fedoraproject.org/wiki/SELinux/IndependentPolicy and use the %pre/%post macros documented there?

 - See the post by DWalsh on the -devel ML: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/GK3KPEDHX5NLDW32X7RIAP2IVEYHIAX3/

(In reply to Robert-André Mauchin from comment #12)
>  - Could you follow the rules specified at
> https://fedoraproject.org/wiki/SELinux/IndependentPolicy and use the
> %pre/%post macros documented there?
> 
>  - See the post by DWalsh on the -devel ML:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/
> message/GK3KPEDHX5NLDW32X7RIAP2IVEYHIAX3/

Ah, I was not aware of it. I will check, thanks!

The Independent policy guide (https://fedoraproject.org/wiki/SELinux/IndependentPolicy) should cover all you need in terms of packaging the policy.
As for the policy itself, please try to follow https://github.com/SELinuxProject/refpolicy/wiki/StyleGuide. Also, using "permissive bitcoin_t;" disables the SELinux confinement on the domain (AVC denials are logged, but not enforced) and should therefore only be used for debugging/testing.

(In reply to Vit  Mojzis from comment #14)
> The Independent policy guide
> (https://fedoraproject.org/wiki/SELinux/IndependentPolicy) should cover all
> you need in terms of packaging the policy.
> As for the policy itself, please try to follow
> https://github.com/SELinuxProject/refpolicy/wiki/StyleGuide. Also, using
> "permissive bitcoin_t;" disables the SELinux confinement on the domain (AVC
> denials are logged, but not enforced) and should therefore only be used for
> debugging/testing.

Ok, thanks for the info!

(In reply to Simone Caronni from comment #10)
> (In reply to Robert-André Mauchin from comment #7)
> > Also create a logrotate file for the log:
> > https://docs.fedoraproject.org/en-US/packaging-guidelines/
> > #_logrotate_config_file
> 
> A couple of notes here:
> 
> - The log file after months of a running daemon is only 9.7 Mb, so I'm not
> sure there is really any benefit here.

I think it is not matter how big is log file.
Let user to decide switch it off on demand.

Some questions.

* In the spec file, what exactly is `Source4` used for?

* Why is the Bitcoin Core package called `bitcoin` and not `bitcoin-core` like in other package managers. E.g. https://snapcraft.io/bitcoin-core or https://flathub.org/apps/details/org.bitcoincore.bitcoin-qt

* Is the package going to be made available in CentOS? If yes, on what schedule is CentOS/RHEL going to update the package? Is there a chance that EOL version are offered in the package manager? See https://bitcoincore.org/en/lifecycle/

(In reply to marco from comment #17)
> * Why is the Bitcoin Core package called `bitcoin` and not `bitcoin-core`
> like in other package managers. E.g. https://snapcraft.io/bitcoin-core or
> https://flathub.org/apps/details/org.bitcoincore.bitcoin-qt

I can try to answer this.
* "bitcoin-core" includes bitcoind (bitcoin-server, but it is "client" of bitcoin network), bitcoin-utils (pure "client" as client) and bitcoin-qt ("server" as client + partially "client" as client all-in-one :-)
* all of them are absolutely indepent from each another on runtime.
So, this "-core" is not "core" and not includes runtime core (and this is big problem - it's impossible to select "-lib" from these tonns of code).
And as for me I think current packaging (bitcoin => -server + -utils + -qt) is very logical.
PS. in addition some of docs say "..then run bitcoin-core (they mean bitcoin-qt binary) and..."

(In reply to Eugene A. Pivnev from comment #16)
> (In reply to Simone Caronni from comment #10)
> > (In reply to Robert-André Mauchin from comment #7)
> > > Also create a logrotate file for the log:
> > > https://docs.fedoraproject.org/en-US/packaging-guidelines/
> > > #_logrotate_config_file
> > 
> > A couple of notes here:
> > 
> > - The log file after months of a running daemon is only 9.7 Mb, so I'm not
> > sure there is really any benefit here.
> 
> I think it is not matter how big is log file.
> Let user to decide switch it off on demand.

Exactly, but -shrinkdebugfile set to 1 automatically when in debug mode, the user will get truncated logs at restart. So in my opinion is better to disable the default shrinkdebugfile when turning on debug if we want to use logrotate.

(In reply to marco from comment #17)
> * In the spec file, what exactly is `Source4` used for?

Contains some things related to packaging (icon, desktop menu, etc.). I might remove it entirely at some point.
It's unpacked in the %autosetup macro:

%autosetup -a 4 -p1

> * Is the package going to be made available in CentOS? If yes, on what
> schedule is CentOS/RHEL going to update the package? Is there a chance that
> EOL version are offered in the package manager? See
> https://bitcoincore.org/en/lifecycle/

I'm planning to build it also on CentOS/RHEL 7+ (stuff is already in the spec file).
I just plan to build the latest version. Maintaining minor releases of previous version in EPEL might be an option (ex. 0.20.1 in EPEL when 0.21 gets released in Fedora), but I don't really see the point honestly. Might be that at one point due to dependencies it's not possible to build the latest for EPEL, so in that case a maintained old release or bundling of some libraries could be an option.

Regarding EOL releases, no, no plan to keep EOL releases alive.

Don't think you have to build multiple different SELinux policies, one should work on all variants.

Cryptocurrency wallets are very juicy targets for criminals, so it's paramount that you do everything you can to prevent and detect attempts to inject malware into the package.

First, never use insecure HTTP if HTTPS is available.

Second, verify upstream's signature before unpacking the tarball. Unfortunately they sign it in an indirect way that our handy verifier script doesn't expect. That makes the verification code a bit tricky, so I have written it for you.

These are the changes you need to make:

--- bitcoin.spec.old	2020-06-30 12:57:18.000000000 +0200
+++ bitcoin.spec	2020-07-06 15:48:51.656323998 +0200
@@ -7,9 +7,9 @@
 Release:    2%{?dist}
 Summary:    Peer to Peer Cryptographic Currency
 License:    MIT
-URL:        http://bitcoin.org/
+URL:        https://bitcoin.org/
 
-Source0:    http://github.com/%{name}/%{name}/archive/v%{version}/%{name}-%{version}.tar.gz
+Source0:    https://github.com/%{name}/%{name}/archive/v%{version}/%{name}-%{version}.tar.gz
 Source1:    %{name}-tmpfiles.conf
 Source2:    %{name}.sysconfig
 Source3:    %{name}.service
@@ -20,12 +20,16 @@
 Source8:    README.server.redhat
 Source9:    README.utils.redhat
 Source10:   README.gui.redhat
+Source11:   https://bitcoin.org/bin/bitcoin-core-%{version}/SHA256SUMS.asc
+Source12:   https://bitcoin.org/laanwj-releases.asc
 
 BuildRequires:  autoconf
 BuildRequires:  automake
 BuildRequires:  boost-devel
 BuildRequires:  checkpolicy
 BuildRequires:  desktop-file-utils
+BuildRequires:  gnupg2
+BuildRequires:  grep
 BuildRequires:  java
 BuildRequires:  libdb4-cxx-devel
 BuildRequires:  libevent-devel
@@ -76,7 +80,7 @@
 may be used by third party software to provide consensus verification
 functionality.
 
-Unless you know need this package, you probably do not.
+Unless you know you need this package, you probably do not.
 
 %package devel
 Summary:    Peer-to-peer digital currency
@@ -126,6 +130,15 @@
 need this package.
 
 %prep
+gpgworkdir="$(mktemp --directory)"
+# Decode the ASCII armor on the keyring.
+gpg2 --homedir="${gpgworkdir}" --yes --output="${gpgworkdir}/keyring.gpg" --dearmor '%{SOURCE12}'
+# Verify the signature on the checksums file using the decoded keyring.
+gpgv2 --homedir="${gpgworkdir}" --keyring="${gpgworkdir}/keyring.gpg" '%{SOURCE11}'
+# Verify the tarball using the checksums file minus the signature.
+( cd '%{_sourcedir}' && grep bitcoin '%{SOURCE11}' | sha256sum --check --ignore-missing - )
+rm --recursive --force ${gpgworkdir}
+
 %autosetup -a 4 -p1
 mv packaging-*/debian/* contrib/debian/

If you fetch the key from the same website the binaries are taken from, there is no security. Anyone replacing the binaries can trivially replace the key.

Also, bitcoincore.org is the official download site (bitcoin.org is a mirror site unrelated to the Bitcoin Core project). So I recommend to use the steps to verify from their download page: https://bitcoincore.org/en/download/

The instructions recommend to fetch the key based on its fingerprint (01EA5486DE18A882D4C2684590C8019E36C2E964).

(In reply to marco from comment #23)
> If you fetch the key from the same website the binaries are taken from,
> there is no security. Anyone replacing the binaries can trivially replace
> the key.

That would be true if the upstream project would generate a new key for every release, and also wouldn't sign their keys.

When a new release is signed with the same key that the developers have been using for years, then that increases our confidence that it is from the same source as all the previous releases. When a key needs replacing, then the project can maintain continuity by signing the new key with the old key. We packagers must be very careful when a release-signing key changes, and not blindly replace the key like we replace a tarball.

In this particular case you're also off the mark because my patched spec *doesn't* fetch the key from the same website as the binaries. To my slight surprise I found that the tarball from Github is identical to the one on bitcoin.org (and on bitcoincore.org). If that's reliable, then we can improve security by fetching the tarball from Github and the signed checksum file from bitcoin.org or bitcoincore.org, and verifying them with the key we already have. An attacker will then have to acquire the secret key and compromise *both* websites before they can sneak malicious changes past the verification step.

> Also, bitcoincore.org is the official download site (bitcoin.org is a mirror
> site unrelated to the Bitcoin Core project).

In that case the URL field in the package should also point to https://bitcoincore.org/en/about/.

> The instructions recommend to fetch the key based on its fingerprint
> (01EA5486DE18A882D4C2684590C8019E36C2E964).

Hmm, they refer to keyserver.ubuntu.com, which runs Hockeypuck. I don't see any statement that Hockeypuck has a solution to the spam attack (https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f) that led to keys.fedoraproject.org being turned off (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/COEYWJBQDAWRSYNQW7Y7TD2EKEGBWOAY/) in February this year. If it doesn't then you expose yourself to a denial of service when you fetch from the keyserver.

In case somebody thinks that fetching a key from a keyserver is more secure than fetching it from the project's website: It's not, because anyone can write somebody else's name on a key and upload it to a keyserver. Only the fingerprint ensures that you get the right key, and someone who can replace files on the project's website can replace the fingerprint too.

> packagers must be very careful when a release-signing key changes

Source12 simply downloads the key from https://bitcoin.org/laanwj-releases.asc without checking the hash or fingerprint, so there is no way to detect changes. What am I missing?

> To my slight surprise I found that the tarball from Github is identical to the one on bitcoin.org (and on bitcoincore.org)

I think this is only a coincidence for the 0.20.0 release. All other releases should not match, which is why I assumed the download sources are identical.

> I don't see any statement that Hockeypuck has a solution to the spam attack

Good point, personally I can recommend https://keys.openpgp.org/vks/v1/by-fingerprint/01EA5486DE18A882D4C2684590C8019E36C2E964, which claim to be resistant to those attacks ( https://keys.openpgp.org/about/faq#sks-pool )

Not sure, but keyserver.ubuntu.com might have solved the attack by disabling key updates, which could lead to problems should the key ever be revoked.

Though generally, as long as the fingerprint matches, it should be possible to download the key from any source with reliable uptime.

(In reply to marco from comment #25)
> Source12 simply downloads the key from
> https://bitcoin.org/laanwj-releases.asc without checking the hash or
> fingerprint, so there is no way to detect changes. What am I missing?

You're missing the fact that RPMbuild doesn't download anything and the Koji builders are isolated from Internet access. All sources and patches are taken from the Fedora Project's Git repository and lookaside cache, and change only when a package maintainer uploads a new file. Our source file verification policy says that the keyring shall be committed to Git:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_source_file_verification

The URL is there to document where the keyring came from, so that anyone can download it and verify that it's identical to the one in Git.

(In reply to Björn Persson from comment #26)
> (In reply to marco from comment #25)
> > Source12 simply downloads the key from
> > https://bitcoin.org/laanwj-releases.asc without checking the hash or
> > fingerprint, so there is no way to detect changes. What am I missing?
> 
> You're missing the fact that RPMbuild doesn't download anything and the Koji
> builders are isolated from Internet access. All sources and patches are
> taken from the Fedora Project's Git repository and lookaside cache, and
> change only when a package maintainer uploads a new file. Our source file
> verification policy says that the keyring shall be committed to Git:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/
> #_source_file_verification
> 
> The URL is there to document where the keyring came from, so that anyone can
> download it and verify that it's identical to the one in Git.

This looks too dependent on Fedora infrastructure.
What about those who want to re-build the package from the spec file on their computer (and download all necessary sources using spectool)?
What about those who want to re-build the package using OBS and download_files source service?
I think, the main PGP public key's checksum should be embedded into spec file and checked against to make sure all re-downloaded sources are correct.

(In reply to Oleg Girko from comment #27)
> What about those who want to re-build the package from the spec file

I would recommend rebuilding from the source RPM package. Rebuilding from only a spec isn't possible in the general case. Many packages have patches, and sometimes there is no working URL to a source. Those will be missing if you try to rebuild from only the spec.

> I think, the main PGP public key's checksum should be embedded into spec
> file and checked against to make sure all re-downloaded sources are correct.

That wouldn't hurt. You would want a GnuPG command – or a series of commands – to verify that the given keyring contains a key with the given fingerprint, and also that it doesn't contain any other keys. Can you propose such a command? Don't forget to ensure that GnuPG will look only in the specified keyring even if the user has a default keyring.

(In reply to Björn Persson from comment #28)
> (In reply to Oleg Girko from comment #27)
> > I think, the main PGP public key's checksum should be embedded into spec
> > file and checked against to make sure all re-downloaded sources are correct.
> 
> That wouldn't hurt. You would want a GnuPG command – or a series of commands
> – to verify that the given keyring contains a key with the given
> fingerprint, and also that it doesn't contain any other keys. Can you
> propose such a command? Don't forget to ensure that GnuPG will look only in
> the specified keyring even if the user has a default keyring.

Something like this in %prep section:

    echo 123456789abcdef... %{SOURCE12} | sha256sum -c

Thanks, I've added signature verification which is a bit from all comments above.
The packaging guidelines are pretty clear about signatures, so:

- Key is downloaded from the keyserver (as also suggested by upstream) and instructions are in the SPEC file.
- Key is added to the Fedora SCM (aka it's in git).
- Detached signed checksum is in the lookaside cache (aka it's in the sources file).
- Since /usr/lib/rpm/redhat/gpgverify (aka %gpgverify) does not support signed sums files I've replaced it with gpgv2/sha256sum commands.

I will also add the SHA256UM.asc file in the .gitignore file once approved so there is no chance that the hashed checksum gets into SCM and can only go into the lookaside cache.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.20.0-3.fc32.src.rpm

* Sat Jul 18 2020 Simone Caronni <negativo17> - 0.20.0-3
- Add signature verification.
- Trim changelog.
- Fix typo in the libs description.

I will start working on the SELinux part hopefully soon (terribly busy in real life).

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.20.0-4.fc32.src.rpm

* Sun Jul 19 2020 Simone Caronni <negativo17> - 0.20.0-4
- Fix tests on RHEL/CentOS 7.

Thank you for adding the signature verification, but you're still using HTTP instead of HTTPS in URL and Source20 for no reason I can see.

(In reply to Björn Persson from comment #32)
> Thank you for adding the signature verification, but you're still using HTTP
> instead of HTTPS in URL and Source20 for no reason I can see.

Just forgot it to change it in %url which get expanded in the Source20 line.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.20.0-5.fc32.src.rpm

More cleanups:

* Tue Jul 21 2020 Simone Caronni <negativo17> - 0.20.0-5
- Use HTTPS for url tag.
- Reorganize sources. Add cleaned files from the packaging repository directly;
  bash completion snippets are now supported in the main sources.
- Move check section after install and include desktop file validating in
  there.

Room for improvements:
- Check hardening measures for the systemd unit (differences between the various distributions).
  Ex.: https://github.com/bitcoin/bitcoin/blob/master/contrib/init/bitcoind.service#L53-L74
- Rework SELinux policy
- Check about bundling libdb4 (deprecated everywhere in Fedora, not available in EPEL), upstream accepts any libdb versions for building but wallets are not compatible across different libdb versions and need to be converted. Debian builds with the latest libdb: https://salsa.debian.org/cryptocoin-team/bitcoin/-/blob/master/debian/rules#L31

I'll be away from the 25th of July for holidays until the 17th of August.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.20.0-6.fc32.src.rpm

* Tue Jul 21 2020 Simone Caronni <negativo17> - 0.20.0-6
- Update systemd unit.
- Update configuration options.
- Declared bundled libraries/forks.

Room for improvements:
- Rework SELinux policy
- Check about bundling libdb4 (deprecated everywhere in Fedora, not available in EPEL), upstream accepts any libdb versions for building but wallets are not compatible across different libdb versions and need to be converted. Debian builds with the latest libdb: https://salsa.debian.org/cryptocoin-team/bitcoin/-/blob/master/debian/rules#L31

I've run out of procrastination items for the SELinux policy :D

I've tested with a few wallets and everything is fine, no conversion needed.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.20.0-7.fc32.src.rpm

* Wed Jul 22 2020 Simone Caronni <negativo17> - 0.20.0-7
- Use libdb 5.x instead of deprecated 4.x. Fixes build on RHEL/CentOS 8.

Room for improvements:
- Rework SELinux policy

> BuildRequires:  openssl-devel
> BuildRequires:  protobuf-devel

I believe neither of them are needed at all

Getting back to this over the weekend. I just moved and got back internet today.

House move, renovations, virus, work, kids, you name the issue.
No time yet for looking at the SELinux policy.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.20.1-2.fc32.src.rpm

* Thu Nov 19 2020 Simone Caronni <negativo17> - 0.20.1-2
- Remove openssl/protobuf from build requirements.

* Wed Oct 21 2020 Simone Caronni <negativo17> - 0.20.1-1
- Update to 0.20.1.

(In reply to Simone Caronni from comment #39)

Keep in mind that now:
- bitcoins service does not covers DATA_DIR (/etc/sysconfig/bitcoin) with "datadir=" variable from /etc/bitcoin/bitcoin.conf.
  So if you want to move working data (~400GB) out from /var/lib you have to edit right (and only) DATA_DIR
- permissions of /etc/bitcoin folder (0750) too strict - ordinary users cannot view conf
- permissions of bitcoind data folder (/var/lib/bitcoin if "out of box") too strict too.

(In reply to Eugene A. Pivnev from comment #40)
> (In reply to Simone Caronni from comment #39)
> 
> Keep in mind that now:
> - bitcoins service does not covers DATA_DIR (/etc/sysconfig/bitcoin) with
> "datadir=" variable from /etc/bitcoin/bitcoin.conf.
>   So if you want to move working data (~400GB) out from /var/lib you have to
> edit right (and only) DATA_DIR

Keep in mind that there is an SELinux policy that covers all these folders, so you would need to edit that as well. Let's say that if you have SELinux policies covering the specifics of a service you better think about it two times before shuffling things around.

> - permissions of /etc/bitcoin folder (0750) too strict - ordinary users
> cannot view conf
> - permissions of bitcoind data folder (/var/lib/bitcoin if "out of box") too
> strict too.

Which is fine, as the former might contain the RPC password and the latter a wallet.

Anyone up for review? And maybe help with the SELinux policy as well.

(In reply to Simone Caronni from comment #42)
> Anyone up for review? And maybe help with the SELinux policy as well.

I can review but just after selinux fix.

1. Please add "--tmpdirprefix %{_tmppath}" to command line args of functional tests in "%check" section. By default, "/tmp" is used by tests for temporary data dirs, usually it's mounted as tmpfs, but some tests use quite large amount of space (like 7 or 8 gigabytes), so it can lead to test failures on systems with limited amount of memory.

2. What's the reason to include java as build requirement?

Everything builds OK if I remove java from build requirements.

I've built updated packages with the following changes:
- Update to 0.21.0.
- Use /var/tmp instead of /tmp as tmpdirprefix for functional tests.
- Remove java from build requirements.

See https://obs.infoserver.lv/package/show/cryptocurrency/bitcoin for details.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.21.0-2.fc32.src.rpm

* Wed Jan 20 2021 Simone Caronni <negativo17> - 0.21.0-2
- Update to 0.21.0.
- Remove java build requirement.
- Use local folder for test output.

(In reply to Oleg Girko from comment #45)
> - Update to 0.21.0.

For the 0.21 release something has changed in the boost code, so Boost as provided by the base distribution in CentOS/RHEL 7 is no longer enough.

I'm struggling on how to make the build system consider -I%{_includedir}/boost169 both at %configure time and at make time.

If I add the BOOST_CPPFLAGS I can pass configure but not the build, if I add it to CXXFLAGS or CPPFLAGS it breaks the configure detection (with or without BOOST_CPPFLAGS declared).

Any hint? Everything is in the SPEC file above.

Sorry I've missed this discussion. Please do not proceed with this package in Fedora until we've had a chance to speak. I am concerned that we have a long-term plan in line with upstream's recommendations.

Concerned parties please contact me at warren on Freenode or wtogami. Let's schedule a meeting to discuss this. See historical discussion in Bug #1020292.

> - permissions of /etc/bitcoin folder (0750) too strict - ordinary users cannot view conf

You mean bitcoin.conf? You absolutely do not want other users to be able to read that. It can contains secrets.

Overall I have concerns that this package shoehorns non-default config and datadir. This is not the "normal" way of using bitcoind. This package entering the repository could be a surprise to some with several external repo packages that people have installed in the past years.

I wrote related concerns of packaging conflicts at https://bugzilla.redhat.com/show_bug.cgi?id=1020292#c45

> # FIXME This is less than ideal, but until dwalsh gives me a better way...

Mitigating factor is if all the non-upstream-default stuff is in an optional subpackage. In your case -server does seem to self-contain the .service, selinux, non-default config and non-default datadir. If people want to use it in the upstream way they could install only the main package. (I have not fully reviewed if these things are fully contained in the -server subpackage.)

> * Wed Jul 22 2020 Simone Caronni <negativo17> - 0.20.0-7
> - Use libdb 5.x instead of deprecated 4.x. Fixes build on RHEL/CentOS 8.

Sorry please do not do this. Upstream strongly recommends against using DB5 for a reason.

db4 is the official upstream wallet.dat format. They plan to migrate away from db4 to sqlite in the next year. Building against db5 is in the build system as only a convenience but it is strongly discouraged all these years because db5 wallet.dat is not supported by the upstream distribution, and it could leave users stuck without a supported migration path.

https://salsa.debian.org/debian/guix/-/tree/debian/devel/debian
I'm told this is how Debian packaged Guix. It appears to be a proper bootstrap starting from guile.

oops wrong ticket

> For the 0.21 release something has changed in the boost code, so Boost as provided by the base distribution in CentOS/RHEL 7 is no longer enough.

I have a plan to fix this and more for RHEL7+ and Fedora in a uniform way. It might take a few weeks for this to be ready. I can explain it sooner if you would like to talk.

Sorry for injecting myself into this after you've put half a year of work into this. You disregarded critical warnings in Bug #1020292 as to why packaging this is hazardous. There are risks you are not familiar with as to why historically none of the leading distros (Fedora, Debian, Ubuntu) have packaged Bitcoin Core. Tldr: Builds dynamic linked to system libraries have previously been vulnerable to catastrophic network divergence. Distribution packages wouldn't be dangerous if only a few people used it. But should they become the most common way of using Bitcoin Core then it would be a systemic risk. This was not only a hypothetical problem. BIP66 is one such historical example that could have been exacerbated by distro packages becoming the most common way to run Bitcoin Core full nodes.

The safer way for downstream distributions to handle this not become possible until recent upstream work (Guix-related). There's three step needed to make this usable for Fedora/RHEL.

1) Guix-based deterministic builds of Bitcoin Core to become the official release process (replacing their previous Ubuntu-based Gitian). This work is now 99% complete.

2) Add rpmbuild to upstream's Guix build process. It would generate deterministic binary RPMS alongside their binary tarballs.

https://salsa.debian.org/debian/guix/-/tree/debian/devel/debian
https://github.com/pjotrp/guix-notes/blob/master/GUIX-NO-ROOT.org

3) Package Guix for Fedora much in the same way as Debian did it. This would allow us to have a known deterministic build system that is capable of building identical binaries.

I did not appreciate how you closed Bug #1020292 and disregarded the warnings written there. Out of respect I am not unilaterally closing this bug.

Step #2 above is an opportunity to collaborate. I assigned one of my engineers to work on this. We should collaborate on what exactly we want to be in a bitcoincore RPM package. For example instead of your -server package we may want to consider systemd service @ units as an official way to configure and launch multiple nodes.

First of all, I've been using these packages myself for years, but I'm all in for one that is fine by everyone, including your concerns.

(In reply to Warren Togami from comment #52)
> I did not appreciate how you closed Bug #1020292 and disregarded the
> warnings written there. Out of respect I am not unilaterally closing this
> bug.

The original ticket has been closed according to guidelines and the original poster even jumped in, so respect or not, this does not give you any entitlement to close tickets arbitrarily.

All you've written it's all nice but I'm not even sure if builds based on GUIX that don't follow guidelines are allowed in Fedora/EPEL. Have you checked that? If not, that as well would grant people to ignore your comments and just proceed.

> Step #2 above is an opportunity to collaborate. I assigned one of my
> engineers to work on this. We should collaborate on what exactly we want to
> be in a bitcoincore RPM package. For example instead of your -server package
> we may want to consider systemd service @ units as an official way to
> configure and launch multiple nodes.

I'm fine with your proposal, but the original bug report for the review is open since 2013, and apart high level things that should be done or not done I've not seen much from you on the topic.
If you think the various components (guix etc.) are mature enough why don't you post a review for guix, a guix-based RPM or the parts that in your opinion are important? You can take over, I don't mind. I can take the reviews straight away if you're fine so we can get this sorted out. Even if the various components are not 100% ok like you said, already it's a start.

I think that threat of unintentional fork due to slightly different versions of system libraries is significantly exaggerated. So far this threat is purely theoretical, there were no incidents related to that. The problem fixed with BIP66 was caused not by variations of OpenSSL behaviour between different versions, but by using OpenSSL incorrectly in Bitcoin Core client.

https://github.com/bitcoin/bips/blob/master/bip-0066.mediawiki

Relying on specific implementation details of particular versions of libraries is harmful because it introduces undocumented implicit consensus rules nobody knows about. And requiring using one true build of a single client does not help at all, it just hides a ticking bomb until it explodes later. An incident with unintended hard fork when migrating Bitcoin Core client from BerkleyDB to LevelDB has shown this quite prominently.

https://github.com/bitcoin/bips/blob/master/bip-0050.mediawiki

Another lesson from this incident is that unintended forks caused by bugs in consensus rules do not cause catastrophic consequences and are resolved very quickly. And I'm pretty sure that they are beneficial in the long run because they fix hidden critical bugs that (also theoretically) can be exploited by malicious actors.

Approach taken by Ethereum is much healthier. They not only don't require one true build of a single client, but they even encourage multiple client implementations. Consensus rules of Ethereum 2 beacon chain make using the most popular validator client riskier than less popular ones: more clients misbehave synchronously, bigger penalty they get. Of course, this doesn't guarantee from unintended forks caused by bugs of consensus implementation. But this causes most of these bugs to be caught early in testnet.

Hence, building Bitcoin client with system libraries is only slightly riskier than using one true build, but this risk is very minor (and only theoretical so far), and this is acceptable risk. If Linux distributions adopt this approach, this will be beneficial in the long run because it will help to detect and fix bugs in consensus implementation earlier.

I'm voting for adding properly built (using system libraries) Bitcoin Core client to Fedora and accepting minor risks caused by this for bigger benefit of Bitcoin ecosystem's diversity.

And to show that I've already put my money where my mouth is, I use Dash Core client built the same way for almost 4 years without a single incident.

https://obs.infoserver.lv/package/show/cryptocurrency/dash-core

(Dash Core codebase is originally a fork of Bitcoin's one.)

Oleg wrote:
> So far this threat is purely theoretical, there were no incidents related to that. 

This is incorrect. Some alt coins like PPC suffered exactly a catastrophic fork because they disregarded warnings in this regard.

The diversity argument is counter to the opinion of among Bitcoin Core developers. Debating the diversity of implementations issue is not productive here. I ask that folks please defer to upstream Bitcoin Core developer opinions on the wisdom and safety of how to ship Bitcoin Core in downstream distributions.

Simone wrote:
> I'm fine with your proposal, but the original bug report for the review is open since 2013, and apart high level things that should be done or not done I've not seen much from you on the topic.

Upstream did not have a feasible build method until recently. I am sorry this took so long. I also recognize this is extremely weird compared to the normal way software is built in Fedora. The upstream developers felt so strong about this that it was preferable to have no Bitcoin in the leading Linux distros all these years. Now however we are close to a satisfactory solution. I ask for a bit more patience.

(In reply to Warren Togami from comment #55)
> Oleg wrote:
> > So far this threat is purely theoretical, there were no incidents related to that. 
> 
> This is incorrect. Some alt coins like PPC suffered exactly a catastrophic
> fork because they disregarded warnings in this regard.

I was not talking about minor altcoins with very low adoption rate. They suffer from more serious problems, like being vulnerable to 51% attacks. Also, they suffer from lack of developer resources and accumulate technical debt much quicker as a result. Hence, I'm not surprised that among thousands of altcoins you can find one that suffered because of this problem. You can find even more altcoins that suffered much worse problems because of bugs in their code anyway.

> The diversity argument is counter to the opinion of among Bitcoin Core
> developers.

Yes, I know. But this doesn't automatically mean that Bitcoin Core developers are right. Ethereum developers have radically different opinion, for example.

> Debating the diversity of implementations issue is not
> productive here.

I'm not debating diversity of implementations. I'm just using this argument to point to the fact that insisting on using particular versions of libraries is not an acceptable way to build reliable software. Unfortunately, it's becoming quite common mindset among developers: "I've tested my program with these particular versions of libraries, hence everyone should use them". Yes, this makes life easier for developers, but complicates life for everyone else.

There is a good reason Fedora has a strong attitude against bundling of libraries. Upstream developers are usually not proficient enough to update all libraries they use in a timely manner. Maintainers of library packages are more proficient in this. If a remotely exploitable vulnerability is found in one of many libraries Bitcoin Core uses, it will take much more time for new version of Bitcoin Core to be released and packaged than for new version of this particular library packaged in Fedora. Very often Fedora gets patched package even before upstream releases a new version.

Hence, if my argument about beneficial diversity is not enough for you, I have a much stronger argument: vulnerability in a bundled library.

Remotely exploitable vulnerability is a true nightmare scenario for a cryptocurrency software, because it can lead to massive theft of funds. Unintended forks are just a nuisance compared to that. Even large-scale fork can lead just to temporary outage and increased time to reach consensus. Some transactions may be remaining in the mempool for a long time or even lost, so they have to be repeated later, but it's very unlikely that funds will be lost or stolen as a result. Single cases of double spend that can happen during an outage like this (that will be resolved when the fork is over anyway) are nothing compared to mass theft of funds.

Considering a very minor fraction of nodes that will run Fedora-packaged Bitcoin Core, a theoretical fork involving these nodes will not be even large-scale. In worst case scenario, small percentage of nodes will be temporarily banned from the network until the problem is fixed. And probability of this is still lower than an unpatched vulnerability in a library Bitcoin uses. And packaging procedures in Fedora are well suited exactly to mitigate risks like this.

> I ask that folks please defer to upstream Bitcoin Core
> developer opinions on the wisdom and safety of how to ship Bitcoin Core in
> downstream distributions.

As a former Dash Core developer, I strongly disagree. Bitcoin Core deveopers' "wisdom" prevented Bitcoin Core client from being shipped in downstream distributions at all for 12 years already.

> Simone wrote:
> > I'm fine with your proposal, but the original bug report for the review is open since 2013, and apart high level things that should be done or not done I've not seen much from you on the topic.
> 
> Upstream did not have a feasible build method until recently. I am sorry
> this took so long. I also recognize this is extremely weird compared to the
> normal way software is built in Fedora. The upstream developers felt so
> strong about this that it was preferable to have no Bitcoin in the leading
> Linux distros all these years. Now however we are close to a satisfactory
> solution. I ask for a bit more patience.

I don't think that building with all libraries bundled is satisfactory solution. At least, not for me. So, let's package Bitcoin Core client according to Fedora guidelines for now. However, I'm not against providing a bundled version as another option when it's available. I'm just against having this as the only option (and no option at all for quite a long time before it's available). Let users decide which slightly increased risk they prefer to take: minor fork and temporary ban or remotely exploitable vulnerability.

There were multiple misstatements of fact in what you wrote that I will not dissect here because this is the wrong forum for ideological debate between competitors. 

The only question here is if Fedora will follow Bitcoin Core upstream's safety recommendations. It was not possible in past years but now the toolchain is 99.99% ready. I'm working on this.

Simone, I'm interested in collaborating with you.

> Yes, I know. But this doesn't automatically mean that Bitcoin Core developers are right. Ethereum developers have radically different opinion, for example.

Ethereum has had numerous emergency updates due to catastrophic errors through the years. Bitcoin has had significantly fewer emergency updates. You call this a difference of opinion. We call this wisdom.
Let's not go deeper into the ideological debate.

Regarding the bundled library argument I was unfair that I did not respond. I am well aware of the bundled library reasoning. I am among the people who wrote Fedora's packaging standards and zealously defended it years ago. There is more recently an exceptions process where bundled libraries are allowed in Fedora. Bitcoin Core through the years has been working towards progressively eliminating external library dependencies because they have been the source of most dangerous vulnerabilities. That being said the libraries that it continues to rely upon are carefully tracked by the upstream project.

https://fedoraproject.org/wiki/Bundled_Libraries?rd=Packaging:Bundled_Libraries
Fedora now has a process to help keep track of bundled libraries. This package will follow these guidelines and anything else required by FESCO.

Seems hardcoded selinux dependency not resolved.
What about moving selinux things into -selinux subpackage?

(In reply to Robert-André Mauchin 🐧 from comment #7)
> Also create a logrotate file for the log:
> https://docs.fedoraproject.org/en-US/packaging-guidelines/
> #_logrotate_config_file

And this is not solved yet too.

> %package server
> Requires:           %{name}-utils%{_isa} = %{version}

Really?

(In reply to Eugene A. Pivnev from comment #60)
> (In reply to Robert-André Mauchin 🐧 from comment #7)
> > Also create a logrotate file for the log:
> > https://docs.fedoraproject.org/en-US/packaging-guidelines/
> > #_logrotate_config_file
> 
> And this is not solved yet too.

I replied here, not planning to change it at the moment: https://bugzilla.redhat.com/show_bug.cgi?id=1834731#c19

(In reply to Eugene A. Pivnev from comment #61)
> > %package server
> > Requires:           %{name}-utils%{_isa} = %{version}
> 
> Really?

Well, to interact with the daemon you require the cli, but is not strictly required per se. I've removed the requirement.

(In reply to Eugene A. Pivnev from comment #59)
> Seems hardcoded selinux dependency not resolved.
> What about moving selinux things into -selinux subpackage?

Replied here: https://bugzilla.redhat.com/show_bug.cgi?id=1834731#c9

(In reply to Simone Caronni from comment #64)
> (In reply to Eugene A. Pivnev from comment #59)
> > Seems hardcoded selinux dependency not resolved.
> > What about moving selinux things into -selinux subpackage?
> 
> Replied here: https://bugzilla.redhat.com/show_bug.cgi?id=1834731#c9

Nevermind, saw a note in the package guidelines, will separate it from the main package.

*** Bug 1020292 has been marked as a duplicate of this bug. ***

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.21.0-3.fc32.src.rpm

* Wed Mar 10 2021 Simone Caronni <negativo17> - 0.21.0-3
- Remove requirements for utils subpackage in server subpackage.
- Separate SELinux package in its own subpackage and use RPM rich booleans on
  Fedora and RHEL/CentOS 8+ to install the SELinux package if the base policy is
  installed.
- Update server README.

Separate SELinux policy package here: https://bugzilla.redhat.com/show_bug.cgi?id=1937302

Spec URL: https://slaanesh.fedorapeople.org/bitcoin.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-0.21.0-4.fc33.src.rpm

* Wed Mar 10 2021 Simone Caronni <negativo17> - 0.21.0-4
- Fix build on RHEL/CentOS 8.
- Adjust SELinux requirement for server subpackage.

I looked into packaging Guix for Fedora. It would be possible but difficult. For now I give up on the reproducible build goal as that is a problem that needs to be solved for the entire Fedora build system.

These are my remaining recommendations to align best with upstream's intent.

* Fedora's package should be named "bitcoincore". It should conflict with "bitcoin". This would allow a popular feature-fork "bitcoinknots" would have the same binary and configuration files and would thus conflict with these other names.
* Ask FESCO to disallow any package named "bitcoin". There are multiple reasons for this including unexpected upgrade conflicts with ways it was previously packaged. It is also convenient for distributors to entirely sidestep political fights over what has the right to be called "bitcoin".
* Less important: Another upstream concern is the risk of old bitcoin binaries in the wild when Fedora goes EOL. The simplest safeguard is to ship a final RPM update before a Fedora release's EOL that simply removes the binary. We would ask FESCO if they're OK with this.

Thoughts?

FYI: Years ago the linked library dependencies were a terrible risk of causing consensus failure. It was beyond hypothetical risk, it actually happened to unmaintained clones who failed to heed CVE's. That previous risk was mostly mitigated by the removal of openssl. Upstream aims to eventually eliminate the boost dependency which would further reduce risk. In any case the risk is low enough now that it might be OK to ship in downstream distros. Don't mistake this as endorsement. I intend for upstream to distribute a reproducibly built RPM that would Epoch override the Fedora package for those who prefer static libraries exactly as tested by upstream. Upstream opposes automatic upgrades of Bitcoin Core so this would be a way for Fedora users to opt-in to upstream's recommended deployment method. This isn't Fedora's concern but just explaining the line of reasoning here.

(In reply to Warren Togami from comment #70)
> * Fedora's package should be named "bitcoincore". It should conflict with
> "bitcoin". This would allow a popular feature-fork "bitcoinknots" would have
> the same binary and configuration files and would thus conflict with these
> other names.

Fine, I will make the necessary adjustments.

> * Ask FESCO to disallow any package named "bitcoin". There are multiple
> reasons for this including unexpected upgrade conflicts with ways it was
> previously packaged. It is also convenient for distributors to entirely
> sidestep political fights over what has the right to be called "bitcoin".

Fine for me as well.

> * Less important: Another upstream concern is the risk of old bitcoin
> binaries in the wild when Fedora goes EOL. The simplest safeguard is to ship
> a final RPM update before a Fedora release's EOL that simply removes the
> binary. We would ask FESCO if they're OK with this.

I think this solves nothing (someone could still avoid installing the updates, fetching an old package or we could just miss a release retirement because of a person being on holiday, etc. Same applies for manually installed binaries, there is no way to enforce that. If you want it I'm fine with that, but again I think it's completely useless.

> FYI: Years ago the linked library dependencies were a terrible risk of
> causing consensus failure. It was beyond hypothetical risk, it actually
> happened to unmaintained clones who failed to heed CVE's. That previous risk
> was mostly mitigated by the removal of openssl. Upstream aims to eventually
> eliminate the boost dependency which would further reduce risk. In any case
> the risk is low enough now that it might be OK to ship in downstream
> distros.

I'm fine with this as well and will try to keep it in sync as possible with these dependencies slated for removal.

> Don't mistake this as endorsement. I intend for upstream to
> distribute a reproducibly built RPM that would Epoch override the Fedora
> package for those who prefer static libraries exactly as tested by upstream.
> Upstream opposes automatic upgrades of Bitcoin Core so this would be a way
> for Fedora users to opt-in to upstream's recommended deployment method. This
> isn't Fedora's concern but just explaining the line of reasoning here.

I'm willing to keep it updated / in sync with upstream for the time being, so we can ensure a smooth upgrade path for whoever wants to use official binaries. Count me in if you think I should contribute to the upstream's SPEC file and SELinux policies. I'm fine also in integrating Guix as conditionals in the SPEC file to allow building from the same SPEC file (e.g. hosted upstream, non-guix build in fedora, higher epoch and guix build upstream).

Are you ok with these points?

Current package (0.21.0-4) _requires_ selinux subpackage.
So - no way to install bitcoin server without selnux.
What about 'Recommends' or 'Suggests'?

(In reply to Eugene A. Pivnev from comment #72)
> Current package (0.21.0-4) _requires_ selinux subpackage.
> So - no way to install bitcoin server without selnux.
> What about 'Recommends' or 'Suggests'?

Well it's requiring it only if the selinux-policy-targeted package is installed, otherwise not even suggesting it. I was using this guide which is still in draft: https://fedoraproject.org/wiki/PackagingDrafts/SELinux_Independent_Policy#Adding_dependency_to_the_spec_file_of_corresponding_package

Do we have something a bit more up to date or official than this?

Points of Agreement:
* Rename package to "bitcoincore"
* Conflicts: bitcoin
* Ask FESCO to disallow any package named "bitcoin".

Advice:
* EL7's boost is too old while EL8+ and Fedora are easy to support. Let's drop EL7 and documentation should say to use bitcoincore.org's builds on EL7. The plan to maintain a work-alike package suitable for upstream builds so this should be convenient enough for end-users.
* How about rename the 'core' subpackage to 'qt' since that is what it contains?
* bitcoin-wallet is actually a utility? It isn't a required part for the server to function.

Hard Line:
* The official supported file format of wallet.dat is db4. "./configure --with-incompatible-bdb" is named as such because it is not intended to be used by distributors. It exists as an unsupported option so the software is possible to use where db4 does not conveniently exist. I strongly advise against shipping an incompatible data format in a package that may be widely used as it will be an inevitable point of support confusion. The software should work identically when switching between Fedora or upstream's build. The only maintainable choice here is to ship exactly the db4 that upstream maintains and tests against within this package. Future Bitcoin Core will migrate to a different database but it will forever need to link to db4 to allow for automatic wallet migration.

Regarding README.utils.redhat:
"By default bitcoin-cli looks for configuration at /etc/bitcoin/bitcoin.conf. This must be readable only by users authorized to communicate with bitcoind."

These binaries should behave the same as upstream's builds. The $HOME/.bitcoin/bitcoin.conf default path should remain supported so it behaves identically to upstream builds. I haven't tested if this isn't the case now.

Other suggestions for further discussion:
* server vs core subpackage kind of bother me because while you don't use bitcoind and bitcoin-qt simultaneously they are functionally identical. I don't know what to suggest about this.
* The bitcoin.service file bothers me in that a single system service is one of many ways in which bitcoind is used. I'd prefer if it was a .service type that allowed for multiple admin "@" definable instances. There could also be a different .service file like "bitcoincore-homedir-service" that uses $HOME/.bitcoin as the datadir in the way many have already deployed the upstream binary? It is also my strong preference for the different types of .service files to be in their own subpackage. That would also help the above concern regarding a single hard-coded /etc/bitcoin/bitcoin.conf not being how many use this software.

Is there a benefit to 'utils' being a subpackage separate from `bitcoind`? Many expect -cli to exist in order to manually send commands to their bitcoind so it is unexpected for it to be installed separately IMO.

I would expect the only reason to not ship them together is if it bloats a minimal server with deps but that isn't the case?

The test suite is *excessively slow*. IMO we should not run these extensive tests during these builds. Upstream official builds do not. We developers should instead run these tests on the target platforms independently.

Regarding bitcoin.service:

Restart=on-failure
TimeoutStopSec=120
TimeoutStartSec=60
StartLimitInterval=240
StartLimitBurst=5

There exists a corner case where "stop" can take significantly more time than any amount hardcoded here. Force kill in that situation causes data corruption and an even more time consuming recovery the next time indexing begins again. I asked my engineers here if they came up with a better way to detect a safe stop.

(In reply to Warren Togami from comment #74)
> * server vs core subpackage kind of bother me because while you don't use
> bitcoind and bitcoin-qt simultaneously they are functionally identical.

No.
Bitcoind runs as service by 'bitcoin' user.
Bitcoin-qt is running by ordinary user with his creditentials.
PS. users that are not familiar with this system can run both of them and create two bitcoin services simultaneously but this is another question.

(In reply to Warren Togami from comment #75)
> Is there a benefit to 'utils' being a subpackage separate from `bitcoind`?
> Many expect -cli to exist in order to manually send commands to their
> bitcoind so it is unexpected for it to be installed separately IMO.
> 
> I would expect the only reason to not ship them together is if it bloats a
> minimal server with deps but that isn't the case?

E.g. I run bitcoind at one host (to store this huge database inplace) and use bitcoin-cli in another hosts (without any bitcoin service).

> E.g. I run bitcoind at one host (to store this huge database inplace) and use bitcoin-cli in another hosts (without any bitcoin service).

OK good point.

(In reply to Warren Togami from comment #77)
> Regarding bitcoin.service:
> 
> Restart=on-failure
> TimeoutStopSec=120
> TimeoutStartSec=60
> StartLimitInterval=240
> StartLimitBurst=5
> 
> There exists a corner case where "stop" can take significantly more time
> than any amount hardcoded here. Force kill in that situation causes data
> corruption and an even more time consuming recovery the next time indexing
> begins again. I asked my engineers here if they came up with a better way to
> detect a safe stop.

PrivateTmp=true
TimeoutStopSec=1200s
TimeoutStartSec=5s
# Fail when 10 tries within 1 minute fail (never)
StartLimitInterval=60s
StartLimitBurst=10
# Limit attempts to 1 per 10 seconds
Restart=always
RestartSec=10

This is what we've been using in prod here. We didn't find a more intelligent solution. Under normal operation it stops relatively fast but in rare situations it needs the extra time to cleanly shut down.

Will respond and continue during the week.

Thanks.

bitcoin-22.0 released

Sorry had some real life tasks that I had to deal with. Posting updated packages later today.
Bitcoin release 22 (versioning change) has also a different tarball signature verification procedure, so I'm introducing that as well.

I'll reply in the detail after.

Well, it's more complicated than it seems:

https://github.com/bitcoin/bitcoin/tree/master/contrib/builder-keys

I need to download all of those keys in one GPG keyring and then verify the signatures inside the SPEC file. I'm thinking of adding a separate script and waiting until the official instructions go live:

https://github.com/bitcoin-core/bitcoincore.org/commit/4bf8149e600ce9eb044dd0b87726e8341521883b

Release 22.0 is not live yet on bitcoincore.org with the new instructions at the time of writing this.

Well not on bitcoin.org, but is live on bitcoincore.org: https://bitcoincore.org/en/download/

(In reply to Simone Caronni from comment #86)
> Well not on bitcoin.org, but is live on bitcoincore.org:
> https://bitcoincore.org/en/download/

I believe bitcoincore.org is the canonical place maintained by core devs, whereas bicoin.org is maintained by Cøbra Bitcoin and other contributors.  The key point is who controls the domain name.  The releases are made available on bitcoincore.org, which is mirrored on bitcoin.org at some point.  It has been this way for several years.  The download link from the github repo also points to bitcoincore.org.

(In reply to Suvayu from comment #87)
> (In reply to Simone Caronni from comment #86)
> > Well not on bitcoin.org, but is live on bitcoincore.org:
> > https://bitcoincore.org/en/download/
> 
> I believe bitcoincore.org is the canonical place maintained by core devs,
> whereas bicoin.org is maintained by Cøbra Bitcoin and other contributors. 
> The key point is who controls the domain name.  The releases are made
> available on bitcoincore.org, which is mirrored on bitcoin.org at some
> point.  It has been this way for several years.  The download link from the
> github repo also points to bitcoincore.org.

Good point, I've switched only to bitcoincore.org for all references in the SPEC file.

So here's the issue with the signature.

1- The tarball contains a file with all the PGP keys used to sign SHA256SUM (contrib/builder-keys/keys.txt).
2- The signatures are all in SHA256SUM.asc.
3- The keys can be on keyserver.ubuntu.com, keys.openpgp.org or both.
4- All keys need to be downloaded and put into the public keyring, or the signature verification fails (return code 2 instead of 0).
5- Some keys MIGHT be revoked when downloading them for the first time (which is the case now).

So verifying signatures always fails with the bundled key file, to make it right someone needs to clean the SHA256SUM.asc files of all the signatures made with revoked files, which in the end does not match with what is downloaded.

Steps to reproduce:

tar -xzf bitcoin-22.0.tar.gz --strip-components=3 bitcoin-22.0/contrib/builder-keys/keys.txt

rm -f bitcoin-22.0.gpg

while read fingerprint keyholder_name; do
  gpg2 -q --no-default-keyring --keyring ./bitcoin-22.0.gpg --keyserver hkps://keyserver.ubuntu.com --recv-keys ${fingerprint}
  gpg2 -q --no-default-keyring --keyring ./bitcoin-22.0.gpg --keyserver hkps://keys.openpgp.org --recv-keys ${fingerprint}
  gpg2 --no-default-keyring --keyring ./bitcoin-22.0.gpg --export --export-options export-minimal ${fingerprint} >> bitcoin-22.0-pubring.gpg
done < keys.txt

rm -fr keys.txt

gpgv2 -q --keyring=`pwd`/bitcoin-22.0-pubring.gpg  SHA256SUMS.asc SHA256SUMS

Any idea how to solve it?

- Editing SHA256SUM.asc to remove signatures with revoked keys involves prior manual work and then does not match the one included in the release folder.
- Just checking all keys throws an error due to revoked keys.
- Just checking one key throws an error due to missing keys (same as above).

I think the best is to add a script like the one above and add a comment in the SPEC file on how to verify the signature and not actually check it. This means the package maintainer must do the proper due diligence before pushing sources to the lookaside cache.

Basically verification requires at least one signature to be valid.

I will pipe the output of a script like the one above and just grep for at least one occurrence of "^gpgv: Good signature from".

This way, also in the future, after lots of revocations, as long as one of the signatures is valid the release can be verified, and the one "grepped" can also be different every time.

Hi Simone,

Here's my suggestion:

1. the instructions on bitcoincore.org say "7. It is recommended that you choose a few individuals from this list who you find trustworthy and import their keys ..."
2. I suggest you choose the person who signed and tagged the release.

To do that, you can do something like this:

  $ git clone https://github.com/bitcoin/bitcoin.git 
  $ cd bitcoin.git
  $ grep --color=never $(git log -1 --format=%aL v22.0) contrib/builder-keys/keys.txt
  71A3B16735405025D447E8F274810B012346C9A6 Wladimir J. van der Laan (laanwj)

You can then verify with only this key.

This should make the verification much simpler, what do you think?

Cheers,

(In reply to Suvayu from comment #92)
>   $ cd bitcoin.git

should be `$ cd bitcoin`

(In reply to Suvayu from comment #92)
> 2. I suggest you choose the person who signed and tagged the release.

Very good idea! Thanks!

Well, on second thought this key might as well be revoked after a release and there is no .git information in the release tarball to derive it from, so anyway would need a prior script to execute before the build runs in koji/mock without network.
I'll come up with something.

(In reply to Suvayu from comment #92)
> You can then verify with only this key.

...and it would require editing the asc file to remove all other signatures or gpgv will complain anyway.

I think you can use the GH public API: https://api.github.com/repos/bitcoin/bitcoin/releases

Something like the following in a script should work; I'm using curl and python (no build dependencies)

  $ curl https://api.github.com/repos/bitcoin/bitcoin/releases > bitcoin-releases.json
  $ python 
  >>> import json
  >>> with open("bitcoin-releases.json") as f:
  ...     data = json.load(f)
  ...
  >>> data[0]["author"]["login"]
  'laanwj'
  >>> data[0]["tag_name"]
  'v22.0'

If it's okay to add `jq` as a build dependency, you could also device a oneliner by piping the output of curl.  Using the above JSON file, the following works:

  $ cat bitcoin-releases.json | jq '.[0]["author"]["login"], .[0]["tag_name"]'
  "laanwj"
  "v22.0"

This does require network being enabled during the prebuild stage, does that match with how koji is setup?  I would imagine it works, because downloading source tarballs is a common requirement.

Hope this helps

(In reply to Simone Caronni from comment #96)
> (In reply to Suvayu from comment #92)
> > You can then verify with only this key.
> 
> ...and it would require editing the asc file to remove all other signatures
> or gpgv will complain anyway.

I think the expectation is to filter for the key you are using to verify.  So just grepping for the verified signature should be okay.

  $ gpg --keyserver hkp://keyserver.ubuntu.com --recv-keys \
      $(grep --color=never laanwj keys.txt | cut -d' ' -f1)
  $ gpg --verify SHA256SUMS.asc |& grep -C 2 'Good signature'
  gpg: Signature made Friday 10 September 2021 07:33:30 PM CEST
  gpg:                using RSA key 9DEAE0DC7063249FB05474681E4AED62986CD25D
  gpg: Good signature from "Wladimir J. van der Laan <laanwj>" [unknown]
  gpg:                 aka "Wladimir J. van der Laan <laanwj>" [unknown]
  gpg:                 aka "Wladimir J. van der Laan <laanwj>" [unknown]

Ideally failures should go to stderr, and success to stdout, but it seems both go to stderr. I wish gpg had an option to separate the failures (maybe there is, my quick look in the man page didn't turn up anything).

Maybe this helps

Hey, thanks for all the help, but I think I solved it while making it an offline process for mock/koji.

All changes that I will post for review are here: https://github.com/negativo17/bitcoin-core/commits/master

In particular: https://github.com/negativo17/bitcoin-core/commit/b2763c75931fbac5eebd4838ae549e642c2885bd

Will be good to download fresh srpm. I build f34.rpm for myself (periodically)

(In reply to Simone Caronni from comment #99)
> 
> In particular:
> https://github.com/negativo17/bitcoin-core/commit/
> b2763c75931fbac5eebd4838ae549e642c2885bd

Looks good +1

(In reply to Warren Togami from comment #74)
> Points of Agreement:
> * Rename package to "bitcoincore"

Actually "bitcoin-core", but same result.

> * EL7's boost is too old while EL8+ and Fedora are easy to support. Let's
> drop EL7 and documentation should say to use bitcoincore.org's builds on
> EL7. The plan to maintain a work-alike package suitable for upstream builds
> so this should be convenient enough for end-users.

The only problem is how Boost is searched, it expects ../include/boost from where the libraries are found. Theoretically it should work with a bit of fiddling but there are too many compromises anyway, so it has been dropped.

> * How about rename the 'core' subpackage to 'qt' since that is what it
> contains?

I've renamed it to 'desktop', so now all RPMS are subpackages of the main 'bitcoin-core' name for clarity (bitcoin-core-desktop).

> * bitcoin-wallet is actually a utility? It isn't a required part for the
> server to function.

It's a tool that might come handy and allows manipulating and creating wallets from the command line, so I'm not deleting it.

> "By default bitcoin-cli looks for configuration at
> /etc/bitcoin/bitcoin.conf. This must be readable only by users authorized to
> communicate with bitcoind."
> 
> These binaries should behave the same as upstream's builds. The
> $HOME/.bitcoin/bitcoin.conf default path should remain supported so it
> behaves identically to upstream builds. I haven't tested if this isn't the
> case now.

As already discussed this is the default. By default it looks in /etc/bitcoin/bitcoin.conf and if not found it looks in $HOME/.bitcoin/bitcoin.conf, so no changes here.

> * server vs core subpackage kind of bother me because while you don't use
> bitcoind and bitcoin-qt simultaneously they are functionally identical. I
> don't know what to suggest about this.

Already answered above, server is headless and might have client on another system, the main QT application is for desktops, so makes sense to have them as three separate packages depending on usage.

> * The bitcoin.service file bothers me in that a single system service is one
> of many ways in which bitcoind is used. I'd prefer if it was a .service type
> that allowed for multiple admin "@" definable instances. There could also be
> a different .service file like "bitcoincore-homedir-service" that uses
> $HOME/.bitcoin as the datadir in the way many have already deployed the
> upstream binary? It is also my strong preference for the different types of
> .service files to be in their own subpackage. That would also help the above
> concern regarding a single hard-coded /etc/bitcoin/bitcoin.conf not being
> how many use this software.

That's actually not true, in the source code you also have a sample systemd unit with explicit hardcoded paths.
I will investigate if it's feasible to include _also_ systemd user instantiable units in the server package without requiring to destroy completely the SELinux policies.

(In reply to Warren Togami from comment #76)
> The test suite is *excessively slow*. IMO we should not run these extensive
> tests during these builds. Upstream official builds do not. We developers
> should instead run these tests on the target platforms independently.

The tests make sure that the binaries built are working, if they fail some test then the package build fails. They are not part of what the users see.
I would say they are CRITICAL to make sure you have fully functioning binaries.

> Hard Line:
> * The official supported file format of wallet.dat is db4. "./configure
> --with-incompatible-bdb" is named as such because it is not intended to be
> used by distributors. It exists as an unsupported option so the software is
> possible to use where db4 does not conveniently exist. I strongly advise
> against shipping an incompatible data format in a package that may be widely
> used as it will be an inevitable point of support confusion. The software
> should work identically when switching between Fedora or upstream's build.
> The only maintainable choice here is to ship exactly the db4 that upstream
> maintains and tests against within this package. Future Bitcoin Core will
> migrate to a different database but it will forever need to link to db4 to
> allow for automatic wallet migration.

BDB 4.8 is not in fact "conveniently existing" in Fedora, it has been retired for quite some time, it's dated 2010-04-12 (!) and requires some effort to actually bundle: https://github.com/bitcoin/bitcoin/blob/master/contrib/install_db4.sh
I will see what I can do, but there's not even guarantee it builds with recent compilers in Fedora.

> * Conflicts: bitcoin
> * Ask FESCO to disallow any package named "bitcoin".

Not asked yet.

(In reply to Warren Togami from comment #81)
> (In reply to Warren Togami from comment #77)
> > Regarding bitcoin.service:
> > 
> > Restart=on-failure
> > TimeoutStopSec=120
> > TimeoutStartSec=60
> > StartLimitInterval=240
> > StartLimitBurst=5
> > 
> > There exists a corner case where "stop" can take significantly more time
> > than any amount hardcoded here. Force kill in that situation causes data
> > corruption and an even more time consuming recovery the next time indexing
> > begins again. I asked my engineers here if they came up with a better way to
> > detect a safe stop.
> 
> PrivateTmp=true
> TimeoutStopSec=1200s
> TimeoutStartSec=5s
> # Fail when 10 tries within 1 minute fail (never)
> StartLimitInterval=60s
> StartLimitBurst=10
> # Limit attempts to 1 per 10 seconds
> Restart=always
> RestartSec=10
> 
> This is what we've been using in prod here. We didn't find a more
> intelligent solution. Under normal operation it stops relatively fast but in
> rare situations it needs the extra time to cleanly shut down.

Will check along the user systemd unit.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin-core.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-core-22.0-1.fc34.src.rpm

* Wed Sep 22 2021 Simone Caronni <negativo17> - 22.0-1
- Update to 22.0, versioning convention change.
- Implement signature verification with a public GPG keyring and at least one
  valid signature.
- Also the relative selinux package has been renamed to bitcoin-core-selinux.
- Add bitcoin-util to devel subpackage.
- Update docs.
- Add SQLite as dependency for descriptor wallets.
- Drop RHEL/CentOS 7 support.

* Tue Sep 21 2021 Simone Caronni <negativo17> - 0.21.1-2
- Rename package to bitcoin-core.
- Conflicts with bitcoin.
- Desktop subpackage renamed from "core" to "desktop".


TODO:
- Investigate systemd user units along with the system one.
- Check systemd unit parameters.
- Figure out if BDB 4.8 builds in Fedora 35/rawhide and integrate it.

(In reply to Simone Caronni from comment #85)
> I need to download all of those keys in one GPG keyring

I think it may be better to keep each key in a separate file in Git.

Several of those keys will need their expiry dates updated periodically, and any key that expires or gets revoked must be removed from the next release of the package. The keys that are stored in the Fedora Git repository are the trust anchors that the whole verification process hinges on, so changes to the keys should be as transparent as possible. Other packagers should be able to check whether you updated just one key or replaced all the keys with an entirely different set of keys.

With a combined keyring, every change looks like the whole keyring is replaced. Comparing the contents of two keyrings requires some pretty advanced GnuPG expertise. With each key as a separate file, any packager can easily see in the Git diff that one key was updated and all the others are unchanged.

(In reply to Simone Caronni from comment #90)
> - Just checking all keys throws an error due to revoked keys.

According to its manpage, gpgv2 does not check whether keys are revoked or expired. Removing those will be your responsibility when you update the package.

(In reply to Simone Caronni from comment #91)
> I will pipe the output of a script like the one above and just grep for at
> least one occurrence of "^gpgv: Good signature from".

That will miss any signatures that don't match. A few missing keys is not a problem, but if only one signature matches and all the others come out "bad signature", that should be a big flashing red light. If instead most of the signatures are good but one is bad, that's less alarming but should be investigated before the package is distributed. It may indicate that that person shouldn't be trusted to carefully verify releases, so their key should be removed.

(In reply to Suvayu from comment #92)
> 1. the instructions on bitcoincore.org say "7. It is recommended that you
> choose a few individuals from this list who you find trustworthy and import
> their keys ..."
> 2. I suggest you choose the person who signed and tagged the release.

That makes it equivalent to how they did it before, when only one person signed the releases. The whole point of multiple signatures is to not be reliant on a single person.


Here's my proposal for how to check multiple signatures, ignoring missing keys but failing if there are any bad signatures:

Source10: key-Someone.gpg
Source11: key-Somebody.gpg
Source12: key-Other.gpg

%global minimum_good_signatures 10

%prep
# Collect those sources that are keys for signature verification.
for filename in %{sources} ; do case "${filename}" in (*/key-*.gpg) cat "${filename}" ;; esac ; done >combined_keyring.gpg

# Verify the signatures. Write machine-readable results to signature_status. Don't fail if a key is missing.
gpgv2 --status-fd=3 --keyring=./combined_keyring.gpg SHA256SUMS.asc SHA256SUMS 3>signature_status || true

# Fail if there are any signatures that don't match.
! grep --quiet '^\[GNUPG:\] BADSIG ' signature_status

# Fail if fewer than the minimum number of signatures are valid.
test `grep --count '^\[GNUPG:\] GOODSIG ' signature_status` -ge %{minimum_good_signatures}

(In reply to Björn Persson from comment #104)
> Here's my proposal for how to check multiple signatures, ignoring missing
> keys but failing if there are any bad signatures:
> 
> Source10: key-Someone.gpg
> Source11: key-Somebody.gpg
> Source12: key-Other.gpg
> 
> %global minimum_good_signatures 10
> 
> %prep
> # Collect those sources that are keys for signature verification.
> for filename in %{sources} ; do case "${filename}" in (*/key-*.gpg) cat
> "${filename}" ;; esac ; done >combined_keyring.gpg
> 
> # Verify the signatures. Write machine-readable results to signature_status.
> Don't fail if a key is missing.
> gpgv2 --status-fd=3 --keyring=./combined_keyring.gpg SHA256SUMS.asc
> SHA256SUMS 3>signature_status || true
> 
> # Fail if there are any signatures that don't match.
> ! grep --quiet '^\[GNUPG:\] BADSIG ' signature_status
> 
> # Fail if fewer than the minimum number of signatures are valid.
> test `grep --count '^\[GNUPG:\] GOODSIG ' signature_status` -ge
> %{minimum_good_signatures}

Sounds reasonable. I will make sure the script downloads all available keys and puts them into the appropriate files and into the SPEC file.
Maybe 10 signatures is a bit too much, in less than a week from the release there is already a bunch of keys not available. I'll make some tests.

Here is a repository which will contain the latest packages being built as part of the review:

https://negativo17.org/repos/bitcoin/
https://negativo17.org/repos/epel-bitcoin.repo
https://negativo17.org/repos/fedora-bitcoin.repo

Current state is as per my last comment above, so none of the GPG changes just mentioned in it yet.

@Björn Persson here is the updated script and the sample output it generates:

https://github.com/negativo17/bitcoin-core/commit/2546a307014a5ae669bc0128077e07949e85c350

The key names are explicitly saved with no Bitcoin version in the name as they would not necessarily change across releases, so then it's easy to spot differences.
All keys are deleted before regenerating, so at every release it's clear what must go and disappear from git (git status).

$ ./bitcoin-gpg.sh 22.0

Make sure the directory is clean...
Extracting list of GPG keys used to sign the release...
Attempting to download all listed GPG keys for release 22.0...

  keyserver.ubuntu.com - 9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C - Aaron Clauson (sipsorcery)
  keys.openpgp.org     - 9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C - Aaron Clauson (sipsorcery)
  keyserver.ubuntu.com - 617C90010B3BD370B0AC7D424BB42E31C79111B8 - Akira Takizawa (akx20000)
  keys.openpgp.org     - 617C90010B3BD370B0AC7D424BB42E31C79111B8 - Akira Takizawa (akx20000)
  keyserver.ubuntu.com - E944AE667CF960B1004BC32FCA662BE18B877A60 - Andreas Schildbach (aschildbach)
  keys.openpgp.org     - E944AE667CF960B1004BC32FCA662BE18B877A60 - Andreas Schildbach (aschildbach)
  keyserver.ubuntu.com - 152812300785C96444D3334D17565732E08E5E41 - Andrew Chow (achow101)
  keys.openpgp.org     - 152812300785C96444D3334D17565732E08E5E41 - Andrew Chow (achow101)
  keyserver.ubuntu.com - 590B7292695AFFA5B672CBB2E13FC145CD3F4304 - Antoine Poinsot (darosior)
  keys.openpgp.org     - 590B7292695AFFA5B672CBB2E13FC145CD3F4304 - Antoine Poinsot (darosior)
  keyserver.ubuntu.com - 0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8 - Ben Carman (benthecarman)
  keys.openpgp.org     - 0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8 - Ben Carman (benthecarman)
  keyserver.ubuntu.com - 912FD3228387123DC97E0E57D5566241A0295FA9 - BtcDrak (btcdrak)
  keys.openpgp.org     - 912FD3228387123DC97E0E57D5566241A0295FA9 - BtcDrak (btcdrak)
  keyserver.ubuntu.com - C519EBCF3B926298946783EFF6430754120EC2F4 - Christian Decker (cdecker)
  keys.openpgp.org     - C519EBCF3B926298946783EFF6430754120EC2F4 - Christian Decker (cdecker)
  keyserver.ubuntu.com - 18AE2F798E0D239755DA4FD24B79F986CBDF8736 - Chun Kuan Le (ken2812221)
  keys.openpgp.org     - 18AE2F798E0D239755DA4FD24B79F986CBDF8736 - Chun Kuan Le (ken2812221)
  keyserver.ubuntu.com - 101598DC823C1B5F9A6624ABA5E0907A0380E6C3 - CoinForensics (CoinForensics)
gpg: keyserver receive failed: No data
  keys.openpgp.org     - 101598DC823C1B5F9A6624ABA5E0907A0380E6C3 - CoinForensics (CoinForensics)
gpg: key A5E0907A0380E6C3: new key but contains no user ID - skipped
gpg: WARNING: nothing exported
  keyserver.ubuntu.com - F20F56EF6A067F70E8A5C99FFF95FAA971697405 - centaur (centaur)
  keys.openpgp.org     - F20F56EF6A067F70E8A5C99FFF95FAA971697405 - centaur (centaur)
  keyserver.ubuntu.com - C060A6635913D98A3587D7DB1C2491FFEB0EF770 - Cory Fields (cfields)
  keys.openpgp.org     - C060A6635913D98A3587D7DB1C2491FFEB0EF770 - Cory Fields (cfields)
  keyserver.ubuntu.com - BF6273FAEF7CC0BA1F562E50989F6B3048A116B5 - Dev Random (devrandom)
  keys.openpgp.org     - BF6273FAEF7CC0BA1F562E50989F6B3048A116B5 - Dev Random (devrandom)
  keyserver.ubuntu.com - 6D3170C1DC2C6FD0AEEBCA6743811D1A26623924 - Douglas Roark (droark)
  keys.openpgp.org     - 6D3170C1DC2C6FD0AEEBCA6743811D1A26623924 - Douglas Roark (droark)
  keyserver.ubuntu.com - 1C6621605EC50319C463D56C7F81D87985D61612 - Emanuele Cisbani (cisba)
  keys.openpgp.org     - 1C6621605EC50319C463D56C7F81D87985D61612 - Emanuele Cisbani (cisba)
  keyserver.ubuntu.com - 9A1689B60D1B3CCE9262307A2F40A9BF167FBA47 - Erik Mossberg (erkmos)
  keys.openpgp.org     - 9A1689B60D1B3CCE9262307A2F40A9BF167FBA47 - Erik Mossberg (erkmos)
  keyserver.ubuntu.com - D35176BE9264832E4ACA8986BF0792FBE95DC863 - fivepiece (fivepiece)
  keys.openpgp.org     - D35176BE9264832E4ACA8986BF0792FBE95DC863 - fivepiece (fivepiece)
  keyserver.ubuntu.com - 6F993B250557E7B016ADE5713BDCDA2D87A881D9 - Fuzzbawls (Fuzzbawls)
  keys.openpgp.org     - 6F993B250557E7B016ADE5713BDCDA2D87A881D9 - Fuzzbawls (Fuzzbawls)
  keyserver.ubuntu.com - 01CDF4627A3B88AAE4A571C87588242FBE38D3A8 - Gavin Andresen (gavinandresen)
  keys.openpgp.org     - 01CDF4627A3B88AAE4A571C87588242FBE38D3A8 - Gavin Andresen (gavinandresen)
  keyserver.ubuntu.com - D1DBF2C4B96F2DEBF4C16654410108112E7EA81F - Hennadii Stepanov (hebasto)
  keys.openpgp.org     - D1DBF2C4B96F2DEBF4C16654410108112E7EA81F - Hennadii Stepanov (hebasto)
  keyserver.ubuntu.com - A2FD494D0021AA9B4FA58F759102B7AE654A4A5A - Ilyas Ridhuan (IlyasRidhuan)
gpg: keyserver receive failed: No data
  keys.openpgp.org     - A2FD494D0021AA9B4FA58F759102B7AE654A4A5A - Ilyas Ridhuan (IlyasRidhuan)
gpg: keyserver receive failed: No data
gpg: WARNING: nothing exported
  keyserver.ubuntu.com - D3F22A3A4C366C2DCB66D3722DA9C5A7FA81EA35 - Jarol Rodriguez (jarolrod)
gpg: keyserver receive failed: No data
  keys.openpgp.org     - D3F22A3A4C366C2DCB66D3722DA9C5A7FA81EA35 - Jarol Rodriguez (jarolrod)
  keyserver.ubuntu.com - 7480909378D544EA6B6DCEB7535B12980BB8A4D3 - Jeffri H Frontz (jhfrontz)
  keys.openpgp.org     - 7480909378D544EA6B6DCEB7535B12980BB8A4D3 - Jeffri H Frontz (jhfrontz)
  keyserver.ubuntu.com - D3CC177286005BB8FF673294C5242A1AB3936517 - jl2012 (jl2012)
  keys.openpgp.org     - D3CC177286005BB8FF673294C5242A1AB3936517 - jl2012 (jl2012)
  keyserver.ubuntu.com - 82921A4B88FD454B7EB8CE3C796C4109063D4EAF - Jon Atack (jonatack)
gpg: keyserver receive failed: No data
  keys.openpgp.org     - 82921A4B88FD454B7EB8CE3C796C4109063D4EAF - Jon Atack (jonatack)
  keyserver.ubuntu.com - 32EE5C4C3FA15CCADB46ABE529D4BCB6416F53EC - Jonas Schnelli (jonasschnelli)
  keys.openpgp.org     - 32EE5C4C3FA15CCADB46ABE529D4BCB6416F53EC - Jonas Schnelli (jonasschnelli)
  keyserver.ubuntu.com - 4B4E840451149DD7FB0D633477DFAB5C3108B9A8 - Jorge Timon (jtimon)
  keys.openpgp.org     - 4B4E840451149DD7FB0D633477DFAB5C3108B9A8 - Jorge Timon (jtimon)
  keyserver.ubuntu.com - C42AFF7C61B3E44A1454CD3557AF762DB3353322 - Karl-Johan Alm (kallewoof)
  keys.openpgp.org     - C42AFF7C61B3E44A1454CD3557AF762DB3353322 - Karl-Johan Alm (kallewoof)
  keyserver.ubuntu.com - 30DE693AE0DE9E37B3E7EB6BBFF0F67810C1EED1 - Lisa Neigut (niftynei)
  keys.openpgp.org     - 30DE693AE0DE9E37B3E7EB6BBFF0F67810C1EED1 - Lisa Neigut (niftynei)
  keyserver.ubuntu.com - E463A93F5F3117EEDE6C7316BD02942421F4889F - Luke Dashjr (luke-jr)
  keys.openpgp.org     - E463A93F5F3117EEDE6C7316BD02942421F4889F - Luke Dashjr (luke-jr)
  keyserver.ubuntu.com - B8B3F1C0E58C15DB6A81D30C3648A882F4316B9B - Marco Falke (marco)
  keys.openpgp.org     - B8B3F1C0E58C15DB6A81D30C3648A882F4316B9B - Marco Falke (marco)
  keyserver.ubuntu.com - 07DF3E57A548CCFB7530709189BBB8663E2E65CE - Matt Corallo (BlueMatt)
  keys.openpgp.org     - 07DF3E57A548CCFB7530709189BBB8663E2E65CE - Matt Corallo (BlueMatt)
  keyserver.ubuntu.com - CA03882CB1FC067B5D3ACFE4D300116E1C875A3D - MeshCollider (meshcollider)
  keys.openpgp.org     - CA03882CB1FC067B5D3ACFE4D300116E1C875A3D - MeshCollider (meshcollider)
  keyserver.ubuntu.com - E777299FC265DD04793070EB944D35F9AC3DB76A - Michael Ford (fanquake)
  keys.openpgp.org     - E777299FC265DD04793070EB944D35F9AC3DB76A - Michael Ford (fanquake)
  keyserver.ubuntu.com - AD5764F4ADCE1B99BDFD179E12335A271D4D62EC - Michael Tidwell (miketwenty1)
  keys.openpgp.org     - AD5764F4ADCE1B99BDFD179E12335A271D4D62EC - Michael Tidwell (miketwenty1)
  keyserver.ubuntu.com - 9692B91BBF0E8D34DFD33B1882C5C009628ECF0C - Michagogo (michagogo)
  keys.openpgp.org     - 9692B91BBF0E8D34DFD33B1882C5C009628ECF0C - Michagogo (michagogo)
  keyserver.ubuntu.com - C57E4B42223FDE851D4F69DD28DF2724F241D8EE - midnightmagic (midnightmagic)
  keys.openpgp.org     - C57E4B42223FDE851D4F69DD28DF2724F241D8EE - midnightmagic (midnightmagic)
  keyserver.ubuntu.com - F4FC70F07310028424EFC20A8E4256593F177720 - Oliver Gugger (guggero, Oliver Gugger)
  keys.openpgp.org     - F4FC70F07310028424EFC20A8E4256593F177720 - Oliver Gugger (guggero, Oliver Gugger)
  keyserver.ubuntu.com - D62A803E27E7F43486035ADBBCD04D8E9CCCAC2A - Paul Rabahy (prab)
  keys.openpgp.org     - D62A803E27E7F43486035ADBBCD04D8E9CCCAC2A - Paul Rabahy (prab)
  keyserver.ubuntu.com - 37EC7D7B0A217CDB4B4E007E7FAB114267E4FA04 - Peter Todd (petertodd)
  keys.openpgp.org     - 37EC7D7B0A217CDB4B4E007E7FAB114267E4FA04 - Peter Todd (petertodd)
  keyserver.ubuntu.com - D762373D24904A3E42F33B08B9A408E71DAAC974 - Pieter Wuille [Location: Leuven, Belgium] (sipa)
  keys.openpgp.org     - D762373D24904A3E42F33B08B9A408E71DAAC974 - Pieter Wuille [Location: Leuven, Belgium] (sipa)
  keyserver.ubuntu.com - 133EAC179436F14A5CF1B794860FEB804E669320 - Pieter Wuille (sipa)
  keys.openpgp.org     - 133EAC179436F14A5CF1B794860FEB804E669320 - Pieter Wuille (sipa)
  keyserver.ubuntu.com - 6A8F9C266528E25AEB1D7731C2371D91CB716EA7 - Sebastian Falbesoner (theStack)
gpg: keyserver receive failed: No data
  keys.openpgp.org     - 6A8F9C266528E25AEB1D7731C2371D91CB716EA7 - Sebastian Falbesoner (theStack)
gpg: keyserver receive failed: No data
gpg: WARNING: nothing exported
  keyserver.ubuntu.com - A8FC55F3B04BA3146F3492E79303B33A305224CB - Sebastian Kung (TheCharlatan)
  keys.openpgp.org     - A8FC55F3B04BA3146F3492E79303B33A305224CB - Sebastian Kung (TheCharlatan)
  keyserver.ubuntu.com - ED9BDF7AD6A55E232E84524257FF9BDBCC301009 - Sjors Provoost (sjors)
  keys.openpgp.org     - ED9BDF7AD6A55E232E84524257FF9BDBCC301009 - Sjors Provoost (sjors)
  keyserver.ubuntu.com - 867345026B6763E8B07EE73AB6737117397F5C4F - Stephan Oeste (Emzy)
  keys.openpgp.org     - 867345026B6763E8B07EE73AB6737117397F5C4F - Stephan Oeste (Emzy)
  keyserver.ubuntu.com - 9EDAFF80E080659604F4A76B2EBB056FD847F8A7 - Stephan Oeste (Emzy)
  keys.openpgp.org     - 9EDAFF80E080659604F4A76B2EBB056FD847F8A7 - Stephan Oeste (Emzy)
  keyserver.ubuntu.com - 6DEEF79B050C4072509B743F8C275BC595448867 - Tomas Kanocz (KanoczTomas)
  keys.openpgp.org     - 6DEEF79B050C4072509B743F8C275BC595448867 - Tomas Kanocz (KanoczTomas)
  keyserver.ubuntu.com - AEC1884398647C47413C1C3FB1179EB7347DC10D - Warren Togami (wtogami)
  keys.openpgp.org     - AEC1884398647C47413C1C3FB1179EB7347DC10D - Warren Togami (wtogami)
  keyserver.ubuntu.com - 79D00BAC68B56D422F945A8F8E3A8F3247DBCBBF - Willy Ko (willyko)
  keys.openpgp.org     - 79D00BAC68B56D422F945A8F8E3A8F3247DBCBBF - Willy Ko (willyko)
  keyserver.ubuntu.com - 71A3B16735405025D447E8F274810B012346C9A6 - Wladimir J. van der Laan (laanwj)
  keys.openpgp.org     - 71A3B16735405025D447E8F274810B012346C9A6 - Wladimir J. van der Laan (laanwj)

List of source files to add to the SPEC file:

Source21: bitcoin-9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C-pubring.gpg
Source22: bitcoin-617C90010B3BD370B0AC7D424BB42E31C79111B8-pubring.gpg
Source23: bitcoin-E944AE667CF960B1004BC32FCA662BE18B877A60-pubring.gpg
Source24: bitcoin-152812300785C96444D3334D17565732E08E5E41-pubring.gpg
Source25: bitcoin-590B7292695AFFA5B672CBB2E13FC145CD3F4304-pubring.gpg
Source26: bitcoin-0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8-pubring.gpg
Source27: bitcoin-912FD3228387123DC97E0E57D5566241A0295FA9-pubring.gpg
Source28: bitcoin-C519EBCF3B926298946783EFF6430754120EC2F4-pubring.gpg
Source29: bitcoin-18AE2F798E0D239755DA4FD24B79F986CBDF8736-pubring.gpg
Source30: bitcoin-101598DC823C1B5F9A6624ABA5E0907A0380E6C3-pubring.gpg
Source31: bitcoin-F20F56EF6A067F70E8A5C99FFF95FAA971697405-pubring.gpg
Source32: bitcoin-C060A6635913D98A3587D7DB1C2491FFEB0EF770-pubring.gpg
Source33: bitcoin-BF6273FAEF7CC0BA1F562E50989F6B3048A116B5-pubring.gpg
Source34: bitcoin-6D3170C1DC2C6FD0AEEBCA6743811D1A26623924-pubring.gpg
Source35: bitcoin-1C6621605EC50319C463D56C7F81D87985D61612-pubring.gpg
Source36: bitcoin-9A1689B60D1B3CCE9262307A2F40A9BF167FBA47-pubring.gpg
Source37: bitcoin-D35176BE9264832E4ACA8986BF0792FBE95DC863-pubring.gpg
Source38: bitcoin-6F993B250557E7B016ADE5713BDCDA2D87A881D9-pubring.gpg
Source39: bitcoin-01CDF4627A3B88AAE4A571C87588242FBE38D3A8-pubring.gpg
Source40: bitcoin-D1DBF2C4B96F2DEBF4C16654410108112E7EA81F-pubring.gpg
Source41: bitcoin-A2FD494D0021AA9B4FA58F759102B7AE654A4A5A-pubring.gpg
Source42: bitcoin-D3F22A3A4C366C2DCB66D3722DA9C5A7FA81EA35-pubring.gpg
Source43: bitcoin-7480909378D544EA6B6DCEB7535B12980BB8A4D3-pubring.gpg
Source44: bitcoin-D3CC177286005BB8FF673294C5242A1AB3936517-pubring.gpg
Source45: bitcoin-82921A4B88FD454B7EB8CE3C796C4109063D4EAF-pubring.gpg
Source46: bitcoin-32EE5C4C3FA15CCADB46ABE529D4BCB6416F53EC-pubring.gpg
Source47: bitcoin-4B4E840451149DD7FB0D633477DFAB5C3108B9A8-pubring.gpg
Source48: bitcoin-C42AFF7C61B3E44A1454CD3557AF762DB3353322-pubring.gpg
Source49: bitcoin-30DE693AE0DE9E37B3E7EB6BBFF0F67810C1EED1-pubring.gpg
Source50: bitcoin-E463A93F5F3117EEDE6C7316BD02942421F4889F-pubring.gpg
Source51: bitcoin-B8B3F1C0E58C15DB6A81D30C3648A882F4316B9B-pubring.gpg
Source52: bitcoin-07DF3E57A548CCFB7530709189BBB8663E2E65CE-pubring.gpg
Source53: bitcoin-CA03882CB1FC067B5D3ACFE4D300116E1C875A3D-pubring.gpg
Source54: bitcoin-E777299FC265DD04793070EB944D35F9AC3DB76A-pubring.gpg
Source55: bitcoin-AD5764F4ADCE1B99BDFD179E12335A271D4D62EC-pubring.gpg
Source56: bitcoin-9692B91BBF0E8D34DFD33B1882C5C009628ECF0C-pubring.gpg
Source57: bitcoin-C57E4B42223FDE851D4F69DD28DF2724F241D8EE-pubring.gpg
Source58: bitcoin-F4FC70F07310028424EFC20A8E4256593F177720-pubring.gpg
Source59: bitcoin-D62A803E27E7F43486035ADBBCD04D8E9CCCAC2A-pubring.gpg
Source60: bitcoin-37EC7D7B0A217CDB4B4E007E7FAB114267E4FA04-pubring.gpg
Source61: bitcoin-D762373D24904A3E42F33B08B9A408E71DAAC974-pubring.gpg
Source62: bitcoin-133EAC179436F14A5CF1B794860FEB804E669320-pubring.gpg
Source63: bitcoin-6A8F9C266528E25AEB1D7731C2371D91CB716EA7-pubring.gpg
Source64: bitcoin-A8FC55F3B04BA3146F3492E79303B33A305224CB-pubring.gpg
Source65: bitcoin-ED9BDF7AD6A55E232E84524257FF9BDBCC301009-pubring.gpg
Source66: bitcoin-867345026B6763E8B07EE73AB6737117397F5C4F-pubring.gpg
Source67: bitcoin-9EDAFF80E080659604F4A76B2EBB056FD847F8A7-pubring.gpg
Source68: bitcoin-6DEEF79B050C4072509B743F8C275BC595448867-pubring.gpg
Source69: bitcoin-AEC1884398647C47413C1C3FB1179EB7347DC10D-pubring.gpg
Source70: bitcoin-79D00BAC68B56D422F945A8F8E3A8F3247DBCBBF-pubring.gpg
Source71: bitcoin-71A3B16735405025D447E8F274810B012346C9A6-pubring.gpg

I'll make the necessary adjustments if I can use the %include macro, to keep the SPEC file less cluttered.

OK, this is much better: https://github.com/negativo17/bitcoin-core/blob/1c3ee00c999b0ed8b3e497c7d9019ab1d8bc006b/bitcoin-gpg.sh

There is also another caveat, of all the GPG keys listed for the developers, only few have actually signed the release. The SHA256SUMS.asc file contains just 9 signatures.
The script now makes sure to download only the valid keys listed in list of keys that have been used to sign.

This is the output of the script:

$ ./bitcoin-gpg.sh 22.0

Prepare directory and list of GPG keys used to sign the release... done.
Attempting to download all listed GPG keys for release 22.0... done.
List of valid keys used to sign the release:

# 9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C - Aaron Clauson (sipsorcery)
Source21: bitcoin-9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C-pubring.gpg
# 152812300785C96444D3334D17565732E08E5E41 - Andrew Chow (achow101)
Source22: bitcoin-152812300785C96444D3334D17565732E08E5E41-pubring.gpg
# 590B7292695AFFA5B672CBB2E13FC145CD3F4304 - Antoine Poinsot (darosior)
Source23: bitcoin-590B7292695AFFA5B672CBB2E13FC145CD3F4304-pubring.gpg
# 0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8 - Ben Carman (benthecarman)
Source24: bitcoin-0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8-pubring.gpg
# D1DBF2C4B96F2DEBF4C16654410108112E7EA81F - Hennadii Stepanov (hebasto)
Source25: bitcoin-D1DBF2C4B96F2DEBF4C16654410108112E7EA81F-pubring.gpg
# 82921A4B88FD454B7EB8CE3C796C4109063D4EAF - Jon Atack (jonatack)
Source26: bitcoin-82921A4B88FD454B7EB8CE3C796C4109063D4EAF-pubring.gpg
# E777299FC265DD04793070EB944D35F9AC3DB76A - Michael Ford (fanquake)
Source27: bitcoin-E777299FC265DD04793070EB944D35F9AC3DB76A-pubring.gpg
# 9EDAFF80E080659604F4A76B2EBB056FD847F8A7 - Stephan Oeste (Emzy)
Source28: bitcoin-9EDAFF80E080659604F4A76B2EBB056FD847F8A7-pubring.gpg
# 71A3B16735405025D447E8F274810B012346C9A6 - Wladimir J. van der Laan (laanwj)
Source29: bitcoin-71A3B16735405025D447E8F274810B012346C9A6-pubring.gpg

Now every signature (all 9) are checked for validity:

https://github.com/negativo17/bitcoin-core/blob/master/bitcoin-core.spec#L39-L40
https://github.com/negativo17/bitcoin-core/blob/master/bitcoin-core.spec#L140-L145

And now git status clearly shows the differences when bumping releases (here are all new, they will be listed as untracked/deleted/modified):

$ git status
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
  (use "git push" to publish your local commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	bitcoin-0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8-pubring.gpg
	bitcoin-152812300785C96444D3334D17565732E08E5E41-pubring.gpg
	bitcoin-590B7292695AFFA5B672CBB2E13FC145CD3F4304-pubring.gpg
	bitcoin-71A3B16735405025D447E8F274810B012346C9A6-pubring.gpg
	bitcoin-82921A4B88FD454B7EB8CE3C796C4109063D4EAF-pubring.gpg
	bitcoin-9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C-pubring.gpg
	bitcoin-9EDAFF80E080659604F4A76B2EBB056FD847F8A7-pubring.gpg
	bitcoin-D1DBF2C4B96F2DEBF4C16654410108112E7EA81F-pubring.gpg
	bitcoin-E777299FC265DD04793070EB944D35F9AC3DB76A-pubring.gpg
	bitcoin-gpg.inc

nothing added to commit but untracked files present (use "git add" to track)

Spec URL: https://slaanesh.fedorapeople.org/bitcoin-core.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-core-22.0-2.fc34.src.rpm

* Sat Sep 25 2021 Simone Caronni <negativo17> - 22.0-2
- Prepare all keys with a script and verify all keys against the signature file.
  Add reasoning on the process in the SPEC file.

I'm in the process of rebuilding all packages in the above mentioned repository.

TODO:
- Investigate systemd user units along with the system one.
- Check systemd unit parameters.
- Figure out if BDB 4.8 builds in Fedora 35/rawhide and integrate it.

(In reply to Simone Caronni from comment #110)
> Spec URL: https://slaanesh.fedorapeople.org/bitcoin-core.spec
> SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-core-22.0-2.fc34.src.rpm

Do you have koji builds for F34?
To avoid unwanted koji builds from my side.

(In reply to Eugene A. Pivnev from comment #111)
> Do you have koji builds for F34?
> To avoid unwanted koji builds from my side.

No, it's all here:

(In reply to Simone Caronni from comment #105)
> Here is a repository which will contain the latest packages being built as
> part of the review:
> 
> https://negativo17.org/repos/bitcoin/
> https://negativo17.org/repos/epel-bitcoin.repo
> https://negativo17.org/repos/fedora-bitcoin.repo

The directories are browseable, so feel free to look around.

(In reply to Warren Togami from comment #81)
> > There exists a corner case where "stop" can take significantly more time
> > than any amount hardcoded here. Force kill in that situation causes data
> > corruption and an even more time consuming recovery the next time indexing
> > begins again. I asked my engineers here if they came up with a better way to
> > detect a safe stop.
> 
> TimeoutStopSec=1200s

This is way too big. In the case of a shutdown/reboot of a system, all the units depending on network will shut off and then basically there will be the network up and bitcoin trying to stop for 20 minutes. Assuming you reach that point, you would wait with a pingable only host that is trying to reboot for 20 minutes.

I think it's much more reasonable to have a shorter timeout, and in corner cases where you need more, you can just edit the systemd override configuration with what you need (systemctl edit bitcoin).

> Restart=always

This is wrong, it should be "on-failure". If I run "bitcoin-cli stop" then the daemon would just restart endlessly. If I shut it down correctly it should not come up again, only in case of failures. Again, like above, you can tune it for specific cases.

> # Fail when 10 tries within 1 minute fail (never)
> StartLimitInterval=60s
> StartLimitBurst=10
> # Limit attempts to 1 per 10 seconds
> Restart=always
> RestartSec=10

This is as well wrong. You're basically allowing it to restart once maximum every 10 seconds and then you check if it fails 10 times in one minute, which is an impossible condition. Combined with the above restart always you make the daemon restart in a loop even in case of issues. This is definitely not a sane configuration, might work in your case but should not be shipped as the default. I'm not changing the parameters here.

On a side note, now that RHEL/CentOS 7 is no longer a target, I have now added more hardening to the systemd unit.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin-core.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-core-22.0-3.fc34.src.rpm

* Sat Sep 25 2021 Simone Caronni <negativo17> - 22.0-3
- Remove obsolete scriptlets.
- With RHEL/CentOS 7 no longer a target, improve systemd unit security.
- Add systemd user unit to start bitcoind in your user session.
- Move bitcoin-wallet to utils subpackage for offline wallet manipulation.
- Update README files.

TODO:
- Figure out if BDB 4.8 builds in Fedora 35/rawhide and integrate it.

I'm in the process of rebuilding all packages here, as usual:

https://negativo17.org/repos/bitcoin/
https://negativo17.org/repos/epel-bitcoin.repo
https://negativo17.org/repos/fedora-bitcoin.repo

This is probably the last version of the package with Berkeley DB 5.3 from the repositories, the next one will use 4.8.
If you had a wallet running with these packages that was created with the incompatible Berkeley DB (I have since 2016), then you will need to dump the wallet and reimport it when the next package is installed:

walletpassphrase <supersecret> 60
dumpwallet dump.txt

And then with the next package:

createwallet <wallet_name>
encryptwallet <new_password>
walletpassphrase <new_password> 60
importwallet dump.txt
walletlock

You can use the console in bitcoin-qt, bitcoin-cli when connected to a bitcoind or bitcoin-wallet for offline editing.

(In reply to Warren Togami from comment #74)
> * The bitcoin.service file bothers me in that a single system service is one
> of many ways in which bitcoind is used. I'd prefer if it was a .service type
> that allowed for multiple admin "@" definable instances. There could also be
> a different .service file like "bitcoincore-homedir-service" that uses
> $HOME/.bitcoin as the datadir in the way many have already deployed the
> upstream binary? It is also my strong preference for the different types of
> .service files to be in their own subpackage. That would also help the above
> concern regarding a single hard-coded /etc/bitcoin/bitcoin.conf not being
> how many use this software.

This is part of the systemd user unit, and you can start it with all defaults (it just contains bitcoind without parameters): systemctl --user start bitcoin.service

(In reply to Simone Caronni from comment #105)
> Maybe 10 signatures is a bit too much

You can change that number with every release if you want. Ten happened to be the number of signatures I could validate.

(In reply to Simone Caronni from comment #106)
> All keys are deleted before regenerating, so at every release it's clear
> what must go and disappear from git (git status).

(In reply to Simone Caronni from comment #108)
> The script now makes sure to download only the valid keys listed in list of
> keys that have been used to sign.

That's a good approach if the set of people who sign releases will be mostly the same every time. If it turns out that several people sign some releases and not others, then it will cause their keys to be dropped and added back repeatedly. In the latter case it may be better to add a key to the package the first time that key signs a release, and remove only revoked and expired keys.

(In reply to Simone Caronni from comment #108)
> OK, this is much better:
> https://github.com/negativo17/bitcoin-core/blob/
> 1c3ee00c999b0ed8b3e497c7d9019ab1d8bc006b/bitcoin-gpg.sh

bitcoin-gpg.sh relies on the tarball to tell it which keys should be used to verify the tarball. A manipulated tarball will of course contain a manipulated keys.txt that lists fake keys generated by the attacker. This makes it all the more important to not remove and re-add keys in the package unnecessarily. The continuity of the keys in the Git history becomes the only thing that can show that the tarball is genuine.

bitcoin-gpg.sh will include a revoked or expired key if it signs a release. Such keys must be weeded out.

bitcoin-gpg.sh fails for me because the string "Good signature" is locale-specific. The locale-independent solution is to use --status-fd and grep for "^\[GNUPG:\] GOODSIG ".

Why in the name of sanity did Bugzilla change all those fields? I'm trying to restore them. I hope this won't count as me taking over as the reviewer just because I restored the field values. Bugs bugs bugs ...

>> The test suite is *excessively slow*. IMO we should not run these extensive
>> tests during these builds. Upstream official builds do not. We developers
>> should instead run these tests on the target platforms independently.

> The tests make sure that the binaries built are working, if they fail some test then the package build fails.
> They are not part of what the users see.
> I would say they are CRITICAL to make sure you have fully functioning binaries.

It is so excessively slow that it inteferes with the ability of packagers to incrementally improve this package.
The test suite is NOT USED FOR THE OFFICIAL UPSTREAM RELEASE PROCESS. That strongly suggests that we shouldn't do it here either.

(In reply to Björn Persson from comment #117)
> bitcoin-gpg.sh relies on the tarball to tell it which keys should be used to
> verify the tarball. A manipulated tarball will of course contain a
> manipulated keys.txt that lists fake keys generated by the attacker. This
> makes it all the more important to not remove and re-add keys in the package
> unnecessarily. The continuity of the keys in the Git history becomes the
> only thing that can show that the tarball is genuine.

I can also move the SHA256SUMS checks in the script, but this is true for any package for which we verify hashes/signatures in Fedora. That's why you have the URL pointing to external files to check if they're the same.

> bitcoin-gpg.sh will include a revoked or expired key if it signs a release.
> Such keys must be weeded out.

They can be removed only by running it at a specific time. Might be that keys are valid today and not valid in a few days. So up to the packager to execute when at least bumping release.

> bitcoin-gpg.sh fails for me because the string "Good signature" is
> locale-specific. The locale-independent solution is to use --status-fd and
> grep for "^\[GNUPG:\] GOODSIG ".

Good point, will adjust.

(In reply to Warren Togami from comment #119)
> It is so excessively slow that it inteferes with the ability of packagers to
> incrementally improve this package.
> The test suite is NOT USED FOR THE OFFICIAL UPSTREAM RELEASE PROCESS. That
> strongly suggests that we shouldn't do it here either.

I don't get what's issue you have with those.
It is not excessively slow on recent hardware; I'm mostly using a 5 year old desktop and they all run fine. It's the builders who do the work, not the packager. And they can also be disabled for a quick test.
These are handy to catch specific bugs introduced by compilers, compiler flags etc, so there is a lot of value in those. And packaging guidelines as well state that if the software supports checks we should add them.

Would you be more comfortable knowing that actually functional tests fail every time but we package the software anyway?

Additional Sources to include in the .src.rpm
=============================================
depends/sources/db-4.8.30.NC.tar.gz
depends/sources/download-stamps/.stamp_fetched-bdb-db-4.8.30.NC.tar.gz.hash

Build (tested on Fedora 34 x86_64)
==================================
# cd depends
# make bdb

Installing DB include files: /home/rpmdev/rpmbuild/BUILD/bitcoin-22.0/depends/work/staging/x86_64-pc-linux-gnu/bdb/4.8.30-09103229d28/home/rpmdev/rpmbuild/BUILD/bitcoin-22.0/depends/x86_64-pc-linux-gnu/include ...
make[1]: Leaving directory '/home/rpmdev/rpmbuild/BUILD/bitcoin-22.0/depends/work/build/x86_64-pc-linux-gnu/bdb/4.8.30-09103229d28/build_unix'
Postprocessing bdb...
Caching bdb...

Newly Created Files
===================
depends/built/x86_64-pc-linux-gnu/bdb/bdb-4.8.30-09103229d28.tar.gz
depends/built/x86_64-pc-linux-gnu/bdb/bdb-4.8.30-09103229d28.tar.gz.hash

Contents of that tarball
========================
./
./include/
./include/db_cxx.h
./include/db.h
./lib/
./lib/libdb-4.8.a
./lib/libdb.a
./lib/libdb_cxx-4.8.a
./lib/libdb_cxx.a
./.stamp_postprocessed

How to link against it
======================
https://github.com/bitcoin/bitcoin/blob/master/contrib/install_db4.sh
Some clues in here. Seems to be:
* Unpack the above tarball somewhere.
* Define BDB_PREFIX using absolute pathname and add certain ./configure flags.

At the moment the only thing I'm not certain about is how to handle the different arch-specific pathnames seen above.

Thanks. This week I became father again, so I'm a bit distracted. Will take care in the next days.

> This week I became father again

Congrats!

(In reply to Warren Togami from comment #123)
> > This week I became father again
> 
> Congrats!

Thanks! Back to work now.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin-core.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-core-22.0-4.fc34.src.rpm

* Sun Oct 03 2021 Simone Caronni <negativo17> - 22.0-4
- Switch to bundled statically linked Berkeley DB 4.8.30 (NC).

I reused as much as possible the hardening and compiler settings that are part of the default policies:

https://github.com/negativo17/bitcoin-core/commit/34af3b5d70328287a61624d14c97afa5862f0ca8

I'm in the process of rebuilding all packages here, as usual:

https://negativo17.org/repos/bitcoin/
https://negativo17.org/repos/epel-bitcoin.repo
https://negativo17.org/repos/fedora-bitcoin.repo

I will leave also the previous build (22.0-3) for people that want to rebuild their wallets.

Actually, digging a bit deep into the issue, there is no need to migrate wallet if you shutdown your client/node correctly before updating, as this excludes the chance for pending transaction logs (the only part which has changed in format from BDB 4.8): https://github.com/bitcoin/bitcoin/pull/18870

For those interested, here is the proposed (very likely) migration to the descriptor wallets: https://github.com/bitcoin/bitcoin/issues/20160#issue-722711719

(In reply to Simone Caronni from comment #120)
> (In reply to Björn Persson from comment #117)
> > bitcoin-gpg.sh will include a revoked or expired key if it signs a release.
> > Such keys must be weeded out.
> 
> They can be removed only by running it at a specific time. Might be that
> keys are valid today and not valid in a few days. So up to the packager to
> execute when at least bumping release.

Either you misunderstood me, or I'm misunderstanding you.

The signature verification in the spec does not check whether keys have expired. That is as it should be, because we don't want expiring keys to act as time bombs and cause rebuilds to fail some time after a package has been released.

The signature verification in the spec also does not check whether keys have been revoked. If it did, it would help the packager catch mistakes, but that's all it would do. The only way that a revocation certificate could get into the package would be if the packager would add it while updating the package.

Since the spec doesn't check, it's the packager's responsibility to remove expired and revoked keys. Obviously this will only happen when the packager updates the package. It would be helpful if bitcoin-gpg.sh would help with this. Currently it does not, so each key must be checked manually using gpg2 --list-keys.

The current version of bitcoin-gpg.sh uses gpgv2 to check whether each key signed the current release. It does not check whether the keys are expired or revoked. Not unless the gpgv2 manpage lies when it says:

| gpgv assumes that all keys in the keyring are trustworthy. That does also
| mean that it does not check for expired or revoked keys.

Thus, if one of the keys that signed the release has expired or been revoked, bitcoin-gpg.sh will still add that key to the package, so the packager must discover and remove it manually.

(In reply to Björn Persson from comment #127)
> The signature verification in the spec also does not check whether keys have
> been revoked. If it did, it would help the packager catch mistakes, but
> that's all it would do. The only way that a revocation certificate could get
> into the package would be if the packager would add it while updating the
> package.
>
> | gpgv assumes that all keys in the keyring are trustworthy. That does also
> | mean that it does not check for expired or revoked keys.

Isn't this the correct behaviour? Keys might have been revoked and expired and still being left on the key servers. We pick all those up, regardless of their expired/revoked status and use them to validate the signed release. They were valid at the moment of the signature, so we should check them. I could extend the script to print which ones are expired/revoked, but this does not change the fact that I would need to use them to check the signature.
 
> Thus, if one of the keys that signed the release has expired or been
> revoked, bitcoin-gpg.sh will still add that key to the package, so the
> packager must discover and remove it manually.

With a new release (let's say 0.23) everything gets reset and we just check again the entire set of keys used for signing. Expired keys will of course not be in this list, as they will not be use for signing. So there is nothing for the packager to discover. This is what I meant with the packager adapting for the release.

Practical example

- New release
  - Lots of keys sign the release
  - Keys might be revoked and still be on the keyservers
  - Keys might be expired and still be on the keyservers
  - Keys might not be on the keyservers (revoked, expired, or simply removed)

- New package
  - All keys are attempted to be downloaded from keyserver
  - All keys downloaded are used to verify the signature, regardless of their state
  - List of keys is regenarated, so old keys no longer used (which includes expired and revoked) are removed
  - No need for the packager to check which ones are expired/removed

Why would it be beneficial to know which ones are expired/revoked? I can make the script print it if you prefer, but it would just be a cosmetic thing and not actually used for checking anything.

I think this answer your concern. Does it make sense?

This is completely regenarated when calling the script: https://github.com/negativo17/bitcoin-core/blob/master/bitcoin-gpg.inc

And we don't want that list to be empty because of mass revocations or expirations. Expired/revoked keys will of course not be used in 0.23 to sign, so they will not be in that file when we call the script again for the new release.

If you think this does not answer your concern please provide a patch/diff to the script so I can understand what you mean. Thanks.

(In reply to Simone Caronni from comment #128)
> Keys might have been revoked and expired and still being left on the key
> servers. We pick all those up, regardless of their expired/revoked status
> and use them to validate the signed release.

When somebody revokes their key they send the message "I have lost control of my secret key. Malicious actors may have acquired it, and may be signing things with it. Do not trust this key anymore!"

Heed that warning! When you see a signature made with a revoked key you should assume that an attacker made that signature.

> They were valid at the moment of the signature

An attacker can easily change their clock while making a signature to make it look like the signature was made before the key was revoked.

(In reply to Simone Caronni from comment #129)
> And we don't want that list to be empty because of mass revocations or
> expirations.

If many of the developers revoke their keys, then that's because there has been a huge security breach. By ignoring such an event you'll probably end up distributing malware to Fedora users. If there is a purported new release, but all of the signatures are made with revoked keys, then that release is malicious and you should not package it.

> Expired/revoked keys will of course not be used in 0.23 to sign, so they
> will not be in that file when we call the script again for the new release.

That's true only as long as nobody attempts a supply-chain attack. If you could know that there is no attacker, then there would be no need to verify any signatures at all. For security you must always assume that somebody is trying to attack you any way they can.

(In reply to Simone Caronni from comment #130)
> If you think this does not answer your concern please provide a patch/diff
> to the script so I can understand what you mean. Thanks.

I don't have tested code ready right now but I think you can use gpg2 instead of gpgv2 – only in bitcoin-gpg.sh, not in the spec – and (using --status-fd) grep for "^\[GNUPG:\] GOODSIG " only, excluding REVKEYSIG, EXPKEYSIG, BADSIG et cetera. That pattern matches only at the beginning of a line to ensure that it matches a keyword and not some other part of the output. The pattern includes a trailing space to ensure that it matches a whole keyword, not just a prefix.

Got stuck in work stuff. Will resume work on it this weekend or next week.

Created attachment 1843870 [details]
patch to filter out revoked and expired keys

(In reply to Björn Persson from comment #131)
> (In reply to Simone Caronni from comment #130)
> > If you think this does not answer your concern please provide a patch/diff
> > to the script so I can understand what you mean. Thanks.
> 
> I don't have tested code ready right now but I think you can use gpg2
> instead of gpgv2 – only in bitcoin-gpg.sh, not in the spec – and (using
> --status-fd) grep for "^\[GNUPG:\] GOODSIG " only, excluding REVKEYSIG,
> EXPKEYSIG, BADSIG et cetera. That pattern matches only at the beginning of a
> line to ensure that it matches a keyword and not some other part of the
> output. The pattern includes a trailing space to ensure that it matches a
> whole keyword, not just a prefix.

I took the time to write a patch. Here's how to avoid trusting a key whose owner says not to trust it.

(In reply to Björn Persson from comment #133)
> I took the time to write a patch. Here's how to avoid trusting a key whose
> owner says not to trust it.

Thank you!

I finally have time to work on it again.

Spec URL: https://slaanesh.fedorapeople.org/bitcoin-core.spec
SRPM URL: https://slaanesh.fedorapeople.org/bitcoin-core-22.0-5.fc35.src.rpm

* Sun Jan 23 2022 Simone Caronni <negativo17> - 22.0-5
- Update GPG verification script (thanks Björn Persson).

I'm in the process of rebuilding all packages here, as usual:

https://negativo17.org/repos/bitcoin/
https://negativo17.org/repos/epel-bitcoin.repo
https://negativo17.org/repos/fedora-bitcoin.repo

Do we have anything else to cover as part of the review?

miniupnpc packages for epel 8+ on the way: https://bodhi.fedoraproject.org/updates/?packages=miniupnpc

Two main things continue to bother me (although don't consider this to be a veto):

1) "bitcoin-core" is not a good main package name because it looks like "core" is a subpackage. "bitcoincore" would be a fine single word name that matches its website bitcoincore.org.

2) Simone wrote:
> Would you be more comfortable knowing that actually functional tests fail every time but we package the software anyway?

The tests fail on my laptop but succeeds on my server. Why? Timeouts.
The tests are too extensive. They are NOT used during the official release process. They are not needed here.

(In reply to Warren Togami from comment #137)
> 1) "bitcoin-core" is not a good main package name because it looks like
> "core" is a subpackage. "bitcoincore" would be a fine single word name that
> matches its website bitcoincore.org.

It won't look like a subpackage because it's just SRPM package name, but none of RPM packages built from it will be called "bitcoin-core".

However, from aesthetic point of view, I think, it would be nicer is RPM packages would be called "bitcoin-server", "bitcoin-utils" etc. instead of "bitcoin-core-server", "bitcoin-core-utils" etc.
I use similar approach in my Dash package: SRPM is called "dash-core" (to avoid a conflict with "dash" package that is a shell), but RPMS built from it are called "dash-server", "dash-utils" etc.

Also, I'm not sure whether it makes sense to build libbitcoinconsensus as a shared library. Does it have any ABI stability guarantees? Are there any uses of this library outside of this project envisioned?

(In reply to Warren Togami from comment #137)
> The tests are too extensive. They are NOT used during the official release
> process. They are not needed here.

IMHO to skip all tests is not good idea.
Recent example - https://github.com/estraier/tkrzw/issues/24. Only koji builds opened very special bug (or feature?) in upstream code.
From other side executing _all_ of tests is not good idea too.
Maybe limiting tests with set of light ones will be better: https://github.com/estraier/tkrzw/issues/23

(In reply to Oleg Girko from comment #138)
> (In reply to Warren Togami from comment #137)
> > 1) "bitcoin-core" is not a good main package name because it looks like
> > "core" is a subpackage. "bitcoincore" would be a fine single word name that
> > matches its website bitcoincore.org.
> 
> It won't look like a subpackage because it's just SRPM package name, but
> none of RPM packages built from it will be called "bitcoin-core".
> 
> However, from aesthetic point of view, I think, it would be nicer is RPM
> packages would be called "bitcoin-server", "bitcoin-utils" etc. instead of
> "bitcoin-core-server", "bitcoin-core-utils" etc.

This was discussed before, it was the case when I started the review (as it would be my preference as well) and I was asked to change it as the package should not represent bitcoin as a whole but one of the different implementations.

(In reply to Warren Togami from comment #137)
> 1) "bitcoin-core" is not a good main package name because it looks like
> "core" is a subpackage. "bitcoincore" would be a fine single word name that
> matches its website bitcoincore.org.

It's referenced everywhere with the dash when the words are not separated by space: https://bitcoin.org/en/bitcoin-core/

(In reply to Oleg Girko from comment #139)
> Also, I'm not sure whether it makes sense to build libbitcoinconsensus as a
> shared library. Does it have any ABI stability guarantees? Are there any
> uses of this library outside of this project envisioned?

Well, the package is new, so no users of the libraries yet, but I can disable if needed. Doesn't hurt to leave the options open though.

(In reply to Simone Caronni from comment #142)
> (In reply to Warren Togami from comment #137)
> > 1) "bitcoin-core" is not a good main package name because it looks like
> > "core" is a subpackage. "bitcoincore" would be a fine single word name that
> > matches its website bitcoincore.org.
> 
> It's referenced everywhere with the dash when the words are not separated by
> space: https://bitcoin.org/en/bitcoin-core/

That's maintained by other people outside of the upstream project (and often lags behind releases).  The upstream releases happen at https://bitcoincore.org/en/releases/.  This has been pointed out earlier in this issue (see comment #23).

(In reply to Suvayu from comment #144)
> (In reply to Simone Caronni from comment #142)
> > (In reply to Warren Togami from comment #137)
> > > 1) "bitcoin-core" is not a good main package name because it looks like
> > > "core" is a subpackage. "bitcoincore" would be a fine single word name that
> > > matches its website bitcoincore.org.
> > 
> > It's referenced everywhere with the dash when the words are not separated by
> > space: https://bitcoin.org/en/bitcoin-core/
> 
> That's maintained by other people outside of the upstream project (and often
> lags behind releases).  The upstream releases happen at
> https://bitcoincore.org/en/releases/.  This has been pointed out earlier in
> this issue (see comment #23).

I know, it's also in the SPEC file. But the package is spelled with the dash everywhere: https://bitcoincore.org/bin/

The bitcoin-core-selinux package review has been approved, and we have no blockers.
@Warren can you set the review flags? Thanks.

Many review pauses from everyone (me included) but this review has lasted 21 months and 2 weeks :)

(In reply to Simone Caronni from comment #147)
> this review has lasted 21 months and 2 weeks :)

Longer then CentOS7 after RHEL7 released?

This is an automatic action taken by review-stats script.

The ticket reviewer failed to clear the NEEDINFO flag in a month.
As per https://fedoraproject.org/wiki/Policy_for_stalled_package_reviews
we reset the status and the assignee of this ticket.

Oh geez, this is nuts.

I'm going to go ahead and fix this...


PACKAGE APPROVED!

Also, please add metainfo files for the GUI software subpackage. :)

(In reply to Neal Gompa from comment #151)
> Also, please add metainfo files for the GUI software subpackage. :)

Sure will do before issuing the first update in Bodhi! many thanks.

(fedscm-admin):  The Pagure repository was created at https://src.fedoraproject.org/rpms/bitcoin-core

FEDORA-2022-5a7b799dbd has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-5a7b799dbd

FEDORA-EPEL-2022-03592f538a has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-03592f538a

FEDORA-2022-70521803fa has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-70521803fa

FEDORA-2022-2c587477ac has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-2c587477ac

FEDORA-EPEL-2022-69091367ab has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-69091367ab

FEDORA-2022-5a7b799dbd has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2022-5a7b799dbd \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-5a7b799dbd

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-70521803fa has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2022-70521803fa \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-70521803fa

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-2c587477ac has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf install --enablerepo=updates-testing --advisory=FEDORA-2022-2c587477ac \*`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-2c587477ac

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-EPEL-2022-69091367ab has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-69091367ab

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-EPEL-2022-03592f538a has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-03592f538a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-EPEL-2022-69091367ab has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-2c587477ac has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-70521803fa has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-EPEL-2022-03592f538a has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.