Bug 2073385

Thank you for the report.  This is weird:

  ...
  Verifying        : rpm-build-4.14.3-22.el8.x86_64                       21/21 
  ...
  bash: /usr/bin/rpmbuild: No such file or directory
  ...

This appears to be the problem (missing rpmbuild binary).  Going to switch
this against the RPM component for the help.

For some reason, the upgrade step decides to remove stuff:

> Removing dependent packages:
> annobin                  x86_64      10.29-3.el8         @appstream      135 k
> gcc                      x86_64      8.5.0-10.el8        @appstream       59 M
> gcc-c++                  x86_64      8.5.0-10.el8        @appstream       31 M
> glibc-gconv-extra        x86_64      2.28-189.el8        @appstream      6.1 M
> libstdc++-devel          x86_64      8.5.0-10.el8        @appstream       11 M
> redhat-rpm-config        noarch      127-1.el8           @appstream      147 k
> rpm-build                x86_64      4.14.3-22.el8       @appstream      292 k

Based on the list of removed components, I'd hazard a guess this is related to annobin vs gcc version dependencies one way or the other, and based on timing, probably caused by bug 2067153.

It's another OVirt COPR repository issue. (See bug 2072397 for another example.) Bugzilla is simply not the right place for reporting them.

Regular CentOS 8 Stream is at:

annobin-10.58-1.el8.x86_64
gcc-8.5.0-12.el8.x86_64
redhat-rpm-config-127-1.el8.noarch
rpm-build-4.14.3-22.el8.x86_64

So it means we have broken CentOS Stream mirrors?  Copr just does what normal
users would do;  or IOW, if this bug affects Copr builds it must affect the
end-users, too.

(In reply to Pavel Raiskup from comment #4)
> So it means we have broken CentOS Stream mirrors?  Copr just does what normal
> users would do;  or IOW, if this bug affects Copr builds it must affect the
> end-users, too.

That COPR isn't configured to use the regular CentOS repositories, instead I see: https://buildlogs.centos.org/centos/8-stream/virt/x86_64/ovirt-45/

It looks like a custom buildroot.

I think the regular CentOS Stream repositories are not affected by this.

> That COPR isn't configured to use the regular CentOS repositories, instead I see

Indeed, I missed this.  OK, but those are just "additional" repositories.  Everything
in that RPM transaction actually comes from baseos or @appstream.  I can see now that
there are also some modules enabled, but despite that - _no_ modular package is installed
in any of RPM transactions.  I don't get it.

At this point in time, gcc-8.5.0-12 should be always installed right away, not "updated".

Honestly, neither bug #2067153 seems clear to me.  I simply don't get how the
additional repository affects the transaction.  Can anyone elaborate?

Indeed, that transaction has nothing to do with additional repos or modules enabled.
Also, if bugzilla is not the right place to report this, can you please point me to the right place?

(In reply to Sandro Bonazzola from comment #7)
> Indeed, that transaction has nothing to do with additional repos or modules
> enabled.
> Also, if bugzilla is not the right place to report this, can you please
> point me to the right place?

I would ask on centos-devel.

It could be a problem with the mirror-manager configuration because the primary hashes might not be verified, so that outdated mirrors are used.

(In reply to Florian Weimer from comment #8)
> (In reply to Sandro Bonazzola from comment #7)
> > Indeed, that transaction has nothing to do with additional repos or modules
> > enabled.
> > Also, if bugzilla is not the right place to report this, can you please
> > point me to the right place?
> 
> I would ask on centos-devel.
> 
> It could be a problem with the mirror-manager configuration because the
> primary hashes might not be verified, so that outdated mirrors are used.

centos-devel is not responsible for centos stream, they pointed me to RHEL bugzilla for centos stream related issues.

(In reply to Sandro Bonazzola from comment #9)
> (In reply to Florian Weimer from comment #8)
> > (In reply to Sandro Bonazzola from comment #7)
> > > Indeed, that transaction has nothing to do with additional repos or modules
> > > enabled.
> > > Also, if bugzilla is not the right place to report this, can you please
> > > point me to the right place?
> > 
> > I would ask on centos-devel.
> > 
> > It could be a problem with the mirror-manager configuration because the
> > primary hashes might not be verified, so that outdated mirrors are used.
> 
> centos-devel is not responsible for centos stream, they pointed me to RHEL
> bugzilla for centos stream related issues.

Do you have a link to the discussion? Thanks.

It has been discussed on centos-devel IRC channel, here's the transcript:

Conversation with #centos-devel at gio 04 feb 2021 17:00:51 CET on sbonazzo.net (irc)

(17:06:32) Arrfab: is that for 8 or 8-stream ?
(17:06:35) sbonazzo: 3.5 is default one, but for some reason in cbs it isnt
(17:06:41) sbonazzo: 8
(17:06:45) Arrfab: and also, would it be possible to create a bug reporton correct tracker ?
(17:07:05) sbonazzo: Arrfab: I would be happy to file it in the right tracker, can you point me to the right one?
(17:07:09) Arrfab: hmm, listing maven shows 3.5.4 by default
(17:07:25) Arrfab: sbonazzo: as announced multiple times on centos-devel, it's on https://pagure.io/centos-infra/issues
(17:07:36) Arrfab: stream issues should be reported on RH bugzilla

Conversation with #centos-devel at gio 14 gen 2021 08:18:47 CET on sbonazzo.net (irc)
(17:22:26) mjturek: The syslinux-tftpboot package is going to be available on all arches for RHEL 8.4 but as far as jeremyfreudberg and I can tell, it is marked only for x86_64 on centos (at least in 8.3 and Stream). Can anyone give us some guidance on how to get it marked for release on all arches in 8.4 onward? We have this bug open https://bugs.centos.org/view.php?id=17917 (though it does need an update)
(17:24:45) ykarel è ora conosciuto come ykarel|away
(17:24:56) geppetto [~james.147.233] è entrato nella stanza.
(17:25:00) Evolution: mjturek: for stream, bugs should be in RH's bugzilla. there's a 'CentOS Stream' version for RHEL

Conversation with #centos-devel at mer 07 ott 2020 07:39:30 CEST on sbonazzo.net (irc)
(14:13:13) ykarel: @mantis 17139
(14:13:16) centbot: Bug 17139 - jcapitao - open - new
(14:13:17) centbot: Missing subpackage openssl-static in BaseOS CentOS8 - https://bugs.centos.org//view.php?id=17139
(14:28:39) bstinson: ykarel: we can try opening a bugzilla to get it included in CentOS Stream (PowerTools). that might jump-start the conversation
(14:31:30) ykarel: bstinson, can't it be added to Devel repo http://mirror.centos.org/centos/8/Devel/?
(14:31:49) ykarel: other subpacakges are available in BaseOS repo, just -static missing
(14:32:50) bstinson: we want to be sure that a package is appropriate to be shipped in RHEL before we ship it in Devel
(14:33:17) bstinson: a bugzilla against CentOS Stream would help continue the conversation that we've been having


I can dig more in my IRC client logs but every time something had to do with CentOS Stream the reply was to move it to RH Bugzilla.
And on RH Bugzilla most of the times I got bumped back as notabug, wontfix, worksforme claiming that on RHEL it works, things will be better with CentOS Stream 9 as it has a better pipeline and so on.

So, I'm a bit frustrated and I'm working in Red Hat. I can only guess how users how there can feel.

FTR, I am used to filling C9S bugs here in RHBZ as well: https://wiki.centos.org/ReportBugs

(In reply to Pavel Raiskup from comment #12)
> FTR, I am used to filling C9S bugs here in RHBZ as well:
> https://wiki.centos.org/ReportBugs

This is supposedly for CentOS 7 only per a recent centos-devel discussion.

(In reply to Sandro Bonazzola from comment #11)
> I can dig more in my IRC client logs but every time something had to do with
> CentOS Stream the reply was to move it to RH Bugzilla.

This issue is either specific to oVirt, or it's a CentOS mirror-manager/CDN issue. I'm simply not aware of a Bugzilla component for that.

I'll ask around internally.

For the record, we are getting similar reports these days, like:
https://github.com/rpm-software-management/mock/issues/893

(In reply to Florian Weimer from comment #13)
> This issue is either specific to oVirt, or it's a CentOS mirror-manager/CDN issue. I'm simply not aware of a Bugzilla component for that.

In the mock issue above, no additional repository is enabled.  This really
seems to be a repo mirroring issue.

> I'll ask around internally.

Any progress on this?

Note the weirdness, there are two DNF transactions (if we exclude the third
bootstrap-related one) -- 1) first installs the minimal build deps, and 2)
does dnf update.  Mock precisely caches the metadata, yet the metadata
was re-downloaded for the 2) transaction.  This means DNF picked a different
mirror in the 2) transaction which broke the build.

Per the dnf.librepo.log below, it seems the broken mirror is
http://mirrors.xmission.com/centos/8-stream/BaseOS/x86_64


First transaction:

No matches found for the following disable plugin patterns: local, spacewalk, versionlock
Copr repository                                 2.7 kB/s | 257  B     00:00
CentOS Stream 8 - BaseOS                        8.7 MB/s |  21 MB     00:02
CentOS Stream 8 - AppStream                      12 MB/s |  20 MB     00:01
CentOS Stream 8 - Extras                         74 kB/s |  18 kB     00:00
CentOS Stream 8 - PowerTools                     17 MB/s | 4.5 MB     00:00
Dependencies resolved.
======================================================================================
 Package                       Arch    Version                        Repo        Size
======================================================================================
Installing:
 bash                          x86_64  4.4.20-3.el8                   baseos     1.5 M
 bzip2                         x86_64  1.0.6-26.el8                   baseos      60 k
 centos-stream-release         noarch  8.6-1.el8                      baseos      22 k
...

Second transaction:

Copr repository                                  37 kB/s | 3.3 kB     00:00
CentOS Stream 8 - BaseOS                        3.4 MB/s |  21 MB     00:06
CentOS Stream 8 - AppStream                      13 MB/s |  20 MB     00:01
CentOS Stream 8 - Extras                        5.0 kB/s |  18 kB     00:03
CentOS Stream 8 - PowerTools                    9.0 MB/s | 4.5 MB     00:00
Dependencies resolved.
================================================================================
 Package                  Arch        Version             Repository       Size
================================================================================
Upgrading:
 glibc                    x86_64      2.28-196.el8        baseos          2.2 M
 glibc-all-langpacks      x86_64      2.28-196.el8        baseos           26 M
...


Logs from the second transaction (not sure why we don't have all dumped here):
https://download.copr.fedorainfracloud.org/results/jered/trafficserver/centos-stream-8-x86_64/04199835-trafficserver/chroot_scan/var/lib/mock/centos-stream-8-x86_64-1649802754.722392/root/var/log/dnf.librepo.log
...
2022-04-12T22:35:14+0000 INFO Downloading: http://mirror.facebook.net/centos/8-stream/PowerTools/x86_64/os/repodata/557a1df4fd8f29d0ec41777aa4b5f3d862ddb1ee0c2908a9735699cc36f57320-comps-PowerTools.x86_64.xml
2022-04-12T22:35:14+0000 INFO Downloading: http://mirror.facebook.net/centos/8-stream/PowerTools/x86_64/os/repodata/a4629df40556a947ce599692de8e8448aeaf77d4c6ce5d8a52f43e7f99ea5b9a-modules.yaml.xz
2022-04-12T22:35:15+0000 INFO Downloading: http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=BaseOS&infra=stock
2022-04-12T22:35:15+0000 INFO Downloading: http://mirrors.xmission.com/centos/8-stream/BaseOS/x86_64/os/Packages/glibc-2.28-196.el8.x86_64.rpm
2022-04-12T22:35:15+0000 INFO Downloading: http://mirrors.xmission.com/centos/8-stream/BaseOS/x86_64/os/Packages/glibc-all-langpacks-2.28-196.el8.x86_64.rpm
...

Still... upgrade of glibc related packages shouldn't remove rpm-build right?

(In reply to Sandro Bonazzola from comment #16)
> Still... upgrade of glibc related packages shouldn't remove rpm-build right?

From the DNF perspective, it's the expected outcome for this repository configuration because there is no way to satisfy the installation request otherwise.

(In reply to Florian Weimer from comment #17)
> (In reply to Sandro Bonazzola from comment #16)
> > Still... upgrade of glibc related packages shouldn't remove rpm-build right?
> 
> From the DNF perspective, it's the expected outcome for this repository
> configuration because there is no way to satisfy the installation request
> otherwise.

AFAIK `--allowerasing` wasn't passed. In this condition I would have expected dnf to fail asking to use `--allowerasing` or `--nobest` in order to perform the update.

(In reply to Sandro Bonazzola from comment #18)
> AFAIK `--allowerasing` wasn't passed. In this condition I would have
> expected dnf to fail asking to use `--allowerasing` or `--nobest` in order
> to perform the update.

It's there?

2022-04-12T22:35:02+0000 DDEBUG Extra commands: ['--installroot', '/var/lib/mock/centos-stream-8-x86_64-1649802754.722392/root/', '-y', '--releasever', '8', '--setopt=deltarpm=False', '--allowerasing', '--disableplugin=local', '--disableplugin=spacewalk', '--disableplugin=versionlock', 'update', '--setopt=tsflags=nocontexts']

Yes, since: https://github.com/rpm-software-management/mock/issues/251

I guess next step is opening an issue on mock to protect rpm-build from being accidentally removed then?