Bug 2046690

Summary: CentOS Stream 9 repos are broken
Product: Red Hat Enterprise Linux 9 Reporter: Alfredo Moralejo <amoralej>
Component: distributionAssignee: Brian Stinson <bstinson>
Status: CLOSED CURRENTRELEASE QA Contact: Release Test Team <release-test-team-automation>
Severity: urgent Docs Contact:
Priority: urgent    
Version: CentOS StreamCC: adrian, alex.iribarren, apevec, bstinson, carl, jhughes, jwboyer, riehecky
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-28 16:10:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2021809    
Bug Blocks:    

Description Alfredo Moralejo 2022-01-27 09:03:23 UTC
Description of problem:

compose-id is CentOS-Stream-9-20220126.0

After last compose CS9 promotion, BaseOS/AppStream repos seems broken:

- [root@6b5a57fab5b4 /]# dnf install gcc
- Last metadata expiration check: 0:00:25 ago on Thu Jan 27 08:45:21 2022.
- Error:
-  Problem: cannot install the best candidate for the job
-   - nothing provides libgomp = 11.2.1-7.6.el9 needed by
gcc-11.2.1-7.6.el9.x86_64
-   - nothing provides libgcc >= 11.2.1-7.6.el9 needed by
gcc-11.2.1-7.6.el9.x86_64
- (try to add '--skip-broken' to skip uninstallable packages or '--nobest'
to use not only best candidate packages)
- [root@6b5a57fab5b4 /]# dnf install gcc-c++
- Last metadata expiration check: 0:01:56 ago on Thu Jan 27 08:45:21 2022.
- Error:
-  Problem: cannot install the best candidate for the job
-   - nothing provides libstdc++ = 11.2.1-7.6.el9 needed by
gcc-c++-11.2.1-7.6.el9.x86_64
- (try to add '--skip-broken' to skip uninstallable packages or '--nobest'
to use not only best candidate packages)
- [root@6b5a57fab5b4 /]#

Repoclosure is also complaining:

- package: gcc-11.2.1-7.6.el9.x86_64 from appstream
-   unresolved deps:
-     libgcc >= 11.2.1-7.6.el9
-     libgomp = 11.2.1-7.6.el9
- package: gcc-c++-11.2.1-7.6.el9.x86_64 from appstream
-   unresolved deps:
-     libstdc++ = 11.2.1-7.6.el9
- package: gcc-gfortran-11.2.1-7.6.el9.x86_64 from appstream
-   unresolved deps:
-     libquadmath = 11.2.1-7.6.el9
-     libgfortran = 11.2.1-7.6.el9
- package: libgomp-offload-nvptx-11.2.1-7.6.el9.x86_64 from appstream
-   unresolved deps:
-     libgomp = 11.2.1-7.6.el9
- package: libquadmath-devel-11.2.1-7.6.el9.i686 from appstream
-   unresolved deps:
-     libquadmath = 11.2.1-7.6.el9
- package: libquadmath-devel-11.2.1-7.6.el9.x86_64 from appstream
-   unresolved deps:
-     libquadmath = 11.2.1-7.6.el9
- package: libstdc++-devel-11.2.1-7.6.el9.i686 from appstream
-   unresolved deps:
-     libstdc++(x86-32) = 11.2.1-7.6.el9
- package: libstdc++-devel-11.2.1-7.6.el9.x86_64 from appstream
-   unresolved deps:
-     libstdc++(x86-64) = 11.2.1-7.6.el9

Comment 1 Alfredo Moralejo 2022-01-27 09:27:34 UTC
Issue seems similar to https://bugzilla.redhat.com/show_bug.cgi?id=2021809

Issue may be related to picking different mirrors for different repos:

Last metadata expiration check: 0:00:09 ago on Thu Jan 27 09:23:05 2022.
Repo-id            : appstream
Repo-name          : CentOS Stream 9 - AppStream
Repo-revision      : 9-stream
Repo-distro-tags      : [cpe:/o:centos-stream:centos-stream:9]:  ,  , 9, C, O, S, S, a, e, e, m, n, r, t, t
Repo-updated       : Wed Jan 26 21:19:21 2022
Repo-pkgs          : 9507
Repo-available-pkgs: 9507
Repo-size          : 13 G
Repo-metalink      : https://mirrors.centos.org/metalink?repo=centos-appstream-9-stream&arch=x86_64&protocol=https,http
  Updated          : Thu Jan 27 09:23:05 2022
Repo-baseurl       : http://mirror.netsite.dk/centos-stream/9-stream/AppStream/x86_64/os/ (27 more)
Repo-expire        : 21600 second(s) (last: Thu Jan 27 09:23:05 2022)
Repo-filename      : /etc/yum.repos.d/centos.repo

Repo-id            : baseos
Repo-name          : CentOS Stream 9 - BaseOS
Repo-revision      : 9-stream
Repo-distro-tags      : [cpe:/o:centos-stream:centos-stream:9]:  ,  , 9, C, O, S, S, a, e, e, m, n, r, t, t
Repo-updated       : Tue Jan 25 15:31:18 2022
Repo-pkgs          : 2761
Repo-available-pkgs: 2761
Repo-size          : 3.0 G
Repo-metalink      : https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http
  Updated          : Thu Jan 27 09:23:00 2022
Repo-baseurl       : https://mirror1.hs-esslingen.de/pub/Mirrors/centos-stream/9-stream/BaseOS/x86_64/os/ (33 more)
Repo-expire        : 21600 second(s) (last: Thu Jan 27 09:23:00 2022)
Repo-filename      : /etc/yum.repos.d/centos.repo
Total packages: 12268


If i force to use the same mirror for both repos it seems to work fine.

IMO, there is something wrong with the way we sync and pick mirrors.

Comment 2 Adrian Reber 2022-01-27 09:55:26 UTC
Should be fixed 20 after the hour. This is the known problem of scanning the primary mirror for changes while it is being updated.

This scan was triggered by the COMPOSE_ID in SIGs telling that there are changes in 'SIGs/'. The scan, however, happened during a time that '9-stream/' was being updated so that we scanned the primary mirror while files are being copied to the primary mirror which resulted in an incomplete scan.

This is known and we still have not implemented a fix currently.

Comment 4 Alfredo Moralejo 2022-01-28 14:36:51 UTC
Similar issue today for CentOS Stream8:

Error: 
 Problem: cannot install the best candidate for the job
  - nothing provides libgcc >= 8.5.0-8.el8 needed by gcc-8.5.0-8.el8.x86_64
  - nothing provides libgomp = 8.5.0-8.el8 needed by gcc-8.5.0-8.el8.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Comment 5 Adrian Reber 2022-01-28 14:41:34 UTC
(In reply to Alfredo Moralejo from comment #4)
> Similar issue today for CentOS Stream8:
> 
> Error: 
>  Problem: cannot install the best candidate for the job
>   - nothing provides libgcc >= 8.5.0-8.el8 needed by gcc-8.5.0-8.el8.x86_64
>   - nothing provides libgomp = 8.5.0-8.el8 needed by gcc-8.5.0-8.el8.x86_64
> (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to
> use not only best candidate packages)

As this is CentOS Stream 8 it does not really belong in this ticket as the backend systems are different.