Bug 2102548

(In reply to Wong Hoi Sing Edison from comment #0)
> Created attachment 1893558 [details]
> nothing provides glibc(x86-64) = 2.28-203.el8 needed by glibc-gconv-extra
> 
> Description of problem:
> 
> From
> https://linux.cc.iitk.ac.in/mirror/centos/8/AppStream/x86_64/os/Packages/ we
> have
> https://linux.cc.iitk.ac.in/mirror/centos/8/AppStream/x86_64/os/Packages/
> glibc-gconv-extra-2.28-206.el8.i686.rpm, but missing
> https://linux.cc.iitk.ac.in/mirror/centos/8/AppStream/x86_64/os/Packages/
> glibc-gconv-extra-2.28-206.el8.x86_64.rpm; it is only available from BaseOS
> with
> https://linux.cc.iitk.ac.in/mirror/centos/8/BaseOS/x86_64/os/Packages/glibc-
> gconv-extra-2.28-206.el8.x86_64.rpm.
> 
> This lead to https://build.opensuse.org report error message as below (in
> case we setup AppStream as higher priority than BaseOS, which usually
> working well):
> 
> > nothing provides glibc(x86-64) = 2.28-203.el8 needed by glibc-gconv-extra, (got version 2.28-206.el8 provided by glibc), nothing provides glibc-common = 2.28-203.el8 needed by glibc-gconv-extra, (got version 2.28-206.el8)

This really shouldn't cause any problems for DNF. I suspect the error is coming from something else. Could you share a longer log message with the full error message? Thanks.

Created attachment 1893658 [details]
https://build.opensuse.org/projects/home:alvistack/meta

> This really shouldn't cause any problems for DNF. I suspect the error is coming from something else. Could you share a longer log message with the full error message? Thanks.

Agree that may not affect DNF, because we will load in ALL repos with whatever order, then calculate the highest available and suitable version, pick it up and install correctly. So even if AppStream unexpectedly missing glibc-gconv-extra-2.28-206.el8.x86_64.rpm, we will pick it up from BaseOS so don't break anything.

But sorry that with OBS limitation as mentioned from https://en.opensuse.org/openSUSE:Build_Service_Tips_and_Tricks#Adding_multiple_repositories_to_a_project:

> The order of the included repositories is important: The BuildService will try to use a package from the first repository that includes the package, even if the version doesn't match.

As screenshot attached, since my project setup with AppStream > BaseOS, in case AppStream already have glibc-gconv-extra with legacy version 2.28-203 only, OBS will pick it up because 2.28-206 is now missing, and will not fallback to BaseOS for searching available package.

IMHO, since we have all expected legacy version for glibc-gconv-extra with both i686 and x86_64 but nwo only missing for 2.28-206.el8.x86_64, it seems to be a simple careless bug for me which could be fixed easily?

Interesting. CentOS Koji publishes buildroot repositories here:

https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/

These repositories are not split along the BaseOS/AppStream lines and should therefore work with OBS. The package versions should match what is available in the compose (but the compose may not have hit the CentOS mirror network yet).

Can you switch to these repositories as a workaround? Thanks.

> Can you switch to these repositories as a workaround? Thanks.

First of all I am not an OBS admin but a normal user, so I have no right for switching the repo...

Moreover, from https://build.opensuse.org/projects/CentOS:CentOS-8:Stream/meta, their AppStream repo is now pointing to http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/, which also missing http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/glibc-gconv-extra-2.28-206.el8.x86_64.rpm...

Finally, I always checking package availability with https://pkgs.org/search/?q=glibc-gconv-extra, but also show that missing glibc-gconv-extra-2.28-206.el8.x86_64.rpm from CentOS 8 Stream officially...

@fweimer I give different workaround with my own https://build.opensuse.org/projects/home:alvistack/meta, but:

* If AppStream > BaseOS (the style which working fine for last year until this bug report), only glibc-gconv-extra hit the version miss match issue

* If BaseOS > AppStream, now libxml2-devel get version miss match with libxml2...

* If remove AppStream but only keep BaseOS, now multiple packages get version miss match, too...

Both behavior happened with CentOS 8 Stream and CentOS 9 Stream, which clearly show that CentOS Stream now a day heavily depends on AppStream repo (as a replacement for CentOS 7 update repo).

Understand that CentOS Koji didn't split BaseOS/AppStream lines, but package is now disappeared from all mirror network, which seems like some unexpected pipeline error?

glibc-gconv-extra hasn't disappeared though, it moved to BaseOS:
http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/glibc-gconv-extra-2.28-206.el8.x86_64.rpm

I also notice that the i686 package stayed in AppStream:
http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/glibc-gconv-extra-2.28-206.el8.i686.rpm

@fweimer do we need to address that?

From https://pkgs.org/search/?q=glibc and https://pkgs.org/search/?q=glibc-gconv-extra, showing that:

* For BaseOS, originally glibc-gconv-extra <= 2.28-180 (but now, unexpectedly 2.28-206 push to here)

* For AppStream, originally glibc-gconv-extra >= 2.28-181 (but now, unexpectedly 2.28-206 is missing)

Since now glibc and glibc-common from BaseOS already go to 2.28-206, if OBS repo order config as AppStream > BaseOS, normally it should pick up glibc-gconv-extra = 2.28-206 from AppStream, and so no error happened.

IMHO, the move of glibc-gconv-extra = 2.28-206 from BaseOS to AppStream shouldn't be a feature but a bug?

Also see https://lists.opensuse.org/archives/list/buildservice@lists.opensuse.org/thread/UJ4ZNOKJ2AN3NUUUYUH4M4K2FSZUI3D4/

(In reply to Wong Hoi Sing Edison from comment #5)
> * If BaseOS > AppStream, now libxml2-devel get version miss match with
> libxml2...

I asked internally, and prioritizing BaseOS over AppStream is supposed to work because we never remove packages from BaseOS and add them to AppStream. This means that the version in BaseOS is always newer.

If that's not true for libxml2-devel, that is a separate bug, but I do not immediately see that this is the case.

> Both behavior happened with CentOS 8 Stream and CentOS 9 Stream, which
> clearly show that CentOS Stream now a day heavily depends on AppStream repo
> (as a replacement for CentOS 7 update repo).

AppStream has always been an integral part.  I think the difference relative to 7 is that CentOS now follows the Red Hat Enterprise Linux repository structure more closely, which leads to fewer surprises for those of us who live in both worls.

> Understand that CentOS Koji didn't split BaseOS/AppStream lines, but package
> is now disappeared from all mirror network, which seems like some unexpected
> pipeline error?

Which package has disappeared? I don't think CentOS removes packages.

(In reply to Brian Stinson from comment #6)
> glibc-gconv-extra hasn't disappeared though, it moved to BaseOS:
> http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/glibc-
> gconv-extra-2.28-206.el8.x86_64.rpm
> 
> I also notice that the i686 package stayed in AppStream:
> http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/glibc-
> gconv-extra-2.28-206.el8.i686.rpm
> 
> @fweimer do we need to address that?

The packages should be in BaseOS for all architectures. This is what we requested for Red Hat Enterprise Linux 8 & 9, and the repositories seem to match.

(In reply to Wong Hoi Sing Edison from comment #7)
> Also see
> https://lists.opensuse.org/archives/list/buildservice@lists.opensuse.org/
> thread/UJ4ZNOKJ2AN3NUUUYUH4M4K2FSZUI3D4/

I commented on that thread as well, thanks.

Brian, do we need to keep this bug open to track the CentOS change, or can we close it? Thanks.

If there are no further issues, let's reassign this to the distribution component and I'll take assignment. We'll complete the move to BaseOS in any case.

Created attachment 1894009 [details]
nothing provides libxml2(x86-64) = 2.9.7-11.el8 needed by libxml2-devel, (got version 2.9.7-14.el8 provided by libxml2)

For your reference if I move BaseOS > AppStream (which means if they both have a specific package with different version, OBS will ALWAYS install it and its dependency from BaseOS ONLY), now most packages could compile successfully.

BTW, still 15 packages failed due to "nothing provides libxml2(x86-64) = 2.9.7-11.el8 needed by libxml2-devel, (got version 2.9.7-14.el8 provided by libxml2)", which I am going to report as another ticket.

Moreover, if "glibc-gconv-extra should move from AppStream to BaseOS" as official decision, could we move ALL LEGACY AppStream glibc-gconv-extra packages to BaseOS, too?

CentOS 8 Stream builds are still breaking on OBS for us for the above reason. This is pretty frustrating because we don't know who can fix this — the OBS folks or CentOS, or us? Has anyone identified a workaround for folks who just want to package code for a bunch of platforms on OBS?

Fwiw, here are related threads:

https://lists.opensuse.org/archives/list/buildservice@lists.opensuse.org/thread/2CWV33W5XZNKT6WJRDBUPAANGWQHDH5H/
https://lists.opensuse.org/archives/list/buildservice@lists.opensuse.org/thread/A2T7CATUGIDFCO7JFTNKWFTFBPTY3I7Z/

And the gist of the matter on OBS:

https://build.opensuse.org/project/monitor/security:zeek?arch_x86_64=1&defaults=0&repo_CentOS_8_Stream=1&unresolvable=1

Thanks!

(In reply to Florian Weimer from comment #3)
> Interesting. CentOS Koji publishes buildroot repositories here:
> 
> https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/
> 
> These repositories are not split along the BaseOS/AppStream lines and should
> therefore work with OBS. The package versions should match what is available
> in the compose (but the compose may not have hit the CentOS mirror network
> yet).
> 
> Can you switch to these repositories as a workaround? Thanks.

It unlikely that the openSUSE Build Service admins would use that, since neither the RPMs nor the repodata are signed. OBS relies on at least repodata signing to verify the integrity of build inputs.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.

This ticket is not visible in Jira — neither at the linked https://issues.redhat.com/browse/RHEL-3756 nor via the https://issues.redhat.com/issues/?jql=%22Bugzilla%20Bug%22%20%3D%202102548 search.

(In reply to Christian from comment #17)
> This ticket is not visible in Jira — neither at the linked
> https://issues.redhat.com/browse/RHEL-3756 nor via the
> https://issues.redhat.com/issues/?jql=%22Bugzilla%20Bug%22%20%3D%202102548
> search.

Please open a support ticket at <https://access.redhat.com/support/cases/> if you have repository access issues. Thanks.