Bug 2127325

(In reply to Maxwell G from comment #0)
> Description of problem:
> 
> The new python39 module that was pushed out in the CentOS Stream 8 compose a
> couple days ago is missing almost all of its packages. This makes it
> impossible for me to build any python39 packages in EPEL 8 Next.
> 
> This is the third or fourth time that one of the python modules has broken
> in Stream 8. I understand that these issues are an unfortunate product of
> the poor CentOS Stream 8 workflow and modularity, but it would be great to
> have testing in place to prevent this in the future

Hi Maxwell,
apologies for the problems, I hope they'll be addressed promptly by the CentOS Stream team.

However, speaking on behalf of the Python-maint team that's developing these alternative Python stacks, we're working on a new solution for the next alternative stack in RHEL/CentOS Stream 8 that will hopefully improve upon this.

Thank you for the response, Tomáš. If I may, I'd like to suggest making the next alternative python stack non-modular (if you weren't already planning that). In my opinion, there's not any point for the alternative Python stacks to be modular. Each python module is already parallel install-able and split into its own module with one default stream. This already defeats the purpose of modularity. Besides the broken CentOS Stream modules, the python38 and python39 modules have caused various other problems for both RHEL and EPEL packages:

- For EPEL, there was an issue with grobisplitter not handling the python38-devel and python39-devel CRB modules, which hampered us from building Python 3.8 packages in EPEL for three months.
- The python3X-devel modules not being enabled by default also causes a lot of friction when building packages locally in mock, because you need a special mock configuration. Multiple packagers have had issues with this.

- For RHEL, this has caused problems with ansible-core. In CentOS Stream 8, ansible-core requires a special buildtag, because it depends on python39 packages but isn't part of the python39 module. This wasn't set up properly at first, and the ansible-core package failed to build in c8s for months. This happened once with the python38 version and again after the python39 rebase.
- For rhel-system-roles and the other ansible-core dependents in RHEL 8, the maintainers have had to resort to hacky workarounds to build Ansible collections due to ansible-core not being part of the default buildroot. These workarounds have leaked into the Fedora versions of these packages and caused issues.

I doubt the average customer/user cares whether the alt Python stacks are modular or not.

I apologize if my previous message was a bit harsh/ranty. I appreciate your work on this and understand that the issues aren't your fault.

Thank you for your kind message, Maxwell.

I am similarly of the opinion that modularity does not greatly align with the use cases of Python alternative stacks. For a while we've been exploring options to de-modularize Python 3.11, but as yet I cannot make any promises. Fingers crossed!

Carl, do you know who can fix this?

I check the recent python39 module builds, and the python39:3.9:8070020220908144956:be1f0497 one was missing components.  It was part of the following composes:

CentOS-Stream-8-20220912.n.0
CentOS-Stream-8-20220913.n.0
CentOS-Stream-8-20220915.n.0

I don't have a record of exactly when (or even if) each compose was pushed to the mirrors, but based on timing (the original comment was on 2022-09-16) it's likely it was one of these that shipped the broken module.  A new module export happened on 2022-09-20, which Johnny then built as python39:3.9:8070020220920211220:be1f0497.  Thankfully this build included all the necessary components, and was included in the following composes:

CentOS-Stream-8-20220921.n.0
CentOS-Stream-8-20220922.n.0
CentOS-Stream-8-20220922.n.1

The last one is what is currently on the mirrors (see http://mirror.centos.org/centos/8-stream/COMPOSE_ID).  I spot checked a few packages and they install fine, so this appears to be resolved.  If something is missing please re-open this bug for further investigation.

I believe the root cause of the issue (and why it keeps happening) is the old versions of koji and MBS that are deployed for c8s infrastructure.  The inside-out workflow for c8s doesn't help.  No one has worked on fixing it because the Stream team is focused on migrating c8s to the c9s workflows, which is expected to be completed sometime this year.  That change should stop the problem from happening again because of the newer koji and MBS versions used there.