Can you provide an installation log that shows rhc being installed?  It is not listed in the @core comps group at all, and from what I can tell it is not pulled in by dependencies.

It is part of @core in RHEL; do RHEL and CentOS use the same comps groups?

This was an oversight on my part. insights-client is not shipped in CentOS. I'll switch the dependency to Recommends. I think that should prevent the breakage in dependency resolution.

(In reply to Link Dupont from comment #2)
> It is part of @core in RHEL; do RHEL and CentOS use the same comps groups?

No it isn't.  It's part of @standard/@base, not @core.  That's only in RHEL, not in CentOS Stream, which doesn't reference rhc in any installation group.  That makes me wonder how it got installed on Jesper's system to begin with.

> This was an oversight on my part. insights-client is not shipped in CentOS.
> I'll switch the dependency to Recommends. I think that should prevent the
> breakage in dependency resolution.

Josh, when i list the packages in the core group, I see `rhc` listed under "default packages" for both Stream 8 and 9.  So I'm not sure what you mean, when saying it doesn't reference `rhc`? 

Only thing that was also done to the system I have access to right now, is that the @^Workstation was added after installation in order to provide desktop functionality to the developpers.
I have added the %packages section from the Kickstart file below, but I can currently only convince myself that it got installed from the core group.  It doesn't seem that any other packages depend on it, despite the `rhc-devel` package, and I don't have that one installed.


```
$ cat /etc/redhat-release 
CentOS Stream release 8

$ dnf group info core | grep rhc
   rhc
# its listed under the default packages.
```

```
$ cat /etc/redhat-release 
CentOS Stream release 9

$ dnf group info core | grep rhc
   rhc
# Also listed under default packages

$ dnf group info core
Last metadata expiration check: 3:02:10 ago on Thu 17 Mar 2022 05:27:29 PM CET.
Group: core
 Description: Minimal host installation
 Mandatory Packages:
   audit
   basesystem
   bash
   coreutils
   cronie
   crypto-policies
   crypto-policies-scripts
   curl
   dnf
   e2fsprogs
   filesystem
   firewalld
   glibc
   grubby
   hostname
   iproute
   iproute-tc
   iputils
   irqbalance
   kbd
   kexec-tools
   less
   logrotate
   man-db
   ncurses
   openssh-clients
   openssh-server
   p11-kit
   parted
   passwd
   policycoreutils
   procps-ng
   rootfiles
   rpm
   rpm-plugin-audit
   rsyslog
   selinux-policy-targeted
   setup
   shadow-utils
   sssd-common
   sssd-kcm
   sudo
   systemd
   util-linux
   vim-minimal
   xfsprogs
   yum
 Default Packages:
   NetworkManager
   NetworkManager-team
   NetworkManager-tui
   authselect
   dnf-plugins-core
   dracut-config-rescue
   initscripts-rename-device
   iwl100-firmware
   iwl1000-firmware
   iwl105-firmware
   iwl135-firmware
   iwl2000-firmware
   iwl2030-firmware
   iwl3160-firmware
   iwl5000-firmware
   iwl5150-firmware
   iwl6000g2a-firmware
   iwl6050-firmware
   iwl7260-firmware
   kernel-tools
   libsysfs
   linux-firmware
   lshw
   lsscsi
   microcode_ctl
   prefixdevname
   python3-libselinux
   rhc
   sg3_utils
   sg3_utils-libs
 Optional Packages:
   dracut-config-generic
   dracut-network
   initial-setup
   rdma-core
   selinux-policy-mls
   tboot

```


```
$ dnf group list --installed
Last metadata expiration check: 0:00:15 ago on Thu 17 Mar 2022 08:36:07 PM CET.
Installed Environment Groups:
   Minimal Install
   Workstation

$ dnf group info Workstation
Last metadata expiration check: 0:00:24 ago on Thu 17 Mar 2022 08:36:07 PM CET.
Environment Group: Workstation
 Description: Workstation is a user-friendly desktop system for laptops and PCs.
 Mandatory Groups:
   GNOME
   Internet Browser
   Workstation product core
   base-x
   core
   fonts
   guest-desktop-agents
   hardware-support
   multimedia
   networkmanager-submodules
   print-client
   standard
 Optional Groups:
   Backup Client
   GNOME Applications
   Headless Management
   Internet Applications
   Office Suite and Productivity
   Remote Desktop Clients
   Smart Card Support
```


It looks like we didn't enable any logging in the Kickstart file, but the following are the %packages section from the file, and then it only removes `*-firmware` in the %post section.

```
%packages
@^minimal-environment # Includes @core

python3

# vmware tools agent
open-vm-tools

# make sure realmd is installed
realmd
sssd
adcli
oddjob
oddjob-mkhomedir

bash-completion

# remove cockpit
-cockpit
-cockpit-*

# Pull firmware packages out, as we don't need it in virt.
#
# Apparently its not valid to remove the `linux-firmware` package
# here.  This will result in Anaconda failing on the selected package
# group.  Thus we remove it in the %post step.
##-linux-firmware
-aic94xx-firmware
-alsa-firmware
-alsa-lib
-alsa-tools-firmware
-ivtv-firmware
-iwl1000-firmware
-iwl100-firmware
-iwl105-firmware
-iwl135-firmware
-iwl2000-firmware
-iwl2030-firmware
-iwl3160-firmware
-iwl3945-firmware
-iwl4965-firmware
-iwl5000-firmware
-iwl5150-firmware
-iwl6000-firmware
-iwl6000g2a-firmware
-iwl6000g2b-firmware
-iwl6050-firmware
-iwl7260-firmware
-libertas-sd8686-firmware
-libertas-sd8787-firmware
-libertas-usb8388-firmware

%end
```

[root@localhost ~]# cat /etc/os-release 
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
[root@localhost ~]# dnf group info core | grep rhc
[root@localhost ~]# 
[root@localhost ~]# dnf group info core 
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Last metadata expiration check: 0:57:22 ago on Thu 17 Mar 2022 02:54:32 PM EDT.
Group: Core
 Description: Minimal host installation
 Mandatory Packages:
   audit
   basesystem
   bash
   coreutils
   cronie
   crypto-policies
   crypto-policies-scripts
   curl
   dnf
   e2fsprogs
   filesystem
   firewalld
   glibc
   grubby
   hostname
   iproute
   iproute-tc
   iputils
   irqbalance
   kbd
   kexec-tools
   less
   logrotate
   man-db
   ncurses
   openssh-clients
   openssh-server
   p11-kit
   parted
   passwd
   policycoreutils
   procps-ng
   rootfiles
   rpm
   rpm-plugin-audit
   rsyslog
   selinux-policy-targeted
   setup
   shadow-utils
   sssd-common
   sssd-kcm
   sudo
   systemd
   util-linux
   vim-minimal
   xfsprogs
   yum
 Default Packages:
   NetworkManager
   NetworkManager-team
   NetworkManager-tui
   authselect
   dnf-plugins-core
   dracut-config-rescue
   initscripts-rename-device
   iwl100-firmware
   iwl1000-firmware
   iwl105-firmware
   iwl135-firmware
   iwl2000-firmware
   iwl2030-firmware
   iwl3160-firmware
   iwl5000-firmware
   iwl5150-firmware
   iwl6000g2a-firmware
   iwl6050-firmware
   iwl7260-firmware
   kernel-tools
   libsysfs
   linux-firmware
   lshw
   lsscsi
   microcode_ctl
   prefixdevname
   python3-libselinux
   sg3_utils
   sg3_utils-libs
 Optional Packages:
   dracut-config-generic
   dracut-network
   initial-setup
   rdma-core
   selinux-policy-mls
   tboot
[root@localhost ~]# 


I cannot recreate what you see.  Perhaps you have a stale mirror?  If you look at the comps file directly on the mirrors, rhc is not mentioned.

http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/repodata/060f9c4a2b0e177e4b2f0b19faa7f23ddd6b8a5bf35f166f3b74c4d9c840aa26-comps-BaseOS.x86_64.xml
http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/repodata/83a8b2b4081703932220fb51fbcaa0fbbf2f6268105a41fc28fff7b388a94115-comps-AppStream.x86_64.xml

If you look at a non-mirror production compose you will also find rhc is not mentioned.

https://odcs.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/repodata/060f9c4a2b0e177e4b2f0b19faa7f23ddd6b8a5bf35f166f3b74c4d9c840aa26-comps-BaseOS.x86_64.xml
https://odcs.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/repodata/83a8b2b4081703932220fb51fbcaa0fbbf2f6268105a41fc28fff7b388a94115-comps-AppStream.x86_64.xml

What repos is your machine pointing to?

I walked through a different machine and discovered that while rhc isn't included in Stream 9 comps at all, it is included in the EPEL 9 comps groups for some reason I cannot possibly explain.

https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/repodata/8a0d561b583b60a7c6aae31c6342fd8b961a58e26e1b7bfc3db84d364e240922-comps-Everything.x86_64.xml

So, Link is correct this should be a Recommends and we can use this bug to track the rhc changes.  The EPEL comps groups make no sense though.  Carl, can you look at why EPEL even has a "core" group, and why on earth it has a package it doesn't ship listed in there?

Josh, 

Awesome explanation.  I would newer have imagined that EPEL would overlap in such a way.  We are using it to get some packages, however that is added later and would not really explain why I ended up with the rhc package in the first place.  Unless it pulls it in during an `dnf update` after the epel repo has been added.

I will have to try and add some logging output from our Kickstart file and see if I can have it dump info on what packages are installed.  and try and pinpoint exactly when in the process it goes wrong.

I should hopefully not be using stale mirrors, as I'm using the official one:

```
$ cat /etc/yum.repos.d/centos.repo
[baseos]
name=CentOS Stream $releasever - BaseOS
metalink=https://mirrors.centos.org/metalink?repo=centos-baseos-$stream&arch=$basearch&protocol=https,http                                                                                                        
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
gpgcheck=1
repo_gpgcheck=0
metadata_expire=6h
countme=1
enabled=1

[...]
```

However the dvd1 I'm installing from was from a compose that added the secure boot signing of the bootloader, which I got from a another BZ.  I unpacked the iso, and looked at the BaseOS/repodata/40232bada337516d6e75012012436d8138a529f3ac7171f07fbef5bd6c6e6a3a-comps-BaseOS.x86_64.xml and AppStream/repodata/7ef9f9e67bbfab203ee8b351b625ece9b09f212d651dd423eb2e69b1d1e4eb25-comps-AppStream.x86_64.xml and truly there is no mention of rhc in there.

I just confirmed with a fresh install, that it doesn't install the rhc package.  It gets installed when I'm installing the Workstation environment group, **after** enabling the Epel repository, just as Josh pointed out:


```
$ sudo dnf group install workstation | head -10 # Test that it works without errors
Last metadata expiration check: 0:22:09 ago on Sat 19 Mar 2022 03:26:58 PM CET.
No match for group package "insights-client"
No match for group package "redhat-release"
No match for group package "redhat-release-eula"
Dependencies resolved.
======================================================================================================
 Package                                     Arch    Version                          Repo        Size
======================================================================================================
Upgrading:
 NetworkManager                              x86_64  1:1.37.2-1.el9                   baseos     2.2 M
 NetworkManager-libnm                        x86_64  1:1.37.2-1.el9                   baseos     1.7 M
 NetworkManager-team                         x86_64  1:1.37.2-1.el9                   baseos      33 k
 NetworkManager-tui                          x86_64  1:1.37.2-1.el9                   baseos     227 k


$ sudo dnf install -y --enablerepo crb epel-release epel-next-release
Last metadata expiration check: 0:03:47 ago on Sat 19 Mar 2022 03:46:39 PM CET.
Dependencies resolved.
================================================================================
 Package                 Architecture Version         Repository           Size
================================================================================
Installing:
 epel-next-release       noarch       9-2.el9         extras-common       8.1 k
 epel-release            noarch       9-2.el9         extras-common        17 k

[...]
Complete!


$ sudo dnf group install workstation
Last metadata expiration check: 0:05:44 ago on Sat 19 Mar 2022 03:47:07 PM CET.
No match for group package "insights-client"
No match for group package "redhat-release"
No match for group package "redhat-release-eula"
Error: 
 Problem: cannot install the best candidate for the job
  - nothing provides insights-client needed by rhc-1:0.2.1-4.el9.x86_64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

```


Note that the `insights-client` and some other RedHat specific packages show up when trying to install the @Workstation group.

From what I can see the `insights-client` is listed in the comps AppStream under the: base and standard groups.  

I assume that the `redhat-release` should be `centos-stream-release`, and that there is no equivalent stream eula (atleast no such package exists)? These two are listed in the comps BaseOS under the: base, server-product, standard and workstation-product groups.


Oh, and just for information. I noticed that the description of some of the groups also still has the "Red Hat Enterprise Linux" branding in their descriptions.  I guess this should be its own BZ, but I'm not quite sure where to file that against.  For example the base group:

```
# dnf group info base | head -3
Last metadata expiration check: 0:11:28 ago on Sat 19 Mar 2022 03:47:07 PM CET.
Group: Base
 Description: The standard installation of Red Hat Enterprise Linux.
```

Where my Stream 8 setup doesn't:
```
$ dnf group info base | head -4
Last metadata expiration check: 1:28:29 ago on Sat 19 Mar 2022 02:31:50 PM CET.

Group: Base
 Description: The standard installation of CentOS Linux.
```

Hi, I just found out about this bug today.  This is my fault, and I'm in the middle of cleaning it up.  I'm sorry about this.
Let me explain.

For the KDE section of the epel8 and epel9 comps, I take the kde sections from the corresponding Fedora comps file.  For epel9, that meant the F34 kde sections.
In the kde-desktop-environment section there are several groups listed.
When pungi creates the final comps.xml for a repo (in this case the epel9 repo) it goes through the comps-epel9.xml.in file and removes any groups that are not available in the comps.xml file.
The investigation into this can be found here.
https://pagure.io/releng/issue/10228

As a result of this feature/bug, I had to create small stubs of each group that I wanted to get pulled in.  The groups I created had to have at least one package listed in them for pungi to use them.
dnf would then combine any groups with the same names together.
It's a pretty straightforward solution.

Here's where the problem came about.
I created these small groups on epel8 initially.  Both core and standard.
If you look at the RHEL 8 (not centos8) appstream comps.xml file, the only "default" package is rhc
My mind didn't even think about what rhc was, I just grabbed it, and put it into the comps-epel8.xml.in
Then, when I was creating the comps-epel9.xml.in I just cut and pasted those same groups over, not even noticing that I was pulling in rhc.
And, thus this bug came about.

This problem just came to my attention today.
I have created a pull request to fix this in epel9.  I removed both core and standard mini-groups from the epel9 (and epel9-next) comps files, removed them from the groups that kde pulls in, as well as double checked all the other mini-groups that are listed there.
https://pagure.io/fedora-comps/pull-request/720
This pull request has already been merged.
I don't know how long it will take before it makes it into the epel repo's, but hopefully it's just a day or two.

Again, I apologize for my mistake.

If you need to do something that requires you to use the core group, adding --exclude=rhc will fix the problem.  Such as
  dnf --exclude=rhc group install core

Thanks for the detailed explanation Troy.  It looks like the change took effect in the latest compose.  I no longer see the rhc package in the core group.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rhc bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8084