Bug 1554010

I'm currently working on module installation support in Anaconda and I think the new PoC API mhatina added for this can fix this issue with environments as well.

For some background:
- when the Anaconda DNF payload was first written DNF apparently did not support group exclusion in it's API
- Anaconda needs group exclusion to work, even just for the (--nobase and --nocore options for the %packages section)
- this was implemented by resolving all groups requested for installation to a list and then removing all excluded groups from this list (all this in DNF payload on the Anaconda side)
- this included resolving groups belonging to the currently selected environment (if any)
- we never told DNF to install an environment, as that would pull all its groups in unconditionally, nullifying our group exclusion logic

For the new solution:
In the proposed DNF API addition mhatina developed there is now a function called install_specs(), which takes two iterables of package/group/module/environment specifications, one including items to include and the other to exclude from the installation transaction. This is how a call to it could look like:

def install_specs(install=["pkg", "@group", "@group/optional", "@module", "@module:stream", "@module/profile", "@module:stream/profile", "@^environment"],
            exclude=["pkg", "@group"])

The main idea is that we drop our own group exclusion code and just tell DNF what to include and exclude in the transaction. This way we can also tell DNF to install the environment, as the group exclusion no longer happens in Anaconda code, so it should not break anything anymore.

Anaconda side PR:
https://github.com/rhinstaller/anaconda/pull/1391

mhatinas PoC API:
https://github.com/mhatina/dnf/tree/install_specs

Feedback welcome!

This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

This is being worked on a part of module installation support together with DNF.

This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Unfortunately still valid for Fedora 31 (tested with official workstation iso dvd). Please could you change the component to component that sets kickstarts for disks?

Proposed as a Blocker for 32-beta by Fedora user jmracek using the blocker tracking app because:

 The presence of environmental group is critical part to keep integrity of the system. When user on fresh systems (workstation distribution) installs additional env. group (KDE) and then removes it it also removes most of the part of Workstation (also it tries to remove dnf). Official distributions must always mark one env. group as installed.

Based on my tests this is happening only on Live media where we don't install packages using DNF but rsync the Live image.

Switching to fedora kickstarts because when the image will have environment set we will rsync that to the resulting system. Please set the environment in the kickstart used for live image creation.

Can you expand on what exactly should be set in the kickstart here?

Let me use an example with 'Fedora Workstation'. It consist from following components.
Environment Group: Fedora Workstation
 Description: Fedora Workstation is a user friendly desktop system for laptops and PCs.
 Mandatory Groups:
   Common NetworkManager Submodules
   Container Management
   Core
   Fedora Workstation product core
   Firefox Web Browser
   Fonts
   GNOME
   Guest Desktop Agents
   Hardware Support
   LibreOffice
   Multimedia
   Printing Support
   base-x
 Optional Groups:
   ARM Tools

I suggest that spin-kickstarts to setup workstation do not ask to install 'Fedora Workstation' but it sends a list of groups that consist env. group 'Fedora Workstation' because not all groups want to install. I am proposing oposite approach that is supported by Anaconda and DNF => send request to anaconda to install env. group 'Fedora Workstation', and also add excluded groups that you want to skip.

Is it clear?

I'm inclined to -1 blocker. We've shipped this way for several releases, after all, and it doesn't violate any criteria really. It should be fixed, but I don't see that it blocks F32.

Sure, that makes sense.

I'm not sure I have time/energy to fix all of them, but we can urge maintainers to do so.

-1 blocker from me as well.

So, IIRC, *historically*, the kickstarts for live images didn't use environment groups because livecd-creator actually couldn't - it didn't understand them, it just didn't work. That's why we always wound up effectively duplicating the environment groups in the kickstarts, by listing all the package groups from them individually.

It may be the case that livemedia-creator *does* accept environment group syntax, in which case we can certainly look at changing at least the more prominent kickstarts to use that. We'd also have to check that this really does solve the problem (there's still the question of whether the information on which groups are installed actually makes it into the generated live image, and then from the generated live image to the installed system).

I think the LMC should accept the groups as expected but lets ask the correct person here.

Brian could you please confirm if livemedia-creator does or does not accept environment groups? See comment 14.

(In reply to Jiri Konecny from comment #15)
> I think the LMC should accept the groups as expected but lets ask the
> correct person here.
> 
> Brian could you please confirm if livemedia-creator does or does not accept
> environment groups? See comment 14.

Yes, lmc is using Anaconda, and is just passing the kickstart over to it so environment groups should work just fine.

I guess the other question is whether we want the kickstarts to remain compatible with livecd-creator as we know some folks are still using it, including Ben Williams for the live respins, I think. CCing him.

to me i dont care as long as something works to let folks like me create updates isos.
i only use LCC now when i run into issues with LMC (they both use the same ks)

I'm confused why this is broken in livecd-tools? We mostly just hand over to DNF to do group handling: https://github.com/livecd-tools/livecd-tools/blob/master/imgcreate/dnfinst.py#L120-L153

It may not be any more. I didn't test. I just know it didn't work *at one time* and that's why the live kickstarts didn't use it.

*** Bug 1780280 has been marked as a duplicate of this bug. ***

Discussed at 2020-02-03 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2020-02-03/f32-blocker-review.2020-02-03-17.13.html . We agreed that improving this would be a great idea and definitely worth doing, but it doesn't really violate any criteria, and the thing it causes problems with (removing an entire desktop environment) is a thing we've previously kinda indicated you shouldn't really try to do. We've also been shipping this way for years. So it was rejected as a blocker.

We did approve it as a freeze exception issue on the grounds it can't really be fixed with an update (you can change comps in the update repo, but the comps in the stable repo will not change, and the interactions there are not super predictable).

This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.

I'm trying to figure out why new packages added to e.g. "Fedora Workstation product core" are not brought in on 'dnf systemd-upgrade', so I found myself reading this bug, but I don't understand it.

The title says "No env. group installed after Fedora installation"

But on a clean install of 'Fedora-Workstation-Live-x86_64-31-1.9.iso' I see this:

# dnf group list --installed 
Last metadata expiration check: 0:09:01 ago on Mon 09 Mar 2020 10:00:01 AM MDT.
Installed Environment Groups:
   Fedora Workstation
Installed Groups:
   Container Management
   LibreOffice

And the subset of 'dnf group list --hidden' includes

Installed Groups:
   base-x
   Container Management
   Core
   Firefox Web Browser
   Fonts
   GNOME
   Guest Desktop Agents
   Hardware Support
   LibreOffice
   Multimedia
   Common NetworkManager Submodules
   Printing Support
   Fedora Workstation product core


If the idea is to drop 'installed environment groups: Fedora Workstation' and just have 'installed groups' (set explicitly) then should this bug be retitled?

Since 'dnf system-upgrade' does distro-sync, I'd expect that dnf should install any newly added packagereq type="mandatory" or packagereq type="default" from comps for the new release version; true/false?

A nevermind is in order because the man page for distro-sync says it only applies to installed packages. I guess we don't have a facility for bringing in new packages other than through adding a dependency to something already installed, or a clean install.

*** Bug 1841241 has been marked as a duplicate of this bug. ***

Please is there any progress with the issue? Today we got another report, but this is not the only one. I probably did a mistake that I didn't close all of the reports as a duplicate of this one to highlight the issue.

The issue is still valid for Fedora 32. Hope that it will be fixed in Fedora 33.

Proposed as a Blocker for 33-beta by Fedora user jmracek using the blocker tracking app because:

 The problem is that Fedora is shipped permanently with broken content. The fix is quite easy - mark environmental group as installed and all unwanted group will be marked as excluded. DNF and Anaconda team invested a lot of time in deliver perfect product and all is wasted by incorrect recipe to create Fedora images.
The issue is there for several years, and users of Fedora that plays with env. groups periodicity reports the problem. Therefore it has negative impact on Fedora project and it consume limited resources of DNF team.

We already discussed this as a blocker and rejected it for the last release cycle. Nothing about it has changed. There's very little point in discussing it again. I am still -1.

the desired changes here are straightforward, though, so why not just go ahead and send a PR for https://pagure.io/fedora-kickstarts ?

I mention in comment 24 that 'dnf group list --installed' does list 'Fedora Workstation' on a clean installed system. So I can't tell what this bug is still complaining about. Upgrades? Or some other indication of environment other than what I'm using? I don't think it can be fixed during an upgrade.

Chris Murphy: dnf group list --installed is empty after installation of Fedora 31 KDE Spin (F31-KDE-x86_64-LIVE-20200113.iso).

[akik@localhost ~]$ sudo -i dnf group list --installed

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

[sudo] password for akik: 
Fedora Modular 31 - x86_64                   562 kB/s | 5.2 MB     00:09    
Fedora Modular 31 - x86_64 - Updates         260 kB/s | 4.1 MB     00:16    
Fedora 31 - x86_64 - Updates                 949 kB/s |  26 MB     00:27    
Fedora 31 - x86_64                           3.4 MB/s |  71 MB     00:20    
Last metadata expiration check: 0:00:01 ago on Sun 28 Jun 2020 12:45:29 AM EEST.
[akik@localhost ~]$

I guess it should be 'KDE Plasma Workspaces'?

Chris Murhpy: Yes that's listed in sudo -i dnf group list.

This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

Please note this bug can be seen when installing some other spins too. I've just installed XFCE spin of Fedora-33-20200911.n.0 and observed similar behaviour:

# dnf groups list --installed
Last metadata expiration check: 0:10:50 ago on Sat 12 Sep 2020 06:54:06 PM CEST.
Installed Groups:
   Administration Tools

I create PR - that should fix the issue - https://pagure.io/fedora-kickstarts/pull-request/729

This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

This message is a reminder that Fedora 33 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 33 on 2021-11-30.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '33'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 33 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

well, the PR that intended to fix this got merged. So I think we can let it die, unless someone found a case where this is still true in F34/F35.

Fedora 33 changed to end-of-life (EOL) status on 2021-11-30. Fedora 33 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days