Bug 1199868 - Anaconda does not exclude packages specified in kickstart files
Summary: Anaconda does not exclude packages specified in kickstart files
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 23
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
: 1266952 (view as bug list)
Depends On:
Blocks: F23FinalBlocker
TreeView+ depends on / blocked
 
Reported: 2015-03-09 03:14 UTC by Yu Watanabe
Modified: 2015-10-23 17:20 UTC (History)
21 users (show)

Fixed In Version: dnf-1.1.3-1.fc23 dnf-1.1.3-1.fc22
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-10-17 15:51:36 UTC


Attachments (Terms of Use)
kickstart file generated by anaconda (3.27 KB, text/plain)
2015-03-09 03:14 UTC, Yu Watanabe
no flags Details
anaconda.log (7.00 KB, text/plain)
2015-03-09 03:15 UTC, Yu Watanabe
no flags Details
dnf.log (331.85 KB, text/plain)
2015-03-09 03:15 UTC, Yu Watanabe
no flags Details
dnf.rpm.log (102.39 KB, text/plain)
2015-03-09 03:16 UTC, Yu Watanabe
no flags Details
journal.log of the installation process (975.76 KB, text/x-vhdl)
2015-03-09 03:16 UTC, Yu Watanabe
no flags Details
packaging.log of anaconda (5.93 KB, text/plain)
2015-03-09 03:17 UTC, Yu Watanabe
no flags Details
program.log of anaconda (153.45 KB, text/plain)
2015-03-09 03:18 UTC, Yu Watanabe
no flags Details
anaconda-ks.cfg (3.17 KB, text/plain)
2015-09-10 09:41 UTC, Yu Watanabe
no flags Details
anaconda.log (8.50 KB, text/plain)
2015-09-10 09:41 UTC, Yu Watanabe
no flags Details
dnf.log (91.44 KB, text/plain)
2015-09-10 09:42 UTC, Yu Watanabe
no flags Details
dnf.rpm.log (100.16 KB, text/plain)
2015-09-10 09:43 UTC, Yu Watanabe
no flags Details
packaging.log (3.41 KB, text/plain)
2015-09-10 09:43 UTC, Yu Watanabe
no flags Details
program.log (128.24 KB, text/plain)
2015-09-10 09:44 UTC, Yu Watanabe
no flags Details
exclude-test.py (926 bytes, text/plain)
2015-09-11 21:00 UTC, David Shea
no flags Details

Description Yu Watanabe 2015-03-09 03:14:44 UTC
Created attachment 999424 [details]
kickstart file generated by anaconda

Description of problem:

The package selection section of Anaconda kickstart file, I specify several packages to exclude. However, these packages are actually installed.

Version-Release number of selected component (if applicable):

Fedora 22 Server Alpha TC8

How reproducible:

Steps to Reproduce:
1. install fedora 22 by using kickstart file.
2.
3.

Actual results:

Packages explicitly excluded in the kickstart file are installed.

Expected results:

These packages are not installed.

Additional info:

Comment 1 Yu Watanabe 2015-03-09 03:15:17 UTC
Created attachment 999425 [details]
anaconda.log

Comment 2 Yu Watanabe 2015-03-09 03:15:45 UTC
Created attachment 999426 [details]
dnf.log

Comment 3 Yu Watanabe 2015-03-09 03:16:09 UTC
Created attachment 999427 [details]
dnf.rpm.log

Comment 4 Yu Watanabe 2015-03-09 03:16:52 UTC
Created attachment 999428 [details]
journal.log of the installation process

Comment 5 Yu Watanabe 2015-03-09 03:17:48 UTC
Created attachment 999429 [details]
packaging.log of anaconda

Comment 6 Yu Watanabe 2015-03-09 03:18:17 UTC
Created attachment 999430 [details]
program.log of anaconda

Comment 7 Yu Watanabe 2015-03-09 03:21:04 UTC
In package.log I attached above, there exists the following INFO messages
%%%%%%%%%%%%%%%%
07:24:06,454 INFO packaging: skipped removing nonexistant package: realmd
07:24:06,469 INFO packaging: skipped removing nonexistant package: abrt-cli
07:24:06,485 INFO packaging: skipped removing nonexistant package: sssd
07:24:06,501 INFO packaging: skipped removing nonexistant package: fprintd-pam
07:24:06,517 INFO packaging: skipped removing nonexistant package: mlocate
07:24:06,533 INFO packaging: skipped removing nonexistant package: mcelog
%%%%%%%%%%%%%%%%
which program output these messages? If Anaconda output them, it might be Anaconda's bug.

Comment 8 Radek Holy 2015-03-09 08:12:59 UTC
TBH, I have no idea how Anaconda uses DNF. It would be nice if someone can tell us. Depending on what it does, this bug may be a duplicate of bug 1185408.

Comment 9 Honza Silhan 2015-03-09 12:37:47 UTC
The message is not from DNF. Reassigning to Anaconda. If you find it's really DNF bug, put there some reproducer calling DNF so we can fix it. Thanks.

Comment 10 Yu Watanabe 2015-03-11 02:25:10 UTC
Thank you for reassigning to the suitable package.

I've tested Fedora 22 Server Alpha. However, the situation is the same.

%%%%%%%%%%%%
01:56:58,945 INFO packaging: skipped removing nonexistant package: realmd
01:56:58,961 INFO packaging: skipped removing nonexistant package: abrt-cli
01:56:58,977 INFO packaging: skipped removing nonexistant package: sssd
01:56:58,993 INFO packaging: skipped removing nonexistant package: fprintd-pam
01:56:59,008 INFO packaging: skipped removing nonexistant package: mlocate
01:56:59,024 INFO packaging: skipped removing nonexistant package: mcelog
%%%%%%%%%%%%
package.log say above. However, these package actually exist on the repository, and are installed by this installation process.

Comment 11 Chris Lumens 2015-09-09 20:04:56 UTC
Can you please try again with F23 beta?  I think I fixed this in anaconda-23.1-1, which would be a version of anaconda that will be in F23.

Comment 12 Yu Watanabe 2015-09-10 09:38:53 UTC
I got the same result on F23 beta TC4. Excluded packages in ks file are still installed.

Comment 13 Yu Watanabe 2015-09-10 09:41:30 UTC
Created attachment 1072068 [details]
anaconda-ks.cfg

Comment 14 Yu Watanabe 2015-09-10 09:41:58 UTC
Created attachment 1072069 [details]
anaconda.log

Comment 15 Yu Watanabe 2015-09-10 09:42:54 UTC
Created attachment 1072070 [details]
dnf.log

Comment 16 Yu Watanabe 2015-09-10 09:43:14 UTC
Created attachment 1072071 [details]
dnf.rpm.log

Comment 17 Yu Watanabe 2015-09-10 09:43:51 UTC
Created attachment 1072072 [details]
packaging.log

Comment 18 Yu Watanabe 2015-09-10 09:44:14 UTC
Created attachment 1072073 [details]
program.log

Comment 19 Chris Lumens 2015-09-10 13:14:02 UTC
Okay, which packages that you requested for removal are being installed?

Also note that when you list that you want a package removed, you really are just requesting it.  If it is required by some other package, it will still get pulled in.  anaconda doesn't break dependencies to force removals.

Comment 20 Yu Watanabe 2015-09-10 14:49:08 UTC
As you can see in the attached ks file,
https://bugzilla.redhat.com/attachment.cgi?id=1072068
I've tried to exclude the following packages:
-NetworkManager
-abrt-cli
-cifs-utils
-coolkey
-cryptsetup
-cyrus-sasl-plain
-fprintd-pam
-irda-utils
-mlocate
-ntfs-3g
-ntfsprogs
-pam_krb5
-pam_pkcs11
-pcmciautils
-plymouth
-policycoreutils
-realmd
-selinux-policy-targeted
-sssd
-vconfig
-wireless-tools

I have not check that all the above packages are not required by the selected packages, which are, also as you can see in the attached ks file, @core, @standard, and mainly development tools. However, at least, realmd is not required. After the installation, realmd can be removed independently.

Comment 21 Yu Watanabe 2015-09-10 14:59:02 UTC
I confirm that irda-utils, mlocate and vconfig can be independently removed on the installed system.

Comment 22 David Shea 2015-09-11 20:59:08 UTC
I'm able to reproduce this with

%packages
@standard
-abrt-cli
%end

which installs abrt-cli. Anaconda is calling dnf's base.group_install(<group object>, {'default', 'mandatory'}, exclude=['abrt-cli']), which does not appear to be working as expected.

Comment 23 David Shea 2015-09-11 21:00:06 UTC
Created attachment 1072636 [details]
exclude-test.py

Reproducer. The output of packages selected in the transaction includes:

('abrt-cli', 'x86_64', '0', '2.6.2', '7.fc24')

Comment 24 David Shea 2015-09-28 17:05:33 UTC
*** Bug 1266952 has been marked as a duplicate of this bug. ***

Comment 25 Adam Williamson 2015-09-28 22:11:50 UTC
Discussed at 2015-09-28 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-09-28/f23-blocker-review.2015-09-28-16.01.html . We are (still...last discussed at F18...sigh) lacking detailed kickstart criteria, so we're still operating on the basis that we evaluate kickstart issues case-by-case with the intent of setting some precedent for writing better criteria. There was a consensus at the meeting that this is a sufficiently core and commonly-used element of kickstart functionality that it should be release blocking, hence this bug is accepted as a blocker. We noted that the Cloud images in fact rely on this feature to trim some unneeded packages, and thus may be larger than intended at present.

Comment 26 Adam Williamson 2015-09-30 23:03:04 UTC
Hmm, so I think I see a possible cause here. I think it may not be possible to exclude 'mandatory' packages...and libcomps considers any package whose type is not explicitly stated to be 'mandatory'.

Viz libcomps comps_elem.c:

    tmp = comps_dict_get(elem->attrs, "type");
    if (!tmp)
        package->type = parsed->def_options->default_pkgtype;
    else
        package->type = comps_package_get_type(tmp);

and comps_default.c:

   .default_pkgtype = COMPS_PACKAGE_MANDATORY

Now in dnf, you have to get all the way to dnf/comps.py Solver.group_install() , which is ultimately what gets called by base's group_install (it calls dnf.comps.install_or_skip() which is a thin wrapper around the function that's passed as the first argument, that function is solver.group_install, which is Solver.group_install).

Solver.group_install() does this:

        types = pkg_types & MANDATORY
        mandatory = self._pkgs_of_type(group, types, exclude)
        types = pkg_types & (DEFAULT | OPTIONAL)
        trans.install_opt = self._pkgs_of_type(group, types, exclude)

...so we go look at _pkgs_of_type(), which does this:

    def _pkgs_of_type(group, pkg_types, exclude):
        pkgs = set()
        if pkg_types & MANDATORY:
            pkgs.update(pkg.name for pkg in group.mandatory_packages)
        if pkg_types & DEFAULT:
            pkgs.update(pkg.name for pkg in group.default_packages
                        if pkg.name not in exclude)
        if pkg_types & OPTIONAL:
            pkgs.update(pkg.name for pkg in group.optional_packages
                        if pkg.name not in exclude)
        return pkgs

there's some bitwise crap going on there which I'm kinda handwaving, but point is basically that, when that first condition is hit, exclude is *ignored*: presumably the idea is you shouldn't be able to exclude 'mandatory' packages, but I don't think that's compatible with current expectations.

Comment 27 Orion Poplawski 2015-10-01 02:01:40 UTC
I take it then that a packagereq without a type is "mandatory" in comps?  apper is:
      <packagereq>apper</packagereq>

and I can't exclude it.  I'm think it's reasonable to not be able to exclude "mandatory" packages from comps groups, but if that's going to be the case we're going to need to change a lot of packages in comps to "default".

Comment 28 Adam Williamson 2015-10-01 05:10:18 UTC
"I take it then that a packagereq without a type is "mandatory" in comps?"

That's part of my theory, yeah.

"I'm think it's reasonable to not be able to exclude "mandatory" packages from comps groups, but if that's going to be the case we're going to need to change a lot of packages in comps to "default"."

That's *only* my theory. I might be wrong on any point, I haven't tested it for sure.

Comment 29 Honza Silhan 2015-10-02 16:21:27 UTC
(In reply to awilliam@redhat.com from comment #28)
> "I take it then that a packagereq without a type is "mandatory" in comps?"
> 
> That's part of my theory, yeah.

Probably. If you think this behavior is wrong -> reassign it to libcomps component, please.

> "I'm think it's reasonable to not be able to exclude "mandatory" packages
> from comps groups, but if that's going to be the case we're going to need to
> change a lot of packages in comps to "default"."

It's exactly like that. If you want to exclude mandatory package you can't use group_install. You have to use the workaround of iterating over packages in groups and add call "base.install" for non-excluded packages. This was already discussed in some bug report and AFAIK Anaconda uses this workaround. Keeping the mandatory packages from `group_install` is intentional.

Comment 30 Adam Williamson 2015-10-02 16:35:44 UTC
Hmm, so looking at dnfpayload.py anaconda is indeed doing something like that workaround in one place, as part of `_apply_selections`:

        for pkg_name in set(self.data.packages.packageList) - set(self.data.packages.excludedList):
            try:
                self._install_package(pkg_name)
                log.info("selected package: '%s'", pkg_name)
            except packaging.NoSuchPackage as e:
                self._miss(e)

but several *other* places in the same file call `self._select_group()`, which calls `self._base.group_install(grp, types, exclude=exclude)` . The most recent change which added a `self._select_group()` call is https://github.com/rhinstaller/anaconda/commit/079e643683d3673cac2ee5cc8481bb9ecc269c99 , from April - the others have all been there since last December or earlier, I think.

Comment 31 Adam Williamson 2015-10-02 18:06:54 UTC
jsilhan: I can see the thinking, there, but it *is* causing anaconda quite a lot of trouble, especially since dnf doesn't communicate that there's any kind of issue with the 'exclude' list, it simply silently ignores what you told it.

couldn't group_install have some sort of 'really-exclude' arg, or some other mechanism to allow a caller to force exclusion of 'mandatory' packages?

I think the historical context here is valuable. AIUI, the 'mandatory'/'default'/'optional' thing was originally intended for *anaconda*, it really wasn't so much for yum: it defined what packages on the old package selection screen were selected by default and could not be unselected, which were selected by default but be unselected, and which were not selected by default (for each group).

Pre-dnf, including all the time anaconda had the old package selection UI, you could always de-select whatever you liked with a kickstart (assuming dependencies were satisfied). 'mandatory' was never enforced to the extent dnf is now trying to enforce it, it was more a cue for the anaconda UI.

Comment 32 Kevin Kofler 2015-10-04 00:47:10 UTC
IMHO, being able to say, I want group @foo without packages bar and baz, no matter whether bar and baz are considered "mandatory" in @foo (which is a pretty much arbitrary decision), is essential kickstart functionality.

Comment 33 Honza Silhan 2015-10-06 09:09:25 UTC
Hmm... now we even let the users to be able to exclude mandatory packages from the group in CLI (dnf install @<group> -x <mandatory_pkg> --setopt=strict=0). Maybe we can allow it in python API too.

I will throw it into discussion during software management meeting - whats the meaning of mandatory for package manager.

PR: https://github.com/rpm-software-management/dnf/pull/369

Comment 34 Michael Catanzaro 2015-10-12 13:58:20 UTC
(In reply to awilliam@redhat.com from comment #28)
> "I take it then that a packagereq without a type is "mandatory" in comps?"
> 
> That's part of my theory, yeah.
> 
> "I'm think it's reasonable to not be able to exclude "mandatory" packages
> from comps groups, but if that's going to be the case we're going to need to
> change a lot of packages in comps to "default"."

Got to say, I would never have guessed that packagereqs are not "default" by default....

Comment 35 Fedora Update System 2015-10-14 16:59:43 UTC
dnf-1.1.3-1.fc23 dnf-plugins-extras-0.0.11-1.fc23 dnf-plugins-core-0.1.13-1.fc23 hawkey-0.6.2-1.fc23 libsolv-0.6.14-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-a011ac001a

Comment 36 Fedora Update System 2015-10-14 17:02:16 UTC
dnf-1.1.3-1.fc22 dnf-plugins-extras-0.0.11-1.fc22 dnf-plugins-core-0.1.13-1.fc22 hawkey-0.6.2-1.fc22 libsolv-0.6.14-2.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-02efefba0e

Comment 37 Fedora Update System 2015-10-14 22:52:07 UTC
dnf-1.1.3-1.fc23, dnf-plugins-core-0.1.13-1.fc23, dnf-plugins-extras-0.0.11-1.fc23, hawkey-0.6.2-1.fc23, libsolv-0.6.14-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update hawkey dnf-plugins-core dnf-plugins-extras dnf libsolv'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-a011ac001a

Comment 38 Fedora Update System 2015-10-15 05:51:59 UTC
dnf-1.1.3-1.fc22, dnf-plugins-core-0.1.13-1.fc22, dnf-plugins-extras-0.0.11-1.fc22, hawkey-0.6.2-1.fc22, libsolv-0.6.14-2.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update hawkey dnf-plugins-core dnf-plugins-extras dnf libsolv'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-02efefba0e

Comment 39 Kamil Páral 2015-10-16 12:02:07 UTC
Should this start working once a new test compose has the new dnf, or are anaconda changes required as well?

Comment 40 David Shea 2015-10-16 13:57:58 UTC
Anaconda changes should not be necessary.

Comment 41 Orion Poplawski 2015-10-16 17:44:47 UTC
Looks fixed to me with F23_TC11

Comment 42 Adam Williamson 2015-10-16 19:00:28 UTC
yeah, upstream DNF basically changed to allow what anaconda's trying to do to work, so this should work without anaconda changes. Kamil, can you confirm it works for you? Then we'll get the DNF update pushed stable and this can be closed.

Comment 43 Fedora Update System 2015-10-17 15:51:07 UTC
dnf-1.1.3-1.fc23, dnf-plugins-core-0.1.13-1.fc23, dnf-plugins-extras-0.0.11-1.fc23, hawkey-0.6.2-1.fc23, libsolv-0.6.14-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 44 Fedora Update System 2015-10-23 17:19:53 UTC
dnf-1.1.3-1.fc22, dnf-plugins-core-0.1.13-1.fc22, dnf-plugins-extras-0.0.11-1.fc22, hawkey-0.6.2-1.fc22, libsolv-0.6.14-2.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.