Bug 1632518 - PackageKit installs packages from default module streams, but does not enable the module stream
Summary: PackageKit installs packages from default module streams, but does not enable...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libdnf
Version: rawhide
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F29FinalBlocker 1649307
TreeView+ depends on / blocked
 
Reported: 2018-09-25 01:25 UTC by Adam Williamson
Modified: 2018-11-13 11:15 UTC (History)
18 users (show)

Fixed In Version: libdnf-0.22.0-5.fc29 libdnf-0.22.0-6.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1632520 1641003 1649307 (view as bug list)
Environment:
Last Closed: 2018-10-23 18:37:37 UTC


Attachments (Terms of Use)

Description Adam Williamson 2018-09-25 01:25:22 UTC
In https://bugzilla.redhat.com/show_bug.cgi?id=1629397#c11 , jmracek reports:

"According to my investigation, packages installed from default module by microdnf trigger enablement of module. Unfortunately this is not true for PackageKit. Tested with command "pkcon install dwm". I think that people from PackageKit can investigate the issue more closely and find the issue."

That is, we now have this thing in Rawhide called 'default module streams', which essentially means that the default source for certain packages is not the non-modular distribution repo, but a defined module stream. Basically the idea is that if you do 'dnf install dwm' (one of the packages now in a default module stream), you will get the 'dwm' from a specific module stream, and that module stream should be enabled.

dnf has specific code to do this (using a call to libdnf, so it's not a *lot* of code):

https://github.com/rpm-software-management/dnf/blob/master/dnf/base.py#L765-L770

however, there does not appear to be equivalent code in PackageKit, and jmracek's report confirms that this is a problem. It seems that if you do 'pkcon install dwm' on Rawhide, you will get the dwm from the default module stream, but *that module stream will not be enabled* so it is used by future operations. This could potentially cause significant problems with installation of the wrong packages, or inability to install things due to dependency errors.

Proposing as a Fedora 30 Beta blocker per criterion "The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated" - for packages in default module streams I would say this constitutes a violation of "appropriately", there. Not filing for F29 as I believe we are not using default module streams for F29.

Comment 1 Adam Williamson 2018-09-25 01:27:01 UTC
sgallagh, is there anything we can do to improve communication here to ensure that changes like these get made in PackageKit as well as dnf? (And probably dnfdragora, too; I'm about to go test what happens if you install dwm in Rawhide with dnfdragora...)

Comment 2 Adam Williamson 2018-09-25 02:14:22 UTC
dnfdragora actually works OK, I think because dnfdaemon actually winds up using the dnf `resolve()` method where this is done in dnf.

Comment 3 Fedora Blocker Bugs Application 2018-10-15 15:40:16 UTC
Proposed as a Freeze Exception for 29-final by Fedora user zbyszek using the blocker tracking app because:

 @adawm wrote:
> Proposing as a Fedora 30 Beta blocker per criterion "The installed system must be able appropriately to install, remove, and update software with the default console tool for the relevant software type (e.g. default console package manager). This includes downloading of packages to be installed/updated" - for packages in default module streams I would say this constitutes a violation of "appropriately", there. Not filing for F29 as I believe we are not using default module streams for F29.

But that last sentence is not correct: we now have "default module streams" in F29 (even if just two).

Comment 4 Adam Williamson 2018-10-15 18:17:48 UTC
Note, see also https://bugzilla.redhat.com/show_bug.cgi?id=1636184 , which was proposed as a blocker but 'fixed' before it could be voted on. It now seems that while *that* default module stream was removed, two others still exist (dwm and stratis).

Comment 5 Geoffrey Marr 2018-10-15 19:23:56 UTC
Discussed during the 2018-10-15 blocker review meeting: [1]

The decision to classify this bug as an "AcceptedBlocker" was made as default module streams appearing in F29 was not expected by all parties involved. We agree at least that this bug must be fixed *OR* default module streams must be delayed to Fedora 30. We will also file a FESCo ticket to ensure FESCo is aware of this situation.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-10-15/f29-blocker-review.2018-10-15-16.00.txt

Comment 6 Stephen Gallagher 2018-10-15 19:25:15 UTC
Just to note: upgrading to libdnf 0.22.0 does *not* fix this issue.

Comment 7 Adam Williamson 2018-10-15 22:43:08 UTC
CCing Matt just for info here, regarding the whole 'default module streams were a surprise to some of us' angle. There's an extended discussion in the full meeting log.

Comment 8 Richard Hughes 2018-10-19 11:24:45 UTC
This isn't possible to fix in PackageKit. The only way to trigger moduleContainer->save() seems to be from dnf_context_run(), which PK can't use as it needs more control over when packages are downloaded. In C it's possible to call dnf_context_get_sack() to get the context sack, but dnf_sack_get_module_container() isn't exported, and ModulePackageContainer isn't available from C. We can't switch to dnf_context_run() as we need to run dnf_transaction_depsolve() as a unique step so we know what permissions are required before choosing an appropriate PolicyKit action.

If moduleContainer->save() is moved from dnf_context_run() to dnf_transaction_commit() then it magically will start working for PackageKit with no changes to the PackageKit backend required. It also doesn't require exporting any more API if it's solved this way. Kalev agrees, and added "commit seems like semantically better place for this anyway" which I agree with.

Comment 9 Jaroslav Mracek 2018-10-19 14:03:59 UTC
I create PR that should solve the issue (https://github.com/rpm-software-management/libdnf/pull/619). But before merge we need an explicit confirmation from  Richard Hughes that the solution works with PackageKit. Please Richard can you test it to ensure that it works correctly?

Comment 10 Jaroslav Mracek 2018-10-19 16:10:52 UTC
The patch is part of libdnf-0.22.0-4.fc29, but I cannot edit bodhi update https://bodhi.fedoraproject.org/updates/FEDORA-2018-2789f6b6e7 to get it into Fedora 29. Please can anyone help me?

Comment 11 Petr Šabata 2018-10-19 16:19:11 UTC
(In reply to Jaroslav Mracek from comment #10)
> The patch is part of libdnf-0.22.0-4.fc29, but I cannot edit bodhi update
> https://bodhi.fedoraproject.org/updates/FEDORA-2018-2789f6b6e7 to get it
> into Fedora 29. Please can anyone help me?

Any provenpacager can edit the update but you could also file a new update that will obsolete this one.

Comment 12 Fedora Update System 2018-10-19 18:03:04 UTC
anaconda-29.24.7-1.fc29 dnf-4.0.4-1.fc29 dnf-plugins-core-4.0.0-2.fc29 libblockdev-2.20-2.fc29 libdnf-0.22.0-4.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2789f6b6e7

Comment 13 Nils Philippsen 2018-10-19 19:27:16 UTC
This crashes on me when run in a PackageKit context:

--- 8< ---
root@gibraltar:~> pkcon install dwm
Resolving                     [=========================]         
Loading cache                 [=========================]         
Testing changes               [=========================]         
Finished                      [                         ] (0%)  
The following packages have to be installed:
 dmenu-4.8-2.fc29.x86_64	Generic menu for X
 dwm-6.1-8.module_1995+c3e93812.x86_64	Dynamic window manager for X
 st-0.8.1-3.fc29.x86_64	A simple terminal implementation for X
Proceed with changes? [N/y] y

                              [=========================]         
Installing                    [=========================]         
Querying                      [=========================]         
Downloading packages          [=========================]         
Testing changes               [=========================]         
Installing packages           [======================== ] (99%)  The daemon crashed mid-transaction!
root@gibraltar:~>
--- >8 ---

It crashes because the sack in the DNF context isn't initialized (this is with a debugging build of libdnf):

--- 8< ---
(gdb) bt
#0  0x00007fffe830bcf1 in dnf_sack_get_module_container(_DnfSack*) (sack=0x0)
    at /home/nils/src/libdnf/libdnf/dnf-sack.cpp:2018
#1  0x00007fffe8326c25 in dnf_transaction_commit(DnfTransaction*, HyGoal, DnfState*, GError**) (transaction=0x5555555ccab0, 
    goal=0x55555592a9d0, state=0x7fffd4001910, error=0x7fffda4f7cd0)
    at /home/nils/src/libdnf/libdnf/dnf-transaction.cpp:1440
#2  0x00007ffff7fc0d90 in pk_backend_transaction_download_commit
    (error=0x7fffda4f7cd0, state=0x7fffd40019f0, job=<optimized out>)
    at pk-backend-dnf.c:2363
--- >8 ---

Comment 14 Jaroslav Mracek 2018-10-19 22:01:17 UTC
How it is possible after testing?

Comment 15 Jaroslav Mracek 2018-10-19 23:19:45 UTC
I create an additional patch for PackageKit (https://github.com/rpm-software-management/libdnf/pull/620), that should solve the issue from comment 13. Again before merge we need an explicit confirmation from  Richard Hughes that the solution works with PackageKit. Please Richard can you test it to ensure that it works correctly?

Comment 16 Fedora Update System 2018-10-20 19:21:17 UTC
anaconda-29.24.7-1.fc29, dnf-4.0.4-1.fc29, dnf-plugins-core-4.0.0-2.fc29, libblockdev-2.20-2.fc29, libdnf-0.22.0-5.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2789f6b6e7

Comment 17 Nils Philippsen 2018-10-20 20:34:04 UTC
Unfortunately, this fixes just the crash but doesn't enable the module for me:

root@gibraltar:~> pkcon -y install dwm
Resolving                     [=========================]         
Installing                    [=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading repository information[=========================]         
Loading cache                 [=========================]         
Downloading packages          [=========================]         
Testing changes               [=========================]         
Installing packages           [=========================]         
Finished                      [=========================]         
root@gibraltar:~> dnf module list --enabled
Copr repo for pulseaudio-dlna owned by cygn     173  B/s | 345  B     00:01    
RCM Tools for Fedora 29 (RPMs)                  0.0  B/s |   0  B     00:01    
Failed to synchronize cache for repo 'cygn-pulseaudio-dlna', ignoring this repo.
Failed to synchronize cache for repo 'rcm-tools-fedora-rpms', ignoring this repo.
Last metadata expiration check: 0:15:07 ago on Sat 20 Oct 2018 22:03:40 CEST.
Fedora Modular 29 - x86_64
Name     Stream     Profiles                                Summary             
gimp     2.10 [e]   devel [i], default [i]                  GIMP                
nodejs   8 [e]      development, minimal, default [d] [i]   Javascript runtime  

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled
root@gibraltar:~>

This is with a manually started packagekitd from the Fedora package, against a local build of libdnf with this patch.

Comment 18 Nils Philippsen 2018-10-20 21:02:29 UTC
I'm not too familiar with either code bases but I think the sack must be present in order for the module to be enabled to be recorded.

Just a shot in the dark, but would this involve a call to dnf_context_setup_sack() in the right place?

Comment 19 Jaroslav Mracek 2018-10-21 18:58:34 UTC
It looks like that the problem is little bit more complex and should be solved in cooperation with PackageKit (probably changes in PackageKit required).

Comment 20 Jaroslav Mracek 2018-10-22 13:20:50 UTC
I update a patch for PackageKit (https://github.com/rpm-software-management/libdnf/pull/620), that should solve the issue from comment 13. Again before merge we need an explicit confirmation from  Richard Hughes that the solution works with PackageKit. Please Richard can you test it to ensure that it works correctly?

It uses sack from goal, therefore it should be present.

Comment 21 Nils Philippsen 2018-10-22 14:02:16 UTC
https://github.com/rpm-software-management/libdnf/pull/620#issuecomment-431842106

TL;DR: this updated patch works in my tests. Thanks!

Comment 22 Fedora Update System 2018-10-22 16:11:54 UTC
anaconda-29.24.7-1.fc29, dnf-4.0.4-1.fc29, dnf-plugins-core-4.0.0-2.fc29, libblockdev-2.20-2.fc29, libdnf-0.22.0-5.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 23 Fedora Update System 2018-10-23 10:19:41 UTC
libdnf-0.22.0-6.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5a90628a95

Comment 24 Lukas Ruzicka 2018-10-23 13:53:07 UTC
This version of libdnf works as expected. If I install a module using packagekit, the stream gets automatically enabled. Verified.

Comment 25 Fedora Update System 2018-10-23 17:10:34 UTC
libdnf-0.22.0-6.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5a90628a95

Comment 26 Fedora Update System 2018-10-23 18:37:37 UTC
libdnf-0.22.0-6.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.