Bug 1427365

Summary: DNF (and thus anaconda, livemedia-creator...) skips packages that are missing or cannot be installed due to conflicts or dependency issues with no warning, error or information
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: dnfAssignee: rpm-software-management
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: medium    
Version: rawhideCC: anaconda-maint-list, awilliam, dennis, dmach, jkonecny, jmracek, jonathan, kellin, kparal, mblaha, ngompa13, packaging-team-maint, pbrobinson, releng, rpm-software-management, samuel-rhbugs, sbueno, vanmeeuwen+fedora, vmukhame, v.podzimek+fedora, vponcova, wwoods
Target Milestone: ---Keywords: FutureFeature, Reopened, Tracking, Triaged
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: dnf-2.6.2-1.fc26 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-04 13:22:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
output from 'dnf install @web-server' run none

Description Adam Williamson 2017-02-28 02:24:30 UTC
Currently in Fedora Rawhide, if you do a clean install of Server DVD and then do:

dnf install @freeipa-server

dnf will list and install 14 packages - opendnssec , bind-dyndb-ldap , ldns , softhsm , bind-pkcs11 , bind-pkcs11-utils , bind , bind-pkcs11-libs , python3-ply , libxslt , opencryptoki , opencryptoki-libs , sqlite and opencryptoki-icsftok - and then happily exit. It prints no further errors, warnings or even information about any skipped packages.

However, the package freeipa-server is 'mandatory' for the freeipa-server group, and the packages freeipa-server-dns and freeipa-server-trust-ad are 'default'. None of those three is installed. If you do 'dnf install freeipa-server' you can see that this is due to https://bugzilla.redhat.com/show_bug.cgi?id=1405790 - freeipa-server requires freeipa-client which can't be installed, due to that problem.

But just from the `dnf install @freeipa-server` command you'd have no idea there was anything wrong, unless you happen to know what packages are included in the group. It gives you no indication whatsoever that packages - even mandatory packages! - from the group were actually skipped.

Comment 1 Fedora End Of Life 2017-02-28 12:28:09 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 2 Honza Silhan 2017-03-08 11:55:44 UTC
We will either make it strict again (bug 1292892) or report skipping mandatory packages. We are more inclined to the first solution which must be coordinated with relengs.

Comment 3 Adam Williamson 2017-03-29 04:33:30 UTC
See also:

https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.org/message/CHUSNPRN65I6DVOKQQVAKRRPFRSP3WUG/

this silent skipping of non-installable group packages has caused ABRT to be missing from all Fedora 26 Alpha desktop images, because the abrt-desktop package can't be installed in F26 at present. In the past, the image compose would've failed and that would've caused us to notice the problem.

I don't recall - why was this change from yum's behaviour made in the first place? I vaguely remember maybe releng requested it because it was interfering with something?

Comment 4 Dennis Gilmore 2017-03-29 14:01:01 UTC
the only times this has come up releng has always asked for yum's behaviour to be restored. It is a issue that has broken buildroots in the past.

Comment 5 Adam Williamson 2017-04-05 18:13:05 UTC
Ah. Now I remember the wrinkle here, and to be fair, it's kinda all my fault :P

https://bugzilla.redhat.com/show_bug.cgi?id=1337731

During F24 cycle, dnf was using the 'strict' behaviour, and it caused us problems, because of arch issues.

Basically there are some packages listed in comps that are only available on certain arches. The 'expected' behaviour there was that on arches where the package doesn't exist, it would be silently skipped during installs and image composes.

I haven't checked, but I *think* the way this worked in Ye Olde Days was that different tools had different behaviours. The tools that created the live images and the installer environment would ignore missing packages like this (I haven't dug into the details of what they did about packages with dependency issues or other cases). But anaconda and yum-for-regular-users choked on them - if a package specified in a selected group couldn't be find, or had dependency issues, the install would fail.

We're now having problems because all this has been changed a bit. Particularly, live images are built using anaconda now.

So we kinda have a few different considerations to juggle here. On the one hand, we don't want image composes or installs to fail because some package specified in comps or a kickstart is only available on certain arches. On the other hand, we don't want to be building images or running installs that are missing expected packages, especially with absolutely no warning about it.

The more we get experience with the non-strict behaviour, the more I'm concerned about the possibility for havoc caused by 'silently skipped' packages, so I do think we need to switch back to the 'strict' behaviour by default. But we'll need to figure out something to do with comps. This comment:

https://bugzilla.redhat.com/show_bug.cgi?id=1337731#c22

suggests that for F26+, we *should* now be able to specify arches in comps, which should allow us to fix up the offending entries in comps and hopefully switch back to using strict DNF behaviour and catch missing / broken packages during image composes, system installs and post-install package group installs.

I think we have a window to do this ahead of Fedora 26 Beta, but we should do it *now* so we can sort out any messes comfortably before the Beta freeze. Does this sound doable to folks?

Does anyone know what 'switches' we'd need to flip to go back to strict behaviour? Is everything relying on dnf's default, or do we know of any tools (anaconda, pungi etc.) that are explicitly setting 'non-strict' mode?

Comment 6 Adam Williamson 2017-04-05 18:44:29 UTC
So Dennis says this:

<dgilmore> adamw: the main difference between yum and dnf comes down to silently ignoring brokenness
<dgilmore> yum skips packages missing entirely
<dgilmore> but if a package is listed with broken deps it spews noise saying thisis busted

so basically, with dnf, we've tried two different behaviours, but neither actually matches what yum used to do. It seems like dnf lumps 'package doesn't exist at all' and 'package exists but can't be installed' together, and you can only choose to treat *both* of those cases or fatal, or *neither* of them. dnf doesn't give the option to treat "package doesn't exist" as non-fatal but "packag exists but can't be installed" as fatal (at least, so far as we know).

At least for the medium term it would be nice to have this third option, because we suspect it's going to be difficult to whip comps and kickstarts into shape such that they only list existent packages per arch. But we *do* want installs and image composes to fail on listed packages that are *there* but broken.

Comment 7 Jaroslav Mracek 2017-06-13 09:34:27 UTC
Please can you test it with DNF-2.5.0? We refactored errors reports.

Comment 8 Adam Williamson 2017-06-13 16:30:09 UTC
Can you say what behaviour you would *expect* with 2.5.0? Does the fundamental behaviour change, or is it only a change to informational messages?

Comment 9 Jaroslav Mracek 2017-06-13 16:43:35 UTC
If the package is not installed due to broken dependency it will be reported.

Comment 10 Jaroslav Mracek 2017-06-13 17:11:23 UTC
Also I created a pull-request that should report skipped packages (https://github.com/rpm-software-management/dnf/pull/840)

Comment 11 Adam Williamson 2017-06-14 17:31:15 UTC
Unless I'm missing something, this (the reporting of skipped packages) isn't working. I tested with dnf 2.5.1 in current Rawhide, in a mock chroot. I installed the @web-server group, because it contains crypto-utils:

  <group>
    <id>web-server</id>
    <_name>Basic Web Server</_name>
    <_description>These tools allow you to run a Web server on the system.</_description>
    <default>false</default>
    <uservisible>false</uservisible>
    <packagelist>
      <packagereq type="mandatory">httpd</packagereq>
      <packagereq type="default">crypto-utils</packagereq>

which has broken dependencies in current Rawhide:

[crypto-utils]
	crypto-utils-2.4.1-65.fc26.x86_64 requires libperl.so.5.24()(64bit)
	crypto-utils-2.4.1-65.fc26.x86_64 requires perl(:MODULE_COMPAT_5.24.1)

indeed in the resulting transaction crypto-utils isn't installed. AFAICS there is no message about this printed in the entire transaction log. If I look at /var/log/dnf*.log , I can see a line mentioning crypto-utils:

/var/log/dnf.log:2017-06-14T17:26:33Z DEBUG Adding packages from group 'web-server': {<libcomps.Package object 'httpd' at 0x7fa544760fc0>, <libcomps.Package object 'mod_ssl' at 0x7fa52c230060>, <libcomps.Package object 'mod_perl' at 0x7fa52c230048>, <libcomps.Package object 'crypto-utils' at 0x7fa52c230030>, <libcomps.Package object 'php-mysqlnd' at 0x7fa52c2300a8>, <libcomps.Package object 'squid' at 0x7fa52c2300c0>, <libcomps.Package object 'httpd-manual' at 0x7fa544760fd8>, <libcomps.Package object 'php' at 0x7fa52c230078>, <libcomps.Package object 'webalizer' at 0x7fa52c2300d8>, <libcomps.Package object 'php-ldap' at 0x7fa52c230090>}

so clearly it did *try* to include crypto-utils in the transaction then kicked it out due to the dependency problem...but AFAICS it did not actually report this to me in the output at all. I will attach the full output.

Comment 12 Adam Williamson 2017-06-14 17:32:11 UTC
Created attachment 1287727 [details]
output from 'dnf install @web-server' run

Comment 13 Adam Williamson 2017-06-14 17:50:42 UTC
I also filed a new bug to request, if possible, a DNF mode which replicates yum's behaviour (be OK with non-existent packages, fail on existent-but-broken packages):

https://bugzilla.redhat.com/show_bug.cgi?id=1461539

Comment 14 Jaroslav Mracek 2017-06-19 18:18:42 UTC
Please can you try behavior of groups with "dnf-0:2.5.1_1-5gdfc02f7.fc*" form our repository (dnf copr enable rpmsoftwaremanagement/dnf-nightly)? I think you should be satisfied with implementation. Thanks a lot for your feedback.

Comment 15 Adam Williamson 2017-06-20 19:19:35 UTC
Well, if the change is this:

https://github.com/rpm-software-management/dnf/commit/453769b364b7de3c7d26e967cbcb37f43473295c

you can see right there that it only appears to log *missing* packages. I don't see anything in that commit that would log packages that exist, but have dependency errors. Or did I miss a commit?

Comment 16 Jaroslav Mracek 2017-06-21 12:50:29 UTC
The reports about packages with broken dependency was implemented and merged in PR https://github.com/rpm-software-management/dnf/pull/782. The patches were released in dnf-2.5.0. The PR https://github.com/rpm-software-management/dnf/pull/840 just only add last functionality. 

Here is an example where 3 packages are missing (No match for group package "python-catkin_tools") and some packages has broken dependency (in table section Skipping packages with broken dependencies):

$ sudo dnf group install "Robotics" --exclude eclipse-tm-terminal
Last metadata expiration check: 2:30:21 ago on Wed 21 Jun 2017 12:17:29 PM CEST.
No match for group package "python-catkin_tools"
No match for group package "python-catkin_lint"
No match for group package "python-rosinstall_generator"
Dependencies resolved.
==========================================================================
 GroupPackages                                                              
==========================================================================
Marking packages as installed by the group:
 @Roboticsfawkes-plugin-laser      player-examples        fawkes-plugin-xmlrpc  
 rcssmonitor              fawkes-plugin-pantilt  gearbox               
 fawkes-plugin-webview    gazebo                 player-ruby           
 player-doc               libphidget             player-python         
 fawkes-plugin-skiller    catkin                 fawkes-plugin-bblogger
 mrpt-doc                 urg                    opencv                
 eclipse-egit             fawkes-plugin-bbsync   fawkes-plugin-worldmodel
 fawkes-plugin-ttmainloop fawkes-plugin-katana   player                
 python-rosdep            python-wstool          fawkes                
 fawkes-plugin-refboxcomm pcl                    fawkes-core           
 rcssserver3d-doc         fawkes-plugin-player   fawkes-firevision-tools
 arduino                  rcssserver-gui         stage                 
 rcssserver3d             eclipse-subclipse      gazebo-doc            
 fawkes-plugin-luaagent   fawkes-lua             openni-primesense     
 mrpt-apps                fawkes-plugin-laserht  mrpt-libs             
 python-vcstools          rcsslogplayer          stage-playerplugin    
 python-bloom             fawkes-firevision      fawkes-guis           
 fawkes-doc               fawkes-plugin-joystick python-rosinstall     
 fawkes-plugin-flite      rcssserver                                   

 Problem 1: conflicting requests
  - package eclipse-cdt-1:9.0.0-1.fc25.x86_64 requires osgi(org.eclipse.remote.core), but none of the providers can be installed
  - nothing provides osgi(org.eclipse.tm.terminal.control) needed by eclipse-cdt-1:9.2.1-2.fc25.x86_64
  - nothing provides osgi(org.eclipse.tm.terminal.control) needed by eclipse-remote-2.1.1-1.fc25.noarch
 Problem 2: problem with installed package python2-catkin_pkg-0.3.1-1.fc25.noarch
  - installed package python2-catkin_pkg-0.3.1-1.fc25.noarch obsoletes python-catkin_pkg < 0.3.1-1.fc25 provided by python-catkin_pkg-0.2.10-4.fc25.noarch
  - package python2-catkin_pkg-0.3.1-1.fc25.noarch obsoletes python-catkin_pkg < 0.3.1-1.fc25 provided by python-catkin_pkg-0.2.10-4.fc25.noarch
  - conflicting requests
 Problem 3: problem with installed package python2-rosdistro-0.6.2-1.fc25.noarch
  - installed package python2-rosdistro-0.6.2-1.fc25.noarch obsoletes python-rosdistro < 0.6.2-1.fc25 provided by python-rosdistro-0.4.7-2.fc25.noarch
  - package python2-rosdistro-0.6.2-1.fc25.noarch obsoletes python-rosdistro < 0.6.2-1.fc25 provided by python-rosdistro-0.4.7-2.fc25.noarch
  - conflicting requests
 Problem 4: problem with installed package python2-rospkg-1.1.0-1.fc25.noarch
  - installed package python2-rospkg-1.1.0-1.fc25.noarch obsoletes python-rospkg < 1.1.0-1.fc25 provided by python-rospkg-1.0.38-5.fc25.noarch
  - package python2-rospkg-1.1.0-1.fc25.noarch obsoletes python-rospkg < 1.1.0-1.fc25 provided by python-rospkg-1.0.38-5.fc25.noarch
  - conflicting requests
==========================================================================
 Package                         Arch   Version             Repository
                                                                     Size
==========================================================================
Installing dependencies:
 arduino                         noarch 1:1.6.4-10.fc25     updates 1.7 M
 catkin                          noarch 0.4.5-13.gitd4f1f24.fc25
                                                            fedora  214 k
 eclipse-egit                    noarch 4.7.0-1.fc25        updates 8.7 M
 eclipse-subclipse               noarch 1.10.11-3.fc25      fedora  4.7 M
 fawkes                          x86_64 0.5.0-30.fc25       fedora   14 k
 fawkes-core                     x86_64 0.5.0-30.fc25       fedora  788 k
 fawkes-doc                      noarch 0.5.0-30.fc25       fedora   25 M
 fawkes-firevision               x86_64 0.5.0-30.fc25       fedora  349 k
 fawkes-firevision-tools         x86_64 0.5.0-30.fc25       fedora   64 k
 fawkes-guis                     x86_64 0.5.0-30.fc25       fedora  636 k
 fawkes-lua                      x86_64 0.5.0-30.fc25       fedora  437 k
 fawkes-plugin-amcl              x86_64 0.5.0-30.fc25       fedora   61 k
 fawkes-plugin-bblogger          x86_64 0.5.0-30.fc25       fedora   61 k
 fawkes-plugin-bbsync            x86_64 0.5.0-30.fc25       fedora   34 k
 fawkes-plugin-festival          x86_64 0.5.0-30.fc25       fedora   24 k
 fawkes-plugin-flite             x86_64 0.5.0-30.fc25       fedora   22 k
 fawkes-plugin-joystick          x86_64 0.5.0-30.fc25       fedora   42 k
 fawkes-plugin-katana            x86_64 0.5.0-30.fc25       fedora   91 k
 fawkes-plugin-laser             x86_64 0.5.0-30.fc25       fedora   55 k
 fawkes-plugin-laserht           x86_64 0.5.0-30.fc25       fedora   26 k
 fawkes-plugin-luaagent          x86_64 0.5.0-30.fc25       fedora   40 k
 fawkes-plugin-map-lasergen      x86_64 0.5.0-30.fc25       fedora   27 k
 fawkes-plugin-openni            x86_64 0.5.0-30.fc25       fedora   51 k
 fawkes-plugin-openni-data       x86_64 0.5.0-30.fc25       fedora   41 k
 fawkes-plugin-pantilt           x86_64 0.5.0-30.fc25       fedora  110 k
 fawkes-plugin-player            x86_64 0.5.0-30.fc25       fedora   41 k
 fawkes-plugin-refboxcomm        x86_64 0.5.0-30.fc25       fedora   58 k
 fawkes-plugin-roomba            x86_64 0.5.0-30.fc25       fedora   84 k
 fawkes-plugin-rrd               x86_64 0.5.0-30.fc25       fedora   40 k
 fawkes-plugin-rrdweb            x86_64 0.5.0-30.fc25       fedora   23 k
 fawkes-plugin-skiller           x86_64 0.5.0-30.fc25       fedora   92 k
 fawkes-plugin-static-transforms x86_64 0.5.0-30.fc25       fedora   29 k
 fawkes-plugin-tabletop-objects  x86_64 0.5.0-30.fc25       fedora   67 k
 fawkes-plugin-ttmainloop        x86_64 0.5.0-30.fc25       fedora   21 k
 fawkes-plugin-webview           x86_64 0.5.0-30.fc25       fedora   95 k
 fawkes-plugin-worldmodel        x86_64 0.5.0-30.fc25       fedora   59 k
 fawkes-plugin-xmlrpc            x86_64 0.5.0-30.fc25       fedora   34 k
 gazebo                          x86_64 7.4.0-2.fc25        fedora  2.3 M
 gazebo-doc                      noarch 7.4.0-2.fc25        fedora  2.6 M
 gazebo-media                    noarch 7.4.0-2.fc25        fedora   40 M
 gearbox                         x86_64 10.11-13.fc24       fedora  105 k
 libphidget                      x86_64 2.1.8.20140319-5.fc24
                                                            fedora  240 k
 libqhull_p                      x86_64 2015.2-1.fc25       fedora  163 k
 mrpt-2d-slam                    x86_64 1.3.2-9.fc25        fedora  228 k
 mrpt-apps                       x86_64 1.3.2-9.fc25        fedora  4.7 M
 mrpt-camera-calibration         x86_64 1.3.2-9.fc25        fedora  131 k
 mrpt-doc                        noarch 1.3.2-9.fc25        fedora   57 M
 mrpt-gridmap-navigation         x86_64 1.3.2-9.fc25        fedora  147 k
 mrpt-libs                       x86_64 1.3.2-9.fc25        fedora  6.1 M
 mrpt-navlog-viewer              x86_64 1.3.2-9.fc25        fedora  162 k
 mrpt-rawlog-viewer              x86_64 1.3.2-9.fc25        fedora  663 k
 mrpt-reactive-navigation        x86_64 1.3.2-9.fc25        fedora  112 k
 mrpt-robotic-arm-kinematics     x86_64 1.3.2-9.fc25        fedora  154 k
 mrpt-scene-viewer               x86_64 1.3.2-9.fc25        fedora  215 k
 mrpt-stereo-camera-calibration  x86_64 1.3.2-9.fc25        fedora  169 k
 opencv                          x86_64 3.1.0-8.fc25        fedora  1.8 M
 openni-primesense               x86_64 5.1.6.6-6.fc24      fedora  343 k
 pcl                             x86_64 1.8.0-2.fc25        fedora   11 M
 player                          x86_64 3.0.2-54.fc25       fedora  1.1 M
 player-doc                      noarch 3.0.2-54.fc25       fedora  1.9 M
 player-examples                 noarch 3.0.2-54.fc25       fedora   43 k
 player-python                   x86_64 3.0.2-54.fc25       fedora  1.8 M
 player-ruby                     x86_64 3.0.2-54.fc25       fedora  2.0 M
 python-bloom                    noarch 0.5.21-1.fc25       fedora  1.2 M
 python-rosdep                   noarch 0.11.4-4.fc25       fedora  315 k
 python-rosinstall               noarch 0.7.7-4.fc25        fedora  250 k
 python-vcstools                 noarch 0.1.38-4.fc25       fedora  242 k
 python-wstool                   noarch 0.1.13-1.fc25       fedora   92 k
 rcsslogplayer                   x86_64 15.1.1-13.fc25      fedora  347 k
 rcssmonitor                     x86_64 15.1.1-15.fc25      fedora  234 k
 rcssserver                      x86_64 15.2.2-18.fc25      fedora  574 k
 rcssserver-gui                  x86_64 15.2.2-18.fc25      fedora   15 k
 rcssserver3d                    x86_64 0.6.10-7.fc25       fedora  1.8 M
 rcssserver3d-doc                noarch 0.6.10-7.fc25       fedora  1.8 M
 stage                           x86_64 4.1.1-14.fc25       fedora  855 k
 stage-playerplugin              x86_64 4.1.1-14.fc25       fedora   39 k
 urg                             x86_64 0.8.18-13.fc24      fedora  116 k
Installing weak dependencies:
 eclipse-collabnet-merge         noarch 4.1.0-1.fc25        fedora  451 k
 eclipse-subclipse-graph         noarch 1.10.11-3.fc25      fedora  132 k
Skipping packages with broken dependencies:
 eclipse-cdt                     x86_64 1:9.0.0-1.fc25      fedora   43 M
 eclipse-cdt                     x86_64 1:9.2.1-2.fc25      updates  38 M
 eclipse-remote                  noarch 2.1.1-1.fc25        fedora  1.0 M

Transaction Summary
==========================================================================
Install  79 Packages
Skip      3 Packages

Total download size: 187 M
Installed size: 769 M

Comment 17 Adam Williamson 2017-06-21 16:52:55 UTC
But as mentioned above, I tested with 2.5.1 installing a group that contained a package with a dependency issue, and saw no notification about it...

Comment 18 Fedora Update System 2017-07-24 14:50:56 UTC
libdnf-0.9.3-1.fc26 dnf-plugins-core-2.1.3-1.fc26 dnf-2.6.2-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-6f4c06b2d7

Comment 19 Fedora Update System 2017-07-25 04:26:42 UTC
dnf-2.6.2-1.fc26, dnf-plugins-core-2.1.3-1.fc26, libdnf-0.9.3-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-6f4c06b2d7

Comment 20 Fedora Update System 2017-07-25 16:55:32 UTC
dnf-2.6.2-1.fc26, dnf-plugins-core-2.1.3-1.fc26, libdnf-0.9.3-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Adam Williamson 2018-02-20 00:00:16 UTC
This isn't really fixed in practical terms.

Couple of contemporary examples, from the Fedora-Rawhide-20180218.n.0 compose:

Cinnamon live image - compose task https://koji.fedoraproject.org/koji/taskinfo?taskID=25132075 - is missing the 'cinnamon' package itself, which is part of the '@cinnamon-desktop' group. The logs from anaconda (hence from dnf) tell us nothing at all about why the package isn't included, and we actually have not been able to figure this out yet.

KDE live image - compose task https://koji.fedoraproject.org/koji/taskinfo?taskID=25132023 - is missing calligra and plasma-nm-openswan (due to dependency issues in packages they require).

Note the original report specifically referred to the compose process: the most difficult case here is noticing and debugging when important packages are left out of image composes.

Comment 22 Jaroslav Mracek 2018-03-06 14:08:47 UTC
If there is no report from Anaconda, lets look into anaconda point of view.

Comment 23 Jiri Konecny 2018-03-08 08:44:20 UTC
How do you want to report this by Anaconda? Should we kill the installation process due to missing package or should we continue and just log this into a packaging.log?

Jirka

Comment 24 Adam Williamson 2018-03-09 18:10:22 UTC
I think logging would be a safer choice for now (see the notes in comment #5). Here's another example of this being a problem:

https://bugzilla.redhat.com/show_bug.cgi?id=1553807

gnome-software is missing from the Workstation images, but we have no idea why. I suspect there may be quite a *lot* of cases of this happening - no-one's going around inspecting every image and making sure it actually contains everything it's supposed to contain, after all.

I'm gonna try and look at improving this on the comps side, but it may be a bit of a project. We'd have to use the feature where you can say 'include package only for particular arch', which I don't think we've used before.

Logging would be very *helpful* in updating comps, in fact, as then I could just grab all the logs from all image composes to find all the problematic cases that need tweaking in comps (and kickstarts).

Comment 25 Adam Williamson 2018-03-10 08:33:54 UTC
So I remembered about https://bugzilla.redhat.com/show_bug.cgi?id=1461539 too. Note https://bugzilla.redhat.com/show_bug.cgi?id=1461539#c2 , which is the result of my attempt to figure out how the hell dnf works. I think I'm going to bed.

Comment 26 Jiri Konecny 2018-03-12 10:10:03 UTC
Shouldn't this be part of DNF logs already.

We have dnf logger in our code already so all the logs from DNF should be there:

# Increase dnf log level to custom DDEBUG level
# Do this here to prevent import side-effects in anaconda_logging
dnf_logger = get_dnf_logger()
dnf_logger.setLevel(dnf.logging.DDEBUG)

Comment 27 Adam Williamson 2018-03-13 21:38:00 UTC
Yeah, I kinda concur with Jiri. The dnf code is incredibly convoluted and difficult to follow, but as near as I can make out, it never logs non-fatal dependency errors.

Look at resolve() in base.py:

        if not self._run_hawkey_goal(goal, allow_erasing):
            if self.conf.debuglevel >= 6:
                goal.log_decisions()
            msg = dnf.util._format_resolve_problems(goal.problem_rules())
            exc = dnf.exceptions.DepsolveError(msg)
        else:
            self._transaction = self._goal2transaction(goal)

if self._run_hawkey_goal(goal, allow_erasing) *fails*, it sets an error message and does a detailed debug log, then raises that exception. But that's only if it *fails*. When we hit this path - dnf decides it's OK to just leave the thing with the dependency issue out of the transaction entirely - there's no failure, no debug logging is done there or anywhere in resolve() .

I also looked at how this gets reported in the CLI, and it's quite telling:

https://bugzilla.redhat.com/show_bug.cgi?id=1461539#c2

the CLI ultimately pulls out broken dependency information from its base object's private _goal attribute, and displays that. It seems like it'd be clearly wrong for anaconda to duplicate that behaviour, but I don't see any other way anaconda could actually get at the information about what non-fatal dependency errors occurred.

Comment 28 Adam Williamson 2018-03-14 01:55:46 UTC
Note I've sent a dnf PR which would (I believe) cause it to act as yum previously did, as requested all the way back in #1292892 , and also in #1461539:

https://github.com/rpm-software-management/dnf/pull/1038

Lots more details in the commit message there.

Comment 29 Jiri Konecny 2018-03-14 10:53:57 UTC
We were looking into a packaging.log with Jaroslav and we found that logs from DNF are there already we even saw there lines about skipping missing group packages.

So everything should be logged properly now AFAICT.

Comment 30 Adam Williamson 2018-03-24 00:04:06 UTC
Per notes on that PR and in #1461539 , packages skipped due to broken dependencies are only logged *by the dnf CLI*, using a private attribute of the Base instance. It is not possible for users of the dnf API to discover or log these skips, AFAICS.

Comment 31 Adam Williamson 2018-04-13 18:58:07 UTC
Just ran into yet another case of this biting us in real life: the plasma-workspace package was silently left out of the KDE live image in Fedora-Rawhide-20180412.n.0 . This meant the image didn't work properly at all (the whole desktop doesn't work).

Comment 32 Adam Williamson 2018-04-17 00:08:47 UTC
Another case: basically all of GNOME was left off the Fedora-28-20180416.n.0 Workstation images due to gnome-desktop3 dependencies not being satisfied. https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.org/message/PWIFLS6Z5NB5GABNWUYLCO3LB2QWQ2QD/%3E

Comment 33 Fedora End Of Life 2018-05-03 08:14:58 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 34 Fedora End Of Life 2018-05-29 12:33:38 UTC
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26
is no longer maintained, which means that it will not receive any
further security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 35 Neal Gompa 2018-07-09 23:07:46 UTC
Reopened and set so it doesn't get EOL'd again. This is still a major problem.

Comment 36 Adam Williamson 2018-07-09 23:11:59 UTC
Note that I updated my PR a while ago, it is still sitting awaiting review:

https://github.com/rpm-software-management/dnf/pull/1038

Comment 37 Samantha N. Bueno 2018-07-10 08:43:39 UTC
Since Adam's PR is against dnf, I'm going to move this bug over to dnf.

Comment 38 Daniel Mach 2019-01-04 13:22:13 UTC
This has been merged and released couple months back. Closing.