Bug 1197456 - [RFE] implement `strict` config option (defaults=True)
[RFE] implement `strict` config option (defaults=True)
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dnf-plugins-core (Show other bugs)
22
Unspecified Unspecified
low Severity unspecified
: ---
: ---
Assigned To: Honza Silhan
Fedora Extras Quality Assurance
:
: 1208919 1208981 1215339 1224485 1227230 1232986 1253929 1268606 1359810 (view as bug list)
Depends On: 1227952
Blocks:
  Show dependency treegraph
 
Reported: 2015-03-01 06:43 EST by Jens Petersen
Modified: 2016-07-25 11:33 EDT (History)
28 users (show)

See Also:
Fixed In Version: dnf-plugins-core-0.1.10-1.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-08-14 22:14:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jens Petersen 2015-03-01 06:43:38 EST
Description of problem:
Maybe "dnf update" doesn't need --skip-broken but it would still
be useful for "dnf install" to support it.

Version-Release number of selected component (if applicable):
dnf-0.6.4

Steps to Reproduce:
1. eg
$ dnf install ghc*devel

Actual results:
1. fails due to 3 broken packages

Expected results:
Provide --skip-broken to avoid having to --exclude each broken package by hand
and install remaining 300+ packages.

Additional info:
"yum install --skip-broken ghc*devel" handles this fine.
Comment 1 Parag Nemade 2015-03-01 06:51:11 EST
Check if the "--best" switch helps here. 
http://dnf.readthedocs.org/en/latest/cli_vs_yum.html#no-skip-broken
Comment 2 Radek Holy 2015-03-02 06:06:36 EST
Hello, can you elaborate on you use case? Why would you install all the ghc-devel packages and why are you OK if some of them are not installed in the end?

FYI, there is a reason of why DNF does not have the --skip-broken switch which is explained in the link above.
Comment 3 Jens Petersen 2015-03-02 19:49:54 EST
(In reply to Radek Holy from comment #2)
> Hello, can you elaborate on you use case? Why would you install all the
> ghc-devel packages

Because I am a Fedora Haskell developer. :)

> and why are you OK if some of them are not installed in the end?

I am ok if some are not installed with --skip-broken. :)

> FYI, there is a reason of why DNF does not have the --skip-broken switch
> which is explained in the link above.

I understand that, but can't see that having --skip-broken would hurt.
It is useful.
Comment 4 Radek Holy 2015-03-02 19:59:16 EST
I'm not a Fedora Haskell developer. Can you explain me why a Fedora Haskell developer needs to install *all* the ghc-devel packages and why is he OK if some of them are not installed with --skip-broken?

I can imagine you need some of them but why all of them? And if all, why some can be skipped?
Comment 5 Jens Petersen 2015-03-02 20:36:09 EST
(In reply to Radek Holy from comment #4)
> Can you explain me why a Fedora Haskell developer needs to install *all*
> the ghc-devel packages and why is he OK if some of them are not installed
> with --skip-broken?

:-)

It is useful to install all the packages for dependency and consistency
testing for example.  The Haskell packages have extremely tight dependencies
using ABI hashes: having all the packages installed means I can see immediately
if I break any dependencies when updating a package or set of packages.

Of course I would like to install *all* packages if possible, but if there is
some temporary breakage in rawhide say that prevents that, it is still
useful to install most of the packages.

Of course I can use --exclude to do that, but does dnf really have to
insist on forcing me to do that extra work: it knows what the broken
packages are so it can easily exclude them from the transaction with
--skip-broken.

> I can imagine you need some of them but why all of them? And if all, why
> some can be skipped?

Perhaps it is not a "typical use-case", it is nevertheless a use-case
and regression compared to yum.
Comment 6 Radek Holy 2015-03-03 03:36:06 EST
Yes, it's not a typical use-case and yes, it's a documented difference between DNF and YUM.

I'm just guessing now but I think that there might be e.g. a problem if the package specification glob matches two conflicting packages. Which of these two packages should be installed? We don't want DNF to guess what user wants in any situation.

Anyway, let's discuss this request within the team... I'd discuss the name too. Although DNF would behave the same as YUM with the switch, having the switch users might tend to use it even in the situations where it isn't needed because they are used to YUM. But... Let's discuss it.
Comment 7 Radek Holy 2015-03-03 04:21:05 EST
FTR, my problem with this is that the use case is Haskell specific. I cannot think of any other use case. Why would someone else want to install a big number of unneeded (because they can be skipped) packages...
Comment 8 Parag Nemade 2015-03-03 04:40:48 EST
Maybe you can think of another use case where user want to install all gnome development related rpms using
dnf install gnome*devel
Comment 9 Radek Holy 2015-03-03 04:44:17 EST
Sure, but why? And why some/all of them can be skipped?
Comment 10 Parag Nemade 2015-03-03 04:49:04 EST
Ah! right I forgot. We are following basic principle of dnf here that if in a transaction some packages are broken then they must be fixed first. So, next time same command will be executed, it will run successfully :)
Comment 11 Jan Zeleny 2015-03-03 06:01:13 EST
Well, that too :-)

But I believe Radek was genuinely trying to understand the use case. Quite frankly I don't follow it either. I understand it's nice to have a way to do something like this but we need to focus on what's really required and why to select the right feature set for dnf.
Comment 12 Jens Petersen 2015-03-03 21:24:30 EST
I understand. :)

I don't think the use-case is Haskell specific actually.
It was just one example.

I can think of similar examples:

eg I might want to install all Fedora gnome-shell-extension's
but one or more might be temporarily broken say.
Then it is convenient to do "dnf install --skip-broken gnome-shell-ext*".
I am sure there would be other similar cases.

Sure it can be worked around but it is a real convenience to have it.
Comment 13 Jens Petersen 2015-03-03 21:27:01 EST
(In reply to Radek Holy from comment #9)
> Sure, but why? And why some/all of them can be skipped?

Well because some is better than nothing. :)
Comment 14 Jens Petersen 2015-03-03 21:28:43 EST
I think of this RFE as "install as much as possible".
Comment 15 Radek Holy 2015-03-04 04:49:47 EST
Yes, I understand what the feature should do. What I don't understand is when it is useful. So far, I didn't meet a package that I want to be installed but just if possible. Even the case with GNOME extensions (FTR I'm not familiar with GNOME extensions at all), I cannot imagine an extension that I want to be installed but that I don't need in fact because I'm OK if it won't be installed in the end. That means that I don't understand why some is better than nothing in the context of RPM packages :(

Anyway, Jan is going to propose a solution so let's wait for it.
Comment 16 Jens Petersen 2015-03-04 05:51:25 EST
(In reply to Radek Holy from comment #15)
> Anyway, Jan is going to propose a solution so let's wait for it.

Sure, great and thanks for taking the time to listen. :)


(Perhaps I could elaborate on Parag's example which is kind of similar
to my original one.  Let's say I am a gnome developer and want
to test/fix the latest partial gnome tree in rawhide say.
Obviously I don't want to start building everything again from scratch
but I'd like to install the devel packages that have already been updated
so far, so that I can start to fix the remaining packages.
"yum install --skip-broken" helps with exactly this kind of case.
Unfortunately occasionally deps breakage also reaches
the stable updates repos and so it can also be useful then.)
Comment 17 Honza Silhan 2015-03-04 08:59:03 EST
At the meeting we concluded to solution to show tip message with exact command along with all the excludes to make it pass.

Like:
# dnf install ghc*devel
package ghc-1-devel cannot be installed.
package ghc-2-devel cannot be installed.
package ghc-2-devel cannot be installed.
Try: `dnf install ghc*devel -x ghc-1-devel -x ghc-2-devel -x ghc-3-devel`

This should come with better error + solution messages.
Comment 18 Bruno Wolff III 2015-04-07 08:35:34 EDT
This is also useful when trying to reinstall a set of packages that were temporarily broken (typically due to soname bumps). I just want to reinstall as many of them that can be installed. I don't want to parse dnf output to figure out which of them can be reinstalled and reissue the command and I don't want to run a separate dnf install command for each package as I some times have over hundreds of packages that have been temporarily removed from rawhide instances.
Comment 19 Bruno Wolff III 2015-04-07 08:46:38 EDT
This is also somewgat related to bug 1208918 in which it is noted that dnf stops processing the command line at the first unknown package. dnf really needs a best effort mode to replace yum. dnf is no longer faster when you have to run it once for each package rather than being able to process hundreds of packages in one run like one can do with yum.
Comment 20 Radek Holy 2015-04-07 08:55:04 EDT
If they were broken, how did you install them? If they are fixed now, why do they have the same release numbers (so that you can *reinstall* them)? Isn't it against Fedora policies?
Comment 21 Radek Holy 2015-04-07 08:56:35 EDT
And also, you don't want to search for "Try: `dnf install foo -x bar...` but you're OK to search for "bar cannot be installed; skipping" in the DNF's output, do I understand it correctly?
Comment 22 Bruno Wolff III 2015-04-07 09:16:11 EDT
No that isn't correct.
What I do is track packages I have removed and try reinstalling (as install a previously removed package not a reinstall command) and after updating my local mirrors I try reinstalling to put back whatever has been fixed to no longer conflict (typically due to soname bumps) with other packages.
So I would do:
yum install -y `cat putback`
and yum would reibstall what it could.
This is broken in a couple of ways by dnf and currently I have to run dnf once for each package in the file which is much slower than yum is.
I clean up putback using the list available command which still works with dnf.
In my case the request isn't really for a single transaction, but rather one for each listed package, that are bayched together for efficiency, not so they can all succeed or all fail as a group.
Comment 23 Radek Holy 2015-04-07 10:09:02 EDT
*** Bug 1208919 has been marked as a duplicate of this bug. ***
Comment 24 Radek Holy 2015-04-07 10:24:40 EDT
Finally, there is a use case that I can understand:

There is a list of packages (maybe just names, maybe some NEVRAs/globs). All of them should be installed at some moment in the future but some of the packages may be broken/unavailable now so it's enough to install at least the rest of them now and wait for the rest to be fixed.

Let's reconsider the implementation so it can be used in scripts too (without the need to parse the output). We should take into account how common is this use case and thus whether it's worth adding a switch. I think that it also can be a plugin that could do all the work instead of the Bruno's script.
Comment 25 Radek Holy 2015-04-07 10:53:38 EDT
*** Bug 1208981 has been marked as a duplicate of this bug. ***
Comment 26 Honza Silhan 2015-04-07 11:28:41 EDT
I am inclined to having a plugin. It's not actually hard to make it but dnf team has more important bugs on the list to fix.

This could be great start for some dnf collaborator. See [1] and [2] for reference. Anyone: feel free to take it.

[1] https://github.com/rpm-software-management/dnf/blob/3dd3693a53df3081e6c0bb6a6a51a51a014d55e1/dnf/base.py#L1314
[2] http://dnf.readthedocs.org/en/latest/use_cases.html#plugins-cli-api
Comment 27 Radek Holy 2015-04-07 11:48:31 EDT
Also, for those interested in this feature... I can imagine this situation:

1. there is a package "foo" that requires a virtual provide "bar-baz"
2. there is a package "bar" that provides a virtual provide "bar-baz"
3. there is a package "baz" that provides a virtual provide "bar-baz"

but:

4. I want to install "foo"
5. I don't like "baz"
6. "bar" is broken

So if I, as the user, run `dnf skip-install foo bar" I get "foo" and "baz" installed despite the fact that I don't like "baz". So, as the user, I need to count with that and explicitly "--exclude" these unwanted replacements. I think that this makes the feature even less usable but sure, a knowledgeable user might benefit from it. (What I'm trying to say is that I still thing that you should prefer to wait until all the packages get fixed and then install them all together.)
Comment 28 Bruno Wolff III 2015-04-07 12:20:33 EDT
No I don't want to wait until all of the packages I removed are fixed, in order to update other packages are fixed already, before installing them again. I sometimes want to use them now and in rawhide there is always at least one broken package so the wait would be forever.
The request is to coveniently batch together what would normally be separate transactions without having to worry about which ones will fail. This runs a lot faster than running a separate transaction for each package.
While the effect I want is essentially that of:
dnf install -y package1
dnf install -y package2
dnf install -y package3
...
dnf install -y package100

I want to be able to run it as:
dnf install -y package1 package2 package3 ... package100
Comment 29 Honza Silhan 2015-04-07 15:33:45 EDT
*** Bug 1208918 has been marked as a duplicate of this bug. ***
Comment 30 Radek Holy 2015-04-08 01:42:03 EDT
Sure, it's your choice and that's why didn't close the bug.

For the further developer of this feature or for the further users (it depends on the implementation): Consider also the same situation as described above PLUS a conflict between "bar" and "baz". The first call would install "baz" and thus it will never be possible to install "bar" even if it becomes available. In that case, the plugin would need to run in the "allow-erasing" mode. On the other hand, this mode needs to be used carefully since it can lead to some unwanted uninstallations (let's say that e.g. "foo" silently starts conflicting with another already installed package or "bar" starts conflicting with another package on the list passed by the user).
Comment 31 Bruno Wolff III 2015-04-08 11:48:25 EDT
For the foo bar baz case, I would prefer that foo and baz get installed so I could use foo now. And I would worry about bar later. Note that is what would happen for me any way if I ran dnf install on each package separately.
Comment 32 Radek Holy 2015-04-27 06:28:07 EDT
*** Bug 1215339 has been marked as a duplicate of this bug. ***
Comment 33 Radek Holy 2015-06-02 11:25:17 EDT
*** Bug 1227230 has been marked as a duplicate of this bug. ***
Comment 34 Radek Holy 2015-06-18 10:36:41 EDT
*** Bug 1232986 has been marked as a duplicate of this bug. ***
Comment 35 Andrew McNabb 2015-06-18 11:34:27 EDT
It looks like a lot of people have requested functionality like this. It's hard to measure demand accurately from bug report counts, but it may be possible that this is a common enough use-case to warrant a built-in flag rather than a plugin, at least eventually.
Comment 36 Radek Holy 2015-06-23 03:46:35 EDT
*** Bug 1224485 has been marked as a duplicate of this bug. ***
Comment 37 Jakub Filak 2015-06-30 07:26:52 EDT
My use case follows:

1. We have a spec file which we use on all RPM based distributions for our project and we support only RPM based distributions. Some of the build requires have different names on different distributions.

2. The spec file is generated from a "*.spec.in" file by the "./configure" script

3. If I want to successfully run "./configure", I need to have installed all the build dependencies, so I do "grep BuildRequires: *.spec.in | ... | xargs dnf install" (the *.spec.in file is not a valid rpm spec file).

In my use case, it is perfectly OK to not install the full requested set of packages, because the set contains packages from the other distros. Even if a missing package is really necessary on a current distro, I do not need dnf to be so kind to not install anything, because the ./configure will tell what I miss exactly later on.
Comment 38 Radek Holy 2015-06-30 07:38:18 EDT
(In reply to Jakub Filak from comment #37)
> 3. If I want to successfully run "./configure", I need to have installed all
> the build dependencies [...]
> 
> In my use case, it is perfectly OK to not install the full requested set of
> packages, because the set contains packages from the other distros. Even if
> a missing package is really necessary on a current distro, I do not need dnf
> to be so kind to not install anything, because the ./configure will tell
> what I miss exactly later on.

This sounds like a contradiction to me. Anyway, I'd just say that you need different files for different distributions. But yes, it's up to you. The only missing piece for me is why it's better to get the error from the ./configure instead of from DNF?
Comment 39 Jiri Jaburek 2015-06-30 08:05:58 EDT
Our use case is suite-based regression testing;

We have a subset of packages/RPMs (on an FTP server) we're interested in and are testing them against stable or semi-stable distribution bases with the intent of testing "newer or equal" versions to those from the FTP.

The testing is done by ie. LTP and at least one additional suite - we basically do 'yum update --nogpgcheck --skip-broken --cacheonly *.rpm' and hope that most of the packages install/update successfully. Some of them don't because the stable distribution has newer ones, some of them don't because of new missing dependencies, but most will.

In the end, we have a comprehensive testing report including the RPMs and their NVRs used, so we know which ones installed successfully.

The idea is that even if some didn't, we could test the rest and reveal possible regressions.
Comment 40 Jakub Filak 2015-06-30 08:30:34 EDT
(In reply to Radek Holy from comment #38)
> (In reply to Jakub Filak from comment #37)
> > 3. If I want to successfully run "./configure", I need to have installed all
> > the build dependencies [...]
> > 
> > In my use case, it is perfectly OK to not install the full requested set of
> > packages, because the set contains packages from the other distros. Even if
> > a missing package is really necessary on a current distro, I do not need dnf
> > to be so kind to not install anything, because the ./configure will tell
> > what I miss exactly later on.
> 
> This sounds like a contradiction to me.

Sorry, my bad, correction: "I need to have installed all the build dependencies for the current distribution [...]"

> Anyway, I'd just say that you need
> different files for different distributions. But yes, it's up to you.

The biggest spec file has 2757 lines and we need to modify only 5 build requires and update few other lines. Do you really think it is better to have two (or more) files that are identical on 99%?

> The only missing piece for me is why it's better to get the error from the
> ./configure instead of from DNF?

DNF says which package does not exist and the ./configure says which package I have to install.
Comment 41 Bruno Wolff III 2015-06-30 11:09:04 EDT
The reason why people want this feature is that doing an install of most of a set of packages in one run than running seperate run of dnf for each file. I still ocasionally use yum-deprecated to do this. The feature doesn't allow people to do stuff thet couldn't do anyway, it just makes it easier and a lot faster.
Comment 42 Jiri Jaburek 2015-07-01 13:27:42 EDT
(In reply to Bruno Wolff III from comment #41)
> The reason why people want this feature is that doing an install of most of
> a set of packages in one run than running seperate run of dnf for each file.
> I still ocasionally use yum-deprecated to do this. The feature doesn't allow
> people to do stuff thet couldn't do anyway, it just makes it easier and a
> lot faster.

The issue is that installing packages one-by-one is often not an option, ie. when the packages depend on each other. In that case, you need some kind of dependency-resolution-aware thing that can select/install as much as it can based on which packages pass the dependency checks (together, as a set) and ignore the ones that don't.

For simple standalone RPMs, using rpm(8) directly is the easiest "workaround".
Comment 43 Radek Holy 2015-07-01 23:45:26 EDT
Nice to hear it from someone else than me. Thank you.
Comment 46 Dario Lesca 2015-07-18 18:53:13 EDT
If I run "yum-deprecated -y install aaa xxx zzz bbb" and xxx zzz are for some reason missing, the command warn me with "xxx (then zzz) is missing and it's not installed" and install aaa, bbb and all their dependent.

If I run "dnf -y install aaa xxx zzz bbb" and xxx zzz are missing, the command fail and exit.

IMHO, this is an error for a command (dnf) that wants to replace another command (yum).

There is not need to create a new dnf plugin or option to solve this, just ignore this kind of error (missing from repositories one or more line command package), warn the error and install the rest like yum do.
Comment 47 Radek Holy 2015-07-20 04:01:41 EDT
That would be against the use case of the "install" command. The command should make sure that the given package is installed. So it cannot skip any argument even because the package is not available. You could just forget to enable a repository in which the package is available and if the transaction is long you may miss the message that something was skipped. Also if you would use "dnf install" in a script, you'd have to parse the output in order to find the skipped packages. That's why we prefer implementing a new command.
Comment 48 Honza Silhan 2015-07-20 10:26:14 EDT
We plan to work on this soon. In the first stage the option will affects install cmd only.
Comment 49 Honza Silhan 2015-08-05 12:32:35 EDT
PR: https://github.com/rpm-software-management/dnf/pull/328
Comment 50 Fedora Update System 2015-08-10 06:05:15 EDT
dnf-plugins-core-0.1.10-1.fc23,dnf-1.1.0-1.fc23,hawkey-0.6.0-1.fc23 has been submitted as an update for Fedora 23.
https://admin.fedoraproject.org/updates/dnf-plugins-core-0.1.10-1.fc23,dnf-1.1.0-1.fc23,hawkey-0.6.0-1.fc23
Comment 51 Fedora Update System 2015-08-10 06:49:23 EDT
dnf-plugins-core-0.1.10-1.fc22,dnf-1.1.0-1.fc22,hawkey-0.6.0-1.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/dnf-plugins-core-0.1.10-1.fc22,dnf-1.1.0-1.fc22,hawkey-0.6.0-1.fc22
Comment 52 Fedora Update System 2015-08-10 22:10:38 EDT
Package dnf-plugins-core-0.1.10-1.fc22, dnf-1.1.0-1.fc22, hawkey-0.6.0-1.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnf-plugins-core-0.1.10-1.fc22 dnf-1.1.0-1.fc22 hawkey-0.6.0-1.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-13162/dnf-plugins-core-0.1.10-1.fc22,dnf-1.1.0-1.fc22,hawkey-0.6.0-1.fc22
then log in and leave karma (feedback).
Comment 53 Bruno Wolff III 2015-08-14 17:10:50 EDT
This seems to be working pretty well. (It would work better if my restore list didn't currently include a package that has file conflicts with another installed package. But that's a long existing restriction and people should be fixing it at the package level.)
Comment 54 Fedora Update System 2015-08-14 22:14:15 EDT
dnf-plugins-core-0.1.10-1.fc22, hawkey-0.6.0-1.fc22, dnf-1.1.0-2.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 55 Michal Luscon 2015-08-17 03:23:15 EDT
*** Bug 1253929 has been marked as a duplicate of this bug. ***
Comment 56 Fedora Update System 2015-08-19 03:51:06 EDT
dnf-plugins-core-0.1.10-1.fc23, hawkey-0.6.0-1.fc23, dnf-1.1.0-2.fc23 has been pushed to the Fedora 23 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 57 Dario Lesca 2015-08-31 04:56:50 EDT
How to use this "feature"?

If I run dnf (like yum, without options) still not work:

[root@ludo-seashell ~]# dnf install foo banner bar
Last metadata expiration check performed 2:44:04 ago on Mon Aug 31 08:06:14 2015.
Nessun pacchetto foo disponibile.
Nessun pacchetto bar disponibile.
Errore: Unable to find a match.
[root@ludo-seashell ~]# rpm  -q dnf-plugins-core
dnf-plugins-core-0.1.10-1.fc22.noarch

If I pass only "banner", dnf install it.
 
I must use some options?
Many thanks
Comment 58 Michal Luscon 2015-08-31 05:24:06 EDT
You can specify --setopt=strict=0 on command line or edit your dnf.conf accordingly.
Comment 59 Dario Lesca 2015-08-31 07:47:10 EDT
Ok, with this options it work.
Thanks
Comment 60 Honza Silhan 2015-10-05 05:59:06 EDT
*** Bug 1268606 has been marked as a duplicate of this bug. ***
Comment 61 Gilboa Davara 2016-02-25 06:45:11 EST
(In reply to Michal Luscon from comment #58)
> You can specify --setopt=strict=0 on command line or edit your dnf.conf
> accordingly.

Thanks!
Is this documented somehow?

- Gilboa
Comment 63 Gilboa Davara 2016-03-06 03:23:11 EST
Thanks. 
/me knocks head on table.
Comment 64 Honza Silhan 2016-07-25 11:33:32 EDT
*** Bug 1359810 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.