Bug 1486137 - Cannot remove duplicates
Summary: Cannot remove duplicates
Status: CLOSED DUPLICATE of bug 1465292
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: rawhide
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Jaroslav Mracek
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2017-08-29 07:13 UTC by Milan Crha
Modified: 2017-08-30 18:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2017-08-30 10:06:02 UTC
Type: Bug

Attachments (Terms of Use)

Description Milan Crha 2017-08-29 07:13:13 UTC
I've a virtual machine which runs rawhide. It lasts for a long time and I keep it updated. I just realized that there are duplicate packages installed, but I cannot remove them. I even do not understand how that happened, to be honest.

I realized that when checking what gtk3 version I've installed:

   $ rpm -q gtk3

thus I tried dnf to see whether there are more of them:

   $ dnf check
   Error: Check discovered 1696 problem(s)

To see how many duplicates there are:

   $ dnf check | grep --count duplicate
   Error: Check discovered 1696 problem(s)

Then I found what to use to remove duplicates:
   # dnf remove --duplicates
     Problem: The operation would result in removing the following protected
     packages: sudo, systemd, systemd, systemd-udev, systemd-udev

thus bad luck for me. I do not want to have installed duplicate packages, because I want to save space on the disk.

For the info:
   $ rpm -q sudo systemd systemd-udev

I do not see anything useful in `dnf remove --help`. Could there be added a similar argument to --skip-broken, to remove what it can remove and left the rest for the user? Though the error doesn't make much sense, because there will be left one version (I believe the latest) of the "protected package", thus no harm to the system will be made.

Comment 1 Jaroslav Mracek 2017-08-29 10:15:10 UTC
I created a pull-request that should help with the problem (https://github.com/rpm-software-management/dnf/pull/880). Please can you try the new implementation and report if the problem was solved?

Comment 2 Milan Crha 2017-08-29 13:21:24 UTC
(In reply to Jaroslav Mracek from comment #1)
> Please can you try the new implementation and report if the problem was solved?

How do I do that, please? I've no single idea about rpm-software-management tools, the same as github interface is too confusing to me. If you have a test package with it, then it'll be a different story.

Comment 3 Jaroslav Mracek 2017-08-29 15:49:56 UTC
I create a repo that provides the patch. First of all you have to enable dnf-nightly repo to fulfill dependencies ("sudo dnf copr enable rpmsoftwaremanagement/dnf-nightly"). Then enable repo with my patch ("sudo dnf copr enable jmracek/test"). Then install the patch: "sudo dnf install dnf-0:2.6.4-1.git.40.040c4a4.fc28.noarch"

May be it will require to add "--allowerasing"

Hope that it helps

Comment 4 Milan Crha 2017-08-30 06:45:39 UTC
> May be it will require to add "--allowerasing"

Right, it does (some grub2 stuff conflicting, even it came from 'rawhide' repository).

At the moment:
   # dnf remove --duplicates
runs, but what I do not fully understand is that it downloads all the packages again (840MB to download). It doesn't make much sense, because they are already installed, unless you want their scripts to know what they installed and what can be removed (like some /usr/lib64/ links to correct versions are shared by multiple packages), though `rpm --whatprovides` still gets the information, as can be seen here:

$ rpm -qa gtk3*

$ ls -l /usr/lib64/libgtk-3*
...      21 Aug 22 16:19 /usr/lib64/libgtk-3.so -> libgtk-3.so.0.2200.19
...      21 Aug 22 16:19 /usr/lib64/libgtk-3.so.0 -> libgtk-3.so.0.2200.19
... 7479792 May 25 13:17 /usr/lib64/libgtk-3.so.0.2200.15
... 7418320 Jun 21 05:25 /usr/lib64/libgtk-3.so.0.2200.16
... 7471896 Aug 22 16:19 /usr/lib64/libgtk-3.so.0.2200.19

$ for i in `ls /usr/lib64/libgtk-3*`; do echo "   $i:"; \
  rpm -q --whatprovides $i; done


Thus there might be a way to delete files which are not needed without any additional download.

Comment 5 Milan Crha 2017-08-30 07:06:51 UTC
The command finished and this is the result:

# dnf check
> fwupd-labels-0.9.4-1.fc27.noarch is a duplicate with fwupd-labels-0.9.6-1.fc27.x86_64
> pyserial-3.1.1-3.fc26.noarch is obsoleted by python2-pyserial-3.1.1-5.fc28.noarch
> python-enum34-1.1.6-1.fc27.noarch is obsoleted by python2-enum34-1.1.6-3.fc28.noarch
> python-ipaddress-1.0.18-1.fc27.noarch is obsoleted by python2-ipaddress-1.0.18-3.fc28.noarch
> python-xpyb-1.3.1-7.fc26.x86_64 is obsoleted by python2-xpyb-1.3.1-10.fc28.x86_64
> setools-python3-4.1.0-3.fc27.x86_64 is obsoleted by python3-setools-4.1.1-4.fc28.x86_64
> Error: Check discovered 6 problem(s)

# dnf remove --duplicates
> Loading repositories.
> Last metadata expiration check: 0:43:26 ago on Wed 30 Aug 2017 08:21:15 AM CEST.
> Dependencies resolved.
> Nothing to do.
> Complete!

Note of the fwupd-labels, which is claimed as a duplicate, but the command doesn't fix it.

Comment 6 Jaroslav Mracek 2017-08-30 09:26:26 UTC
In your case "sudo dnf shell" should solve the problem.

then add following lines to input:
remove fwupd-labels-0.9.4-1.fc27.noarch
reinstall fwupd-labels-0.9.6-1.fc27.x86_64

Please if fwupd-labels-0.9.6-1.fc27.x86_64 is not available, replace line reinstall fwupd-labels-0.9.6-1.fc27.x86_64 by:

remove fwupd-labels-0.9.6-1.fc27.x86_64
install fwupd-labels-<available version>

Hope that it helps

Comment 7 Jaroslav Mracek 2017-08-30 10:06:02 UTC
What "dnf remove --duplicates" in my patch does:

It removes duplicates expect the last version.
Also it tries to reinstall the latest package to ensure:
- all dependencies are installed (often happen in case of broken transaction)
- ensure that scripts were correctly applied
- it also allows removal of duplicates of protected packages

I believe that this approach is more safe than only removal of duplicates that could result in nonbootable system.

If descriptions above don't help, please don't hesitate to reopen the bug report.

Comment 8 Milan Crha 2017-08-30 12:15:27 UTC
(In reply to Jaroslav Mracek from comment #6)
> In your case "sudo dnf shell" should solve the problem.
> ...

Why did it fail with `dnf remove --duplicates`? That might be the last outstanding bit. I know I can fix it myself, but I have the system in that state in case you'd like to test any further change.

(In reply to Jaroslav Mracek from comment #7)
> If descriptions above don't help, please don't hesitate to reopen the bug
> report.

Nope, this is perfectly fine.

Are you sure you wanted to close this as NotABug? That doesn't seem accurate, especially when you provided a patch which will be available in the future release of dnf/libdnf and the comment #0 is valid until that particular fix is received within usual updates.

Comment 9 Jaroslav Mracek 2017-08-30 18:43:33 UTC
(Reply to Comment 7)
> Why did it fail with `dnf remove --duplicates`?

This duplicate is very special. You updated package from noarch to arch. This is possible but "dnf remove --duplicates" is unable to detect it. I think that this is better than to remove some multiarch packages accidentally. In future we will have a better tools to recover broken transaction, or rerun broken transaction.

Also I will mark this bug as a duplicate, just for tracking purpose.

*** This bug has been marked as a duplicate of bug 1465292 ***

Note You need to log in before you can comment on or make changes to this bug.