Bug 1500370 - DNF becomes intolerably slow when using --exclude
Summary: DNF becomes intolerably slow when using --exclude
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 26
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Jaroslav Rohel
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-10 13:06 UTC by Artem S. Tashkinov
Modified: 2017-10-11 11:36 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-10-11 11:36:02 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
rpm -qa | sort (41.63 KB, text/plain)
2017-10-10 13:06 UTC, Artem S. Tashkinov
no flags Details

Description Artem S. Tashkinov 2017-10-10 13:06:27 UTC
Created attachment 1336761 [details]
rpm -qa | sort

# time dnf update --exclude='selinux-policy*,kernel*,trousers*'
Last metadata expiration check: 1:45:36 ago on Tue 10 Oct 2017 04:16:43 PM +05.



Dependencies resolved.
================================================================================
 Package                          Arch     Version              Repository
                                                                           Size
================================================================================
Upgrading:
 acpica-tools                     x86_64   20170831-1.fc26      updates   957 k
 dnf                              noarch   2.7.3-1.fc26         updates   340 k
 dnf-conf                         noarch   2.7.3-1.fc26         updates    67 k
 dnf-plugins-core                 noarch   2.1.5-1.fc26         updates    50 k
 dnf-yum                          noarch   2.7.3-1.fc26         updates    48 k
 python3                          x86_64   3.6.2-8.fc26         updates    64 k
 python3-devel                    x86_64   3.6.2-8.fc26         updates   861 k
 python3-dnf                      noarch   2.7.3-1.fc26         updates   455 k
 python3-dnf-plugin-leaves        noarch   2.1.5-1.fc26         updates    43 k
 python3-dnf-plugin-show-leaves   noarch   2.1.5-1.fc26         updates    41 k
 python3-dnf-plugins-core         noarch   2.1.5-1.fc26         updates   141 k
 python3-libs                     i686     3.6.2-8.fc26         updates   1.4 M
 python3-libs                     x86_64   3.6.2-8.fc26         updates   1.4 M
 system-python                    x86_64   3.6.2-8.fc26         updates    56 k
 system-python-libs               i686     3.6.2-8.fc26         updates   6.3 M
 system-python-libs               x86_64   3.6.2-8.fc26         updates   6.3 M

Transaction Summary
================================================================================
Upgrade  16 Packages

Total download size: 19 M
Is this ok [y/N]: Operation aborted.

real	1m10.159s
user	1m9.852s
sys	0m0.196s

This is on an Intel i5 Sky Lake CPU running at 2.8GHz.





When --exclude is not specified:

Is this ok [y/N]: Operation aborted.

real	0m2.887s
user	0m2.727s
sys	0m0.138s

Comment 1 Jaroslav Mracek 2017-10-10 16:23:05 UTC

*** This bug has been marked as a duplicate of bug 1499534 ***

Comment 2 Artem S. Tashkinov 2017-10-10 23:27:48 UTC
I'm not sure these bugs are related.

I'm not using groups at all.

I'm running update with --exclude.

Comment 3 Jaroslav Rohel 2017-10-11 06:33:22 UTC
I have some idea how make code quicker.

But the main problem of --exclude and --include is that globs are searched in filenames too. And file query is very slow.

But Jarda Mracek done some optimization in recent patch (https://github.com/rpm-software-management/libdnf/pull/338/commits/2de54548ea974f76ca8d06f8a7a8e604a306fc33). Now, file query is called only for patterns which start "/" or "*/".

So main problem of your example was fixed.

Comment 4 Jaroslav Rohel 2017-10-11 11:36:02 UTC
We made another optimalizations of excludes/includes.

Jarda Mracek PR: https://github.com/rpm-software-management/dnf/pull/948
My PR: https://github.com/rpm-software-management/dnf/pull/950

So I am closing this bug.


Note You need to log in before you can comment on or make changes to this bug.