Bug 1500370

Summary: DNF becomes intolerably slow when using --exclude
Product: [Fedora] Fedora Reporter: Artem S. Tashkinov <aros>
Component: dnfAssignee: Jaroslav Rohel <jrohel>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 26CC: dmach, ignatenko, jmracek, jrohel, mhatina, rpm-software-management, vmukhame
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-11 11:36:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
rpm -qa | sort none

Description Artem S. Tashkinov 2017-10-10 13:06:27 UTC
Created attachment 1336761 [details]
rpm -qa | sort

# time dnf update --exclude='selinux-policy*,kernel*,trousers*'
Last metadata expiration check: 1:45:36 ago on Tue 10 Oct 2017 04:16:43 PM +05.



Dependencies resolved.
================================================================================
 Package                          Arch     Version              Repository
                                                                           Size
================================================================================
Upgrading:
 acpica-tools                     x86_64   20170831-1.fc26      updates   957 k
 dnf                              noarch   2.7.3-1.fc26         updates   340 k
 dnf-conf                         noarch   2.7.3-1.fc26         updates    67 k
 dnf-plugins-core                 noarch   2.1.5-1.fc26         updates    50 k
 dnf-yum                          noarch   2.7.3-1.fc26         updates    48 k
 python3                          x86_64   3.6.2-8.fc26         updates    64 k
 python3-devel                    x86_64   3.6.2-8.fc26         updates   861 k
 python3-dnf                      noarch   2.7.3-1.fc26         updates   455 k
 python3-dnf-plugin-leaves        noarch   2.1.5-1.fc26         updates    43 k
 python3-dnf-plugin-show-leaves   noarch   2.1.5-1.fc26         updates    41 k
 python3-dnf-plugins-core         noarch   2.1.5-1.fc26         updates   141 k
 python3-libs                     i686     3.6.2-8.fc26         updates   1.4 M
 python3-libs                     x86_64   3.6.2-8.fc26         updates   1.4 M
 system-python                    x86_64   3.6.2-8.fc26         updates    56 k
 system-python-libs               i686     3.6.2-8.fc26         updates   6.3 M
 system-python-libs               x86_64   3.6.2-8.fc26         updates   6.3 M

Transaction Summary
================================================================================
Upgrade  16 Packages

Total download size: 19 M
Is this ok [y/N]: Operation aborted.

real	1m10.159s
user	1m9.852s
sys	0m0.196s

This is on an Intel i5 Sky Lake CPU running at 2.8GHz.





When --exclude is not specified:

Is this ok [y/N]: Operation aborted.

real	0m2.887s
user	0m2.727s
sys	0m0.138s

Comment 1 Jaroslav Mracek 2017-10-10 16:23:05 UTC

*** This bug has been marked as a duplicate of bug 1499534 ***

Comment 2 Artem S. Tashkinov 2017-10-10 23:27:48 UTC
I'm not sure these bugs are related.

I'm not using groups at all.

I'm running update with --exclude.

Comment 3 Jaroslav Rohel 2017-10-11 06:33:22 UTC
I have some idea how make code quicker.

But the main problem of --exclude and --include is that globs are searched in filenames too. And file query is very slow.

But Jarda Mracek done some optimization in recent patch (https://github.com/rpm-software-management/libdnf/pull/338/commits/2de54548ea974f76ca8d06f8a7a8e604a306fc33). Now, file query is called only for patterns which start "/" or "*/".

So main problem of your example was fixed.

Comment 4 Jaroslav Rohel 2017-10-11 11:36:02 UTC
We made another optimalizations of excludes/includes.

Jarda Mracek PR: https://github.com/rpm-software-management/dnf/pull/948
My PR: https://github.com/rpm-software-management/dnf/pull/950

So I am closing this bug.