Bug 438625

Summary: [RFE] performance: disable signature/digest checking for rpm query modes
Product: [Fedora] Fedora Reporter: Bill Crawford <billcrawford1970>
Component: rpmAssignee: Fedora Packaging Toolset Team <packaging-team>
Status: CLOSED UPSTREAM QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: low    
Version: rawhideCC: jason.vas.dias, lkardos, n3npq, pnasrat, yaneti
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-20 16:11:58 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bill Crawford 2008-03-23 11:51:47 UTC
[feel free to reassign this to redhat-rpm-config or whatever ...]

Description of problem:
Query of rpm database is pretty slow (especially on my old PATA disk, but 
that's another story). The problem is that all the headers are being 
signature-checked when doing simple queryies like "show me all the packages 
installed". Overhead is obviously fairly small when querying only a single 
package, but "rpm -qa" shouldn't take >15 seconds when all it actually needs 
to do is print out the package names. Verify modes, rebuilddb etc should 
obviously still do the checks, especially the latter. Not sure about install, 
erase etc.

I'd like to suggest (as the minimal change) defining _vsflags_query to 0xf0f00 
either in /usr/lib/rpm/macros or somewhere under /etc/rpm, comme ça:

    %_vsflags_query         0xf0f00

Additional info:
[bill@pikachu ~]$ time rpm -qa --define '_vsflags_query 0xf0000' >/dev/null

real    0m18.063s
user    0m12.861s
sys     0m0.463s
[bill@pikachu ~]$ time rpm -qa --define '_vsflags_query 0xf0f00' >/dev/null

real    0m1.794s
user    0m0.948s
sys     0m0.400s

Comment 1 Yanko Kaneti 2008-03-23 12:21:49 UTC
I bet your second one runs from cache
you should try with  echo 3 > /proc/sys/vm/drop_caches    in between

The difference on my system is negligible. 0.5-1 s

That being said I also find -qa to be too slow, for no reason that I can imagine.

Comment 2 Bill Crawford 2008-03-24 14:54:00 UTC
Caching brings the second run of the first command to between 13.5 and 14.5 
seconds. The second is still approaching ten times faster. Maybe you have 
fewer packages installed?

Comment 3 Bill Crawford 2008-03-24 14:55:24 UTC
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0000' >/dev/null

real    0m17.078s
user    0m12.014s
sys     0m1.062s
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0000' >/dev/null

real    0m13.636s
user    0m11.917s
sys     0m0.583s
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0000' >/dev/null

real    0m14.454s
user    0m12.028s
sys     0m0.460s
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0f00' >/dev/null

real    0m1.513s
user    0m0.972s
sys     0m0.400s
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0f00' >/dev/null

real    0m1.506s
user    0m0.973s
sys     0m0.399s
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0f00' >/dev/null

real    0m1.638s
user    0m1.001s
sys     0m0.408s
[root@pikachu ~]# echo 3 > /proc/sys/vm/drop_caches 
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0f00' >/dev/null

real    0m6.141s
user    0m1.062s
sys     0m0.781s
[root@pikachu ~]# time rpm -qa --define '_vsflags_query 0xf0f00' >/dev/null

real    0m1.451s
user    0m0.988s
sys     0m0.341s


Comment 4 Bill Crawford 2008-03-24 14:56:02 UTC
All that's been proven is that with new flags, from cold cache, is twice as 
fast as old flags, even with warm cache :o)

Comment 5 Yanko Kaneti 2008-03-24 16:13:52 UTC
I can reporoduce the ~50% speeup on two fc8 systems i386 and x86_64 , unable to
reproduce on two rawhide systems i386 and x86_64


Comment 6 Panu Matilainen 2008-04-02 11:36:09 UTC
Yes, disabling signature checks speeds up db queries a lot. But changing
_vsflags_query disables signature checks on all types of queries, including
those of non-installed, untrusted packages which is not so good...

Comment 7 Bill Crawford 2008-04-02 11:49:09 UTC
#       0x00f00 (_RPMVSF_NOHEADER)     --nohdrchk      if set, don't check rpmdb
headers

Should only affect the reads from the database. Although I could be wrong, I'm
only looking at the comments in /usr/lib/rpm/macros.

Comment 8 Jon Stanley 2008-04-23 20:29:48 UTC
Adding FutureFeature keyword to RFE's.

Comment 9 Panu Matilainen 2009-06-03 14:05:46 UTC
*** Bug 501103 has been marked as a duplicate of this bug. ***

Comment 10 Jeff Johnson 2009-06-03 23:01:57 UTC
This is *purely* a configuration issue.

You can disable whether rpm --query checks signatures (or not)
by configuring a macro. No one "default" will ever satisfy everyone.

The macro can be set persistently, on a per-user, per-system, per-vendor basis.
See details in /usr/lib/rpm/macros.

And the issue is not speed. Noone has ever claimed that disabling signature/digest checking
isn't faster. Cold cache, warm cache, It Really Doesn't Matter.

Hint: Signatures/digests are not checked for performance reasons.

Comment 11 Jeff Johnson 2009-06-03 23:08:57 UTC
Re: comment #6: Better read your @rpm.org code. vsflags can be disabled for --query
without affecting any other behavior.

Comment 12 Fedora Admin XMLRPC Client 2012-04-13 23:13:06 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 13 Fedora Admin XMLRPC Client 2012-04-13 23:14:15 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 14 Ľuboš Kardoš 2015-03-20 16:11:58 UTC
Upstream patch adds macro _vsflags_query_rpmdb. If it is defined then it is used for queries when "-p" is not set on commandline. If it is not defined then macro _vsflags_query is used. For queries with "-p" on commadline macro _vsflags_query is always used. Macro _vsflags_query_rpmdb is set to 0xf0f00 (no checking) by default.

Comment 15 Bill Crawford 2016-09-27 13:55:02 UTC
diff --git a/lib/query.c b/lib/query.c
index 739a0de..febb7b9 100644
--- a/lib/query.c
+++ b/lib/query.c
@@ -594,7 +594,7 @@ int rpmcliQuery(rpmts ts, QVA_t qva, char * const * argv)
        qva->qva_queryFormat = fmt;
     }
 
-    if (!(qva->qva_source & RPMQV_RPM) &&
+    if (!(qva->qva_source == RPMQV_RPM) &&
        rpmExpandNumeric("%{?_vsflags_query_rpmdb:1}")) {
 
        vsflags = rpmExpandNumeric("%{?_vsflags_query_rpmdb}");

Comment 16 Bill Crawford 2016-09-27 13:57:16 UTC
(otherwise fails when doing -qa, as RPMQV_ALL == 2, so (qva->qva_source & 3) is non-zero).

Worth re-opening? :)