Bug 2048168 - rpm-4.17.0-3 performance regression
Summary: rpm-4.17.0-3 performance regression
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: rpm
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2047477 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-29 20:57 UTC by Ondrej Mosnacek
Modified: 2022-02-04 01:23 UTC (History)
14 users (show)

Fixed In Version: rpm-4.17.0-4.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-04 01:23:07 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Flamegraph of `rpm --reinstall breeze-icon-theme-*.noarch.rpm` with rpm-4.17.0-3.fc35 (151.07 KB, image/svg+xml)
2022-01-29 20:57 UTC, Ondrej Mosnacek
no flags Details

Description Ondrej Mosnacek 2022-01-29 20:57:37 UTC
Created attachment 1857711 [details]
Flamegraph of `rpm --reinstall breeze-icon-theme-*.noarch.rpm` with rpm-4.17.0-3.fc35

Description of problem:
rpm-4.17.0-3 introduced a performance regression that causes a long delay when updating/removing some packages.

Version-Release number of selected component (if applicable):
rpm-4.17.0-3.fc35

How reproducible:
Reliably.

Steps to Reproduce:
1. Grab an F35 VM.
2. # dnf install -y breeze-icon-theme
3. # dnf download breeze-icon-theme
4. Update rpm to rpm-4.17.0-3.fc35
5. # time rpm --reinstall breeze-icon-theme-*.noarch.rpm

Actual results:
2m49.142s

Expected results:
0m4.731s as with rpm-4.17.0-1.fc35

Additional info:
Not to be confused with BZ 2029709, which is filesystem-related. This one is almost definitely a bug in rpm itself, as most of the time is spent in the rpmtriggersPrepPostUnTransFileTrigs() function (see attached flamegraph), which is modified by a patch introduced in rpm-4.17.0-3.fc35:
https://src.fedoraproject.org/rpms/rpm/c/b3ac7d69fd307e87f35c5db8288d5743e011aff5?branch=f35

Found by following this hint from Vitaly Zaitsev:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/WMSGBUWLQPETC32QPUUJRB7QJU5QQOUR/

Comment 1 Panu Matilainen 2022-01-31 10:14:30 UTC
Ack, easily reproduced. Thanks for the report.

Comment 2 Panu Matilainen 2022-02-01 11:31:49 UTC
Note that the old version was fast because it failed to do a lot of necessary work.

Presumably some/much of the lost time can be gained back by optimizing, but this isn't some random bug/regression making things slower, it's caused by doing previously missing calculations proper file trigger execution.

Comment 3 Vitaly 2022-02-01 11:51:00 UTC
> Note that the old version was fast because it failed to do a lot of necessary work.

It was always fast.

Comment 4 Ondrej Mosnacek 2022-02-01 12:16:04 UTC
I know that it probably won't be equally as fast as before. The thing is, even if it took 3x longer with that patch, it would be still ok. But it takes ~30x longer :)

# strace -f --summary-only rpm --reinstall breeze-icon-theme-*.noarch.rpm
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 97.51   70.382951           1  55545184           pread64
[...]

I mean surely it has to be possible to compute the triggers with less than 55 million DB page reads... The whole database is just ~21MB, so with that many calls you can read the whole DB more than 10 000 times.

Comment 5 Panu Matilainen 2022-02-01 13:07:07 UTC
Yes like I said, it's probably possible to optimize it quite a bit. Just don't expect us to revert the patch because of a slowdown on some packages.

Comment 6 Kevin P. Fleming 2022-02-01 13:11:41 UTC
It appears that this effect is triggered by kernel packages as well (unsurprising since they include a large number of files). That will probably make it more noticeable to most users.

Comment 7 Vitaly 2022-02-01 13:17:32 UTC
> Yes like I said, it's probably possible to optimize it quite a bit. Just don't expect us to revert the patch because of a slowdown on some packages.

Hangs for 5 (five) minutes with 100% usage of 1 CPU core is not good.

Users are already complaining about this: https://bugzilla.redhat.com/show_bug.cgi?id=2032498

Comment 8 Panu Matilainen 2022-02-01 13:27:43 UTC
Nobody's saying it's good, but please understand that executing wrong filetriggers is worse.

Comment 9 Panu Matilainen 2022-02-01 15:00:41 UTC
...however being 30x slower AND still executing wrong stuff is pretty bad, lol.

Seems like a bit of a Monday-morning patch (by yours truly) this one.

Comment 10 Fedora Update System 2022-02-02 12:50:52 UTC
FEDORA-2022-a9a84f2456 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-a9a84f2456

Comment 11 Fedora Update System 2022-02-03 01:35:27 UTC
FEDORA-2022-a9a84f2456 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-a9a84f2456`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-a9a84f2456

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Panu Matilainen 2022-02-03 08:20:19 UTC
*** Bug 2047477 has been marked as a duplicate of this bug. ***

Comment 13 Fedora Update System 2022-02-04 01:23:07 UTC
FEDORA-2022-a9a84f2456 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.