Bug 1418198 - Tear and wear/disk thrashing during DNF clean operation
Summary: Tear and wear/disk thrashing during DNF clean operation
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 27
Hardware: All
OS: Linux
medium
high
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-01 09:26 UTC by Artem S. Tashkinov
Modified: 2018-11-27 21:00 UTC (History)
8 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2018-11-27 14:41:40 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Artem S. Tashkinov 2017-02-01 09:26:22 UTC
I have noticed that during `dnf update` the cleanup operation writes tens of megabytes of data (sometimes up to a hundred and more) while `rpm -U` doesn't do that and uses the disk a lot less.

I wonder if it's possible to optimize this process so that cleanup worked more efficiently and maybe processed multiple packages in one step (i.e. a single writeout to /var/lib/rpm).

Comment 1 Honza Silhan 2017-02-06 13:02:20 UTC
We are using rpm transaction API and just receive callbacks. DNF only deletes packages in the package cache after transaction. No additional cache updates happens.

RPM devs, is there any difference WRT IO operations when using RPM API and RPM from CLI for upgrade?

Comment 2 Panu Matilainen 2017-02-06 13:11:51 UTC
Rpm cli uses the same exact API to do transactions. Only it doesn't do anything at all in the callbacks, whereas dnf does ... I don't know what, but yum used to update the history db and whatnot in there.  And then there are plugins and the like to consider.

But guessing is waste of time, you need to investigate. I'd suggest you start with strace to see what is being written and where.

Comment 3 Artem S. Tashkinov 2017-02-07 12:33:02 UTC
There are people who know DNF source inside our and you tell me to strace? WTF? Seriously?

I update a 30KB package using DNF and my disk writes are over 5MB(!). That's just insane and I'm not gonna dig into megabytes of strace data when someone perfectly knows what DNF does during update.

Let's cehck today's updates:

# dnf update --downloadonly

Last metadata expiration check: 1:13:12 ago on Tue Feb  7 16:08:35 2017.
Dependencies resolved.
==============================================================================================================================================================================================
 Package                                           Arch                                     Version                                           Repository                                 Size
==============================================================================================================================================================================================
Upgrading:
 bash-completion                                   noarch                                   1:2.5-1.fc25                                      updates                                   268 k
 gnutls                                            i686                                     3.5.8-2.fc25                                      updates                                   712 k
 gnutls                                            x86_64                                   3.5.8-2.fc25                                      updates                                   737 k
 gnutls-c++                                        i686                                     3.5.8-2.fc25                                      updates                                    37 k
 gnutls-c++                                        x86_64                                   3.5.8-2.fc25                                      updates                                    36 k
 gnutls-dane                                       i686                                     3.5.8-2.fc25                                      updates                                    38 k
 gnutls-dane                                       x86_64                                   3.5.8-2.fc25                                      updates                                    37 k
 gnutls-devel                                      i686                                     3.5.8-2.fc25                                      updates                                   1.6 M
 gnutls-devel                                      x86_64                                   3.5.8-2.fc25                                      updates                                   1.6 M

Transaction Summary
==============================================================================================================================================================================================
Upgrade  9 Packages

Total download size: 5.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/9): bash-completion-2.5-1.fc25.noarch.rpm                                                                                                                  431 kB/s | 268 kB     00:00    
(2/9): gnutls-dane-3.5.8-2.fc25.x86_64.rpm                                                                                                                    1.0 MB/s |  37 kB     00:00    
(3/9): gnutls-c++-3.5.8-2.fc25.x86_64.rpm                                                                                                                     750 kB/s |  36 kB     00:00    
(4/9): gnutls-3.5.8-2.fc25.i686.rpm                                                                                                                           3.6 MB/s | 712 kB     00:00    
(5/9): gnutls-3.5.8-2.fc25.x86_64.rpm                                                                                                                         725 kB/s | 737 kB     00:01    
(6/9): gnutls-dane-3.5.8-2.fc25.i686.rpm                                                                                                                      692 kB/s |  38 kB     00:00    
(7/9): gnutls-devel-3.5.8-2.fc25.x86_64.rpm                                                                                                                   1.4 MB/s | 1.6 MB     00:01    
(8/9): gnutls-c++-3.5.8-2.fc25.i686.rpm                                                                                                                       411 kB/s |  37 kB     00:00    
(9/9): gnutls-devel-3.5.8-2.fc25.i686.rpm                                                                                                                     3.1 MB/s | 1.6 MB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         1.4 MB/s | 5.0 MB     00:03     
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.

Let's note the disk writes at this point after running `sync`: 689,627,136 bytes (I'm writing this in a text editor, so at the moment there are no applications which are writing anything to the disk).

Now let's run dnf for real:
# dnf update
and Ctrl+Z it right before the cleanup stage begins

Disk writes: 712,798,208 bytes

After cleanup finishes and dnf exists, disk writes: 736,256,000 bytes

IOW DNF cleanup has written over 24 megabytes of data for 9 petty packages.

Comment 4 Panu Matilainen 2017-02-07 12:49:01 UTC
Artem, I was merely pointing out that *somebody* needs to actually investigate what is being written and where. Typically that'd be the developers, assuming they can reproduce.

Comment 5 Honza Silhan 2017-02-13 14:18:41 UTC
DNF team will investigate.

Comment 6 Fedora End Of Life 2017-11-16 19:08:34 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 7 Artem S. Tashkinov 2017-11-16 20:32:55 UTC
DNF install/update/delete all write too much data to the HDD. It would be great if DNF operations could be optimized.

Comment 8 Jaroslav Mracek 2018-11-27 14:41:40 UTC
We will definitely improve a performance of dnf.

But the reason why RPM transaction requires less resources than DNF is quite obvious. We use RPM to perform a transaction therefore this part is same.

Additionally we:
1. Read metadata and creates *.solv files (Required by solver) even for installed packages
2. We have our own database about transactions (history commands)
3. We store information about installed groups, expired repository, downloaded rpms or so on. 

We provide more functionality than RPM, therefore we need more resources.

I think that it is not necessary to keep the bug open, because this is anyway long term plan of DNF team. Thanks for your report.

Comment 9 Ben Cotton 2018-11-27 15:34:56 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 10 Artem S. Tashkinov 2018-11-27 21:00:04 UTC
(In reply to Ben Cotton from comment #9)

Having plans and having an open bug report are close things but this bug report could still serve as a friendly reminder.

If you insist on this bug report being closed I won't object.


Note You need to log in before you can comment on or make changes to this bug.