Bug 1220074 - DNF shouldn't clean up downloaded packages when download is interrupted or transaction test fails
Summary: DNF shouldn't clean up downloaded packages when download is interrupted or tr...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 22
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Honza Silhan
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1223779 1225221 1225342 1225349 1225942 1228960 1243043 1245402 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-05-09 19:41 UTC by Erik van Pienbroek
Modified: 2015-08-18 17:25 UTC (History)
25 users (show)

Fixed In Version: dnf-1.0.2-3.fc22
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-11 02:08:42 UTC
Type: Bug


Attachments (Terms of Use)

Description Erik van Pienbroek 2015-05-09 19:41:20 UTC
When dnf is used to install/update packages and the download process is interrupted (using CTRL+C) or the transaction test fails (for example due to not enough disk space) all packages which were downloaded are automatically removed from the filesystem. This is very annoying as it causes users to re-download everything again the next time they run dnf (which is fun for people with slow internet connections..). With yum the download packages were only removed from the /var/cache/yum folder when the transaction was completed successfully. dnf should also use this behavior.

Here's an example session which illustrates the problem:

[erik@erik ~]$ sudo dnf clean all
Cleaning repos: updates-testing fedora rpmfusion-free-rawhide adobe-linux-x86_64 smock rpmfusion-nonfree-rawhide rawhide google-chrome fedora-steam fedora-chromium-stable updates
Cleaning up Everything
[erik@erik ~]$ du -hs /var/cache/dnf/
9,8M	/var/cache/dnf/
[erik@erik ~]$ sudo dnf update
<<snip>>
Transaction Summary
=============================================================================================================================================================================================================================================
Install    7 Packages
Upgrade  353 Packages
Remove     5 Packages

Total download size: 549 M
Is this ok [y/N]: n
Operation aborted.
[erik@erik ~]$ du -hs /var/cache/dnf/
142M	/var/cache/dnf/
[erik@erik ~]$ sudo dnf update -y
<<snip>>
Transaction Summary
=============================================================================================================================================================================================================================================
Install    7 Packages
Upgrade  353 Packages
Remove     5 Packages

Total download size: 549 M
Downloading Packages:
(1/360): kernel-4.0.1-300.fc22.x86_64.rpm                                                                                                                                                                     46 kB/s |  63 kB     00:01    
(2/360): kernel-modules-extra-4.0.1-300.fc22.x86_64.rpm                                                                                                                                                      195 kB/s | 2.2 MB     00:11    
<<snip>>
(141/360): libcom_err-devel-1.42.12-4.fc22.x86_64.rpm                                                                                                                                                        132 kB/s |  35 kB     00:00    
(142/360): libdca-devel-0.0.5-8.fc22.x86_64.rpm                                                                                                                                                               28 kB/s |  16 kB     00:00    
(143/360): libdca-0.0.5-8.fc22.x86_64.rpm                                                                                                                                                                     75 kB/s | 105 kB     00:01    
^CTerminated.: libdvbpsi-1.2.0-2.fc22.x86_64.rpm                                                    37% [=====================================                                                             ] 670 kB/s | 100 MB     04:12 ETA
[erik@erik ~]$ du -hs /var/cache/dnf/
149M	/var/cache/dnf/
[erik@erik ~]$ 

In this example I pressed CTRL+C after 100MB of package data was downloaded. In this situation I would expect all downloaded package data to be available in /var/cache/dnf but apparently dnf removes everything automatically even when the transaction is unsuccessful.

Comment 1 Honza Silhan 2015-05-11 10:20:49 UTC
Hi, you should set `keepcache=1` in dnf.conf. You don't want to have packages from unsuccessful transaction kept on system everytime. You could have downloaded a package with lot of dependencies then you pressed ctrl+c because you have changed your mind and rather plan to install a more lightweight package providing the same functionality with fewer dependencies. The downloaded packages will not be "ever" used then.

*** This bug has been marked as a duplicate of bug 1218415 ***

Comment 2 Erik van Pienbroek 2015-05-11 11:11:26 UTC
I have tried using the keepcache=1 option but that causes all downloaded data to remain available on the disk even after the transaction was completed successfully. This is expected behavior for the keepcache parameter, but the behavior I'm looking for is that all downloaded data remains on disk when the transaction fails (keepcache=1), but automatically gets removed when the transaction succeeds (keepcache=0).

Comment 3 Antoine Martin 2015-05-30 05:51:33 UTC
+1
Just had to spend many extra hours re-downloading packages because I control-C dnf when it got stuck at 99%.
And it is doing it again right now, 116 packages downloaded, 00:00 ETA, stuck.
keepcache is not the same thing.

Comment 4 Honza Silhan 2015-06-15 18:27:56 UTC
PR: https://github.com/rpm-software-management/dnf/pull/295

Comment 5 Parag Nemade 2015-06-19 04:30:01 UTC
*** Bug 1223779 has been marked as a duplicate of this bug. ***

Comment 6 Parag Nemade 2015-06-19 04:30:34 UTC
*** Bug 1225349 has been marked as a duplicate of this bug. ***

Comment 7 Radek Holy 2015-06-19 08:41:10 UTC
*** Bug 1228960 has been marked as a duplicate of this bug. ***

Comment 8 Honza Silhan 2015-06-19 11:41:36 UTC
*** Bug 1225221 has been marked as a duplicate of this bug. ***

Comment 9 Hedayat Vatankhah 2015-06-22 07:27:46 UTC
*** Bug 1225942 has been marked as a duplicate of this bug. ***

Comment 10 Hedayat Vatankhah 2015-06-22 07:28:45 UTC
*** Bug 1225342 has been marked as a duplicate of this bug. ***

Comment 11 cornel panceac 2015-06-27 09:06:52 UTC
last example of tis when dnf failed to complete the download of one package, than it exited and removed all almost 1GB of downloaded files. dnf should not remove the dowqnloaded files unless the transaction is completed successfully!

Comment 12 Sitsofe Wheeler 2015-06-29 11:54:17 UTC
I've just hit this issue too. I ran
# dnf upgrade
and it downloaded 600MBytes only to say
warning: /var/cache/dnf/x86_64/22/google-chrome/packages/google-chrome-stable-43.0.2357.130-1.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 7fac5991: NOKEY
Error: Public key for google-chrome-stable-43.0.2357.130-1.x86_64.rpm is not installed

At which point all 600MBytes of downloads were deleted.

You can blow away used files after a successful transaction but deleting them after failure is extraordinarily painful. Perhaps you can delete them after the next successful transaction so people have the opportunity to retry with different options?

Neither OS X, Windows nor Debian's apt throw away update downloads just because the update procedure failed so I think dnf is the odd one out...

Comment 13 Honza Silhan 2015-07-15 10:04:47 UTC
*** Bug 1243043 has been marked as a duplicate of this bug. ***

Comment 14 Eduard Vopicka 2015-07-15 16:05:17 UTC
Nearly 300GB to be re-downloaded today after failed trx chech due to lack of free space in root filesystem.

Please consider this seriously.

dnf should probably act just like yum does in this area. Plus, we really need --downloadonly.

Brgds,

Ed

Comment 15 Steeve McCauley 2015-07-15 16:50:41 UTC
I agree that keepcache is not the correct approach for this issue.  It makes no sense to flush downloaded cache after a failed transaction.

Yesterday I had to restart upgrade of 331 packages several times because downloads were stalling (hitting Ctrl-C) and one package was failing a transaction because of a packaging issue (polkit).  It wasn't until the 4th or 5th attempt that I realized that it was downloading EVERY package EVERY time I retried.  Then I spent a 30 minutes researching the issue and updating all my machines with keepcache=1.

I'm a linux user with over 20 years experience, but the average newbie user would not figure out this kind of problem, not easily at least.

The default should be to retain cache until the transaction is successful and only then flush the cache, if keepcache is set to 0.

This is also a bandwidth issue for both clients and servers.

Now I have to ensure that keepcache is configured to 1, and adjust my scripts to run 'dnf clean' after a successful transaction.

If the issue is unwanted packages in the cache after changing your mind (the dependencies scenario from comment #1 above) this is resolved if 'dnf clean' is performed immediately after any successful upgrade/install.  Default to run 'dnf clean' after a successful transaction, unless keepcache=1, never clean after a failed transaction.

Comment 16 Klaus Pedersen 2015-07-16 01:34:02 UTC
Here in China the following behaviour is the norm:

------------------------------
# dnf update 
Last metadata expiration check performed 2:39:53 a[...]
Install    7 Packages
Upgrade  337 Packages

Total download size: 561 M
Is this ok [y/N]: y
Downloading Packages:
(1/344): kernel-4.0.7-300.fc22.x86_64.rpm        73 kB/s |  67 kB     00:00    
(2/344): kernel-devel-4.0.7-300.fc22.x86_64.rpm 150 kB/s | 9.4 MB     01:04    
(3/344): kernel-core-4.0.7-300.fc22.x86_64.rpm  263 kB/s |  19 MB     01:14 
[...]

[MIRROR] autocorr-en-4.4.4.3-3.fc22.noarch.rpm: Curl error (78): Remote file not found for ftp://mirror.rise.ph/fedora/linux//updates/22/x86_64/a/autocorr-en-4.4.4.3-3.fc22.noarch.rpm [RETR response: 550]
[MIRROR] autocorr-en-4.4.4.3-3.fc22.noarch.rpm: Status code: 404 for http://ftp.cuhk.edu.hk/pub/Linux/fedora/updates/22/x86_64/a/autocorr-en-4.4.4.3-3.fc22.noarch.rpm
[FAILED] autocorr-en-4.4.4.3-3.fc22.noarch.rpm: No more mirrors to try - All mirrors were already tried without success
[DRPM] autoconf-2.69-18.fc22_2.69-20.fc22.noarch.drpm: done                    
Error: Error downloading packages:            ] 590 kB/s |  65 MB     07:04 ETA
  Cannot download a/autocorr-en-4.4.4.3-3.fc22.noarch.rpm: All mirrors were tried
------------------------------

It have been a few weeks since update didn't choke on some of the mirrors. At least with yum you could re-run the update until you eventually had all the packages downloaded.

Comment 17 Klaus Pedersen 2015-07-16 01:51:31 UTC
Forgot to mention that "keepcache=1" doesn't solve the [MIRROR] error.

dnf can be happily downloading packages, but when dnf is filling the queue with new packages to download, it will hit the 'maximum connection limit' on good mirrors or mirrors that is not up-to-date and fail.

This is now it looks:
[...]
[SKIPPED] autoconf-2.69-20.fc22.noarch.rpm: Already downloaded 
(15-17/338): bash-4.  5% [=                   ] 591 kB/s |  15 MB     07:07 ETA
[...]
All mirrors were already tried without success
(15-17/338): bash-4.  5% [=                   ] 608 kB/s |  20 MB     06:47 ETA
Error: Error downloading packages:
  Cannot download drpms/bash-4.3.39-1.fc22_4.3.39-3.fc22.x86_64.drpm: All mirrors were tried

Comment 18 Radek Holy 2015-07-22 08:02:34 UTC
*** Bug 1245402 has been marked as a duplicate of this bug. ***

Comment 19 Sitsofe Wheeler 2015-07-22 19:02:47 UTC
Good news everyone! Looks like this "issue" will be fixed in DNF 1.0.2 http://dnf.readthedocs.org/en/latest/release_notes.html#release-notes :

"1.0.2 Release Notes

When a transaction is not successfully finished, DNF preserves downloaded packages until the next successful transaction even if keepcache option is set to False."

Curious that a link to this issue didn't make the release notes though...

Comment 20 Steeve McCauley 2015-07-22 19:34:38 UTC
Excellent news.  Now I guess I get to go around removing keepcache from all my servers :(

;)

Comment 21 Fedora Update System 2015-07-31 13:30:44 UTC
dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/dnf-1.0.2-3.fc22,hawkey-0.5.9-3.fc22

Comment 22 Fedora Update System 2015-08-01 02:28:43 UTC
Package dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnf-1.0.2-3.fc22 hawkey-0.5.9-3.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-12577/dnf-1.0.2-3.fc22,hawkey-0.5.9-3.fc22
then log in and leave karma (feedback).

Comment 23 Steeve McCauley 2015-08-04 13:26:57 UTC
Interrupted the update and after restarting it SKIPPED the packages that were already cached, as expected.  Much better.

BTW, you should probably update your "Fedora Update System" message to use "dnf update --enablerepo=..." instead of yum.

Comment 24 Fedora Update System 2015-08-11 02:08:42 UTC
dnf-1.0.2-3.fc22, hawkey-0.5.9-3.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 25 John Trickey 2015-08-18 10:26:18 UTC
I'm afraid there is still a problem but it might be appropriate to spin it off into a new ticket. Basically dnf's cache management is far too brutal.

I have been following this problem due to it's relevance for 1048433.

Having installed the updates to dnf I ran dnf-automatic. This now behaves and does not wipe the cache when it exits.

To prove the cache was valid I updated just one package from those downloaded. This worked with the package being called from cache.

Buoyed I tried updating the remaining packages. This did not work. It pulled the first package from cache then proceeded to download the rest.

Checking the log I see that the first run of "dnf update" cleaned the cache on exit.

The above scenario is one used to update low bandwidth or data capped sites. Losing the download is bad news as there are times where picking off updates and applying them is more appropriate than updating the entire system in one session.

Comment 26 Radek Holy 2015-08-18 10:44:06 UTC
I agree with you that in your case, it would make sense to remove just the one installed. On the other hand, if we don't want to force users to regularly run "dnf clean packages" and if we don't want to eat all the users disk space (in the worst case), we need to remove the unneeded downloads at some point. Since it is still very hard to read the user's mind and since we want the code as simple as possible (which is beneficial for you as well as for us), the best moment seems to be the first successful transaction. Do you have a better idea when and how should we recognize that a downloaded package will never be needed any more and thus that we can remove it? I mean, people often try to install something but then they cancel the transaction and don't repeat it later any more, or they don't upgrade the system as often as dnf-automatic downloads the updates etc. If you have an (simple) idea, please, file a new report.

Comment 27 John Trickey 2015-08-18 10:59:56 UTC
I agree too but I hope we can write this one off to brain-fade.
I run an update regime where I will fix a high risk flaw as soon as it is available but leave others to a planned cycle.
Been running this with yum for years so much so that I'd forgotten the keepcache=1 hack needed in yum.conf. I had been looking for a command line option in dnf and only just seen dnf.conf also has keepcache=true as an option - OOPS.

Comment 28 cornel panceac 2015-08-18 11:01:43 UTC
I believe it's rare a user treis to install a package (and fails) and later changes its mind. What i expect to happen usually is that either: 1) the update is retried and after the update is installed, the rpm from cache can be removed or 2) the update is installed and a NEWER version of the package is installed, and then the older version from cache can be also removed together with the newer version.

Comment 29 Radek Holy 2015-08-18 11:13:45 UTC
Right, users probably do not often change their mind. On the other hand, what should we do with those who do?
Anyway, I think that your use case is very valid. I think that it deserves to be tracked in its own issue report even if we don't have a proper solution so far.

Comment 30 Radek Holy 2015-08-18 11:19:36 UTC
(BTW, I, for example, in order to diagnose a bug, often do "dnf install foo" just to find out what dependencies would "foo" add to my system but I never intend to install "foo". Right, there might be a better way - a combination of repoquery arguments, --assumeno etc., but you can never stop people doing this no matter how wrong it is.)

Comment 31 cornel panceac 2015-08-18 11:20:20 UTC
ok, so let's see if John Trickey will create a ticket for this. Then i'll propose a solution.

Comment 32 Radek Holy 2015-08-18 11:21:05 UTC
(Oh, right, it does not download packages in this case. Ignore me, sorry.)

Comment 33 Eduard Vopicka 2015-08-18 11:50:49 UTC
Hello. How about one of the following two strategies:

Do just like yum does.

        - OR -

1) Any successful transaction should remove just the processed packages.

2) Unlimited update transaction ("dnf update") should in addition remove packages downoaded earlier than N (configurable parameter) time units ago.

Of course, no strategy will be good enough for all people and situations.


Also please read with great care about metadata_expire and metadata_expire_filter in yum's doc. Especially, description of yum's metadata_expire says: "So that if the current metadata downloaded is less than this many seconds old then yum  will not  update  the  metadata against the repository." -- It does not say that yum should redownload older metadata again there are no newer metadata available. It says that yum should not download metadata too frequently, even when newer metadata are possibly available.

It is my opinion that metadata_expire in dnf has completely different meaning than in yum, causing dnf to download the same metadata again and again each time they local copy is older than metadata_expire value, even when the matadata do not change. Example of this repeated downloading of the "fedora" repo metadata by dnf in contrast to how yum behaves.

Brgds,

Ed

Comment 34 Steeve McCauley 2015-08-18 12:15:57 UTC
(In reply to Eduard Vopicka from comment #33)
> Hello. How about one of the following two strategies:
> 
> Do just like yum does.
> 
>         - OR -
> 
> 1) Any successful transaction should remove just the processed packages.
> 
> 2) Unlimited update transaction ("dnf update") should in addition remove
> packages downoaded earlier than N (configurable parameter) time units ago.
> 
> Of course, no strategy will be good enough for all people and situations.
> 

The latter, especially #1, seems to me to be the correct solution.  If I run 'dnf update foo' the transaction should only apply to cached package foo and its dependences. Although I would prefer to never remove unused cached packages since disk is cheap, and bandwidth is often not, I could live without the implementation of option #2 :)

Comment 35 cornel panceac 2015-08-18 12:30:58 UTC
What i would like to see in any application writing data to disk is this:

Before writing data to disk, check if there's enough available disk space, so that after data is written there's still 10% free disk space left for user (10% shall be change by user, if needed).
If this is not fulfilled, decline to perform the operation, giving user details so that he has to create more free space.

In case of dnf, when there's not enough disk space and cleaning the cache would provide the required disk space, we can ask the user for approval to proceed with disk cache cleanup. (if this is not adding enough free disk spcae, provide user with option to stil lcleanup cache, but make it clear this is not enough).

Comment 36 John Trickey 2015-08-18 13:27:14 UTC
(In reply to cornel panceac from comment #31)

It is highly unlikely I will raise a ticket which will require a solution other than a bugfix. I was too busy proving the original bug had been fixed to notice I had a configuration error. As soon as I have enough updates to create a test case, I will be able to see if "keepcache=true" works and, unless I find anything else, that's all which worries me about dnf.

If you can see a different problem I suggest you raise the ticket and propose the solution.

I agree with Radek in that the user should not have to worry about cache management but I add the caveat "unless he/she chooses otherwise". As long as it all works as described on the tin, I don't think there's a problem.

Strategy 1 in Eduard's is as far as dnf should go. However such behaviour should be configurable. Strategy 2 would be a step too far but acceptable as an option to "dnf clean ..."

Comment 37 Eduard Vopicka 2015-08-18 17:25:01 UTC
The two strategies mentioned in #33 are:

The first, behave like yum does.

The second, 1) and 2) together. 1) cleans sucessfully processed packages only. And 2) performs delayed general cleanup of zombie downloaded packages. Thus, you do not lose downloaded packages that may be used by the next or repeated dnf command, but with some delay the will also be no zombie packages in download directories and no need for explicit maintenance.

And I expect 1) and 2) to work together in harmony.

And those requiring total control are supported via keep_cache=1.

Ed


Note You need to log in before you can comment on or make changes to this bug.