Bug 1256313 - DNF should download packages specified by URL into the cache directory
Summary: DNF should download packages specified by URL into the cache directory
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf   
(Show other bugs)
Version: 25
Hardware: Unspecified
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords: Reopened, Triaged
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-08-24 11:00 UTC by Ali Akcaagac
Modified: 2017-05-05 13:34 UTC (History)
7 users (show)

Fixed In Version: dnf-2.4.0-1.fc26
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-05-05 13:34:35 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
downloader script (3.60 KB, application/x-shellscript)
2015-08-24 11:00 UTC, Ali Akcaagac
no flags Details

Description Ali Akcaagac 2015-08-24 11:00:23 UTC
Created attachment 1066304 [details]
downloader script

I have a script, that downloads newest (but not yet released) wine from Fedora koji. I depend on this because wine is usually being pushed to testing with a delay or so. Anyways this is not just wine related.

Assume running the attached script. This will download wine from the koji locations and install it. This simple script was written a few years back and used to use yum. So basicly this is a s/yum/dnf/g nobrainer.

What I end is getting this:

-bash-4.3$ ls -l /tmp/*.rpm
-rw------- 1 root root    78796 Aug 24 12:44 /tmp/tmp0zWXUX.rpm
-rw------- 1 root root  1737796 Aug 24 12:44 /tmp/tmp1mzm1T.rpm
-rw------- 1 root root   142504 Aug 24 12:44 /tmp/tmp2eD9rK.rpm
-rw------- 1 root root   114440 Aug 24 12:44 /tmp/tmp2VWI1f.rpm
-rw------- 1 root root    52464 Aug 24 12:44 /tmp/tmp3UU378.rpm
-rw------- 1 root root    67896 Aug 24 12:44 /tmp/tmp43s0B1.rpm
-rw------- 1 root root    83180 Aug 24 12:44 /tmp/tmp6RJuLY.rpm
-rw------- 1 root root    73340 Aug 24 12:43 /tmp/tmp9EBbaY.rpm
-rw------- 1 root root    65412 Aug 24 12:44 /tmp/tmp9LKbJA.rpm
-rw------- 1 root root    65388 Aug 24 12:44 /tmp/tmp9OZKlJ.rpm
-rw------- 1 root root   100632 Aug 24 12:44 /tmp/tmpAYzh1k.rpm
-rw------- 1 root root    64772 Aug 24 12:44 /tmp/tmpB2Lg4l.rpm
-rw------- 1 root root    65628 Aug 24 12:44 /tmp/tmp_B6bo1.rpm
-rw------- 1 root root   104860 Aug 24 12:43 /tmp/tmpb8b0OY.rpm
-rw------- 1 root root   158272 Aug 24 12:44 /tmp/tmpdI5qYY.rpm
-rw------- 1 root root   107660 Aug 24 12:44 /tmp/tmpHpNsTG.rpm
-rw------- 1 root root 22168476 Aug 24 12:44 /tmp/tmpioO89J.rpm
-rw------- 1 root root    71844 Aug 24 12:44 /tmp/tmpLeJiD8.rpm
-rw------- 1 root root    84696 Aug 24 12:44 /tmp/tmplPHyVU.rpm
-rw------- 1 root root    63968 Aug 24 12:44 /tmp/tmpM0rI_l.rpm
-rw------- 1 root root    94412 Aug 24 12:44 /tmp/tmpn4w5bd.rpm
-rw------- 1 root root    52932 Aug 24 12:43 /tmp/tmpQjJZNK.rpm
-rw------- 1 root root    58504 Aug 24 12:43 /tmp/tmpqZJL9n.rpm
-rw------- 1 root root   159396 Aug 24 12:44 /tmp/tmpsD8ygd.rpm
-rw------- 1 root root    53128 Aug 24 12:44 /tmp/tmpSIwyom.rpm
-rw------- 1 root root 53136060 Aug 24 12:45 /tmp/tmpvedFOi.rpm
-rw------- 1 root root 31613004 Aug 24 12:45 /tmp/tmpxiISCM.rpm
-bash-4.3$ 

Now running the same script with yum instead of dnf:

-bash-4.3$ su -c "ls -l /var/tmp/yum-root-3eoD0D"
Password: 
total 108200
-rw-r--r-- 1 root root 31613004 Aug 16 23:09 mingw32-wine-gecko-2.40-1.fc22.noarch.rpm
-rw-r--r-- 1 root root    52932 Aug 15 01:27 wine-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root   104860 Aug 15 01:27 wine-alsa-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root   142504 Aug 15 02:00 wine-arial-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    58504 Aug 15 01:27 wine-capi-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root    73340 Aug 15 01:28 wine-cms-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root   107660 Aug 15 02:00 wine-common-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root 22168476 Aug 15 01:28 wine-core-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root   100632 Aug 15 02:00 wine-courier-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root   158272 Aug 15 02:00 wine-desktop-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    83180 Aug 15 02:00 wine-filesystem-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    65628 Aug 15 02:00 wine-fixedsys-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    52464 Aug 15 02:00 wine-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root   114440 Aug 15 01:27 wine-ldap-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root    64772 Aug 15 02:00 wine-marlett-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root 53136060 Jun 10 16:10 wine-mono-4.5.6-4.fc22.noarch.rpm
-rw-r--r-- 1 root root  1737796 Aug 15 02:00 wine-ms-sans-serif-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    65412 Aug 15 01:27 wine-openal-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root    63968 Aug 15 01:27 wine-opencl-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root    94412 Aug 15 01:27 wine-pulseaudio-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root    67896 Aug 15 02:00 wine-small-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    78796 Aug 15 02:00 wine-symbol-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    53128 Aug 15 02:00 wine-systemd-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    71844 Aug 15 02:00 wine-system-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root   159396 Aug 15 02:00 wine-tahoma-fonts-1.7.49-2.fc22.noarch.rpm
-rw-r--r-- 1 root root    84696 Aug 15 01:27 wine-twain-1.7.49-2.fc22.i686.rpm
-rw-r--r-- 1 root root    65388 Aug 15 02:00 wine-wingdings-fonts-1.7.49-2.fc22.noarch.rpm
-bash-4.3$ 

Now:

1) Dumping the downloaded files in /tmp (root of tmp) is not the right path because /tmp could be kept in ram (some ram file system).
2) I believe it should be no problem to name the files correctly even with DNF rather than naming them tmp<somestuff>.rpm in case one like to store the downloaded files to some other place for later usage.
3) Using a different path (and subdir) for storing such downloads would be much nicer. I like the way yum is doing it.

Comment 1 Honza Silhan 2015-08-31 13:44:58 UTC
use `dnf install ... --setopt=keepcache=1` and locate pkgs in "/var/cache/dnf/<repoid>-<hash>/packages/" dir. If you didn't want to install packages -> use download plugin. Do not use stuff from /tmp.

Comment 2 Ali Akcaagac 2015-08-31 15:12:21 UTC
I think you are ignoring or missing the point here. Have you looked at the attachment script ?

dnf install http://koji.someurl.org/somepath/wine-version.rpm

There is no such thing as repoid.hash and so on, because these packages are being downloaded directly into /tmp directory with these strange names and directly processed inside there.

The script run's dnf install <url>. It downloads the file and installs it - as intended. No repository being used. Yum did the same but kept the correct names.

Comment 3 Ali Akcaagac 2015-08-31 15:20:02 UTC
... and ... if you took the time reading my initial bugreport carefully (which you obviously haven), then you understand that I need to use this approach to deal with packages that haven't been pushed to any repository yet. e.g. build but not pushed... this is a different use case.

It makes things hard for us, reporting bugs and issues and then see them being closed just before we can react with a suitable response - or - if the bugreport hasn't been read (or understood) with the necessary care. The answer is out of scope with the report.

We can use wget to grab the packages and use dnf install afterwards but this IS not what I have reported here. We know howto do things differently. But this is not about doing things differently, its about an issue we discover by comparing the behavior of yum vs. dnf. Thus we react and this we report.

If no further bugs, feedback or other things is wanted then please let me know. We are going to spread the word whover is interested.

Comment 4 Michal Luscon 2015-09-01 14:05:37 UTC
Hi Ali,

Jan probably overlooked your script. Could you please explain to us why you need those tmp files to be properly named? 

And regarding the location, I would doubt that we will be able to come up with a better place for storing temporary objects than tmp itself.

Comment 5 Michael Mráka 2015-09-03 13:40:39 UTC
> And regarding the location, I would doubt that we will be able to come up
> with a better place for storing temporary objects than tmp itself.

Well, /var/cache/dnf seems to be a perfect place for such packages ;).

Comment 6 Ali Akcaagac 2015-09-04 09:44:24 UTC
(In reply to Michal Luscon from comment #4)
> Could you please explain to us why you need those tmp files to be properly
> named?
This is easy to answer. We provide packages for offline installation and need them to be named correctly.

The use case is as follows:
- The Wine packages usually show up with a 1-2 week of delay in Fedora 22 updates-(testing) (compared to Fedora 23 or Rawhide as for this example).
- We depend on Wine because we need to run some proprietary stuff on Fedora and thus need to test Wine before forwarding it to other offline installations.
- When we download the Wine packages (around 100-120mb) then we usually know that these koji packages are the ones that get pushed to updates-(testing). So the packages are usually the same as the ones that appear later in the official repositories.
- To avoid downloading these packages again, we usually move the files into --downloaddir=<somepath>. Once we run dnf update --downloaddir=<somepath>.
- We have "keep_cache=1" already enabled by default in /etc/dnf/dnf.conf.
- Therefore naming these files correctly would be a benefit. The changes inside the code should be quite trivial I assume. Rather than naming the files as tmp[0-9|a-z|A-Z]*.rpm, the file names could be retrived from command line (they are retrived anyways, otherwise a download process couldn't be initiated).

(In reply to Michael Mráka from comment #5)
> Well, /var/cache/dnf seems to be a perfect place for such packages ;).
I agree. So "dnf clean all" could be run to remove these temporarily directories as well (as example).

Even if you like to keep /tmp, then please put the files into a subdirectory inside /tmp and have the files named correctly.

e.g.

/tmp/dnf-<date><time>-tmp/wine.*.rpm

or simply

/tmp/dnf-tmp/...

Wine was just one example. There are other packages as well, which might be affected similarily. We might end up in 50-100 tmp*.rpm files in the root of tmp in just a couple of minutest. Not knowing which file belongs to what and how. This also makes readability (in case others store other information in /tmp) quite impossible because they have to dive through all the tmp*.rpm files first.

Comment 7 Radek Holy 2015-09-04 10:02:25 UTC
Right, I believe that the problem here is not where does DNF store its temporary files but that keepcache does not work with packages specified as URLs/paths... We should keep them in the cache as well... At least that's my opinion.

Comment 8 Radek Holy 2015-09-04 10:05:34 UTC
Oh, it works with keepcache but weirdly. It keeps the files in /tmp. Now, it makes sense to me.

Right, I don't think that this is intentional.

Comment 9 Radek Holy 2015-09-07 11:13:21 UTC
OK, so we should create a directory for these packages in the cache directory and place the files there. We can name the directory "@commandline". This is the name which appears in the transaction summary if user installs packages specified by URL. The directory should probably has the same structure as the other directories.

Comment 10 Ali Akcaagac 2015-09-14 11:28:28 UTC
Sounds ok to me. But please save the files in real-name rather than tmp[0-9]*.rpm

Comment 11 Fedora Admin XMLRPC Client 2016-07-08 09:27:45 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 12 Fedora End Of Life 2016-07-19 17:38:36 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 13 Ali Akcaagac 2016-07-19 17:48:22 UTC
Reopened because missing information about actual state.

Comment 14 Jaroslav Mracek 2017-04-03 11:03:43 UTC
I try to implement it in pull request: https://github.com/rpm-software-management/dnf/pull/778 . Hope that it helps

Comment 15 Fedora Update System 2017-05-02 15:40:40 UTC
dnf-plugins-core-2.0.0-1.fc26 libdnf-0.8.2-1.fc26 dnf-2.4.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4e95959f0d

Comment 16 Fedora Update System 2017-05-04 09:48:40 UTC
dnf-2.4.0-1.fc26 dnf-plugins-core-2.0.0-1.fc26 dnf-plugins-extras-2.0.0-1.fc26 libdnf-0.8.2-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-4e95959f0d

Comment 17 Fedora Update System 2017-05-04 22:04:59 UTC
dnf-2.4.0-1.fc26, dnf-plugins-core-2.0.0-1.fc26, dnf-plugins-extras-2.0.0-1.fc26, libdnf-0.8.2-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-4e95959f0d

Comment 18 Fedora Update System 2017-05-05 13:34:35 UTC
dnf-2.4.0-1.fc26, dnf-plugins-core-2.0.0-1.fc26, dnf-plugins-extras-2.0.0-1.fc26, libdnf-0.8.2-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.