Bug 1209638

*** This bug has been marked as a duplicate of bug 1203491 ***

> I am now scared because I detected that dnf does not provide a --downloadonly feature.
> The plugins only offer a "download" (into current directory) feature.

That's not quite true, download plugin provides

dnf download --destdir=/somepath

(In reply to Michael Mráka from comment #2)
> That's not quite true, download plugin provides
> 
> dnf download --destdir=/somepath

Thank you for clarification. I was using dnf which comes shipped with Fedora 20. I am not yet using Fedora 22 (... and don't use Fedora 21 either). dnf on Fedora 20 only said something about downloading into current directory.

But anyways, this is not the use case.

The use case as mentioned above is as follows:

yum --downloaddir=/somepath \
    --downloadonly          \
    --assumeyes             \
    group install           \
    gnome-desktop           \

What it does is quite obvious:

1) Instruct yum to group install "gnome-desktop"
2) Instructing it to answer everything with yes by default

(Important)
3) Resolve all dependencies (the important part)

4) Put all *.rpm packages into downloaddir
5) Download the packages only but do not install them

Basicly it's like mirroring all required packages for the "gnome-desktop" (not entire fedora) into a specific folder for later usage (like offline usage). Same for xfce-desktop and or mate-desktop and or kde-desktop etc.

The thing here is, that I rely on the exact command set. Or with other words, I don't want or intend to change the entire infrastructure (there is more behind that) that I have written.

The requirements to change everything (which comes equal to replace nearly everything written in the past few years) probably justifies rethinking, whether I want to be confronted with future Fedora changes - or - whether it's not better to change the platform entirely (... which I like to avoid. I am quite happy with Fedora to say the truth).

dnf download --destdir=/somepath

Comes equal to yumdownloader. (No resolving ? I just guess this because I haven't tried)

I find it a bit unfair (reading the fedora developer list) to expect "use cases" from users, so things can be implemented, while these "use cases" were already part of "yum", which does not require any explaination.

For dnf: I can say that dnf is definately a good approach. No doubts. But I never expected running into such issues. I am sitting behind my system for the whole day feeling bad and unproductive. Not knowing how to solve *our* requirements with the current provided features by dnf. Knowing the yum is deprecated.

Please keep in mind that other people have work to do as well. Sometimes it relies on functionality that has been there for years. Ripped out from one day to another.

(In reply to Ali Akcaagac from comment #3)
> I find it a bit unfair (reading the fedora developer list) to expect "use
> cases" from users, so things can be implemented, while these "use cases"
> were already part of "yum", which does not require any explaination.

The thing is that YUM also implements some features that are not covered by a real use case. The best approach to avoid these features is to implement features once it turns out that they are needed. If we would implement them first and remove them once it turns out that they are not needed, we would annoy even more users. Also just copy&pasting all the features from YUM into DNF would mean to copy YUM as the whole because sometimes it isn't even clear that a behaviour of YUM is a feature (or that a user considers it a feature) and thus the safest way would be to clone YUM. So, we have rather implemented only those features that have known use case to us and for the rest, we are waiting until the use case becomes known.

> Not knowing how to solve *our*
> requirements with the current provided features by dnf.

Actually, we would be very happy to take this responsibility. We would be very happy if we could know *all* "*your* requirements". This is the most valuable input for us. Based on this knowledge, we can advise you which features you should use then. Also if you share them with us, you can be sure that the use case won't become unsupported without a prior notification/discussion and that it will be covered by our test suite.

Thanks for clarification. Well above is one of our (my) requirements and I would be thankful, if this could be realised by dnf as well.

Atleast there is a step between 4) and 5) missing. Of course packages that has been downloaded into --downloaddir before *and* have the same version as those that get's resolved during the process of resolving are not being downloaded again (makes no sense). So only the differences are downloaded.

Well try above command on your own with yum (I bet there are test environments in your labs) for a few days and you see how yum does handle this. The first time it downloads everything, the upcoming time it only downloads the new packages.

The infrastructure - the part not mentioned here starts another yum download process afterwards with yum -d 3 (and above commands). The log it creates is being stored onto disk

yum -d 3 ... above commands ... &> log.txt

the log is being grepped and all information about "using local copy" is gathered into one list. The new files are moved from --downloaddir=foo to bar.new... this leaves the remaining *old* files in "foo". "foo" is being deleted and "bar.new" renamed to "foo" ... until the same process starts over the next day.

With other words...

1) We like to download everything that belongs to the group "gnome-desktop", skip versions of packages already downloaded in the "downloaddir" folder (we don't need to download them again), and only download the new packages.

2) We re-run the above process again with -d 3 (to get a verbose output) where we grep all "using local copy" informations which we turn into some

mv "currentdir/currentfile.rpm" "newdir/newfile.rpm"
rm -rf "currentdir" (this is run manually, once we look into currentdir)

We of course do this with a list of stuff like. This process repeats every day at 12:00 AM.

yum group install "gnome-desktop" "kde-desktop" "xfce-desktop" "libreoffice" --downloaddir=/somedir --downloadonly --assumeyes --nogpgkeys

So it should be able to process a bunch of input at a time. Same we do repeat for individual packages.

An alternative approach could be:

1) dnf resolve "gnome-desktop"   (or dnf resolve group "gnome-desktop)
2) this spits out a list looking something like this:

http://mirror.fedorapackages.org/../22/i386/../package.rpm
http://mirror.fedorapackages.org/../22/i386/../foo.rpm
...
http://mirror.google.org/../22/i386/../bar.rpm
http://mirror.fedorapackages.org/../22/i386/../lorem.rpm
http://mirror.rpmfusion.org/../22/i386/../ipsun.rpm

...

This could then be handed over to "wget" or "curl" or whatever which then downloads all packages to the directory we like. It would be important to have dnf resolve from the mirrors it had grabbed the metadata from. Otherwise it makes no sense to process metadata from mirror1 and download files from a fixed URL (which might not yet received the updated files).

Yum came quite handy with this because it was just one invocation. No need for external tools. Process, Resolve, Download, done...

The "resolve" would be a new command for dnf (could be a plugin) that only spits out the list of the dependency resolving it has done internally. resolve should be applicable for singular packages as well as meta (group) packages of course. variable arg e.g. able to accept a bunch of packages only or groups only one after another.

The more I think about the 2nd approach, the more I like it. At least it gives a fixed and structured output. Which can be processed with other tools in a unified way. e.g. turned into csv, xml, ... This probably opens more and better possibilities on the longer run. Grepping -d 3 for "using local copy" was never a guarantee to be there forever (I mean the string in the debug log). Having a structured output makes a lot more sense. Using "basename" or "dirname" or even "sed 's/.*\///'" can be used to trim out the mirror and filename (on each own). This would be a guaranteed unified and preceise better solution on the long go... At least how I think about this right now... haven't thought this through. But so far sounds good.

Short question, should I open a new bug-report about this ?

What initially was about missing --downloadonly --downloaddir=<path> has turned into a new idea that could be tracked into a separate bug/feature. I believe that even if downloaddir and downloadonly is being applied, that it will probably do only half of that what we desire. yum -d 3 gave a real verbose debug output. Doing the same with dnf -d 3 gives less debug output (compared to dnf) from what I have figured out. It's most likely - but not impossible - that the string "using local copy of" might be missing anyways. It would be a good idea for a request of structured output as described in comment #6.

I could think of various possibilities:

1) dnf --qf "%{URL}/%{PACKAGE}\n" --assumeno install gnome-desktop &> log.txt
definable output

2) dnf resolve gnome-desktop &> log.txt
optional predefined output

As a possible option. This way one can "construct" it's own output format. Another one maybe as plugin or the command above.

This would open a bunch of possibilities (and even reduces one step) for our use case. Of course this is a different request than the download* stuff, that people requests. But without the debug output of those lines, it might become uelesss anyways (for us) at least.

One problem may be that you'll lose the DNF's ability to choose different mirrors if they become unavailable e.g. in the middle of the download process.

Maybe it could print just package NEVRAs (e.g. foo-2-1.fc22). Then there could be another command that translates a NEVRA into an URL.

Well, I think it's a good idea. On the other hand, since it seems that we are planning to implement the "dnf download @group" feature, I don't see a reason why would you need to duplicate the functionality with e.g. wget. If the only problem is with the "using local copy" message, we can improve the plugin to be more script-friendly.

So, I'd suggest you several options:
A) File an RFE for the feature you've described above OR
B) file an RFE to make the "download" plugin more script-friendly (wrt to the messages about already downloaded packages) OR
C) file an RFE for a plugin that does all you want (synchronizes all the packages needed to install a group off-line in a directory).

Also you can consider using DNF's API, or even more implement it as a plugin (and ideally share it with the rest of the Fedora users) :-)

The least difficult feature for us would be (B), obviously :-)



And just FYI. There is a project, that will be announced soon, that is able to cache all the packages (and metadata) downloaded by YUM/DNF and I really believe that it might replace a part of your infrastructure (but sure, I don't know your infrastructure). But it's just for your interest, I'm not forcing you to migrate. I just thought that it might be interesting for you. You can still file the RFEs mentioned above.

Thanks for the throughout feedback, which is well appreciated. I am currently a bit short of time and therefore will reply to this in a couple of days in a bit more detail. Only as a note:

yum install firefox --downloaddir=<path> --downloadonly --assumeyes

works differently than

yumdownloader firefox

I do assume that

dnf download --destdir=<path>

will work as yumdownloader

Same I do assume now for:

dnf download @group

There is a difference between "downloading" the packages that belong to a group

and

"resolve all packages and see if there are dependency issues" and "download" then (e.g. a package that requires some newer package of something else).

Downloading could be achived by greping through the comps.xml metadata and ripping out all package names otherwise. The desired feature here is "dependency" resolving to make sure that all packages are intact and depend on each other. No version conflicts etc.

The rest I will reply, once I find time and have thought about a proper answer. so far your ideas sounds reasonable and interesting.

Ah, I apologize, I keep forgetting to let you know about a "--resolve" switch of the "download" plugin that should do what you need.

doc for download plugin is here

http://dnf-plugins-core.readthedocs.org/en/latest/download.html

or

man dnf.plugin.download 

or

dnf download --help-cmd

and it is right that dnf download is a replacement for yumdownloader

yum --downloadonly is not very good for downloading packages, it depends on what you have installed on your system, so yum install firefox --downloadonly will not download anything if firefox and deps is installed on your system.

(In reply to Radek Holy from comment #11)
> Ah, I apologize, I keep forgetting to let you know about a "--resolve"
> switch of the "download" plugin that should do what you need.
Thanks.

(In reply to Tim Lauridsen from comment #12)
> yum --downloadonly is not very good for downloading packages, it depends on
> what you have installed on your system, so yum install firefox
> --downloadonly will not download anything if firefox and deps is installed
> on your system.

Right but not necessarily :)

rm -rf /opt/cache             \
       /opt/fedora            \

yum --downloadonly            \
    --downloaddir=/opt/cache  \
    --installroot=/opt/fedora \
    --assumeyes               \
    install                   \
    firefox

The magic here is the installroot. Which is an empty directory :)

So once I run this stuff above, yum downloads all metadata into /opt/fedora/var/.../i386/fedora (and all other directories). It resolves the dependency to firefox (which might end up in 1200 dependencies) and downloads all 1200 files into --downloaddir.

Now assume one day later. You run these lines again:

Cache files are either updated (in case you don't rm -rf anything) or downloaded again. firefox is being resolved again and ends up with 1200 dependencies. This time it doesn't download anything in case *all* versions, release, (epoch) and other criteria matches.

Process ends.

In case it finds 1-2 updates in the list of 1200 dependencies, then it downloads the changed files (rpms: required for firefox). It can even skip firefox but still grabs the changed files the entire rat's tail down. If firefox changes it of course downloads that one as well.

Now imagine this:

yum --downloadonly            \
    --downloaddir=/opt/cache  \
    --installroot=/opt/fedora \
    --assumeyes               \
    group install             \
    gnome-desktop             \
    kde-desktop               \
    xfce-desktop              \
    mate-desktop              \
    virtualization            \
    libreoffice

What this does is creating a new metadatadir in /opt/fedora, downloads all metadata, resolves all groups... ends up in say 2100 files and downloads them into /opt/cache.

End

Next day you run this once again. Say we keep /opt/cache and /opt/fedora. What it does is: resolve all depdenencies for all groups... ends up in say 2100 files again. But skips downloading 2090 of them, because only 10 files had changes (e.g. newer version). It ends up in only downloading the 10 updates into /opt/cache.

Sometimes a dependency doesn't match because from one day to another the mirror didn't catch everything or something got unpushed.

Above command aborts with:

yum ... either run rpm -vsomething or run yum --skip-broken
error ... end

Then the script ends, nothing had been touched until the next day the command runs again.

That way you can ensure that your local mirror of above "meta-packages" or "groups" are *always* installable from the downloaddir (all version matches, no dependency issues, all files locally available).

Of course you are right if you run --downloadonly skips everything in case it has been installed. But together with the installroot it's of course possible. You can be sure that we keep the running system hermetic isolated from grabbing the whole stuff :)

I tested with

sudo dnf download firefox --destdir=/opt/cache --installroot=/opt/iroot 
--releasever=21 --resolve

It download firefox and 156 deps.

So i see 2 problem with dnf download with your usecase

1. It download every package each time
2. dnf download don't support groups yes

So fixing there issues should solve your use-case

(In reply to Tim Lauridsen from comment #14)
> So i see 2 problem with dnf download with your usecase
> 
> 1. It download every package each time
> 2. dnf download don't support groups yes

Ahh ok if I understood you right then 1 and 2 are still left because dnf always downloads (not skipping already downloaded packages). Ok this can be fixed I believe.

> So fixing there issues should solve your use-case

Not necessarily :)

Please have a look at comment #5 again.

When using "yum -d 3" ... and above commands then it gives me quite some deep debug output which the "use case" relies on. The use case don't just download packages. It also moves all "new" packages from A to B by grepping:

"use local copy of NEVRA"

dnf doesn't have such in deep debug output. Therefore it's not possible to get a list of all packages. That's why I suggested a better solution. Rather than using yum -d 3 (to grep for something that might be missing in dnf), it would be better to have some verbose output like a package list (names should be enough).

Otherwise the use case might end up in

cd Download/
ls -1
firefox-31.12.2-i686.fc20.rpm
firefox-31.12.3-i686.fc20.rpm
firefox-31.12.4-i686.fc20.rpm
...
kernel-4.0.0.1-i686.fc20.rpm
kernel-4.0.0.2-i686.fc20.rpm
kernel-4.0.1.1-i686.fc20.rpm
...

with yum -d and grepping for using local copy of, we get a list of all current packages (as current as it can get (sometimes fails)). And use that list to move new packages from A to B leaving old packages in A (which can be rm -rf'ed).

But thanks anyways. I see and learn, that you take care of the use cases. We will be getting there.

(In reply to Tim Lauridsen from comment #14)
> I tested with
> 
> sudo dnf download firefox --destdir=/opt/cache --installroot=/opt/iroot 
> --releasever=21 --resolve
> 
> It download firefox and 156 deps.
> 
> So i see 2 problem with dnf download with your usecase
> 
> 1. It download every package each time
> 2. dnf download don't support groups yes
> 
> So fixing there issues should solve your use-case

I'd like to come back to this. We've been able to move a large part of the infrastructure from yum to dnf. So far things seem to be working. Now we are dealing with the last bit and have been reaching the possibilities of dnf so far:

Regarding point 2). We would really like to see download support for groups because that is what we've been doing (or requiring to do).

With yum we did the stuff as mentioned in comment #13 by executing yum install group somegroupname --downloadonly --installroot.

We've been experimenting with dnf download --resolv so far and it does the thing very good with the packages we handed it over for downloading. Unfortunately we require this to work with groups as well.

What would cover our needs too would be the possibility to capture the downloaded filenames into a log file for further processing or an option that captures this. Sadly piping dnf &> log.txt stays empty.

That would basily be all we need to have our entire framework ported.

dnf download (+resolve) for groups and a possibility to capture the filenames into a text file.

Thanks.