What about to enhance `yum download` command?

With a new behaviour the command will create/overwrite (by default) modules.yaml file with all modules that downloaded rpms belongs. The tricky part will be to also include modular dependencies.

Then work-flow would be:
yum download nodejs
createrepo_c .
modifyrepo modules.yaml repodata/

Then the repository will be fully functional without requirement of additional metadata

What do you think?

I thought about a "yum download" level functionality set as well. The main reason why I avoided it was that would logically be tied to the active module stream of the moment. So, if you were trying to pull multiple streams locally you would have to:

    # yum download nodejs
    # yum module disable nodejs
    # yum module enable nodejs:<stream>
    # yum download nodejs

Whereas, this "modulesync" tooling would allow you to download all module revisions in all streams, and individual stream, or (in the case of a --newest flag) the latest version of either.


That being said, I would be happy to open another RFE for "yum download" functionality described. I even would like an additional RFE for createrepo_c which detects the presence of module metadata alongside RPM artifacts and automatically includes it in the applicable repo metadata...

Thanks Kyle for additional information. Then we have to deliver both options. A new tool and enhanced functionality of the download command.

With a new tool we have still a lot of questions about expected behaviour.

Let start with expectation that user wants to download all latests packages of `nodejs` for all streams. It means the latest nodejs for stream 10, 11, and nonmodular package. In case of `perl-DBI` it will download all `perl-DBI` packages from all latest contexts. It will also store related module documents. So far it is not difficult. But in case that also --resolve capability will be requested then we get a multidimensional task.

Kyle, please could you review my expectations and suggestions?

(In reply to Jaroslav Mracek from comment #6)
> Thanks Kyle for additional information. Then we have to deliver both
> options. A new tool and enhanced functionality of the download command.
> 
> With a new tool we have still a lot of questions about expected behaviour.
> 
> Let start with expectation that user wants to download all latests packages
> of `nodejs` for all streams. It means the latest nodejs for stream 10, 11,
> and nonmodular package. In case of `perl-DBI` it will download all
> `perl-DBI` packages from all latest contexts. It will also store related
> module documents. So far it is not difficult. But in case that also
> --resolve capability will be requested then we get a multidimensional task.
> 
> Kyle, please could you review my expectations and suggestions?

I see the difficulty...

For completeness, I was originally envisioning end users who build repositories doing so with an eye to the following:

       +-------------+
   +---+ Repository  |
   |   +------+------+
   |          v
   |     +----+----+
   |  +--+ Module  |
   |  |  +----+----+
   |  |       v
   |  v    +--+--+
   +--+--->+ RPM |
           +-----+

I still believe that this should be the case, though the problem is, things go in multiple directions at once. If I look at just the nodejs example you raised, I have the following (for just the latest module version):

# MODULEMD='/var/cache/dnf/rhel-8-for-x86_64-appstream-rpms-*/repodata/*-modules.yaml.gz';
# ~/.local/bin/yq '.data | select(.name == "nodejs") | select(.stream == 10) | select(.version == 8020020200617141000) | .name, .version, .stream, .artifacts' <(zcat $MODULEMD)
"nodejs"
8020020200617141000
10
{
  "rpms": [
    "nodejs-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.src",
    "nodejs-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64",
    "nodejs-debuginfo-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64",
    "nodejs-debugsource-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64",
    "nodejs-devel-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64",
    "nodejs-docs-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.noarch",
    "nodejs-full-i18n-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64",
    "nodejs-nodemon-0:1.18.3-1.module+el8+2632+6c5111ed.noarch",
    "nodejs-nodemon-0:1.18.3-1.module+el8+2632+6c5111ed.src",
    "nodejs-packaging-0:17-3.module+el8+2873+aa7dfd9a.noarch",
    "nodejs-packaging-0:17-3.module+el8+2873+aa7dfd9a.src",
    "npm-1:6.14.4-1.10.21.0.3.module+el8.2.0+7071+d2377ea3.x86_64"
  ]
}

If I try and retrieve all of these:

# yum --skip-broken download $(~/.local/bin/yq -r '.data | select(.name == "nodejs") | select(.stream == 10) | select(.version == 8020020200617141000) | .artifacts.rpms[]' <(zcat $MODULEMD))
Updating Subscription Management repositories.
Last metadata expiration check: 0:04:46 ago on Tue 09 Feb 2021 04:32:31 PM EST.
No package nodejs-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.src available.
No package nodejs-debuginfo-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64 available.
No package nodejs-debugsource-1:10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64 available.
No package nodejs-nodemon-0:1.18.3-1.module+el8+2632+6c5111ed.src available.
No package nodejs-packaging-0:17-3.module+el8+2873+aa7dfd9a.src available.
(1/7): nodejs-devel-10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64.rpm                        346 kB/s | 163 kB     00:00    
(2/7): nodejs-10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64.rpm                              1.9 MB/s | 8.9 MB     00:04    
(3/7): nodejs-full-i18n-10.21.0-3.module+el8.2.0+7071+d2377ea3.x86_64.rpm                    1.6 MB/s | 7.3 MB     00:04    
(4/7): nodejs-packaging-17-3.module+el8+2873+aa7dfd9a.noarch.rpm                             135 kB/s |  20 kB     00:00    
(5/7): nodejs-docs-10.21.0-3.module+el8.2.0+7071+d2377ea3.noarch.rpm                         563 kB/s | 3.5 MB     00:06    
(6/7): npm-6.14.4-1.10.21.0.3.module+el8.2.0+7071+d2377ea3.x86_64.rpm                        2.8 MB/s | 3.8 MB     00:01    
(7/7): nodejs-nodemon-1.18.3-1.module+el8+2632+6c5111ed.noarch.rpm                           178 kB/s | 965 kB     00:05    


The missing packages are missing because the module artifacts are split amongst multiple repositories. Though, I would argue that we can apply the same limitations to this offering as we do at the RPM level since the dependency breakage associated with content being spread across repositories is present there as well. In the above, the nodejs-debug{info,debugsource} come from the same rpmbuild operation.

So, the suggested arguments become:

    --alldeps             - Download the dependencies of the module artifacts if available in the available repositories


That being said, there are much more complex abstractions out there. Like RPMs that depend on modular RPMs (via default streams). However, for the purposes of this particular offering, I would say that we should try and keep things as simple as possible and intuitive by mimicking the existing DNF side at the underlying RPM granularity layer.

What do you think Jaroslav?

May I ask for clarification what will be expected outcome from the tool? I suggest a customer wants to create a subset from repository content (group, module, rpm) and optionally including dependencies. Then it is required to download content, create repository with additional metadata (module.yamls) to ensure that repository will be fully functional. Is it correct? Or metadata are not required to be create by the tool, and only creation of related yaml file is required? Or only provide a way how to disable failsafe mechanism that prevents installing modular rpms without modular yamls? Or which approach would you prefer?


In case that a customer wants to download nodejs module, and we will create metadata with modules, shall we propagate also defaults to newly created data?

I investigate the issue with downloaded packages that cannot be installed from created repository and I discovered that there are two cases when fail-safe mechanism is ignored (work-around).

1. Commandline repository
   When I download any package and then I install it not from repository but from commandline (`dnf install /nodejs-doc.rpm`) the fail-safe mechanism is not appleed and operation is allowed

2. Hot-fix repositories
   Hot fix repositories are not as a target for modular filtering and fail-safe mechanism
   When I create my custom repository after `dnf download nodejs --alldeps` I can add to repofile (.repo) `module_hotfixes=1`.
   From commandline using repofrom path - `dnf --repofrompath=<repo_id>,<path> install nodejs-docs  --setopt=<repo_id>.module_hotfixes=1`

I am thinking how to `--resolve` option in case of muticontext stream (perl-DBI) or modules that dependent on multicontext modules.


    When given a module:
      # modulesync perl-DBI --resolve
        > All artifacts from all streams are downloaded for the module
        > All artifacts from dependent modules are downloaded
        > All module YAML for the acquired artifacts are downloaded


Basically I have some basic approaches how to resolve it.
1. Creates all relevant combination to requested module/package and search inside each combination for provides of dependencies
It is complicated but it has a benefit in possibility to add only latest required packages to the repository.

2. Search for all provides of dependencies and add them all to repository. Easy to implement, but the repository will be bigger but prepared for everything.


To know which approach is better depends on requirements:
Is it expected to provide all possible dependencies (versions, related to all context), or only one related to current system state (enabled: perl:5.26, then only context perl-DBI that requires perl:5.26 is handled, downloaded).

I also suggest to provide all modular metadata from all available repositories into the newly created repository. Or do you have a preference to somehow select relevant modules (default, enabled and selected by user)?

Thank you very much for a feedback. Modularity is very complicated therefore to clarify expectation is twice more important.

I create a draft (https://github.com/rpm-software-management/dnf-plugins-core/pull/444) of a new modulesync tool (python3-dnf-plugin-modulesync). For a testing purpose I create a repository https://copr.fedorainfracloud.org/coprs/jmracek/modulesync.

What it does?
1. It downloads rpm modules according to provided spec (according to artifacts and profiles) and creates repository in working directory with all modular metadata
`# dnf modulesync nodejs`
2. It downloads rpm modules according to provided spec (according to artifacts and profiles) and creates repository in DESTDIR with all modular metadata
`# dnf modulesync nodejs --downloadonly --destdir /tmp/modulerepo` (note: `--downloadonly` option is there only temporary, it will be not required in future)
3. It can also enable source or debug repositories and include them in download set (--enable_source_repos, --enable_debug_repos)
4. It can be used with other DNF commands originally used to download and redistribute packages
`# dnf download nodejs --destdir <path>`
`# dnf modulesync --downloadonly --destdir <path>`

`# dnf upgrade --security --downloadonly --destdir <path>`
`# dnf modulesync --downloadonly --destdir <path>`

At this point I really need a feedback about already implemented functionality or missing functionality.

This is fantastic! I think this hits _very_ close to the mark. The only functionality that I see as missing in this revision is the ability to limit the results down to just the latest version of the module (with and without the --resolve flag). Though, you should be able to do just that by selecting the newest Version in the module_spec.

There is an interesting behaviour that I noticed while using the tool this morning. It seems that the following causes the version to be lost:

    https://github.com/rpm-software-management/dnf-plugins-core/pull/444/files#diff-c15be26b960627b42b327e48c1500526578980f7b405e39093edb61193ec3c33R138

That results in the following when issuing "dnf modulesync llvm-toolset:rhel8:8040020210219122224":

    # ls -l llvm-toolset*x86_64.rpm
    -rw-r--r--. 1 root root 10008 Oct 25 12:59 llvm-toolset-10.0.1-1.module+el8.3.0+7459+90c24896.x86_64.rpm
    -rw-r--r--. 1 root root 10184 Oct 25 12:59 llvm-toolset-11.0.0-1.module+el8.4.0+8724+f6b3479d.x86_64.rpm
    -rw-r--r--. 1 root root  9580 Oct 25 12:59 llvm-toolset-7.0.1-1.module+el8+2560+c32c7af1.x86_64.rpm
    -rw-r--r--. 1 root root  9532 Oct 25 12:59 llvm-toolset-7.0.1-1.module+el8.0.0+3181+ec021930.x86_64.rpm
    -rw-r--r--. 1 root root  9692 Oct 25 12:59 llvm-toolset-8.0.1-1.module+el8.1.0+3866+6be7f4d8.x86_64.rpm
    -rw-r--r--. 1 root root  9892 Oct 25 12:59 llvm-toolset-9.0.1-1.module+el8.2.0+5385+5caa90f8.x86_64.rpm

What about this instead?

            if profiles_set:
                for profile in profiles_set:
                    for pkg_name in profile.getContent():
                        query = self.base.sack.query(flags=hawkey.IGNORE_EXCLUDES).filterm(name=pkg_name)
                        if result_query.intersection(query):
                            continue
                        elif query:
                            result_query = result_query.union(query)
                        else:
                            print("No match for profile content", pkg_name)

That checks to see if the profile entry is already present in the resulting query, and if so it doesn't pull it in.

I have a good news and a bad one. The good news is that I finished development of the feature and now we can test it. Also I updated https://copr.fedorainfracloud.org/coprs/jmracek/modulesync repository with new version. The bad news is that I have only builds for Fedora 34,35, and rawhide but not for RHEL8 because I experience an issue with build of libsolv. If it will be important for testing, I will resolve the issue with rhel8 test build. Is it ok for testing only to have only Fedora build?


The feature requires: https://github.com/rpm-software-management/dnf-plugins-core/pull/444, https://github.com/rpm-software-management/dnf/pull/1794.
Additionally for RHEL 8 it requires modified commit - https://github.com/rpm-software-management/libdnf/commit/73868368120ceea97ada628a1aa42236fb42b89d. The commit requires a new modular solver which we are not going to backport to RHEL8.

Please can you try once again?

Hi Jaroslav, apologies for the delay.
I am not sure if testing with Fedora would be proper, however I will give a try on my own.

I want to discuss this further with Renaud, he is currently on multiple assignments so we will be working together next week on this feature.

Please keep the 'needinfo' flag on me.


Thank you,
Aviad N.

For RHEL8 it requires: https://github.com/rpm-software-management/libdnf/pull/1381

Finally I have builds for RHEL8 in https://copr.fedorainfracloud.org/coprs/jmracek/modulesync. I am sorry for the delay but now it also contain modified patch for libdnf.

*** Bug 2025625 has been marked as a duplicate of this bug. ***

The feature is verified for 8.6 therefore I think it will be in 8.6. We have documentation in dnf man pages that describes the feature - https://dnf-plugins-core.readthedocs.io/en/latest/modulesync.html and the feature will be also documented as an enhancement from the documentation team.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (dnf-plugins-core bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2052