Bug 1801747

Given the initial description is private and I'm the reporter behind this feature request, I would like to clarify our expectation (just to be on the safe side):

As of writing we can use "dnf download --releasever=8 --resolve --alldeps foo" to download the RPM package foo including all dependent RPM packages. Once we run "createrepo_c" on this downloaded result, we're able to use this as repository for dnf/yum. If "foo" however is e.g. "python36" (or another package belonging to Modularity/AppStream), this does no longer work, because dnf yells: "Error: No available modular metadata for modular package". Thus I tried to download the YAML metadata manually and tried to inject it using "modifyrepo_c --mdtype=modules ...", however this also didn't lead to any success. Specifically we used these steps (where GSS did not find any mistake except for letting us know that it is technically not supported yet):

0. `dnf remove python36 -y; dnf install createrepo_c -y`

1. Mirror a part of an existing random AppStream
```
dnf download \
  --releasever=8 \
  --resolve \
  --alldeps \
  --disablerepo="*" \
  --repofrompath=c8b,http://mirror.centos.org/centos-8/8/BaseOS/x86_64/os/ \
  --repofrompath=c8a,http://mirror.centos.org/centos-8/8/AppStream/x86_64/os/ \
  --destdir="/tmp/c8t" \
  --arch="x86_64,noarch" \
  python36
```

2. `createrepo_c /tmp/c8t`

3. `curl https://git.centos.org/modules/python36/raw/c8-stream-3.6/f/python36.yaml > /tmp/c8-python36.yaml`

4. `modifyrepo_c --mdtype=modules /tmp/c8-python36.yaml /tmp/c8t/repodata/`

5. `dnf --disablerepo="*" --repofrompath=c8l,file:///tmp/c8t/ install python36 -y --nogpgcheck`

-> No available modular metadata for modular package 'python36-3.6.8-2.module_el8.1.0+245+c39af44f.x86_64', it cannot be installed on the system
-> Error: No available modular metadata for modular package

Note that it's NOT about using python36 from CentOS here, it is just an example that we would like to mirror a part (!) of an existing random AppStream in our own local dnf/yum repository.

Thus our expectation is that e.g. "dnf download --resolve --alldeps python36" downloads not only all RPM packages, but also the required meta data and whatever stuff that is required for Modularity/AppStream. In the end, I would like to be able with "createrepo_c" and "modifyrepo_c" to create a repository where dnf is not yelling "Error: No available modular metadata for modular package". If this requires downloading the whole Modularity/AppStream (maybe including its dependencies again recursively), we expect that dnf is doing that for us either automagically when using "--resolve" or by introducing a new parameter.

The new tool specs:

usage: modulemd-build N:S:V:C:A --summary= [--description[ [--license=] [--requires=] <DIR|--pkglist=FILENAME>

-m, --summary
    mandatory
    using -m, because -s could indicate stream; -m is also frequently used to specify a message, comment
    -> summary

-d, --description
    optional
    -> description
    
-l, --license
    optional, default to MIT
    -> license/module

-r, --requires
    optional
    append option, can be specified multiple times
    -> dependencies/requires

DIR
    directory to recursively scan for RPMs to be included in modulemd

-i, --pkglist=FILENAME
    optional
    conflicts with DIR argument

Recursively read RPMs from DIR or read them from specified pkglist.
If any RPM is missing on unreadable, error out.
Populate artifacts/rpms with RPM NEVRAs (sorted, deduplicated)
Populate license/content with list of RPM licenses (sorted, deduplicated)

Write N:S:V:C:A.modulemd.yaml in the current directory.
Make sure the yaml is in modulemd v2 format.

One more thing.
All the RPMs *MUST* have the MODULARITYLABEL set in the header.
If they don't, the tool must error out.

There should be an option (e.g. --force) to override this check.

Hello Dan,
I have a working prototype here https://github.com/FrostyX/dir2module

Can you do some testing to see what needs to be improved?

I also invited you as a collaborator, so if you find something that
is easier/faster for you to just fix rather than explain it, feel
free to push into that repo.

Seeing my previous comment, I think I should write an update.

The `dir2module` script is no longer a prototype. It is ready to
be used. It was also moved into a more official namespace. We
started a project called modulemd-tools that is going to provide
small scripts dealing with specific modulemd tasks and use-cases.

https://github.com/rpm-software-management/modulemd-tools

It is now packaged into Fedora (see RHBZ 1842042).

Is "dir2module" meanwhile somehow able to satisfy our use case mentioned in comment #1?

Hello Eva,

> dir2module seems to be available in the latest rhel-8.5 nightly
> compose. Can you please update the status of this bug?

Closed in CURRENTRELEASE then?


> Also, can you please answer the question from comment 31? Thanks

I am sorry, I didn't realize the question was meant to me.

> Is "dir2module" meanwhile somehow able to satisfy our use case
> mentioned in comment #1? 

It should but first I would like to suggest an easier solution which
is module hotfixes. Try doing steps 0), 1), and 2). Then omitting the
module metadata steps 3) and 4) and using the following command to
install the packages:

```
dnf --disablerepo="*" --repofrompath=c8l,file:///tmp/c8t/ install \
  python36 -y --nogpgcheck --setopt=c8l.module_hotfixes=true
```

Alternatively, if using module repository is the goal, not an
implementation detail, I would suggest a couple of changes in the
described process.

Do steps 0) and 1) as described.

Unfortunately, the modulemd YAML from step 3) is not very useful because
it is a version of modulemd that hasn't been processed by a build-system 
yet and therefore doesn't contain an `artifacts' section (which is
necessary for DNF to be able to install packages from such module)

Therefore we need to use either `dir2module' or `repo2module' to
generate a modulemd YAML that contains such a section. 

For `dir2module' please see
https://github.com/rpm-software-management/modulemd-tools#creating-a-module-repository-from-a-set-of-rpm-packages

and for `repo2module', please see
https://github.com/rpm-software-management/modulemd-tools#creating-a-module-repository-from-a-regular-repository

Once the modulemd YAML is generated, we can generate repodata. We
don't have to do it in two steps like described in 2) and 4). 
The `createrepo_c' command is now able to generate modulemd
repodata by itself. See `man createrepo_c`

```
If compiled with libmodulemd support modular metadata inside
<directory>   identified   by   the   patterns   below   are
automatically collected, merged and added to the repodata.

The patterns are:

* *.modulemd.yaml       (recommended        file       name:
    N:S:V:C:A.modulemd.yaml)

* *.modulemd-defaults.yaml    (recommended     file    name:
    N.modulemd-defaults.yaml)

* modules.yaml  (recommended   way  of   importing  multiple
    documents at once)
```

At this point, I would expect

```
dnf --disablerepo="*" --repofrompath=c8l,file:///tmp/c8t/ install python36 -y --nogpgcheck
```

comamnd to work. But for some reason, it still doesn't. I tried
everything I could think of but no success. Does anyone have some
explanation what might still be wrong?

PS: Using `dir2module' and `repo2module' is somewhat cumbersome here
since we already have a modulemd YAML available. It just isn't
processed and doesn't contain `artifacts' yet. Too bad we don't have a
way to just do so instead of generating a whole new modulemd YAML from
scratch. Would we be interested in some tool like
`modulemd-generate-artifacts' that would simply modify an existing
YAML file?

(In reply to Robert Scheck from comment #31)
> Is "dir2module" meanwhile somehow able to satisfy our use case mentioned in
> comment #1?

please see answer to your question in comment 33

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (modulemd-tools bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:4152

We never properly documented a customers use case of "I have some RPMs in a
directory, and want to install with yum/dnf from that@, let's do that now.

Just running 'createrepo' on such a directory, one then encounters:
Error: No available modular metadata for modular package

It seems like we have on rhel8.5 now following ways, should we all
of these document in a kbase?  Which should we recommend?

#### Way 1, based on #33

# cd /root && rm -rf /root/localrepo
# mkdir localrepo
# dnf install --downloadonly --downloaddir=/root/localrepo -y java-11-openjdk
# cd /root/localrepo
# createrepo_c .
# dnf --disablerepo="*" --repofrompath=c8l,file:///root/localrepo install \
    -y --nogpgcheck --setopt=c8l.module_hotfixes=true java-11-openjdk

pro: should also work on rhel before 8.5GA ?

#### Way 2, based on  repo2module

# cd /root && rm -rf /root/localrepo
# mkdir localrepo
# dnf install --downloadonly --downloaddir=/root/localrepo -y java-11-openjdk
# cd /root/localrepo
# createrepo_c .

// Create a yaml file
# repo2module -n MODULE_NAME -s MODULE_STREAM -v MODULE_VERSION -c MODULE_CONTEXT /root/localrepo
# createrepo_mod .

// just for verification
# dnf module list --disablerepo=* --repofrompath=local,file:///root/localrepo
# dnf module repoquery --disablerepo=* --repofrompath=local,file:///root/localrepo mypkg-220130
# dnf list --disablerepo=* --repofrompath=local,file:///root/localrepo --available

// now for actual installation from local repo
# dnf install --disablerepo=* --repofrompath=local,file:///root/localrepo java-11-openjdk

con: this requires rhel8.5+
con: many steps

#### Way 3, based on dir2module

# mkdir localrepo
# dnf install --downloadonly --downloaddir=/root/localrepo -y java-11-openjdk
# cd /root/localrepo
# ls *\.module+* > ../repolist
# dir2module mypkg-220130:common:1:rhel8:x86_64 -m "my local repo with module" --pkglist=../repolist
# createrepo_mod .

// just for verification
# dnf module list --disablerepo=* --repofrompath=local,file:///root/localrepo
# dnf module repoquery --disablerepo=* --repofrompath=local,file:///root/localrepo mypkg-220130
# dnf list --disablerepo=* --repofrompath=local,file:///root/localrepo --available

// now we can install from the local repo
# dnf install --disablerepo=* --repofrompath=local,file:///root/myrepo java-11-openjdk

con: this requires rhel8.5+
con: many steps


Any comments on
- whether we should document all 3 ways, or just a selection?
- if one should be preferred, further pros/cons?