Bug 1806303

This is expected behavior.

$ podman pull fedora:31
$ podman run -it fedora:31
$ dnf module list ripgrep
Fedora Modular 31 - x86_64
Name     Stream      Profiles    Summary                                             
ripgrep  latest [d]  default [d] Line oriented search tool using Rust's regex library

Fedora Modular 31 - x86_64 - Updates
Name     Stream      Profiles    Summary                                             
ripgrep  latest [d]  default [d] Line oriented search tool using Rust's regex library

^^^ it shows ripgrep:latest as the default stream

I believe the problem is that fedora-modular still has the default defined.
It must be removed also from this repo.
Module metadata incl. defaults work across repositories by definition.

https://pagure.io/releng/fedora-module-defaults/c/5ebe9aa22cdb63f2b7dddf74c613e5f9c76eff6e?branch=f31

The GA repos are never changed.

In that case, we need to put the defaults back to the updates, but with empty stream:

document: modulemd-defaults
version: 1
data:
    module: ripgrep
    stream:



Steps for testing the solution:

FROM fedora:31
# turn off zchunk so we can easily access yaml content from command line
RUN echo "zchunk=0" >> /etc/dnf/dnf.conf
RUN dnf -y makecache
RUN dnf -y install less createrepo_c
RUN mkdir -p /opt/ripgrep \
    && cd /opt/ripgrep \
    && echo -e "document: modulemd-defaults\nversion: 1\ndata:\n    module: ripgrep\n    stream:\n" > ripgrep-defaults.yaml \
    && createrepo_c . \
    && modifyrepo_c --mdtype=modules ripgrep-defaults.yaml repodata/ \
    && echo -e "[ripgrep-defaults-override]\nbaseurl=file:///opt/ripgrep/\nenabled=1\n" > /etc/yum.repos.d/ripgrep-defaults-override.repo
RUN dnf -y module list ripgrep

One more thing: to be honest, I don't recall how modulemd priorities work when defaults merge.

I tested this with following names in case file name or repoid order matters:
ripgrep-defaults-override (> fedora-modular)
aaa-ripgrep-defaults-override (< fedora-modular)

Both cases seem to work fine.

> This is expected behavior.

Expected by who?

As shown in https://www.reddit.com/r/Fedora/comments/fa9yac/modular_repo_provides_older_package_than_updates/ this is most certainly not expected by our users.

(In reply to Miro Hrončok from comment #5)
> > This is expected behavior.
> 
> Expected by who?

Expected by modularity specifications we were getting from the former Modularity team.
We should check if it's documented properly. If not, we need to fix it.

It doesn't matter from which repos modulemd and related yaml documents come.
They are merged and used globally.

This scenario is similar to:
* package foo is in the fedora repo
* package foo is *not* in the updates repo
* -> package foo is still available, because it wasn't obsoleted

Creating modulemd-defaults document with empty stream is such override/obsolete.

This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

This has slipped through, it still happens this way when defaults are in play.

This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

I also find this behavior nonintuitive. Specification of modulemd-defaults reads:

  When merging, entries with a newer 'modified' value will override any earlier values.

I think it's not clear whether "entries" refer to whole modulemd-defaults documents, or to particular entries fields the documents. Now it behaves per fields. These two documents:

document: modulemd-defaults
version: 1
data:
    modified: 1
    module: ripgrep
    stream: latest
    profiles:
        latest: [more]

document: modulemd-defaults
version: 
data:
    modified: 2
    module: ripgrep
    profiles:
        else: [less]

merge into:

document: modulemd-defaults
version: 1
data:
    modified: 2
    module: ripgrep
    stream: latest
    profiles:
        latest: [more]
        else: [less]

The reasoning is that the "else" stream can be added by a third-party in a different repository without intention to change/define a default stream. It only wants to set a default profile for the new "else" stream. Technically, the third-party supplier could use "modified: 1", but what if he needs to release a new update through out his own updates repository? Then he needs to increase the modified value and will get it higher than what Fedora provides. Hence the modulemd-obsoletes are basically additive.

There three options:

(1) We can claim that distribution is not intended to change a default stream and close this report as NOTABUG. But that will prevent from fixing accidentally set default streams, and upgrading to a new Fedora where the maintainer decided for no default stream (e.g. he demodularized it; the case if a report).

(2) We can change the behavior and unset a default stream if there is none in a document of the highest modified. Then we will need to teach third-parties to copy the default stream from Fedora. I think this is terrible because the third-party will have to watch Fedora an carry that magical value he does not need for anything.

(3) We can introduce a new syntax for unsetting a default stream. Either with "stream:" or "stream: ''". The first undefined value looks like a typo and would be difficult to implement for libmodulemd (because how it uses libyaml). The other one with an empty string is more verbose and easy to handle in libmodulemd. Though with both approaches the unset value would be represented in libmodulemd's API to the applications as no default stream. It would be correctly handled when merging multiple documents but there would be no introspection way to distinguish never-set and unset cases. But that would not affect DNF. Theoretically I could add a new function into libmodulemd to deliver that distinction, if requested.

(3) sounds good. It could be an explicit null value.

Thanks for looking at this.

Moving to libmodulemd component as this the place when the enhancement will be implemented. I guess that DNF does not want to add any special handling.

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

This message is a reminder that Fedora Linux 36 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '36'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 36 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.