Bug 2053213 - Upgrading from an obsolete module stream does not distrosync packages of the new stream [NEEDINFO]
Summary: Upgrading from an obsolete module stream does not distrosync packages of the ...
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Jaroslav Mracek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-10 16:57 UTC by Petr Pisar
Modified: 2022-05-24 11:23 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
jmracek: needinfo? (mcurlej)


Attachments (Terms of Use)

Description Petr Pisar 2022-02-10 16:57:55 UTC
This is result of testing bug #1863049. I have:

dnf-4.9.0-1.fc35.noarch
libdnf-0.64.0-1.fc35.x86_64

In Fedora 35 there is a modulemd-obsoletes document replacing perl:5.30 with perl:5.32. I have enabled perl:5.30:

# dnf module list perl
Last metadata expiration check: 2:09:27 ago on Thu 10 Feb 2022 03:27:16 PM CET.
[...]
Fedora Modular 35 - x86_64 - Updates
Name   Stream     Profiles                       Summary                                  
perl   5.30 [e]   common, default [d], minimal   Practical Extraction and Report Language 
perl   5.32       common [d], minimal            Practical Extraction and Report Language 

# rpm -q perl-libs
perl-libs-5.30.3-457.module_f35+11289+1f4b364e.x86_64


Now I enable modular obsoleting to upgrade to perl:5.32:

# cat /etc/dnf/dnf.conf 
[main]
gpgcheck=1
installonly_limit=3
install_weak_deps=false
clean_requirements_on_remove=True
module_stream_switch=true
module_obsoletes=true
#metadata_expire=never
#excludepkgs=pcre2-10.37-1.fc35,pcre2-devel-10.37-1.fc35,pcre2-syntax-10.37-1.fc35,pcre2-tools-10.37-1.fc35,pcre2-utf16-10.37-1.fc35,pcre2-utf32-10.37-1.fc35

But DNF misbehaves: It immediately report that perl:5.32 is enabled. But the switch has not yet happened:

# dnf module list perl
Last metadata expiration check: 2:15:21 ago on Thu 10 Feb 2022 03:27:16 PM CET.
[...]
Fedora Modular 35 - x86_64 - Updates
Name   Stream     Profiles                       Summary                                  
perl   5.30       common, default [d], minimal   Practical Extraction and Report Language 
perl   5.32 [e]   common [d], minimal            Practical Extraction and Report Language 

I issue an upgrade:

# dnf upgrade
Last metadata expiration check: 2:17:49 ago on Thu 10 Feb 2022 03:27:16 PM CET.
Dependencies resolved.
==========================================================================================
 Package             Architecture      Version                    Repository         Size
==========================================================================================
Reinstalling:
 automake            noarch            1.16.2-5.fc35              fedora            662 k
 paper               x86_64            2.3-3.fc35                 fedora             31 k
Switching module streams:
 perl                                  5.30 -> 5.32                                      

Transaction Summary
==========================================================================================

Total download size: 693 k
Installed size: 1.8 M
Is this ok [y/N]: y

It offers switching the streams. That's fine. But it does not "distrosync" packages of the new stream. And reinstalls two nonmodular packages (automake, paper) for no reason:

# rpm -q perl-libs
perl-libs-5.30.3-457.module_f35+11289+1f4b364e.x86_64

Repeated upgrade results into the same misbehaviour:

# dnf upgrade
Last metadata expiration check: 2:20:30 ago on Thu 10 Feb 2022 03:27:16 PM CET.
Dependencies resolved.
==========================================================================================
 Package             Architecture      Version                    Repository         Size
==========================================================================================
Reinstalling:
 automake            noarch            1.16.2-5.fc35              fedora            662 k
 paper               x86_64            2.3-3.fc35                 fedora             31 k

Transaction Summary
==========================================================================================

Total download size: 693 k
Installed size: 1.8 M
Is this ok [y/N]: y


Only distrosync fixes it:

# dnf distrosync
Last metadata expiration check: 2:20:57 ago on Thu 10 Feb 2022 03:27:16 PM CET.
Dependencies resolved.
==========================================================================================
 Package                Arch   Version                               Repository      Size
==========================================================================================
Upgrading:
 perl-Algorithm-Diff    noarch 1.1903-17.module_f35+11299+2c136bae   fedora-modular  48 k
[...]
Transaction Summary
==========================================================================================
Install   32 Packages
Upgrade  117 Packages

Total download size: 14 M
Is this ok [y/N]: y

# rpm -q perl-libs
perl-libs-5.32.1-471.module_f35+12589+8a7d3254.x86_64

Since then, DNF is content:

# dnf upgrade
Last metadata expiration check: 2:21:59 ago on Thu 10 Feb 2022 03:27:16 PM CET.
Dependencies resolved.
Nothing to do.
Complete!


I think there are bugs in DNF. It probably switches the streams for a modular solver to early and thus forgets to reinstall the modular packages.

Comment 1 Jaroslav Mracek 2022-04-05 13:51:28 UTC
It is really tricky example and it requires deep discussion whether we can call it as a feature or bug. Dnf also shows content of not enabled or default module streams when they are a dependency of enabled or default module. It is the same principle, same strange principle but it is required for proper functionality of enabled module. What do you think?

Comment 2 Petr Pisar 2022-04-05 15:17:07 UTC
(In reply to Jaroslav Mracek from comment #1)
> It is really tricky example and it requires deep discussion whether we can
> call it as a feature or bug.

I expect that if 

  module_stream_switch=true

then distrosync is performed on obsoleting a stream the same way as "dnf module switch-to" does.

> Dnf also shows content of not enabled or
> default module streams when they are a dependency of enabled or default
> module. It is the same principle, same strange principle but it is required
> for proper functionality of enabled module. What do you think?

I believe I had no other stream enabled. And Fedora has no default streams.

Comment 3 Jaroslav Mracek 2022-05-17 08:22:09 UTC
Then run `dnf distrosync --setopt=module_stream_switch=true`. Or system_upgrade will do it for you automatically. It is quite dangerous to use `module_stream_switch=true` as a permanent value on your system. It can trigger unwanted changes even when you trying to install one  package from different major version of your distribution. I guess that we can extend functionality of module-switch command but because all of that fun is already covered by system-upgrade it has very low priority or even it is a good candidate to be close in favor of other issues. What do you think?

Comment 4 Petr Pisar 2022-05-17 08:35:03 UTC
I think that system-upgrade is not enough. What I want is a obsoleting module stream in a middle of a Fedora release life cycle. Hence a normal "dnf upgrade" should handle it.

Comment 5 Jaroslav Mracek 2022-05-17 17:11:17 UTC
Thanks I understand what you want to achieve but it will be a shock for users when `dnf` starts to downgrade packages during upgrade. Personally I do not recommend such a change for `dnf upgrade` command. Or I recommend to create from it a packaging issue. I am 100% sure that a packager can provide an upgrade path on RPM level. Please do not ask for any additional magic in DNF.

If this is a request for change of behavior of upgrade command that it starts to behave like distrosync or module-switch I would like to close it as wont fix.

What do you think?

Comment 6 Petr Pisar 2022-05-18 08:17:05 UTC
I think you are overly conservative. Modules have independent life cycle from the distribution and DNF should respect it and strive for a smooth experience as much as possible. Modular obsoletes which recommends a replacing stream needs to be honored in "dnf upgrade". Otherwise you keep a user stuck on unsupported software. In my opinion considering stream switches as something exceptional which can happen only on dist-upgrade is ill by design and places a user to a dead-end.

If you don't want to implement the stream switching because it's too difficult or to innovative, then at least implement reporting the obsoletes in a DNF output so that a user can benefit from them (i.e. get notified about the EOL and manually switch to the recommended replacement).

Comment 7 Petr Pisar 2022-05-18 09:00:11 UTC
I studied this particular perl:5.32 failure again.

perl:5.32 is missing perl-Pod-Parser package comparing to perl:5.30. That's not a packaging bug. perl-Pod-Parser was simply removed by the upstream from Perl 5.32. Though, it still exists separately from Perl, hence no RPM Obsolete: perl-Pod-Parser" is there. So you are right that perl:5.32 is not an RPM-wise superset of perl:5.30. Thus the "upgrade path on RPM level" was not held. When I uninstalled perl-Pod-Parser, then "dnf upgrade" with module_obsoletes=true and module_stream_switch=true passed.

Technically a packager should create a new perl-Pod-Parser module and extend the modular obsoletes to replace perl:5.30 with perl:5.32 and perl-Pod-Parser:SOMETHING. However modular obsoletes format does not support it yet. Nobody thought that modular obsoletance could be 1:N map. I will note this as an enhancment request for the modular format.

Comment 8 Petr Pisar 2022-05-18 09:19:15 UTC
(In reply to Petr Pisar from comment #7)
> Technically a packager should create a new perl-Pod-Parser module and extend
> the modular obsoletes to replace perl:5.30 with perl:5.32 and
> perl-Pod-Parser:SOMETHING. However modular obsoletes format does not support
> it yet. Nobody thought that modular obsoletance could be 1:N map. I will
> note this as an enhancment request for the modular format.

https://github.com/fedora-modularity/libmodulemd/issues/598

Comment 9 Petr Pisar 2022-05-18 09:23:59 UTC
Meanwhile, I will look whether adding "Obsoletes: perl-Pod-Parser < 0:1.63-441" into some of the perl:5.32 packages would help.

Comment 10 Petr Pisar 2022-05-18 11:29:51 UTC
Adding "Obsoletes: perl-Pod-Parser < 0:1.63-441" into perl-libs of perl:5.32 indeed helps. But it's not a general fix:

(1) RPM-Obsoletes has 2 effects: Uninstalling the obsoleted package, and installing the obsoleting package. And the installation is a problem. If there is no always-installed package, like perl-libs in case of perl streams, adding Obsoletes into any package will be wrong because it will cause installing a unnecessary package. There is "Provides: libsolv-self-destruct-pkg()" hack, but I'm not sure how safe it. Does it need a dedicated package, or can it be placed in an installable package?

(2) Another problem is which version and release to RPM-obsolete. If another module delivering perl-Pod-Parser-0:1.63-440 will be added to the distribution, the RPM-Obsolete from perl:5.32 will interfere.

(3) Finally, the RPM-Obsolete is a hack because the obsoleting package has not relation to obsoleted package. It's similar misuse to a distribution-level fedora-obsolete-packages package. My gut feeling is that maintainer will be against spraying random packages with Obsoletes only to allow upgrading from a module to another module.

Because stream contents are generally incomparable, there is no upgrade path between them maintained. Therefore I'd rather see a distro-syncing module-switch on obsoleting modules.

Maybe a new "dnf module upgrade" subcommand could perform the distro-syncing module-switch.

Comment 11 Petr Pisar 2022-05-18 12:18:06 UTC
(In reply to Petr Pisar from comment #10)
> Adding "Obsoletes: perl-Pod-Parser < 0:1.63-441" into perl-libs of perl:5.32
> indeed helps.

The (2) and (3) issues combined have a problem with third-party modules and packages. If somebody already has built perl-Pod-Parser for Perl 5.32 and it happened to be < 0:1.63-441, then adding the Obsolete will destroy his package.

E.g. When Fedora builds perl-App-cpanminus module, it builds perl-Pod-Parser as a build-only component a never distributes it. It could interfere with the Obsolete if it clashed on the release number of NEVRA. In this particular case there is no clash because the oldest supported perl-App-cpanminus:1.7044 builds perl-Pod-Parser-0:1.63-445. A similar case is when building perl:5.32 we also build a build-only perl-Pod-Parser-1.63-444. I think this illustrates why adding a seldom Obsoletes is a bad idea and it should rather be fixed by DNF with a distro-sync-like operation.


Jitka, do I have your permission to push this change into perl:5.32 (perl-libs subpackage of perl.spec in f34 branch)? I would condition the presence of the Obsolete with a build environment for perl:5.32 module:

%package libs
%if "%{_module_name}" == "perl" && "%{_module_stream}" == "5.32"
Obsoletes: perl-Pod-Parser < 0:1.63-441
%endif

Comment 12 Petr Pisar 2022-05-18 12:21:47 UTC
I only add that perl:5.32 reaches EOL on 2022-06-01. Then the packagers won't be allowed to update that stream.

Comment 13 Jitka Plesnikova 2022-05-19 08:00:32 UTC
> Jitka, do I have your permission to push this change into perl:5.32
> (perl-libs subpackage of perl.spec in f34 branch)? I would condition the
> presence of the Obsolete with a build environment for perl:5.32 module:
> 
> %package libs
> %if "%{_module_name}" == "perl" && "%{_module_stream}" == "5.32"
> Obsoletes: perl-Pod-Parser < 0:1.63-441
> %endif

Yes, please add it.

Comment 14 Petr Pisar 2022-05-24 11:23:41 UTC
(In reply to Jitka Plesnikova from comment #13)
> > Jitka, do I have your permission to push this change into perl:5.32
> > (perl-libs subpackage of perl.spec in f34 branch)? I would condition the
> > presence of the Obsolete with a build environment for perl:5.32 module:
> > 
> > %package libs
> > %if "%{_module_name}" == "perl" && "%{_module_stream}" == "5.32"
> > Obsoletes: perl-Pod-Parser < 0:1.63-441
> > %endif
> 
> Yes, please add it.

Fixed in perl:5.32:NN20220523110914.

I wanted to add a similar fix to perl:5.34 because after perl:5.32 EOL, DNF could upgrade directly from perl:5.30 to perl:5.34. But according to my test, DNF does not evaluate modular obsoletes transitively. Hence you need to do "dnf upgrade" twice to jump from perl:5.30 to perl:5.34. So the fix is not now required for perl:5.34.


Note You need to log in before you can comment on or make changes to this bug.