Bug 1656509 - F29 to F30+ upgrades fail unless --allowerasing is used due to issue with module_platform_id, even explicit --setopt=module_platform_id does not work [NEEDINFO]
Summary: F29 to F30+ upgrades fail unless --allowerasing is used due to issue with mod...
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: libdnf
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
Assignee: Pavla Kratochvilova
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa AcceptedBlocker
Keywords: Reopened, Triaged
Depends On:
Blocks: F30BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2018-12-05 16:32 UTC by Michel Normand
Modified: 2019-05-03 18:52 UTC (History)
29 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2019-03-19 19:38:39 UTC
sgallagh: needinfo? (jmracek)
sgallagh: needinfo? (psabata)


Attachments (Terms of Use)
dnf.log (190.62 KB, text/plain)
2018-12-05 17:56 UTC, Michel Normand
no flags Details

Description Michel Normand 2018-12-05 16:32:17 UTC
Description of problem:
upgrade openQA tests failed for rawhide

as reported in openQA last tests results (compose 20181205)

for ppc64le:
https://openqa.stg.fedoraproject.org/tests/417695#step/upgrade_postinstall/7

for x86_64:
https://openqa.stg.fedoraproject.org/tests/417606#step/upgrade_postinstall/7

related openQA function is check_release:
https://pagure.io/fork/michelmno/fedora-qa/os-autoinst-distri-fedora/blob/master/f/lib/utils.pm#_345

the function is passed with 'rawhide' release name
while /etc/os-release still refer to stable release '29'

Comment 1 Adam Williamson 2018-12-05 16:43:40 UTC
That basically means the upgrade didn't actually happen - we need to figure out why that is.

Comment 2 Adam Williamson 2018-12-05 16:47:50 UTC
If you look at the actual upgrade in the video it doesn't really do anything - it flips straight from the 'preparing' step to the 'complete' step.

Unfortunately we don't get any logs, because installing the tar package fails due to some kinda signature issue. I'm wondering if that's also the problem with the upgrade.

This isn't an openqa bug, the test is failing because the upgrade is failing.

Comment 3 Michel Normand 2018-12-05 17:56 UTC
Created attachment 1511835 [details]
dnf.log

There is another occurence for ppc64le that failed the same way on check_release
but for which openQA is able to upload data
https://openqa.stg.fedoraproject.org/tests/417712#step/upgrade_postinstall/7
https://openqa.stg.fedoraproject.org/tests/417712/file/upgrade_postinstall-var_log.tar.gz

from it there are modular dependency problem as dnf.log extracted from above tarball
===
2018-12-05T12:33:47Z WARNING Modular dependency problems:

 Problem 1: conflicting requests
  - nothing provides module(platform:f30) needed by module stratis:1:20181109160136:a5b0195c-0.ppc64le
 Problem 2: conflicting requests
  - nothing provides module(platform:f30) needed by module ripgrep:latest:20181104174352:a5b0195c-0.ppc64le
 Problem 3: conflicting requests
  - nothing provides module(platform:f30) needed by module libgit2:0.27:20181028172505:a5b0195c-0.ppc64le
 Problem 4: conflicting requests
  - nothing provides module(platform:f30) needed by module gimp:2.10:20181112112217:a5b0195c-0.ppc64le
 Problem 5: conflicting requests
  - nothing provides module(platform:f30) needed by module dwm:6.1:20180831122920:a5b0195c-0.ppc64le
 Problem 6: conflicting requests
  - nothing provides module(platform:f30) needed by module bat:latest:20181106000712:e50d0d19-0.ppc64le
===

Comment 4 Adam Williamson 2018-12-05 20:18:20 UTC
Yeah, those are visible in the video and some of the screenshots too, but I'm not *sure* they're actually the problem. Would be good to reproduce manually and poke around a bit to see.

Comment 5 Adam Williamson 2018-12-14 00:38:46 UTC
The F28 -> Rawhide upgrade tests work fine; that does rather suggest that the problem here is with the modular repos that are enabled in F29.

CCing sgallagh For Great Modularity; the issue here is that all upgrade tests from F29 to Rawhide are failing, what logs we have been able to collect indicate issues with the modular repos / keys / something. Upgrades from F28 to Rawhide work fine. Should be quite easy to reproduce: do a clean F29 install, try and upgrade to Rawhide with 'dnf -y --releasever=30 --nogpgcheck system-upgrade download' and 'dnf --releasever=30 --nogpgcheck system-upgrade reboot'.

Comment 6 Adam Williamson 2018-12-14 00:39:42 UTC
This appears to be a Beta blocker: https://fedoraproject.org/wiki/Fedora_30_Beta_Release_Criteria#Upgrade_requirements - "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from a fully updated, clean default installation of each of the last two stable Fedora releases with that package set installed."

Comment 7 Adam Williamson 2018-12-14 02:26:17 UTC
Reproducing this manually, it really does seem like the "module(platform:f30)" dependencies are the problem. I see those errors during the upgrade boot, which fails to actually run the upgrade.

Comment 8 Stephen Gallagher 2018-12-14 13:42:50 UTC
Hmm, this is actually the opposite of the problem I was expecting we were having.

I was thinking this was caused by us forgetting to do a mass-rebuild after branching, such that the modules in F30 were just tagged from F29 and therefore were expecting platform:f29 and not finding it.

This looks like the virtual platform module is missing entirely from F30...

I forget if this is a DNF issue or an MBS one, so I'm CCing folks from both.

Comment 9 Jan Kaluža 2018-12-14 13:55:55 UTC
I've discussed this briefly with Stephen on IRC and it looks OK from MBS point of view - we have the platform:f30 in the MBS DB - https://mbs.fedoraproject.org/module-build-service/1/module-builds/?nsvc=platform:f30. This virtual module is however never added to Koji as content-generator build and it is never shipped in modular repo by Pungi afaik. DNF uses its own way to set the platform virtual module and use it for resolving.

In case you think MBS needs to do something differently here, please tell me.

Comment 10 Daniel Mach 2018-12-14 17:54:13 UTC
Stephen Gallagher, Igor Gnatenko and me discussed this on #fedora-releng.
I proposed distroverpkg based auto-detection of platform id,
which extends existing releasever detection.

Here's the outcome of the irc chat + my detailed write-up on scenarios and various edge cases.


Problem statement
-----------------
Libdnf handles module dependency resolution.
It is used by dnf, microdnf, PackageKit or any other software.
To resolve module dependencies correctly,
libdnf must handle a virtual pseudo-module 'platform'.
Without 'platform' module or when it's set to an incorrect value,
module dependency resolution fails on unresolved deps.


General approach
----------------
We should focus on data driven solution rather than
interpolating values from the running system:
* Don't derive platform_id from releasever (doesn't always have to work)
* Don't derive platform id from /etc/os-release (may contain irrelevant data)
We should also consider various edge cases
such as running composes for different releasever/platform/arch,
running repoquery on an arbitrary repo, etc.

I propose platform auto-detection in Scenario 6.
Users must enable it willingly or it must turned on
in a tool because it fits into the workflow.
For example during the upgrade to the next distro version.
The auto-detection must not unexpectedly trigger due to
losing /etc/os-release, disabling a repo or any other condition.

The required functionality should be implemented in libdnf
and used by other software such as dnf, microdnf, PackageKit,
Anaconda, Mock, etc.

Scenarios we should care about follow.


Scenario 1: Normal usage (running system, no upgrades)
------------------------------------------------------
The 'platform' pseudo-module is obtained from /etc/os-release:

$ source /etc/os-release; echo $PLATFORM_ID
platform:f29

The prerequisite is to have any system-release package installed,
which includes fedora-release, redhat-release etc.


Scenario 2: Manual system upgrade
---------------------------------
User runs system upgrade by hand:
$ dnf distro-sync --releasever=30 --setopt='module_platform_id=platform:f29'

Value from /etc/os-release is overriden with user input
and dependency resolution works.


Scenario 3: Install into an empty chroot
----------------------------------------
User runs :
$ dnf install --installroot=... --releasever=... --setopt='module_platform_id=...'

There's currently no auto-detection of releasever or module_platform_id.


Scenario 4: Anaconda installs a fresh system
--------------------------------------------
Same as Scenario 3.
module_platform_id hard-coded in install class


Scenario 5: Mock
----------------
Same as Scenario 3.
module_platform_id hard-coded in mock config


Scenario 6: Upgrade an existing system
--------------------------------------
The initial conditions are identical to Scenario 1.
Unfortunately releasever and /etc/os-release contents
are not usable for the upgrade, because they don't carry
any information relevant to the system we're upgrading to.

Increasing releasever and platform by 1 is a guess
that usually works, but we shouldn't consider that
a properly designed solution.

The upgrade tool must read the values from a reliable source.
That is typically a user input or configuration.

All this doesn't work with arbitrary repos.
* You can upgrade to Fedora+2
* You could use Anaconda netinst image from Rawhide to install CentOS (why not?)

There is a feature in YUM and DNF to detect releasever from repodata.
It relies on 'distroverpkg' option (yum) or constant (dnf).

How it works:
* search for RPMs matching distroverpkg list
* pick the first matching RPM in the list (I'm not entirely sure about this step)
* obtain releasever from the "system-release(releasever)" provide
* if there's no such provide, use package version as releasever

More details about distroverpkg can be found in yum3 man page:
$ man yum.conf (YUM v3 on RHEL 7 for example)
distroverpkg  The  package used by yum to determine the "version" of the dis-
tribution, this sets $releasever for use in config. files. This  can  be  any
installed package. Default is `system-release(releasever)', `redhat-release'.
Yum will now look at the version provided by the provide, and if that is non-
empty  then  will  use  the  full V(-R), otherwise it uses the version of the
package.
You can see what provides this manually by using: "yum whatprovides 'system-
release(releasever)' redhat-release" and you can see what $releasever is most
easily by using: "yum version".

We could extend distroverpkg feature to module_platform_id and detect
it from repodata directly.

# An example of a releasever provide:
Provides:       system-release(releasever) = 29

# An example of module_platform_id provide:
# Since the ':' is epoch separator, it's impossible to use it as part of the provide value. It could be replaced with 2 underscores for example:
Provides:       system-release(module_platform_id) = platform__f29


Scenario 7: Repoquery
---------------------
User can use repoquery command to query repositories
that are unrelated to the currently running system.

Auto-detecting releasever and module_platform_id
would improve user experience.

Comment 11 Adam Williamson 2018-12-14 18:08:27 UTC
"Scenario 6: Upgrade an existing system
--------------------------------------
The initial conditions are identical to Scenario 1.
Unfortunately releasever and /etc/os-release contents
are not usable for the upgrade, because they don't carry
any information relevant to the system we're upgrading to."

Uh...what? When you do 'dnf system-upgrade', you are required to pass `--releasever=(target release)`. Upgrading via GNOME Software does the equivalent. I mentioned this in comment 5: "dnf --releasever=30 --nogpgcheck system-upgrade download" - that's the command I tested.

Comment 12 Stephen Gallagher 2018-12-14 18:09:26 UTC
(In reply to Daniel Mach from comment #10)
> Stephen Gallagher, Igor Gnatenko and me discussed this on #fedora-releng.
> I proposed distroverpkg based auto-detection of platform id,
> which extends existing releasever detection.
> 
> Here's the outcome of the irc chat + my detailed write-up on scenarios and
> various edge cases.

...

> Scenario 2: Manual system upgrade
> ---------------------------------
> User runs system upgrade by hand:
> $ dnf distro-sync --releasever=30 --setopt='module_platform_id=platform:f29'
> 

Should this have been `--setopt='module_platform_id=platform:f30'`? (s/f29/f30 in the setopt). I assume that the point of the setopt here is to tell DNF to use the target system's platform module, not the originating system.

...

> Scenario 6: Upgrade an existing system
> --------------------------------------

...

> How it works:
> * search for RPMs matching distroverpkg list
> * pick the first matching RPM in the list (I'm not entirely sure about this
> step)
> * obtain releasever from the "system-release(releasever)" provide
> * if there's no such provide, use package version as releasever
> 


In Fedora 30+, we will have multiple packages that provide `system-release(releasever)`, but they will all have the same releasever number, so that shouldn't be an issue.

Comment 13 Daniel Mach 2018-12-16 15:56:09 UTC
(In reply to Adam Williamson from comment #11)
> "Scenario 6: Upgrade an existing system
> --------------------------------------
> The initial conditions are identical to Scenario 1.
> Unfortunately releasever and /etc/os-release contents
> are not usable for the upgrade, because they don't carry
> any information relevant to the system we're upgrading to."
Sorry for not being clear - what I meant was the releasever value that is
automatically set on runtime when user doesn't override it from the command line.
This runtime value shouldn't be used for making any assumptions about the upgrade,
for example by setting new releasever to releasever+1.

> 
> Uh...what? When you do 'dnf system-upgrade', you are required to pass
> `--releasever=(target release)`. Upgrading via GNOME Software does the
> equivalent. I mentioned this in comment 5: "dnf --releasever=30 --nogpgcheck
> system-upgrade download" - that's the command I tested.

Providing releasever from CLI works exactly as you wrote.
You actually have to do it because $releasever is part of the url
and you don't get to the new repos otherwise.

Comment 14 Adam Williamson 2018-12-17 17:28:57 UTC
The openQA F29-F30 upgrade tests for Fedora-Rawhide-20181216.n.1 nearly all passed. Did someone do something to fix this?

Comment 15 Geoffrey Marr 2018-12-17 20:03:16 UTC
Discussed during the 2018-12-17 blocker review meeting: [1]

The decision to classify this bug as an "AcceptedBlocker" was made as it violates the following criteria:

"For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from a fully updated, clean default installation of each of the last two stable Fedora releases with that package set installed."

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2018-12-17/f30-blocker-review.2018-12-17-17.00.txt

Comment 16 Adam Williamson 2019-01-12 22:09:01 UTC
Well, no-one replied to #c14, but this is still passing in recent tests, so I'm gonna go ahead and close the bug.

Comment 17 Michel Normand 2019-01-14 07:29:42 UTC
unable to verify on ppc64le because compose do not work for ppc64le since a while, so not new Rawhide ppc64le iso to test.

Comment 18 Adam Williamson 2019-01-14 08:02:41 UTC
The ppc64 compose *should* be fixed now, but we haven't had an overall successful Rawhide compose since the ppc64 fix landed in lorax :/ We'll have to see what happens whenever we manage to get a Rawhide compose through next.

Comment 19 Stephen Gallagher 2019-02-08 18:21:39 UTC
Reopening, because I can trivially reproduce this problem today:

Steps:

sudo podman run --privileged  --tty -i --name=upgrades registry.fedoraproject.org/fedora:29

(inside the container)
dnf -y update
dnf -y module install reviewboard:2.5
dnf install 'dnf-command(system-upgrade)'
dnf system-upgrade --releasever=30

Results:
[root@5c6e5b86affa /]# dnf system-upgrade --releasever=30 download
Before you continue ensure that your system is fully upgraded by running "dnf --refresh upgrade". Do you want to continue [y/N]: y
Fedora Modular 30 - x86_64                                                                                                                                         383 kB/s | 2.2 MB     00:05    
Fedora Modular 30 - x86_64 - Updates                                                                                                                               485 kB/s | 2.2 MB     00:04    
Fedora 30 - x86_64 - Updates                                                                                                                                       1.5 MB/s |  61 MB     00:40    
Fedora 30 - x86_64                                                                                                                                                 5.4 MB/s |  61 MB     00:11    
Last metadata expiration check: 0:00:01 ago on Fri 08 Feb 2019 06:06:53 PM UTC.
Modular dependency problems:

 Problem 1: conflicting requests
  - nothing provides module(platform:f30) needed by module avocado:stable:3020190116195115:a5b0195c-0.x86_64
 Problem 2: conflicting requests
  - nothing provides module(platform:f30) needed by module bat:latest:20181220130258:e50d0d19-0.x86_64
 Problem 3: conflicting requests
  - nothing provides module(platform:f30) needed by module django:1.6:20180828135711:a5b0195c-0.x86_64
 Problem 4: conflicting requests
  - nothing provides module(platform:f30) needed by module dwm:6.1:20180831122920:a5b0195c-0.x86_64
 Problem 5: conflicting requests
  - nothing provides module(platform:f30) needed by module fish:3:3020190116191836:602da195-0.x86_64
 Problem 6: conflicting requests
  - nothing provides module(platform:f30) needed by module gimp:2.10:20181223154246:a5b0195c-0.x86_64
 Problem 7: conflicting requests
  - nothing provides module(platform:f30) needed by module libgit2:0.27:3020190128145600:a5b0195c-0.x86_64
 Problem 8: conflicting requests
  - nothing provides module(platform:f30) needed by module meson:latest:3020190123223713:36245242-0.x86_64
 Problem 9: conflicting requests
  - nothing provides module(platform:f30) needed by module ninja:latest:3020190131012415:a5b0195c-0.x86_64
 Problem 10: conflicting requests
  - nothing provides module(platform:f30) needed by module ripgrep:latest:20181215204810:a5b0195c-0.x86_64
 Problem 11: conflicting requests
  - nothing provides module(platform:f30) needed by module stratis:1:20181215204600:a5b0195c-0.x86_64
 Problem 12: conflicting requests
  - module reviewboard:2.5:20180409150256:88f8ec0d-0.x86_64 requires module(django:1.6), but none of the providers can be installed
  - nothing provides module(platform:f30) needed by module django:1.6:20180828135711:a5b0195c-0.x86_64
Error: 
 Problem 1: package python2-djblets-0.9.9-13.module_1631+4353a891.noarch requires python2-django(abi) = 1.6, but none of the providers can be installed
  - problem with installed package python2-djblets-0.9.9-13.module_1631+4353a891.noarch
  - python2-django-1.6.11.7-3.module_2144+ad479803.noarch does not belong to a distupgrade repository
  - package python2-django-1.6.11.7-3.module_2144+ff35d33b.noarch is excluded
  - package python2-djblets-0.9.9-13.module_1631+4353a891.noarch is excluded
  - package python2-djblets-1.0.8-2.module_2438+0f557a29.noarch is excluded
 Problem 2: package ReviewBoard-2.5.17-17.module_1631+4353a891.noarch requires python2-django-haystack(abi) = 2.4, but none of the providers can be installed
  - problem with installed package ReviewBoard-2.5.17-17.module_1631+4353a891.noarch
  - python2-django-haystack-2.4.1-12.module_1631+4353a891.noarch does not belong to a distupgrade repository
  - package ReviewBoard-2.5.17-17.module_1631+4353a891.noarch is excluded
  - package ReviewBoard-3.0.11-1.module_2692+3132a545.noarch is excluded
  - package python2-django-haystack-2.4.1-12.module_1631+4353a891.noarch is excluded
  - package python2-django-haystack-2.4.1-13.module_2438+0f557a29.noarch is excluded

Comment 20 Kevin Fenzi 2019-02-09 23:26:58 UTC
Two things: 

1. rawhide was in a pretty bad state the other day, can you try after todays compose lands?

2. specifying releasever=30 doesn't quite work for modular repo, as the rawhide one seems hard coded. This relates to the plan to make rawhide 'rawhide' and the number, where it's currently inconsistent.

Comment 21 Stephen Gallagher 2019-02-11 12:59:29 UTC
(In reply to Kevin Fenzi from comment #20)
> Two things: 
> 
> 1. rawhide was in a pretty bad state the other day, can you try after todays
> compose lands?
> 

No change today

> 2. specifying releasever=30 doesn't quite work for modular repo, as the
> rawhide one seems hard coded. This relates to the plan to make rawhide
> 'rawhide' and the number, where it's currently inconsistent.

Is there some way to deal with this? --releasever=rawhide doesn't work at all.

Comment 22 Kevin Fenzi 2019-02-11 18:49:19 UTC
It doesn't work currently, but we were going to try and make it work... see: https://pagure.io/releng/issue/7445

Comment 23 Adam Williamson 2019-02-11 21:21:18 UTC
well, one way or the other needs to work, or we can't test upgrades to Rawhide.

openQA uses '--releasever=30', FWIW. this is what Dennis told us to do, IIRC, he didn't think --releasever=rawhide was correct / supportable.

Comment 24 Kevin Fenzi 2019-02-11 21:24:33 UTC
Yes, that is the current state. I am saying we plan to make 'rawhide' work, but I am not sure the status of all that work. I think kparal was trying to get it all done, but ran low on time. 

For now, what happpens if you: 

dnf install fedora-release-rawhide

disable the fedora-modular repos and enable the fedora-rawhide-modular repos and try upgrading with --releasever=30 ?

Comment 25 Stephen Gallagher 2019-02-25 18:38:45 UTC
I was away last week. I just tried again today with --releasever=30 (to upgrade from F29 to F30, taking the "rawhide" issues out of the equation). I get exactly the same results as in Comment #19. Also, I notice that if I install nodejs:8, the upgrade will attempt to update the package to nodejs-10.15.0-1.fc30, which is exactly opposite the expected behavior (which would be to stay on the "8" stream or refuse to upgrade if the "8" stream is unavailable on the target OS).

Reassigning to DNF.

Comment 26 Jaroslav Mracek 2019-02-26 22:08:02 UTC
Pavla Kratochvilova created a patch https://github.com/rpm-software-management/dnf-plugins-extras/pull/143 for the issue.

Comment 27 Sergio Durigan Junior 2019-03-01 17:06:56 UTC
Hi,

I'm seeing the following error when trying to upgrade from F29 to F30:

[root@fedora-stable ~]# dnf system-upgrade download --refresh --releasever=30
Before you continue ensure that your system is fully upgraded by running "dnf --refresh upgrade". Do you want to continue [y/N]: y
Fedora Modular 30 - x86_64                                                                                                                                      11 kB/s |  13 kB     00:01    
Fedora Modular 30 - x86_64 - Updates                                                                                                                            83 kB/s |  15 kB     00:00    
Fedora 30 - x86_64 - Updates                                                                                                                                    83 kB/s |  15 kB     00:00    
Fedora 30 - x86_64                                                                                                                                              38 kB/s |  13 kB     00:00    
Modular dependency problems:

 Problem 1: conflicting requests
  - nothing provides module(platform:f30) needed by module stratis:1:20181215204600:a5b0195c-0.x86_64
 Problem 2: conflicting requests
  - nothing provides module(platform:f30) needed by module standard-test-roles:3.0:3020190214144451:a5b0195c-0.x86_64
 Problem 3: conflicting requests
  - nothing provides module(platform:f30) needed by module ripgrep:latest:3020190214090003:a5b0195c-0.x86_64
 Problem 4: conflicting requests
  - nothing provides module(platform:f30) needed by module ninja:latest:3020190218225555:a5b0195c-0.x86_64
 Problem 5: conflicting requests
  - nothing provides module(platform:f30) needed by module meson:latest:3020190218225437:36245242-0.x86_64
 Problem 6: conflicting requests
  - nothing provides module(platform:f30) needed by module libgit2:0.27:3020190218225305:a5b0195c-0.x86_64
 Problem 7: conflicting requests
  - nothing provides module(platform:f30) needed by module gimp:2.10:3020190218225005:a5b0195c-0.x86_64
 Problem 8: conflicting requests
  - nothing provides module(platform:f30) needed by module fish:3:3020190219150858:602da195-0.x86_64
 Problem 9: conflicting requests
  - nothing provides module(platform:f30) needed by module exa:latest:3020190214120734:e50d0d19-0.x86_64
 Problem 10: conflicting requests
  - nothing provides module(platform:f30) needed by module dwm:6.1:3020190213215420:a5b0195c-0.x86_64
 Problem 11: conflicting requests
  - nothing provides module(platform:f30) needed by module bat:latest:3020190214090936:e50d0d19-0.x86_64
 Problem 12: conflicting requests
  - nothing provides module(platform:f30) needed by module avocado:stable:3020190213205848:a5b0195c-0.x86_64
Error: 
 Problem: package python2-rpkg-1.57-6.fc29.noarch requires rpkg-common = 1.57-6.fc29, but none of the providers can be installed
  - rpkg-common-1.57-6.fc29.noarch does not belong to a distupgrade repository
  - problem with installed package python2-rpkg-1.57-6.fc29.noarch
(try to add '--skip-broken' to skip uninstallable packages)


Using '--skip-broken' doesn't help, of course.  I'm assuming this is the same error reported here, but if it's not, please let me know and I'll file another bug.

Is there any recommended workaround for now?  Thanks.

Comment 28 Sergio Durigan Junior 2019-03-01 17:29:23 UTC
After reading fedora-devel (should have done that earlier), I found out that the recommended way to perform the update is:

dnf --releasever=30 --setopt=module_platform_id=platform:f30 --enablerepo=updates-testing distro-sync

So now I'm just seeing the following error:

Error: 
 Problem: package python2-rpkg-1.57-6.fc29.noarch requires rpkg-common = 1.57-6.fc29, but none of the providers can be installed
  - rpkg-common-1.57-6.fc29.noarch does not belong to a distupgrade repository
  - problem with installed package python2-rpkg-1.57-6.fc29.noarch
(try to add '--skip-broken' to skip uninstallable packages)


Which doesn't seem related to the problem reported on this bug.  Sorry about the noise.

Comment 29 Adam Williamson 2019-03-01 18:38:26 UTC
Well, it's more that "--setopt=module_platform_id=platform:f30" is the *workaround* for this bug. It should not be necessary to manually do that, dnf should be smart enough to figure it out. There still really *is* a bug here :)

Comment 30 Sergio Durigan Junior 2019-03-01 18:41:32 UTC
Right, and agreed.  It took me quite some time to figure it out.  I was able to solve the other issues and now the update is happening without problems, it seems.

Comment 31 Adam Williamson 2019-03-05 20:01:31 UTC
So there was a mail from msuchy on devel@:

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/2WSPKXMAFDB7MUTLPJ6ORWQHHTY3LBJK/

which seemed to suggest that they didn't consider it possible to fix this at present.

To be clear, my working definition of a fix for this bug is: "an upgrade from Fedora 29 to Fedora 30 run with just 'dnf --releasever=30 system-upgrade download' and 'dnf --releasever=30 system-upgrade reboot' on a system with modules enabled works without errors and does appropriate things with the modules.

I've just talked to the folks at the Modularity hackfest - sgallagh, mboddu, ngompa, psabata, langdon etc. - and there was general agreement that we believe this needs to be "fixed" (per my definition above) - it would not be acceptable for the resolution here to be just "people need to pass a magic --setopt arg to make upgrades work".

So AIUI the next step from here is for modularity and DNF folks to work together and figure out a practical solution to this, whatever action that requires on both sides. Neal has a suggestion that it could possibly be solved by (AIUI) having fedora-release-common have a Provides: that indicates the platform ID associated with the release, so that is one avenue to consider.

For now I'm gonna say the expectation here is that this bug should get "fixed" (per my definition above) for F30 Beta, and I'll be checking in for updates from modularity and DNF folks. I'm taking the bug out of POST as it seems clear that the PR referred to above (https://github.com/rpm-software-management/dnf-plugins-extras/pull/143 ) does not actually fix the bug entirely. If anyone at some point thinks we've reached a position where "fixing" this bug per the definition above is not practical, please say so, as we'll need to reconsider things at that point.

Comment 32 Adam Williamson 2019-03-05 20:02:58 UTC
Also sgallagh notes that this bug is AcceptedBlocker for F30 Beta essentially 'by accident' because it had the tag from F29 review - it has not been formally reviewed and accepted as a blocker for F30. I'll make sure we re-consider it at the next blocker review meeting.

Comment 33 Fedora Update System 2019-03-13 13:41:48 UTC
dnf-plugins-core-4.0.6-1.fc29 libdnf-0.28.0-1.fc29 libcomps-0.1.11-1.fc29 createrepo_c-0.12.2-1.fc29 librepo-1.9.5-1.fc29 dnf-4.2.1-1.fc29 dnf-plugins-extras-4.0.4-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-a03be2cebe

Comment 34 Fedora Update System 2019-03-13 13:42:43 UTC
dnf-plugins-core-4.0.6-1.fc30 libdnf-0.28.0-1.fc30 libcomps-0.1.11-1.fc30 createrepo_c-0.12.2-1.fc30 librepo-1.9.5-1.fc30 dnf-4.2.1-1.fc30 dnf-plugins-extras-4.0.4-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 35 Pavla Kratochvilova 2019-03-13 13:52:36 UTC
I agree the "--setopt=module_platform_id=platform:f30" option is not a proper fix for this bug. However, I will make a new bug to track this issue.

As a workaround, dnf saves the module_platform_id, and option "--setopt=module_platform_id=platform:f30" is recommended for system upgrade. This is documented in system-upgrade ( https://github.com/rpm-software-management/dnf-plugins-extras/commit/7b7cc45c9d2d89b944f4256c3b0e590d9e72bf1b ), as well as in the Fedora wiki ( https://fedoraproject.org/wiki/DNF_system_upgrade ).

Comment 36 Stephen Gallagher 2019-03-13 14:06:20 UTC
(In reply to Pavla Kratochvilova from comment #35)
> I agree the "--setopt=module_platform_id=platform:f30" option is not a
> proper fix for this bug. However, I will make a new bug to track this issue.
> 
> As a workaround, dnf saves the module_platform_id, and option
> "--setopt=module_platform_id=platform:f30" is recommended for system
> upgrade. This is documented in system-upgrade (
> https://github.com/rpm-software-management/dnf-plugins-extras/commit/
> 7b7cc45c9d2d89b944f4256c3b0e590d9e72bf1b ), as well as in the Fedora wiki (
> https://fedoraproject.org/wiki/DNF_system_upgrade ).

We (the Modularity WG) discussed this at our Modularity Hackfest last week. We agreed that this workaround is *not* sufficient to clear this blocker. The solution for F30 Beta must not include additional flags beyond what users would have expected for F27->F28 or F28->F29.

Comment 37 Fedora Update System 2019-03-13 15:37:01 UTC
createrepo_c-0.12.2-1.fc30, dnf-4.2.1-1.fc30, dnf-plugins-core-4.0.6-1.fc30, dnf-plugins-extras-4.0.4-1.fc30, libcomps-0.1.11-1.fc30, libdnf-0.28.0-1.fc30, librepo-1.9.5-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 38 Jaroslav Mracek 2019-03-13 15:38:10 UTC
Thanks Stephen Gallagher for information about agreements made in Boston last week. I also considered the behavior as a problematic. But the issue presented by the reporter is solved therefore additional request should be handled by a new bug report.

To resolve the issue completely we need from modularity team:
1. Drop a module platform requirement from yaml (metadata)
2. Or mark requirement of the platform module specifically
3. Or define exactly a format of module platform ID
4. Or provide a module platform provide that could be used for system-upgrade (metadata approach)
5. Or as a partial fix do not require a platform stream

Please could you unblock the solution?

Comment 39 Stephen Gallagher 2019-03-13 15:45:04 UTC
I'll leave it up to Adam Williamson whether he considers this resolution sufficient for this bug and wants to open a new bug to track fixing this without the --setopt hack.

(In reply to Jaroslav Mracek from comment #38)
> To resolve the issue completely we need from modularity team:
> 1. Drop a module platform requirement from yaml (metadata)

This is unlikely at best.

> 2. Or mark requirement of the platform module specifically

I don't know what this means or how it differs from the current situation.

> 3. Or define exactly a format of module platform ID

Does this mean "set a policy on what format the platform ID is allowed to take, so we can assume that `platform:fN+1` is always valid"?

> 4. Or provide a module platform provide that could be used for
> system-upgrade (metadata approach)

Meaning have `fedora-release-common` include a virtual `Provides: platform_id(f30)` or similar?

> 5. Or as a partial fix do not require a platform stream
> 

I'm not sure what this means. Trick the solver to ignore the platform stream when solving?

> Please could you unblock the solution?

Could you describe each of these in greater detail?

Comment 40 Adam Williamson 2019-03-13 17:29:49 UTC
"I'll leave it up to Adam Williamson whether he considers this resolution sufficient for this bug and wants to open a new bug to track fixing this without the --setopt hack."

No, we explicitly do not consider that resolution sufficient. It was already decided that this needs to work without the hack, and this bug was filed on the user experience of the upgrade not working when run as documented. I am going to edit the update to *not* mark this as fixed.

(FWIW I am rather unclear as to what the update is supposed to "fix" at all, because AIUI, explicitly passing --setopt worked already. So what has the update actually changed in regards to this bug at all?)

Comment 41 Adam Williamson 2019-03-13 17:34:47 UTC
Actually, now I re-read through the entire history of the bug, it's a bit more long and confused and may have covered different issues at different times. It'd probably be OK to file a new bug for the specific issue we still consider a problem (to need to manually set --setopt), but I'd still like to know what it is the update actually *fixes* so we can change the description of this bug to match that.

Is it that, without that patch, using `system-upgrade` even with the explicit `--setopt` doesn't work? Only using `distro-sync` works?

Comment 42 Pavla Kratochvilova 2019-03-13 18:43:47 UTC
Yes, without the patch, module_platform_id option is not saved through the upgrade, so even explicit --setopt doesn't work.

Comment 43 Adam Williamson 2019-03-13 19:02:31 UTC
OK, thanks. I will update the summary to make it clear what we're claiming to fix with the update.

Comment 44 Adam Williamson 2019-03-13 19:09:01 UTC
I have filed https://bugzilla.redhat.com/show_bug.cgi?id=1688462 to cover the 'we shouldn't require explicit --setopt' case.

BTW, should the fix for this also go to F28, so F28 - F30 upgrades work?

Comment 45 Kalev Lember 2019-03-13 21:17:03 UTC
F28 afaik didn't ship any modular repos (at least not for Workstation) so it shouldn't be necessary to fix this in F28. Or did Server ship with modular repos enabled?

Comment 46 Petr Pisar 2019-03-14 09:01:14 UTC
Maybe it's time for specifying an upgrade path between module streams. platform:f30 is unsupported in Fedora 31. Thus Fedora 31 modular repository should specify that platform:f31 replaces platform:f30. With this metadata available DNF could switch the stream automatically on system upgrade. Something like Obsoletes in RPM. This feature should be available for all modules. Not only plaform. Because other modules' streams are also subjects of becoming end-of-life.

Comment 47 Adam Williamson 2019-03-14 14:28:46 UTC
Petr: I believe we should have that discussion in the new bug, now: https://bugzilla.redhat.com/show_bug.cgi?id=1688462 .

Comment 48 Adam Williamson 2019-03-14 14:30:51 UTC
Kalev: Yes, there were modules in F28, IIRC. See https://bugzilla.redhat.com/show_bug.cgi?id=1537253 , the tracker bug for the F28 feature, https://fedoraproject.org/wiki/Changes/F28AddonModularity .

I was also considering the case where a default module stream appears in the 'target' release for a package that was installed in the 'source' release. I'm not sure if this bug is a problem for that case, though.

Comment 49 Adam Williamson 2019-03-14 16:42:58 UTC
https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc is marked as fixing this, but that's an F30 update. Is that correct, or to fix F29 -> F30 upgrades does the fix for this actually need to go to F29? Or to both?

Comment 50 Fedora Update System 2019-03-14 18:51:32 UTC
createrepo_c-0.12.2-1.fc29, dnf-4.2.1-1.fc29, dnf-plugins-core-4.0.6-1.fc29, dnf-plugins-extras-4.0.4-1.fc29, libcomps-0.1.11-1.fc29, libdnf-0.28.0-1.fc29, librepo-1.9.5-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-a03be2cebe

Comment 51 Fedora Update System 2019-03-15 13:09:34 UTC
createrepo_c-0.12.2-1.fc30 dnf-4.2.1-1.fc30 dnf-plugins-core-4.0.6-1.fc30 dnf-plugins-extras-4.0.4-1.fc30 libcomps-0.1.11-1.fc30 libdnf-0.28.0-1.fc30 librepo-1.9.5-1.fc30 swid-tools-0.7.3-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 52 František Zatloukal 2019-03-15 14:45:33 UTC
Hmm, I am getting "REDHAT_SUPPORT_PRODUCT_VERSION=30" after upgrade even with dnf-4.1.0-1 (without the https://bodhi.fedoraproject.org/updates/FEDORA-2019-a03be2cebe update).

I' ve upgraded fully updated F29 to F30 with:

sudo dnf system-upgrade download --refresh --releasever=30 --setopt='module_platform_id=platform:f30'

Comment 53 Fedora Update System 2019-03-16 20:17:31 UTC
createrepo_c-0.12.2-1.fc30, dnf-4.2.1-1.fc30, dnf-plugins-core-4.0.6-1.fc30, dnf-plugins-extras-4.0.4-1.fc30, libcomps-0.1.11-1.fc30, libdnf-0.28.0-1.fc30, librepo-1.9.5-1.fc30, swid-tools-0.7.3-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-bc9607a8dc

Comment 54 Pavla Kratochvilova 2019-03-18 10:52:10 UTC
Adam, the bug is both in Fedora 29 and 30, so the fix goes to both of the updates.

Fedora 28 is also affected, but it contains much older version of dnf and there is no simple way to backport the fix. A workaround is to install dnf from the copr repo (rpmsoftwaremanagement/dnf-nightly).

Comment 55 Adam Williamson 2019-03-18 14:16:59 UTC
Pavla: to be clear, which *end* of the transaction does dnf need to be updated on to fix the bug? The original release or the 'target' release, or both?

It would be best to uncheck the 'Close bugs on stable?' box on the updates if this is complicated, because we only want this bug closed once the issue is fully resolved. We don't want it closed when just one of them goes stable, if both of them need to go stable for it to be fully fixed.

Comment 56 Adam Williamson 2019-03-18 18:25:19 UTC
Pavla: also, in any case, both updates have substantial amounts of positive feedback now. Would you be opposed to pushing both to stable at this point, or can we go ahead and do that?

Comment 57 Pavla Kratochvilova 2019-03-19 07:43:42 UTC
The original release is the one requiring the fix. Anyway, I pushed both updates to stable now.

Comment 58 Jaroslav Mracek 2019-03-19 12:57:33 UTC
*** Bug 1686059 has been marked as a duplicate of this bug. ***

Comment 59 Fedora Update System 2019-03-19 19:38:39 UTC
createrepo_c-0.12.2-1.fc30, dnf-4.2.1-1.fc30, dnf-plugins-core-4.0.6-1.fc30, dnf-plugins-extras-4.0.4-1.fc30, libcomps-0.1.11-1.fc30, libdnf-0.28.0-1.fc30, librepo-1.9.5-1.fc30, swid-tools-0.7.3-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 60 Pavla Kratochvilova 2019-03-20 06:06:46 UTC
*** Bug 1685455 has been marked as a duplicate of this bug. ***

Comment 61 Fedora Update System 2019-03-20 22:13:01 UTC
createrepo_c-0.12.2-1.fc29, dnf-4.2.1-1.fc29, dnf-plugins-core-4.0.6-1.fc29, dnf-plugins-extras-4.0.4-1.fc29, libcomps-0.1.11-1.fc29, libdnf-0.28.0-1.fc29, librepo-1.9.5-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 62 Fedora Update System 2019-03-21 14:40:38 UTC
createrepo_c-0.12.2-1.fc29, dnf-4.2.1-1.fc29, dnf-plugins-core-4.0.6-1.fc29, dnf-plugins-extras-4.0.4-1.fc29, libcomps-0.1.11-1.fc29, libdnf-0.28.0-1.fc29, librepo-1.9.5-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 63 Kostya Vasilyev 2019-04-03 18:43:24 UTC
Kind of same thing with Fedora 29 (+ updates + updates testing), not Rawhide.

----------------------

sudo dnf system-upgrade download --refresh --releasever=30
Before you continue ensure that your system is fully upgraded by running "dnf --refresh upgrade". Do you want to continue [y/N]: y
Fedora 30 openh264 (From Cisco) - x86_64                                      5.0 kB/s | 3.0 kB     00:00    
Fedora Modular 30 - x86_64                                                     20 kB/s |  19 kB     00:00    
Fedora Modular 30 - x86_64 - Updates                                           70 kB/s | 3.0 kB     00:00    
Fedora Modular 30 - x86_64 - Test Updates                                      49 kB/s | 2.4 kB     00:00    
Fedora 30 - x86_64 - Test Updates                                             104 kB/s | 4.7 kB     00:00    
Fedora 30 - x86_64 - Updates                                                   25 kB/s |  25 kB     00:00    
Fedora 30 - x86_64                                                             15 kB/s |  14 kB     00:00    
google-chrome                                                                  16 kB/s | 1.3 kB     00:00    
MongoDB Repository                                                            6.7 kB/s | 2.5 kB     00:00    
Scooter Software                                                              8.9 kB/s | 2.9 kB     00:00    
skype (stable)                                                                 14 kB/s | 2.9 kB     00:00    
Sublime Text - x86_64 - Dev                                                   6.2 kB/s | 2.9 kB     00:00    
Visual Studio Code                                                            8.4 kB/s | 2.9 kB     00:00    
Modular dependency problems:

 Problem 1: conflicting requests
  - nothing provides module(platform:f30) needed by module newsboat:latest:3020190307162417:a5b0195c-0.x86_64
 Problem 2: conflicting requests
  - nothing provides module(platform:f30) needed by module newsboat:latest:3020190325084033:a5b0195c-0.x86_64
 Problem 3: conflicting requests
  - nothing provides module(platform:f30) needed by module avocado:stable:3020190304180315:a5b0195c-0.x86_64
 Problem 4: conflicting requests
  - nothing provides module(platform:f30) needed by module bat:latest:3020190307100850:e50d0d19-0.x86_64
 Problem 5: conflicting requests
  - nothing provides module(platform:f30) needed by module dwm:6.1:3020190304180429:a5b0195c-0.x86_64
 Problem 6: conflicting requests
  - nothing provides module(platform:f30) needed by module exa:latest:3020190306064823:e50d0d19-0.x86_64
 Problem 7: conflicting requests
  - nothing provides module(platform:f30) needed by module ffsend:latest:3020190317224628:a5b0195c-0.x86_64
 Problem 8: conflicting requests
  - nothing provides module(platform:f30) needed by module fish:3:3020190301191132:602da195-0.x86_64
 Problem 9: conflicting requests
  - nothing provides module(platform:f30) needed by module gimp:2.10:3020190304180601:a5b0195c-0.x86_64
 Problem 10: conflicting requests
  - nothing provides module(platform:f30) needed by module heatseeker:latest:3020190309110310:a5b0195c-0.x86_64
 Problem 11: conflicting requests
  - nothing provides module(platform:f30) needed by module hyperfine:latest:3020190318171218:a5b0195c-0.x86_64
 Problem 12: conflicting requests
  - nothing provides module(platform:f30) needed by module libgit2:0.27:3020190304180745:a5b0195c-0.x86_64
 Problem 13: conflicting requests
  - nothing provides module(platform:f30) needed by module meson:latest:3020190310183600:36245242-0.x86_64
 Problem 14: conflicting requests
  - nothing provides module(platform:f30) needed by module minetest:5:3020190308194723:a5b0195c-0.x86_64
 Problem 15: conflicting requests
  - nothing provides module(platform:f30) needed by module ninja:latest:3020190304180949:a5b0195c-0.x86_64
 Problem 16: conflicting requests
  - nothing provides module(platform:f30) needed by module ripgrep:latest:3020190306064851:a5b0195c-0.x86_64
 Problem 17: conflicting requests
  - nothing provides module(platform:f30) needed by module rpick:latest:3020190313083345:a5b0195c-0.x86_64
 Problem 18: conflicting requests
  - nothing provides module(platform:f30) needed by module standard-test-roles:3.0:3020190319161255:a5b0195c-0.x86_64
 Problem 19: conflicting requests
  - nothing provides module(platform:f30) needed by module stratis:1:3020190306064421:a5b0195c-0.x86_64
Error: 
 Problem: problem with installed package tortoisehg-4.6.1-1.fc29.noarch
  - package tortoisehg-4.6.1-2.fc30.noarch requires mercurial < 4.7, but none of the providers can be installed
  - tortoisehg-4.6.1-1.fc29.noarch does not belong to a distupgrade repository
  - mercurial-4.5.3-1.fc29.x86_64 does not belong to a distupgrade repository
(try to add '--skip-broken' to skip uninstallable packages)


----------------

Comment 64 Adam Williamson 2019-04-03 18:58:05 UTC
That is https://bugzilla.redhat.com/show_bug.cgi?id=1688462 , not this bug.

Comment 65 RobbieTheK 2019-05-03 18:23:00 UTC
From Fedora 29 --> 30 on i.686 I got this:
Error:
 Problem: package fedora-release-server-30-1.noarch conflicts with system-release provided by fedora-release-workstation-30-1.noarch
  - package fedora-release-workstation-30-1.noarch conflicts with system-release provided by fedora-release-server-30-1.noarch
  - problem with installed package fedora-release-workstation-29-10.noarch
  - problem with installed package fedora-release-server-29-10.noarch
  - fedora-release-workstation-29-10.noarch does not belong to a distupgrade repository
  - fedora-release-server-29-10.noarch does not belong to a distupgrade repository
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

Comment 66 Samuel Sieb 2019-05-03 18:47:31 UTC
That is unrelated to this issue.  Somehow you have two different release files installed.  Remove either the workstation one or the server one before doing the system-upgrade.


Note You need to log in before you can comment on or make changes to this bug.