Bug 1780827 - Migrate Fedora 31 users back to nonmodular content overridden by the eclipse module
Summary: Migrate Fedora 31 users back to nonmodular content overridden by the eclipse ...
Keywords:
Status: ON_QA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 31
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1805260
TreeView+ depends on / blocked
 
Reported: 2019-12-07 11:34 UTC by Miro Hrončok
Modified: 2020-03-24 09:40 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Bug
bcotton: fedora_prioritized_bug+


Attachments (Terms of Use)

Description Miro Hrončok 2019-12-07 11:34:35 UTC
The modular packages from the eclipse module has overridden nonmodular packages on Fedra 31 installations, providing packages with stripped features. This happened implicitly on a regular "dnf upgrade". Nonmodular maintainers were not consulted about this.

One of such packages is protobuf. See:

https://pagure.io/fesco/issue/2285 and https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/WKUK6YSLSELO2VK55EIAKPW4ZJIWSJGO/


This bugzilla is opened to figure a way to migrate the affected users back to the nonmodular content without requiring them to do an explicit action. After all they were migrated to the modular content without an explicit action.

Comment 1 Miro Hrončok 2019-12-07 11:38:36 UTC
Changed to distribution because:

- the eclipse module probably cannot fix this from within
- I cannot use the prioritized bug thing on modular bug

Proposing as prioritized bug. I think this is pretty serious thing and since this happened on a stable Fedora, there is no blocker to propose here.

Comment 2 David Cantrell 2019-12-07 14:58:45 UTC
As undesirable as the Epoch value is, could the non-modular protobuf package increment its Epoch to override the modular one from eclipse?  That could restore the functional protobuf package and then free the system for manual removal of modular RPMs should the user want that?  The protobuf package in the eclipse module is rpmvercmp'ing higher than the regular protobuf package.

Was protobuf the only affected package?  Is it really rpmvercmp'ing higher or do module streams get a different kind of priority or cost in dnf depsolving?

Comment 3 Miro Hrončok 2019-12-07 22:09:35 UTC
If this is only about EVR, protobuf can simply rise the release above 6. But is it? Once the modular protobuf was instated will it be upgraded to nonmodular package simply because it has higher EVR?

Comment 4 David Cantrell 2019-12-08 15:40:33 UTC
That I do not know.  I do not understand how module streams (which in my head are effectively repos) work with regards to depsolving.  If they introduce a repo with a higher repo cost, then I would think it would win the depsolving algorithm depsite having a lower NVR.  Incrementing the epoch may not impact that either, but would would if a list of packages were just handed to RPM.

We should discuss this in the next FESCo meeting.  Default streams are introducing unexpected behavior and leaving systems in a broken state.  Even if default streams are left alone, we now have an unbounded number of repos that can deliver builds for non-modular RPMs.  How are bugs reported for those packages because as we have seen with protobuf, the protobuf maintainer was unaware that the eclipse module was bundling a different protobuf.  Users wouldn't know to report bugs to the module for that since it is not easily discoverable where a package originated.  All that aside, I am still curious why eclipse bundled a less functional protobuf build that still met DT_NEEDED depsolving in RPM.  Why did eclipse not work with the non-modular protobuf package maintainer to ensure the right functionality was there for eclipse?  Or did they and it wasn't something that could be solved?

What I would like to discuss:
1) Whether or not Fedora should have default streams for modules
2) Module bundling policies for dependent packages that conflict with the non-modular system
3) Bug reporting and bug ownership process for bundled packages in modules
4) Testing and review policies for modules
5) Probably more

Comment 5 Zbigniew Jędrzejewski-Szmek 2019-12-08 20:02:03 UTC
> As undesirable as the Epoch value is, could the non-modular protobuf package increment its Epoch to override the modular one from eclipse?

No. My understanding was that Epoch does not matter, and rpms from the module always win with the non-modular rpms.

Comment 6 David Cantrell 2019-12-08 20:07:49 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5)
> > As undesirable as the Epoch value is, could the non-modular protobuf package increment its Epoch to override the modular one from eclipse?
> 
> No. My understanding was that Epoch does not matter, and rpms from the
> module always win with the non-modular rpms.

From a technical standpoint, I view this as violating long standing established and expected behavior of both RPM and yum/dnf.

Comment 7 Miro Hrončok 2019-12-08 20:30:40 UTC
While I agree with that statement 100%, let's not hijack this bugzilla or this obviously higher level discussion?

I suggest we take that either to the FESCo ticket or even better, devel mailing list. And wee keep this bugzilla open to find a solution to this particular problem. While I proposed default modular streams gone, even when we disable all of them, we will need to clean the mess and create a way to migrate users back. That would, however possibly happen on a Fedora release boundary. This problem here is special, because it needs to be fixed within the stable release.

Comment 8 Miro Hrončok 2019-12-08 20:34:03 UTC
s/or this/for this/

Comment 9 David Cantrell 2019-12-08 20:45:47 UTC
Fair point, I was wanting to write down thoughts before I lost them and I'm back and forth between my desk and other things today.  Apologies.

Langdon posted on devel how to revert to the non-modular protobuf.  I have not tried it myself, but plan to tomorrow.  I would like to know if there are packages besides protobuf at the moment that have had the same thing happen just so we don't miss any corner cases.

Comment 10 David Cantrell 2019-12-08 20:48:00 UTC
Also, on Friday I figured out how to revert my system on Friday and posted the clumsy way I did it on my blog.  I would like it if Langdon's method works because it's certainly cleaner in terms of command syntax.

Comment 11 Miro Hrončok 2019-12-09 17:36:31 UTC
Looking at https://src.fedoraproject.org/modules/eclipse/blob/4e3df23f791fd94779eb299ef4cdfb93881aacb9/f/eclipse.yaml



This is the list of components that were not mentioned in the FESCo ticket:

aether-connector-okhttp *
antlr32
apache-commons-collections
apache-commons-compress
apache-commons-discovery
apache-commons-el
apache-commons-jxpath
apache-commons-lang
apache-sshd
args4j
base64coder
batik
bean-validation-api
beust-jcommander
bouncycastle
closure-compiler
docker-client-java
ed25519-java
felix-gogo-command
felix-gogo-runtime
felix-gogo-shell
felix-scr
freemarker
glassfish-annotation-api
glassfish-el
glassfish-hk2
glassfish-jax-rs-api
glassfish-jaxb
glassfish-jaxb-api
glassfish-jaxrpc-api
glassfish-jsp
glassfish-jsp-api
glassfish-servlet-api
google-gson
icu4j
jackson-annotations
jackson-core
jackson-databind
jackson-datatypes-collections
jackson-jaxrs-providers
jackson-modules-base
jakarta-commons-httpclient
java_cup
javaewah
javaparser
javassist
jchardet *
jctools
jersey
jetty
jffi
jgit
jline
jna
jnr-constants
jnr-enxio
jnr-ffi
jnr-netdb
jnr-posix
jnr-unixsocket
jnr-x86asm
jsch-agent-proxy
json_simple
jython
kxml
lpg
lucene
maven-archetype *
maven-archiver
maven-filtering
maven-indexer *
maven-invoker
maven-mapping
maven-osgi *
maven-war-plugin
nekohtml
netty
objectweb-asm
okhttp
okio *
plexus-archiver
plexus-build-api
plexus-io
plexus-velocity
protobuf
rhino
rxtx
sac
sat4j
sequence-library
snakeyaml
sqljet
stringtemplate
svnkit
swt-chart
testng
testng-remote
trilead-ssh2
uddi4j
velocity
ws-commons-util
wsdl4j
wsil4j
xmlgraphics-commons
xmlrpc
xpp3


Most of the packages are available in nonmodular Fedora 31. Except the ones listed with *.

Comment 12 Ben Cotton 2019-12-18 16:21:51 UTC
Accepted as a Prioritized Bug. Default streams should not override the base OS.
https://meetbot.fedoraproject.org/fedora-meeting/2019-12-18/fedora_prioritized_bugs_and_issues.2019-12-18-16.06.log.html#l-29

Comment 13 Miro Hrončok 2020-01-29 09:12:09 UTC
This has been a prioritized bug for 6 weeks without a reply. Is anybody working on this?

Comment 14 Ben Cotton 2020-01-29 12:39:58 UTC
Reassigning to the new development team lead for this.

Comment 15 Daniel Mach 2020-02-17 12:29:55 UTC
Reducing priority to high, because the issue has been there for a while and it doesn't block the release.

We need to organize a meeting with Miro and Java packagers, analyze the situation and propose solution to F33+.
We cannot fix much in the existing releases because it would require changes in how DNF handles package filtering
and any change could cause a regression.

Expected result from the meeting: a proposal for packaging guidelines change incl. explaining the problem and reason for having any policies in place

Comment 16 Miro Hrončok 2020-02-17 15:53:16 UTC
I respectfully disagree with everything that you have just said.

This is affecting a released Fedora version, so we cannot block on it. It was reported immediately after it happened and deserves an immediate fix. It has been accepted as a Fedora Prioritized Bug and it was ignored by the modularity maintainers for month and a half. This being ignored is not a proper reason to lower the priority, it's a reason to escalate it.

This needs to be fixed ASAP (read: within days) in Fedora 31, not Fedora 33+. All users who were migrated to modular protobuf and possibly other packages need to be migrated back within the stable release. Not in a month, not in a week, today. Even if we "screw up" users who explicitly opted in for the eclipse module by disabling this module on "dnf upgrade" for all. Is this fix distributive? Yes it is. So is the problem.

Comment 18 Jaroslav Mracek 2020-02-24 10:09:36 UTC
I would propose a following solution. Because eclipse module is not default any more , the problem could be resolved on distribution by resetting the module. It can be performed by renaming /etc/dnf/module.d/eclipse.module to something like `/etc/dnf/module.d/eclipse.module.deleted` using rpm scriplets (in fedora-release?). The code could look like (python example):

if not os.path.isfile('/etc/dnf/module.d/eclipse.module.deleted') and os.path.isfile('/etc/dnf/module.d/eclipse.module`): #  test presence of files
    os.rename('/etc/dnf/module.d/eclipse.module', '/etc/dnf/module.d/eclipse.module.deleted')

Can you accept the solution?

Comment 19 Miro Hrončok 2020-02-24 10:30:36 UTC
Not sure how /etc/dnf/modules.d/ (not the s!) files work. Could you explain what happens once the file is removed (renamed)?

Comment 20 Miro Hrončok 2020-02-24 10:32:43 UTC
> (not the s!)

I meant to write "notice"

Comment 21 Jaroslav Mracek 2020-02-24 13:18:01 UTC
I am sorry, I provided incorrect path. Correct path is /etc/dnf/modules.d/eclipse.module

Information about state of module or installed profiles is stored in "<installroot>/etc/dnf/modules.d/<module_name>.module". When you remove or rename the file using a different suffix than .module, dnf will not remember the state of module and installed profiles.

if not os.path.isfile('/etc/dnf/modules.d/eclipse.module.deleted'):
    if os.path.isfile('/etc/dnf/modules.d/eclipse.module`): #  test presence of files
        os.rename('/etc/dnf/modules.d/eclipse.module', '/etc/dnf/modules.d/eclipse.module.deleted')
    else:
        #  ensure dir '/etc/dnf/modules.d/' would be also nice
        with open("/etc/dnf/modules.d/eclipse.module.deleted", 'a'): #  to ensure to run script only once
            pass
        
Did I provide requested information?

Comment 22 Miro Hrončok 2020-02-24 21:38:06 UTC
You did. So something like:

%posttrans -p <lua>
-- Migrate users affected by rhbz#1780827 away from the eclipse module
-- But only do it once
original = "%{_sysconfdir}/dnf/modules.d/eclipse.module"
moved = original .. ".rpmmoved"
if posix.stat(original) and not posix.stat(moved)
  os.rename(original, moved)
end

I'll test that.

Comment 23 Miro Hrončok 2020-02-24 21:55:50 UTC
This gets the job done for eclipse module:

%posttrans common -p <lua>
-- Migrate users affected by rhbz#1780827 away from the eclipse module
-- But only do it once
original = "%{_sysconfdir}/dnf/modules.d/eclipse.module"
moved = original .. ".rpmmoved"
if not posix.stat(moved) then
  if posix.stat(original) then
    os.rename(original, moved)
  else
    io.open(moved, "w"):close()
  end
end


However, enabling the eclipse module also enables ant and maven. Hence this needs to be done for all 3 of them :(

Comment 24 Miro Hrončok 2020-03-05 16:02:37 UTC
%posttrans common -p <lua>
-- Migrate users affected by rhbz#1780827 away from the eclipse, ant and maven modules
-- But only do it once
modules = {"eclipse", "ant", "maven"}
for i, module in ipairs(modules) do
  original = "%{_sysconfdir}/dnf/modules.d/" .. module .. ".module"
  moved = original .. ".rpmmoved"
  if not posix.stat(moved) then
    if posix.stat(original) then
      os.rename(original, moved)
    else
      io.open(moved, "w"):close()
    end
  end
end



$ sudo dnf module enable eclipse:latest
Enabling module streams:
 ant     1.10
 eclipse latest
 maven   3.5

$ sudo dnf module list --enabled
Name                                    Stream
ant                                     1.10 [d][e]
eclipse                                 latest [e]
maven                                   3.5 [d][e]
ripgrep                                 latest [d][e]


$ sudo dnf install ./results_fedora-release/31/3/fedora-release{,-common}-31-3.noarch.rpm
Updated:
  fedora-release-31-3.noarch
  fedora-release-common-31-3.noarch

$ sudo dnf module list --enabled
Name                                       Stream
ripgrep                                    latest [d][e]

$ sudo dnf module enable eclipse:latest
Enabling module streams:
 ant     1.10
 eclipse latest
 maven   3.5

$ sudo dnf reinstall ./results_fedora-release/31/3/fedora-release{,-common}-31-3.noarch.rpm
Reinstalled:
  fedora-release-31-3.noarch
  fedora-release-common-31-3.noarch

$ sudo dnf module list --enabled
Name                                    Stream
ant                                     1.10 [d][e]
eclipse                                 latest [e]
maven                                   3.5 [d][e]
ripgrep                                 latest [d][e]

$ sudo rm -v /etc/dnf/modules.d/*.rpmmoved
removed '/etc/dnf/modules.d/ant.module.rpmmoved'
removed '/etc/dnf/modules.d/eclipse.module.rpmmoved'
removed '/etc/dnf/modules.d/maven.module.rpmmoved'

$ sudo dnf reinstall ./results_fedora-release/31/3/fedora-release{,-common}-31-3.noarch.rpm
Reinstalled:
  fedora-release-31-3.noarch
  fedora-release-common-31-3.noarch

$ sudo dnf module list --enabled
Name                                       Stream
ripgrep                                    latest [d][e]

Comment 25 Miro Hrončok 2020-03-11 17:10:05 UTC
Asking FESCo to decide this one: https://pagure.io/fesco/issue/2356

Comment 26 Miro Hrončok 2020-03-11 17:49:59 UTC
Resetting the ant/maven module will have no practical effect. Users who do have packages from them will be re-enabled on next transaction. Users who don't have packages installed from them can have them enabled until they upgrade to Fedora 32. So I'll amend the fesco ticket to only ask for what is proposed in https://bugzilla.redhat.com/show_bug.cgi?id=1780827#c23

Comment 27 Zbigniew Jędrzejewski-Szmek 2020-03-18 18:03:43 UTC
https://src.fedoraproject.org/rpms/fedora-release/pull-request/110

Comment 28 Fedora Update System 2020-03-24 09:40:26 UTC
FEDORA-2020-af8b8d4da8 has been pushed to the Fedora 31 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-af8b8d4da8`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-af8b8d4da8

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.


Note You need to log in before you can comment on or make changes to this bug.