The modular packages from the eclipse module has overridden nonmodular packages on Fedra 31 installations, providing packages with stripped features. This happened implicitly on a regular "dnf upgrade". Nonmodular maintainers were not consulted about this. One of such packages is protobuf. See: https://pagure.io/fesco/issue/2285 and https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/WKUK6YSLSELO2VK55EIAKPW4ZJIWSJGO/ This bugzilla is opened to figure a way to migrate the affected users back to the nonmodular content without requiring them to do an explicit action. After all they were migrated to the modular content without an explicit action.
Changed to distribution because: - the eclipse module probably cannot fix this from within - I cannot use the prioritized bug thing on modular bug Proposing as prioritized bug. I think this is pretty serious thing and since this happened on a stable Fedora, there is no blocker to propose here.
As undesirable as the Epoch value is, could the non-modular protobuf package increment its Epoch to override the modular one from eclipse? That could restore the functional protobuf package and then free the system for manual removal of modular RPMs should the user want that? The protobuf package in the eclipse module is rpmvercmp'ing higher than the regular protobuf package. Was protobuf the only affected package? Is it really rpmvercmp'ing higher or do module streams get a different kind of priority or cost in dnf depsolving?
If this is only about EVR, protobuf can simply rise the release above 6. But is it? Once the modular protobuf was instated will it be upgraded to nonmodular package simply because it has higher EVR?
That I do not know. I do not understand how module streams (which in my head are effectively repos) work with regards to depsolving. If they introduce a repo with a higher repo cost, then I would think it would win the depsolving algorithm depsite having a lower NVR. Incrementing the epoch may not impact that either, but would would if a list of packages were just handed to RPM. We should discuss this in the next FESCo meeting. Default streams are introducing unexpected behavior and leaving systems in a broken state. Even if default streams are left alone, we now have an unbounded number of repos that can deliver builds for non-modular RPMs. How are bugs reported for those packages because as we have seen with protobuf, the protobuf maintainer was unaware that the eclipse module was bundling a different protobuf. Users wouldn't know to report bugs to the module for that since it is not easily discoverable where a package originated. All that aside, I am still curious why eclipse bundled a less functional protobuf build that still met DT_NEEDED depsolving in RPM. Why did eclipse not work with the non-modular protobuf package maintainer to ensure the right functionality was there for eclipse? Or did they and it wasn't something that could be solved? What I would like to discuss: 1) Whether or not Fedora should have default streams for modules 2) Module bundling policies for dependent packages that conflict with the non-modular system 3) Bug reporting and bug ownership process for bundled packages in modules 4) Testing and review policies for modules 5) Probably more
> As undesirable as the Epoch value is, could the non-modular protobuf package increment its Epoch to override the modular one from eclipse? No. My understanding was that Epoch does not matter, and rpms from the module always win with the non-modular rpms.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #5) > > As undesirable as the Epoch value is, could the non-modular protobuf package increment its Epoch to override the modular one from eclipse? > > No. My understanding was that Epoch does not matter, and rpms from the > module always win with the non-modular rpms. From a technical standpoint, I view this as violating long standing established and expected behavior of both RPM and yum/dnf.
While I agree with that statement 100%, let's not hijack this bugzilla or this obviously higher level discussion? I suggest we take that either to the FESCo ticket or even better, devel mailing list. And wee keep this bugzilla open to find a solution to this particular problem. While I proposed default modular streams gone, even when we disable all of them, we will need to clean the mess and create a way to migrate users back. That would, however possibly happen on a Fedora release boundary. This problem here is special, because it needs to be fixed within the stable release.
s/or this/for this/
Fair point, I was wanting to write down thoughts before I lost them and I'm back and forth between my desk and other things today. Apologies. Langdon posted on devel how to revert to the non-modular protobuf. I have not tried it myself, but plan to tomorrow. I would like to know if there are packages besides protobuf at the moment that have had the same thing happen just so we don't miss any corner cases.
Also, on Friday I figured out how to revert my system on Friday and posted the clumsy way I did it on my blog. I would like it if Langdon's method works because it's certainly cleaner in terms of command syntax.
Looking at https://src.fedoraproject.org/modules/eclipse/blob/4e3df23f791fd94779eb299ef4cdfb93881aacb9/f/eclipse.yaml This is the list of components that were not mentioned in the FESCo ticket: aether-connector-okhttp * antlr32 apache-commons-collections apache-commons-compress apache-commons-discovery apache-commons-el apache-commons-jxpath apache-commons-lang apache-sshd args4j base64coder batik bean-validation-api beust-jcommander bouncycastle closure-compiler docker-client-java ed25519-java felix-gogo-command felix-gogo-runtime felix-gogo-shell felix-scr freemarker glassfish-annotation-api glassfish-el glassfish-hk2 glassfish-jax-rs-api glassfish-jaxb glassfish-jaxb-api glassfish-jaxrpc-api glassfish-jsp glassfish-jsp-api glassfish-servlet-api google-gson icu4j jackson-annotations jackson-core jackson-databind jackson-datatypes-collections jackson-jaxrs-providers jackson-modules-base jakarta-commons-httpclient java_cup javaewah javaparser javassist jchardet * jctools jersey jetty jffi jgit jline jna jnr-constants jnr-enxio jnr-ffi jnr-netdb jnr-posix jnr-unixsocket jnr-x86asm jsch-agent-proxy json_simple jython kxml lpg lucene maven-archetype * maven-archiver maven-filtering maven-indexer * maven-invoker maven-mapping maven-osgi * maven-war-plugin nekohtml netty objectweb-asm okhttp okio * plexus-archiver plexus-build-api plexus-io plexus-velocity protobuf rhino rxtx sac sat4j sequence-library snakeyaml sqljet stringtemplate svnkit swt-chart testng testng-remote trilead-ssh2 uddi4j velocity ws-commons-util wsdl4j wsil4j xmlgraphics-commons xmlrpc xpp3 Most of the packages are available in nonmodular Fedora 31. Except the ones listed with *.
Accepted as a Prioritized Bug. Default streams should not override the base OS. https://meetbot.fedoraproject.org/fedora-meeting/2019-12-18/fedora_prioritized_bugs_and_issues.2019-12-18-16.06.log.html#l-29
This has been a prioritized bug for 6 weeks without a reply. Is anybody working on this?
Reassigning to the new development team lead for this.
Reducing priority to high, because the issue has been there for a while and it doesn't block the release. We need to organize a meeting with Miro and Java packagers, analyze the situation and propose solution to F33+. We cannot fix much in the existing releases because it would require changes in how DNF handles package filtering and any change could cause a regression. Expected result from the meeting: a proposal for packaging guidelines change incl. explaining the problem and reason for having any policies in place
I respectfully disagree with everything that you have just said. This is affecting a released Fedora version, so we cannot block on it. It was reported immediately after it happened and deserves an immediate fix. It has been accepted as a Fedora Prioritized Bug and it was ignored by the modularity maintainers for month and a half. This being ignored is not a proper reason to lower the priority, it's a reason to escalate it. This needs to be fixed ASAP (read: within days) in Fedora 31, not Fedora 33+. All users who were migrated to modular protobuf and possibly other packages need to be migrated back within the stable release. Not in a month, not in a week, today. Even if we "screw up" users who explicitly opted in for the eclipse module by disabling this module on "dnf upgrade" for all. Is this fix distributive? Yes it is. So is the problem.
I would propose a following solution. Because eclipse module is not default any more , the problem could be resolved on distribution by resetting the module. It can be performed by renaming /etc/dnf/module.d/eclipse.module to something like `/etc/dnf/module.d/eclipse.module.deleted` using rpm scriplets (in fedora-release?). The code could look like (python example): if not os.path.isfile('/etc/dnf/module.d/eclipse.module.deleted') and os.path.isfile('/etc/dnf/module.d/eclipse.module`): # test presence of files os.rename('/etc/dnf/module.d/eclipse.module', '/etc/dnf/module.d/eclipse.module.deleted') Can you accept the solution?
Not sure how /etc/dnf/modules.d/ (not the s!) files work. Could you explain what happens once the file is removed (renamed)?
> (not the s!) I meant to write "notice"
I am sorry, I provided incorrect path. Correct path is /etc/dnf/modules.d/eclipse.module Information about state of module or installed profiles is stored in "<installroot>/etc/dnf/modules.d/<module_name>.module". When you remove or rename the file using a different suffix than .module, dnf will not remember the state of module and installed profiles. if not os.path.isfile('/etc/dnf/modules.d/eclipse.module.deleted'): if os.path.isfile('/etc/dnf/modules.d/eclipse.module`): # test presence of files os.rename('/etc/dnf/modules.d/eclipse.module', '/etc/dnf/modules.d/eclipse.module.deleted') else: # ensure dir '/etc/dnf/modules.d/' would be also nice with open("/etc/dnf/modules.d/eclipse.module.deleted", 'a'): # to ensure to run script only once pass Did I provide requested information?
You did. So something like: %posttrans -p <lua> -- Migrate users affected by rhbz#1780827 away from the eclipse module -- But only do it once original = "%{_sysconfdir}/dnf/modules.d/eclipse.module" moved = original .. ".rpmmoved" if posix.stat(original) and not posix.stat(moved) os.rename(original, moved) end I'll test that.
This gets the job done for eclipse module: %posttrans common -p <lua> -- Migrate users affected by rhbz#1780827 away from the eclipse module -- But only do it once original = "%{_sysconfdir}/dnf/modules.d/eclipse.module" moved = original .. ".rpmmoved" if not posix.stat(moved) then if posix.stat(original) then os.rename(original, moved) else io.open(moved, "w"):close() end end However, enabling the eclipse module also enables ant and maven. Hence this needs to be done for all 3 of them :(
%posttrans common -p <lua> -- Migrate users affected by rhbz#1780827 away from the eclipse, ant and maven modules -- But only do it once modules = {"eclipse", "ant", "maven"} for i, module in ipairs(modules) do original = "%{_sysconfdir}/dnf/modules.d/" .. module .. ".module" moved = original .. ".rpmmoved" if not posix.stat(moved) then if posix.stat(original) then os.rename(original, moved) else io.open(moved, "w"):close() end end end $ sudo dnf module enable eclipse:latest Enabling module streams: ant 1.10 eclipse latest maven 3.5 $ sudo dnf module list --enabled Name Stream ant 1.10 [d][e] eclipse latest [e] maven 3.5 [d][e] ripgrep latest [d][e] $ sudo dnf install ./results_fedora-release/31/3/fedora-release{,-common}-31-3.noarch.rpm Updated: fedora-release-31-3.noarch fedora-release-common-31-3.noarch $ sudo dnf module list --enabled Name Stream ripgrep latest [d][e] $ sudo dnf module enable eclipse:latest Enabling module streams: ant 1.10 eclipse latest maven 3.5 $ sudo dnf reinstall ./results_fedora-release/31/3/fedora-release{,-common}-31-3.noarch.rpm Reinstalled: fedora-release-31-3.noarch fedora-release-common-31-3.noarch $ sudo dnf module list --enabled Name Stream ant 1.10 [d][e] eclipse latest [e] maven 3.5 [d][e] ripgrep latest [d][e] $ sudo rm -v /etc/dnf/modules.d/*.rpmmoved removed '/etc/dnf/modules.d/ant.module.rpmmoved' removed '/etc/dnf/modules.d/eclipse.module.rpmmoved' removed '/etc/dnf/modules.d/maven.module.rpmmoved' $ sudo dnf reinstall ./results_fedora-release/31/3/fedora-release{,-common}-31-3.noarch.rpm Reinstalled: fedora-release-31-3.noarch fedora-release-common-31-3.noarch $ sudo dnf module list --enabled Name Stream ripgrep latest [d][e]
Asking FESCo to decide this one: https://pagure.io/fesco/issue/2356
Resetting the ant/maven module will have no practical effect. Users who do have packages from them will be re-enabled on next transaction. Users who don't have packages installed from them can have them enabled until they upgrade to Fedora 32. So I'll amend the fesco ticket to only ask for what is proposed in https://bugzilla.redhat.com/show_bug.cgi?id=1780827#c23
https://src.fedoraproject.org/rpms/fedora-release/pull-request/110
FEDORA-2020-af8b8d4da8 has been pushed to the Fedora 31 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-af8b8d4da8` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-af8b8d4da8 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-af8b8d4da8 has been pushed to the Fedora 31 stable repository. If problem still persists, please make note of it in this bug report.
Following up on this, is there a reason why the scriptlet is implemented in lua rather than sh? rpm-ostree today doesn't implement lua: https://github.com/coreos/rpm-ostree/issues/749. And while we can override this if necessary, it'd be great if we didn't have to.
The reason was that RPM has it's own lua, so going with lua seemed like the most natural thing to do.