Description of problem: Since glibc-2.35.9000-16, glibc has stopped being built with DT_HASH data in its ELF binaries. This was recently detected to break a number of games on Steam (mainly ones using Epic Games Easy Anti-Cheat software). It also breaks a number of other libraries (e.g. the libstrangle frame limiter library). Please restore the DT_HASH data to glibc's ELF binaries to restore compatibility. Version-Release number of selected component (if applicable): 2.36-4.fc37 How reproducible: Always Steps to Reproduce: 1. Install Steam 2. Install any EAC-enabled game (Blood Hunt, Elden Ring, MultiVersus, etc.) 3. Try to run the game Actual results: The game fails to load with error "Failed to load anti-cheat module" Expected results: The game loads successfully. Additional info: It is serious competitive disadvantage for Fedora Linux 37 if we cannot play major video games, especially as we are now starting to be taken seriously as platform for enthusiasts and gamers. This has been reported on by the Linux gaming press: https://www.gamingonlinux.com/2022/08/easy-anti-cheat-not-working-on-linux-seems-a-glibc-update-broke-it/ Arch already reverted the breakage: https://github.com/archlinux/svntogit-packages/commit/e1d69d80d07494e3c086ee2c5458594d5261d2e4 Gentoo added a USE flag to deal with it: https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=482fa96f4ed12c6c5fdb8c6aea17340136ecbdac
Pull requests proposed: * Rawhide: https://src.fedoraproject.org/rpms/glibc/pull-request/64 * F37: https://src.fedoraproject.org/rpms/glibc/pull-request/65
Proposed as a Freeze Exception for 37-final by Fedora user ngompa using the blocker tracking app because: Without this fixed, Fedora Linux 37 is basically broken as a gaming platform, which seriously hurts our competitiveness in the desktop Linux space.
Blocker review discussion ticket: https://pagure.io/fedora-qa/blocker-review/issue/920 > Fedora Linux 37 is basically broken as a gaming platform That's a bit exaggerated :) It affects just some games, mostly multiplayer (some of them extremely popular, yes). I found this link that should list those games (sort by Peak to see the popular ones on top): https://steamdb.info/tech/AntiCheat/EasyAntiCheat/
(In reply to Kamil Páral from comment #3) > Blocker review discussion ticket: > https://pagure.io/fedora-qa/blocker-review/issue/920 > > > Fedora Linux 37 is basically broken as a gaming platform > > That's a bit exaggerated :) It affects just some games, mostly multiplayer > (some of them extremely popular, yes). I found this link that should list > those games (sort by Peak to see the popular ones on top): > https://steamdb.info/tech/AntiCheat/EasyAntiCheat/ The reality is that once reviewers get their hands on Fedora Linux 37 and can't play *any* of those games, they'll declare Fedora Linux unsuitable for gaming. Given that we know how to fix it, it is completely irresponsible to allow that to happen.
+5 in https://pagure.io/fedora-qa/blocker-review/issue/920 , marking accepted FE.
Declared a blocker by FESCo: https://pagure.io/fesco/issue/2873
(In reply to Neal Gompa from comment #4) > (In reply to Kamil Páral from comment #3) > > Blocker review discussion ticket: > > https://pagure.io/fedora-qa/blocker-review/issue/920 > > > > > Fedora Linux 37 is basically broken as a gaming platform > > > > That's a bit exaggerated :) It affects just some games, mostly multiplayer > > (some of them extremely popular, yes). I found this link that should list > > those games (sort by Peak to see the popular ones on top): > > https://steamdb.info/tech/AntiCheat/EasyAntiCheat/ > > The reality is that once reviewers get their hands on Fedora Linux 37 and > can't play *any* of those games, they'll declare Fedora Linux unsuitable for > gaming. Given that we know how to fix it, it is completely irresponsible to > allow that to happen. Please note that for EPIC Games EAC, there is already a fix for this in EOS SDK 1.15.2 (August 2022) and the corresponding version of the anti-cheat module. My concern with this request is that this is not the only fix required to get games like Rogue Company working. For example you can see the reverts here: https://github.com/GloriousEggroll/glibc-eac-rc There are a half-dozen reverts listed there, and the DT_HASH revert is just one. The other changes have impact on security (if Intel CET ends up depending on clone3 then we need clone3 enabled) and in-place-upgrade stability (links were simplified to make this easier to support), and diagnostics (always having /usr/bin/ld.so so --list-diagnostics makes crash diagnostic tooling easier to write). I write this here just to make it clear that not all applications are fixed by this change, and that it is only one change in a series of changes spanning more than a year of development. That there are more than a years worth of changes being reviewed and reverted means that there is a disconnected between the Linux gaming ecosystem and the way Linux is developed. Upstream glibc tries very hard keep a stable ABI, and I think we succeed at that. The half-dozen reverted commits feels like a case of Hyrum's law. The Fedora glibc team can put DT_HASH back. Has anyone tested that this change fixes the games in question? Do we have someone who can confirm this?
Setting needinfo on Neal since Carlos has asked for comment in the PR. Setting to POST because a PR exists.
(In reply to Carlos O'Donell from comment #7) > (In reply to Neal Gompa from comment #4) > > (In reply to Kamil Páral from comment #3) > > > Blocker review discussion ticket: > > > https://pagure.io/fedora-qa/blocker-review/issue/920 > > > > > > > Fedora Linux 37 is basically broken as a gaming platform > > > > > > That's a bit exaggerated :) It affects just some games, mostly multiplayer > > > (some of them extremely popular, yes). I found this link that should list > > > those games (sort by Peak to see the popular ones on top): > > > https://steamdb.info/tech/AntiCheat/EasyAntiCheat/ > > > > The reality is that once reviewers get their hands on Fedora Linux 37 and > > can't play *any* of those games, they'll declare Fedora Linux unsuitable for > > gaming. Given that we know how to fix it, it is completely irresponsible to > > allow that to happen. > > Please note that for EPIC Games EAC, there is already a fix for this in EOS > SDK 1.15.2 (August 2022) and the corresponding version of the anti-cheat > module. > > My concern with this request is that this is not the only fix required to > get games like Rogue Company working. > > For example you can see the reverts here: > https://github.com/GloriousEggroll/glibc-eac-rc > > There are a half-dozen reverts listed there, and the DT_HASH revert is just > one. The other changes have impact on security (if Intel CET ends up > depending on clone3 then we need clone3 enabled) and in-place-upgrade > stability (links were simplified to make this easier to support), and > diagnostics (always having /usr/bin/ld.so so --list-diagnostics makes crash > diagnostic tooling easier to write). > > I write this here just to make it clear that not all applications are fixed > by this change, and that it is only one change in a series of changes > spanning more than a year of development. > > That there are more than a years worth of changes being reviewed and > reverted means that there is a disconnected between the Linux gaming > ecosystem and the way Linux is developed. > > Upstream glibc tries very hard keep a stable ABI, and I think we succeed at > that. > > The half-dozen reverted commits feels like a case of Hyrum's law. > > The Fedora glibc team can put DT_HASH back. > > Has anyone tested that this change fixes the games in question? > In the upstream Proton GitHub issue, it is confirmed that fixes the issue. > Do we have someone who can confirm this? I can get MultiVersus from Steam since it's a Free-to-Play game: https://store.steampowered.com/app/1818750/MultiVersus/
(In reply to Carlos O'Donell from comment #7) > (In reply to Neal Gompa from comment #4) > > (In reply to Kamil Páral from comment #3) > > > Blocker review discussion ticket: > > > https://pagure.io/fedora-qa/blocker-review/issue/920 > > > > > > > Fedora Linux 37 is basically broken as a gaming platform > > > > > > That's a bit exaggerated :) It affects just some games, mostly multiplayer > > > (some of them extremely popular, yes). I found this link that should list > > > those games (sort by Peak to see the popular ones on top): > > > https://steamdb.info/tech/AntiCheat/EasyAntiCheat/ > > > > The reality is that once reviewers get their hands on Fedora Linux 37 and > > can't play *any* of those games, they'll declare Fedora Linux unsuitable for > > gaming. Given that we know how to fix it, it is completely irresponsible to > > allow that to happen. > > Please note that for EPIC Games EAC, there is already a fix for this in EOS > SDK 1.15.2 (August 2022) and the corresponding version of the anti-cheat > module. > > My concern with this request is that this is not the only fix required to > get games like Rogue Company working. > > For example you can see the reverts here: > https://github.com/GloriousEggroll/glibc-eac-rc > > There are a half-dozen reverts listed there, and the DT_HASH revert is just > one. The other changes have impact on security (if Intel CET ends up > depending on clone3 then we need clone3 enabled) and in-place-upgrade > stability (links were simplified to make this easier to support), and > diagnostics (always having /usr/bin/ld.so so --list-diagnostics makes crash > diagnostic tooling easier to write). > > I write this here just to make it clear that not all applications are fixed > by this change, and that it is only one change in a series of changes > spanning more than a year of development. > > That there are more than a years worth of changes being reviewed and > reverted means that there is a disconnected between the Linux gaming > ecosystem and the way Linux is developed. > > Upstream glibc tries very hard keep a stable ABI, and I think we succeed at > that. > > The half-dozen reverted commits feels like a case of Hyrum's law. I have no idea what's up with the rest of the changes. Unfortunately, Thomas Crider didn't document the reverts at all, so I have no context for the problems. In the PRs I made, I made sure to document in the commit message why we're reverting an upstream commit so that knowledge doesn't get lost.
I should also note that this is the *only* revert done to the Arch Linux glibc to fix EAC-dependent games. So all the other stuff Thomas is doing for Rouge Company makes no sense to me.
(In reply to Neal Gompa from comment #11) > I should also note that this is the *only* revert done to the Arch Linux > glibc to fix EAC-dependent games. So all the other stuff Thomas is doing for > Rouge Company makes no sense to me. Thanks. I'll try to test MultiVersus myself and propose this as a change upstream can adopt.
Feel free to use my patch upstream, I tried to format it in a way for upstreaming if the project was amenable to it.
Removing the Prioritized Bugs nomination as this is an accepted release blocker.
Ahoy, Carlos - any news here? We're getting close to the wire for F37's first target date; go/no-go is on Thursday, so we really need blockers addressed today or tomorrow. Thanks!
(In reply to Neal Gompa from comment #13) > Feel free to use my patch upstream, I tried to format it in a way for > upstreaming if the project was amenable to it. If package maintainers don't respond, then just use Neal's PR. We don't want to slip due to a bug that's already fixed.
Well, we're pretty much baked in for a slip this week now anyway. But yeah, I think if we don't hear anything by Friday I'll just do a build with Neal's patch and put that out for testing over the weekend so we're ready for next week.
(In reply to Adam Williamson from comment #17) > Well, we're pretty much baked in for a slip this week now anyway. But yeah, > I think if we don't hear anything by Friday I'll just do a build with Neal's > patch and put that out for testing over the weekend so we're ready for next > week. I'm testing out a slightly simpler variant of the patch that was submitted. I would like to just set LDFLAGS with the appropriate hash style for glibc. This is simpler and won't cause any merge issues as we bring in new code from upstream.
Awesome, thanks. Any chance we can get an update with that today (Friday)? It'd be great to have an RC to test over the weekend, versus only being able to test next week. Thanks!
(In reply to Adam Williamson from comment #19) > Awesome, thanks. Any chance we can get an update with that today (Friday)? > It'd be great to have an RC to test over the weekend, versus only being able > to test next week. Thanks! Testing with Fedora Rawhide and F37 rpms (no DT_HASH features) shows that Multiversus under Steam fails. Multiversus just had a recent update, but it still fails with "Launch Error" "Failed to load the anti-cheat module." which confirms the issue. I'm still working on a singular build with DT_HASH enabled that doesn't revert the upstream change. I'll keep this bug updated.
(In reply to Carlos O'Donell from comment #20) > (In reply to Adam Williamson from comment #19) > > Awesome, thanks. Any chance we can get an update with that today (Friday)? > > It'd be great to have an RC to test over the weekend, versus only being able > > to test next week. Thanks! > > Testing with Fedora Rawhide and F37 rpms (no DT_HASH features) shows that > Multiversus under Steam fails. > > Multiversus just had a recent update, but it still fails with "Launch Error" > "Failed to load the anti-cheat module." which confirms the issue. Note: With F36 and glibc 2.35 I can launch Multiversus via Steam. I'll use this to validate the change.
(In reply to Carlos O'Donell from comment #18) > (In reply to Adam Williamson from comment #17) > > Well, we're pretty much baked in for a slip this week now anyway. But yeah, > > I think if we don't hear anything by Friday I'll just do a build with Neal's > > patch and put that out for testing over the weekend so we're ready for next > > week. > > I'm testing out a slightly simpler variant of the patch that was submitted. > > I would like to just set LDFLAGS with the appropriate hash style for glibc. > > This is simpler and won't cause any merge issues as we bring in new code > from upstream. This is incongruent from your earlier suggestion that you were going to apply this revert upstream. In any case, I've rebased my pull requests and they continue to apply properly on f37 and rawhide branches. * Rawhide: https://src.fedoraproject.org/rpms/glibc/pull-request/64 * F37: https://src.fedoraproject.org/rpms/glibc/pull-request/65
We've run out of time here. Carlos, if whatever you're working on isn't going to be ready today, it's time to use Neal's pull requests.
Yes, agreed. I'm also waiting on Georges' improved fixes to the calendar stuff - https://gitlab.gnome.org/GNOME/gnome-calendar/-/merge_requests/266 - but I'm going to aim to file an RC request in the next ~6 hours; if neither of those come through I'll use what we have and do builds myself.
(In reply to Michael Catanzaro from comment #23) > We've run out of time here. Carlos, if whatever you're working on isn't > going to be ready today, it's time to use Neal's pull requests. (In reply to Neal Gompa from comment #22) > (In reply to Carlos O'Donell from comment #18) > > (In reply to Adam Williamson from comment #17) > > > Well, we're pretty much baked in for a slip this week now anyway. But yeah, > > > I think if we don't hear anything by Friday I'll just do a build with Neal's > > > patch and put that out for testing over the weekend so we're ready for next > > > week. > > > > I'm testing out a slightly simpler variant of the patch that was submitted. > > > > I would like to just set LDFLAGS with the appropriate hash style for glibc. > > > > This is simpler and won't cause any merge issues as we bring in new code > > from upstream. > > This is incongruent from your earlier suggestion that you were going to > apply this revert upstream. I can make a public statement that I think that is the direction we should go in, but I can't unilaterally make such changes upstream. Making the changes requires more discussion upstream given the current opinion of developers. September post: https://sourceware.org/pipermail/libc-alpha/2022-September/142353.html October thread: https://sourceware.org/pipermail/libc-alpha/2022-October/142388.html This takes more time than we have in Fedora 37. > In any case, I've rebased my pull requests and they continue to apply > properly on f37 and rawhide branches. > > * Rawhide: https://src.fedoraproject.org/rpms/glibc/pull-request/64 > * F37: https://src.fedoraproject.org/rpms/glibc/pull-request/65 Thanks. (In reply to Michael Catanzaro from comment #23) > We've run out of time here. Carlos, if whatever you're working on isn't > going to be ready today, it's time to use Neal's pull requests. Thanks for the update on the schedule. I'm doing builds right now and I will deliver a fix today for the issue, one way or the other.
Confirmed Steam Multiversus works with the new build. Final scratch builds for F37 (and Rawhide) are in progress. I'll verify the final scratch builds, then push the commit which fixes the issue and kick off the official (non-scratch) final builds.
Could you possibly skip the scratch build at least for F37 and just do a final build? It'll go to updates-testing anyway so we can yank it if something is wrong with it...
(In reply to Adam Williamson from comment #27) > Could you possibly skip the scratch build at least for F37 and just do a > final build? It'll go to updates-testing anyway so we can yank it if > something is wrong with it... Yes. Rawhide finished early with no regressions and passes testing on Steam Multiversus. I expect the Fedora 37 builds to have the same results so I've committed the changes there too and kicked off final builds. Final Rawhide builds here: https://koji.fedoraproject.org/koji/taskinfo?taskID=93151964 Final F37 builds here: https://koji.fedoraproject.org/koji/taskinfo?taskID=93152045
@codonell: Wouldn't this change[1][2] wipe out the other ldflags that the Makefile sets? Part of the reason I didn't do it this way was because I didn't want to accidentally erase the other flags that are set through the build process. [1]: https://src.fedoraproject.org/rpms/glibc/c/1cd731cf293a236f70ed6d54ff711d6a1342a3d3?branch=rawhide [2]: https://src.fedoraproject.org/rpms/glibc/c/a9713abfbd4db4350fbc201e4d5cd6ddc36cfd27?branch=f37
Carlos, can you create an update for this too? Thanks! I can do it if necessary, but if I create the update I cannot also give it karma.
FEDORA-2022-12af0b08c8 has been submitted as an update to Fedora 37. https://bodhi.fedoraproject.org/updates/FEDORA-2022-12af0b08c8
(In reply to Neal Gompa from comment #29) > @codonell: Wouldn't this change[1][2] wipe out the other ldflags > that the Makefile sets? Part of the reason I didn't do it this way was > because I didn't want to accidentally erase the other flags that are set > through the build process. You are correct that we should avoid overriding LDFLAGS. In this case we set "LDFLAGS.so" and "LDFLAGS-rtld" not LDFLAGS, these special purpose variables are used by glibc's build and added to the link lines as required. Today glibc does not actually use LDFLAGS in building glibc, particularly because different LDFLAGS apply very differently to different parts of the build process. (In reply to Adam Williamson from comment #30) > Carlos, can you create an update for this too? Thanks! I can do it if > necessary, but if I create the update I cannot also give it karma. You edited our existing update which works for me.
(In reply to Carlos O'Donell from comment #32) > (In reply to Neal Gompa from comment #29) > > @codonell: Wouldn't this change[1][2] wipe out the other ldflags > > that the Makefile sets? Part of the reason I didn't do it this way was > > because I didn't want to accidentally erase the other flags that are set > > through the build process. > > You are correct that we should avoid overriding LDFLAGS. In this case we set > "LDFLAGS.so" and "LDFLAGS-rtld" not LDFLAGS, these special purpose variables > are used by glibc's build and added to the link lines as required. > I observed in the Makefile that those variables get appended with various different flags as the Makefile gets evaluated. I'm not sure it's a good idea to clobber those...
FEDORA-2022-12af0b08c8 has been pushed to the Fedora 37 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-12af0b08c8` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-12af0b08c8 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Neal gave the bug report a thumbs-up in the update, so I'm gonna count that as indicating VERIFIED.
FEDORA-2022-12af0b08c8 has been pushed to the Fedora 37 stable repository. If problem still persists, please make note of it in this bug report.