Bug 1471427

Hello, I may have found some relevant information for this bug. I'm experiencing the same problem with the two ported Borderlands games.

This arch bug seems to be the same issue:

  https://bugs.archlinux.org/task/54136?project=0&order=id&sort=desc&string=glibc

I'm not clear if their diagnosis that it is a GCC issue is correct. But they also have a work-around.

Please install debuginfo packages for glibc and report the exact fault location using the “disassemble” GDB command.  Please include “info registers” output as well.

Created attachment 1299784 [details]
Civ5 stack trace with glibc-debuginfo package installed

Same issue for me, same program (Civilisation V installed via Steam Linux).  I installed the glibc-debuginfo package and ran gdb against the coredump as per https://fedoraproject.org/wiki/StackTraces.  Is this useful?

(In reply to Joshua Rich from comment #3)
> Created attachment 1299784 [details]
> Civ5 stack trace with glibc-debuginfo package installed
> 
> Same issue for me, same program (Civilisation V installed via Steam Linux). 
> I installed the glibc-debuginfo package and ran gdb against the coredump as
> per https://fedoraproject.org/wiki/StackTraces.  Is this useful?

Looks like you didn't install the 32-bit debuginfo packages.  I'd also need the diassamble output and the register contents.  Thanks.

Created attachment 1300021 [details]
GDB info from Borderlands 2

Does this work? It's the information for Borderlands 2.

GDB suggests installing more debuginfo packages, but dnf says they are already installed.

Created attachment 1300442 [details]
Civ5 stack trace, registers and disassemble output

Check the following compressed archive for the trace (with i686 debuginfo packages installed), registers and disassemble output.  Hope this is helpful, let me know if I need to provide any other info!

Thanks, Dan and Joshua.  This is a stack alignment issue.  The binaries do not comply with the current i386 API, so a lot of SSE2-optimized code in libraries will break.  This issue is not specific to glibc, but shows up mostly with glibc at present because we do not build the whole distribution with SSE2 support.

Do you know what is likely to have caused this? Is it just due to the binaries being compiled with a different GCC version than Fedora uses? Or would it be an actual error in the source code?

(In reply to Dan Doel from comment #8)
> Do you know what is likely to have caused this? Is it just due to the
> binaries being compiled with a different GCC version than Fedora uses? Or
> would it be an actual error in the source code?

The most comment scenario observed is code compiled with Intel's icc compiler in conjunction with options that break the ABI. The developer perhaps doesn't know that the option, specifically -falign-stack=4, will cause problems when entering SSE2 code.

I understand that this might not classify as a "Fedora bug", but what is the plan dealing with this?

Are we expecting Valve and other vendors to recompile binaries or is this going to be addressed on Fedora side?

This is going to be a blocker for many Fedora home users. It should be propably mentioned in the release notes at least. Something like "Expect Steam games to crash unde F26".

Thanks!

(In reply to Martin Povolny from comment #10)
> I understand that this might not classify as a "Fedora bug", but what is the
> plan dealing with this?

There is currently no plan to fix this, the developer has broken the ABI by compiling their application with options that change the ABI.

The glibc team has been discussing that in Fedora we could probably switch glibc to be compiled in a more compatible fashion, disabling SSE2, and building a very vanilla i686 build. This would fix this issue for glibc only.

That still leaves you to find and fix any dependent libraries that use SSE2 and require an aligned stack, but don't automatically align the stack themselves.
 
> Are we expecting Valve and other vendors to recompile binaries or is this
> going to be addressed on Fedora side?

What Valve and the vendors do is up to them.

If Valve and the vendors compile their code with options that don't follow the ABI, then the applications will not run correctly.

They will have the same problem in all the other distributions that use SSE2 optimizations.

> This is going to be a blocker for many Fedora home users. It should be
> propably mentioned in the release notes at least. Something like "Expect
> Steam games to crash unde F26".

Sure, how about:

"In Fedora 26 it has been found that some games distributed under STEAM may fail to start or crash. These 32-bit games crash because they have been incorrectly built (don't follow the i686 stack alignment requirements) and the authors of such games should rebuild as redistribute as needed."

How do we get this into the Fedora 26 release notes?

(In reply to Martin Povolny from comment #10)

> This is going to be a blocker for many Fedora home users.

I feel like I have to choose between Fedora and Gaming. And that is a terrible feeling.

Using a flatpak install of Steam works around the issue for me for at least Civilization Beyond Earth (haven't had time to test other games).  These are the commands I ran to get it installed and running:

1) Add the flathub faltpak repo:

flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

2) Install the steam flatpak:

flatpak install flathub com.valvesoftware.Steam

3) Run the steam flatpak:

flatpak run com.valvesoftware.Steam/x86_64/stable


Assuming this works for other Aspyr games, at least we don't have to choose between Fedora and gaming.

Unfortunately, the Borderlands games crash at other points when run with the flatpak steam (though it does fix the crashes due to videos).

I was able to play Borderlands The Pre-Sequel with flatpak steam for 30 minutes without issues if I set Steam to offline mode before starting Borderlands.  With it in online mode it crashes about when the menu should come up.  I haven't tried to figure out the cause.

Have you tried compiling glibc with -mstackrealign like arch did? That fixed Civ 5 for me.

Do you have a quick how-to for building glibc in 32-bit mode and getting steam games to use it?

I don't know about compiling it in 32-bit mode, but Civ 5 has a folder that it uses for libraries (~/.steam/steam/steamapps/common/Sid Meier's Civilization V/ iirc). So if you compile it with -mstackrealign added, then extract the offending libraries (Civ 5 needs libc-2.25 and libc.so.6) into "~/.steam/steam/steamapps/common/Sid Meier's Civilization V/" and put them there.

Compiling a 32-bit version of glibc and placing the resulting libc.so and libc.so.6 symlink into the ~/.steam/steam/steamapps/common/<game subdir> works for both Civ Beyond Earth and Borderlands The Pre-Sequel.

The config options I used to build it were:

../glibc/configure --prefix=$HOME/glibc/build \
     --host=i686-linux-gnu \
     --build=i686-linux-gnu \
     CC="gcc -m32" CXX="g++ -m32" \
     CFLAGS="-O2 -march=i686 -mstackrealign" \
     CXXFLAGS="-O2 -march=i686 -mstackrealign"

In case that helps someone else figure it out.

I still hope Fedora makes this the default method to build for the 32-bit glibc for Fedora 26.  If a program worked on Fedora 25, it seems like the best thing for users is to sacrifice a little performance in favor of having it continue to work, especially for 32-bit applications.  At least until companies like Aspyr get their build straightened out.

OpenSUSE is also dealing with this issue. https://bugzilla.opensuse.org/show_bug.cgi?id=1048861

glibc-2.25-8.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-92f8958310

(In reply to tbeckman from comment #19)
> Compiling a 32-bit version of glibc and placing the resulting libc.so and
> libc.so.6 symlink into the ~/.steam/steam/steamapps/common/<game subdir>
> works for both Civ Beyond Earth and Borderlands The Pre-Sequel.
> 
> The config options I used to build it were:
> 
> ../glibc/configure --prefix=$HOME/glibc/build \
>      --host=i686-linux-gnu \
>      --build=i686-linux-gnu \
>      CC="gcc -m32" CXX="g++ -m32" \
>      CFLAGS="-O2 -march=i686 -mstackrealign" \
>      CXXFLAGS="-O2 -march=i686 -mstackrealign"
> 
> In case that helps someone else figure it out.
> 
> I still hope Fedora makes this the default method to build for the 32-bit
> glibc for Fedora 26.  If a program worked on Fedora 25, it seems like the
> best thing for users is to sacrifice a little performance in favor of having
> it continue to work, especially for 32-bit applications.  At least until
> companies like Aspyr get their build straightened out.

glibc alone worked? You didn't have to pull / rebuild additional libs?

- Gilboa

@fweimer,

Is this bug fixed or is WONTFIX? I'm a bit confused...

- Gilboa

(In reply to Gilboa Davara from comment #23)
> @fweimer,
> 
> Is this bug fixed or is WONTFIX? I'm a bit confused...

I have submitted an update for Fedora 26, see comment 21.

(In reply to Gilboa Davara from comment #22)
> glibc alone worked? You didn't have to pull / rebuild additional libs?

Other libraries may use SSE2 or other problematic instruction sets after run time detection, but this is still relatively rare.  The bulk of Fedora is compiled in such a way that SSE2 capabilities are not used, so the impact of the new GCC vectorization capabilities is greatly reduced compared to what other distributions see.

(In reply to Florian Weimer from comment #24)
> (In reply to Gilboa Davara from comment #23)
> > @fweimer,
> > 
> > Is this bug fixed or is WONTFIX? I'm a bit confused...
> 
> I have submitted an update for Fedora 26, see comment 21.

I've installed the packages from koji (both x86_64, i686) and it seem to work just fine under Borderlands 2.

Thanks!

- Gilboa

glibc-2.25-8.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-92f8958310

glibc-2.25-8.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

It seems to have fixed zsnes crash as well.