Bug 1440287

This is a known issue in the glibc TLS implementation.  Recompiling applications and libraries with the initial-exec TLS model will work around it.

I can't recompile these apps and libraries, they're proprietary.

I think that https://bugs.freedesktop.org/show_bug.cgi?id=35268 makes the initial-exec model a no-go in any case.

(In reply to Levi Bard from comment #3)
> I think that https://bugs.freedesktop.org/show_bug.cgi?id=35268 makes the
> initial-exec model a no-go in any case.

Then the alternative is that the thread variable needs to be assigned a value during initialization to force allocation outside of a signal-handler context.

(In reply to Levi Bard from comment #3)
> I think that https://bugs.freedesktop.org/show_bug.cgi?id=35268 makes the
> initial-exec model a no-go in any case.

Levi, do you by any chance have a stand-alone reproducer for this? Something that doesn't require a purchased game?

I want to remind people again that there is no easy workaround for this issue.

The glibc limitation for TLS in signal handlers has been present since the first day upstream added thread local storage. It was not well explained or advertised as a limitation though.

The only solution is for applications to *avoid all* TLS in signal handlers.

The Fedora glibc team will be looking into this issue during the year, but it's going to take time to fix.

ok, so Carlos would it be your recommendation that for the time being we try to set up the Steam package to preload jemmaloc by default? I am just worried it might potentially cause problems in other games?

(In reply to Christian Fredrik Kalager Schaller from comment #7)
> ok, so Carlos would it be your recommendation that for the time being we try
> to set up the Steam package to preload jemmaloc by default? I am just
> worried it might potentially cause problems in other games?

I do not recommend that. No allocator is built AS-safe. There is no workaround.

We are working on this and we understand that the glibc limitation on TLS access in signal handlers is a real issue.

Previously in 2015 when we tried to fix this issue it was only Google that showed any real use cases. Unfortunately the fixes we had at the time broke other tooling like valgrind, which is absolutely something we don't want to break.

I can only recommend you reach out to the vendor and explain the situation and hope you get a new build for Linux that fixes the issue.

(In reply to Carlos O'Donell from comment #8)

> I do not recommend that. No allocator is built AS-safe. There is no
> workaround.

Well, this workaround worked for me for every Unity based game. So it works around this very specific issue with this very specific game engine, even though it doesn't necessarily works around *all* the issues that might be caused by using TLS in the signal handler.

> I can only recommend you reach out to the vendor and explain the situation
> and hope you get a new build for Linux that fixes the issue.

This is impossible. Unity3D is one of the most popular game engines out there, with (probably) hundreds of Linux games using it. Even if Unity3D fixes it on their end, the chances of the fix getting deployed to their various customers is slim. Many game developers don't update the engine at all after shipping a game, because it can result in all sort of breakage and a lot of work. We'd have to contact hundreds of different game developers asking them to update the engine, which is unrealistic.

(it's unrealistic to contact so many vendors and unrealistic to expect every single one of them to release a build that fixes the problem, sometimes >4 years after the game was initially released - most games don't get patches so long after the initial release).

I guess we can have a CommonBugs entry for this issue, telling users they have to use 32bit builds of games where these are available (32bit Unity builds are not affected by this issue) or use jemalloc when a 32bit build is not available (and not preload jemalloc automatically, to avoid the slim chance of potential issues it might cause with non-Unity games).

(In reply to Matthew Miller from comment #5)
> (In reply to Levi Bard from comment #3)
> > I think that https://bugs.freedesktop.org/show_bug.cgi?id=35268 makes the
> > initial-exec model a no-go in any case.
> 
> Levi, do you by any chance have a stand-alone reproducer for this? Something
> that doesn't require a purchased game?

http://files.unity3d.com/levi/bug1440287.tar.gz should do it.
(Source project https://github.com/Tak/HelloBugzilla )

(In reply to Levi Bard from comment #10)
> > Levi, do you by any chance have a stand-alone reproducer for this? Something
> > that doesn't require a purchased game?
> 
> http://files.unity3d.com/levi/bug1440287.tar.gz should do it.
> (Source project https://github.com/Tak/HelloBugzilla )

Thanks Levi! I can confirm that the x86_64 version runs perfectly on F25 and fails on F26. I repeated 10x on each platform. Would it be possible to make the test program non-interactive and do a short countdown and then exit with a successful error code? Carlos tells me that he expects that there should still be crashes on F25 at random, and I'd like to get a sense for how frequent that might be (that is, doesn't come up in 10 trials, but maybe in 100 or a 1000).

(In reply to Matthew Miller from comment #11)
> (In reply to Levi Bard from comment #10)
> > > Levi, do you by any chance have a stand-alone reproducer for this? Something
> > > that doesn't require a purchased game?
> > 
> > http://files.unity3d.com/levi/bug1440287.tar.gz should do it.
> > (Source project https://github.com/Tak/HelloBugzilla )
> 
> Thanks Levi! I can confirm that the x86_64 version runs perfectly on F25 and
> fails on F26. I repeated 10x on each platform. Would it be possible to make
> the test program non-interactive and do a short countdown and then exit with
> a successful error code? Carlos tells me that he expects that there should
> still be crashes on F25 at random, and I'd like to get a sense for how
> frequent that might be (that is, doesn't come up in 10 trials, but maybe in
> 100 or a 1000).

Uploaded a batch-runnable version to http://files.unity3d.com/levi/bug1440287-batchable.tar.gz (and updated project repo).

Now it will log and exit with a zero exit code after successfully running for 1s (configurable in the source project).

Run with: ./repro.x86_64 -batchmode
(add "-logfile -" if you want logging to stdout instead of the normal log file location (~/.config/unity3d/DefaultCompany/Hello Bugzilla!/Player.log))
(add "-nographics" if you don't want a window popping up repeatedly)

Thanks Levi. On F26, this reliably hangs on first run. I have it running in a loop on F25, and so far up to 1000 repetitions without hang. I really don't know about the details here and I trust that Carlos and Florian do, but for whatever reason, this code worked reliably on F25 and fails on F26.

Hmm, I would say that if the bug fixed hits in less than 1 out of a 1000 runs, then dealing with having that bug in Fedora 26 seems a lot more tempting than dealing with a bug that crashes every Unity game every time in Fedora 26.

Ideally it would be great to have neither and long term I suspect Carlos will be able to come up with a fix, but for the short/medium term the current fix seems like a bigger problem that the problem it was meant to resolve.

(In reply to Christian Fredrik Kalager Schaller from comment #14)
> Hmm, I would say that if the bug fixed hits in less than 1 out of a 1000
> runs, then dealing with having that bug in Fedora 26 seems a lot more
> tempting than dealing with a bug that crashes every Unity game every time in
> Fedora 26.
> 
> Ideally it would be great to have neither and long term I suspect Carlos
> will be able to come up with a fix, but for the short/medium term the
> current fix seems like a bigger problem that the problem it was meant to
> resolve.

We know *exactly* what needs to be fixed, the problem is entirely about resourcing and priority. As I noted earlier the only users who had ever run into this were Google with a Google-sized project, and they just rebuilt their own internal distros to avoid the problem. We couldn't justify fixing it at that point. However, from a correctness perspective it needs to get fixed. Having a real problem like this bug report goes a long way to raising the priority of the work.

Just to be clear *every* Linux distribution using glibc suffers from this problem, it's not Fedora. The same problem will exist in Debian, Ubuntu, etc. So it's not just specific to Fedora. We are going to fix this upstream in coordination with upstream and the other distributions.

Carlos I have no doubt about your willingness or ability to fix it. What I am trying to say is that I think shipping with the bug is to big of a regression, so since you are not in a position to commit to finding a proper fix (and no blame on you for that) before we ship Fedora 26 we need to find another solution. 
I can't know or say what other distributions will decide to do here, but Ubuntu might escape it simply due to their release schedule (meaning their latest release use the previous glibc version and I this might be fixed in time for their next release), but if it isn't resolved by then I wouldn't be surprised if they too will decide they are not prepared to ship with this regression and either stick with their current version for another release or find some other workaround. 

We worked so hard over the last two years to escape the image of Fedora just being Red Hats development tool and not something anyone but the especially interested should consider running as their main dektop OS, and this is the sort of breakage that would create enough negative noise online to negate that effort.

(In reply to Christian Fredrik Kalager Schaller from comment #16)
> Carlos I have no doubt about your willingness or ability to fix it. What I
> am trying to say is that I think shipping with the bug is to big of a
> regression, so since you are not in a position to commit to finding a proper
> fix (and no blame on you for that) before we ship Fedora 26 we need to find
> another solution. 

The "Fedora Release Criteria" are used to determine if a bug is a blocker to the release. The "Fedora 26 Beta Release Criteria" are being met by glibc and there isn't anything broken in the release criteria by having games that use Unity not work on Fedora 26. There isn't an entry in the release criteria that says Unity-based games need to run before we can make a release. That means to me that the priority of this request is lower than other security and network related bugs we're working on. If Unity-based games and games in general are important to Fedora then you need to work to get them on the release criteria. My guidance is taken from the release criteria.

The decision to mark this a "regression" should not be taken lightly. There were 57 new symbols added to the base glibc ABI/API. The presence of these symbols means we would need to undergo some level of rebuilds to purge the distribution of all such ABI atrifacts before a final release. All of this work would be predicated on the decision that your use case is sufficiently important to Fedora as a whole.

We may also find that simply reverting glibc doesn't fix this problem, and in fact you may have to roll back the compiler, and binutils to get to a point where the generated code doesn't exhibit the issue (and all new ABI/API introduced there). This is a non-trivial amount of work by a significant number of people for an issue that is not part of the release criteria for Fedora.

I sympathize with your request, but we need to prioritize work based on community priorities.

Regarding possible workarounds: the idea of preloading jemalloc in the steam startup script won't work. While it doesn't seem to break any game, it breaks the Steam store itself... so it has to be done for each game individually instead of changing the startup script.

It's clear that reverting glibc in this stage of the release cycle is impossible (see comment #17, and I don't think there's any viable "automatic" workaround  (ie. one that would not require any action from users) we can implement.

I suggest we add a CommonBugs entry which should be something along these lines:

--
Some Unity3D-based video games might freeze on startup, showing only a black screen. To work around this issue, install jemalloc ("dnf install jemalloc") and start the game using jemalloc.sh ./[GameExecutable]. When using Steam to launch your games, right-click on the game on your game list, click properties, then click "set launch options" and write "jemalloc.sh %command%" in the text box that will appear. (note: it must literally be "%command%", there's no need to manually replace that with the executable name)
--

This is probably the most we can do from Fedora's side, especially considering this is all 3rd party software that is not under our control.

We were lucky that in Fedora 26 we introduced tunables, so we have some way to add a Fedora specific tunable to the hot path without much performance impact.

So we could add FEDORA_GLIBC_TUNABLES=glibc.async_signal_safe_tls=1 which enables the use of the signal-safe allocator we had previously for this fix from Google.

So the steps would be:

(1) Re-apply the following commits.

commit 7f507ee17aee720fa423fa38502bc3caa0dd03d7
commit 1f33d36a8a9e78c81bed59b47f260723f56bb7e6
commit 35e8f7ab94c910659de9d507aa0f3e1f8973d914
commit 69a17d9d245dc3551792e95e1823cc2d877592f3
commit 063b2acbce83549df82ab30f5af573f1b9c4bd19
commit b627fdd58554bc36bd344dc40a8787c4b7a9cc46
commit e81c64bba13d2d8b2a4e53254a82cc80f27c8497

And see if the built glibc can run the Unity games.

(2) Add a tunable which can switch between the regular malloc and the new signal safe allocator for TLS.

Then when running steam games we just say you have to set the above env var to change the TLS allocation behaviour (which we don't want to change by default because it would break developer tooling).

The glibc team is currently blocked on another high priority issue, and we would appreciate anyone doing the above backport and verifying that things work.

Thanks Carlos! I will have one of my guys look into the proposed solution.

Nice. Thanks Carlos!

I'm seeing similar Unity hangs, except the sequence of events is:

1. Thread attempts to lazy init a TLS variable.
2. There is a SIGSEGV in _int_malloc() on a movaps %xmm0,(%rsp) instruction because RSP is misaligned due to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58066
3. The Mono SIGSEGV handler gets called.
4. The SIGSEGV handler attempts to lazy init a TLS variable.
5. malloc deadlocks.

LD_PRELOADing the following simple stub fixes the affected Unity games:

#include <dlfcn.h>

__attribute__((force_align_arg_pointer)) void *__tls_get_addr (void *ti)
{
	void *(*tga)(void*) = dlsym(RTLD_NEXT, "__tls_get_addr");

	return tga(ti);
}

Why doesn't glibc's __tls_get_addr() have this attribute to deal with this gcc bug?

Actually, the stack traces I'm seeing are literally identical to comment #0, except the signal is definitely SIGSEGV (from a MOVAPS instruction), not SIGPWR (from Mono's garbage collector).

Created attachment 1288634 [details]
LD_PRELOAD stub for __tls_get_addr() that aligns the stack

unzip align-tls-get-addr.zip
cd align-tls-get-addr
meson BUILD
ninja -C BUILD
LD_PRELOAD=$(pwd)/BUILD/tls_get_addr.so /path/to/bug1440287/repro.x86_64

For Steam, Set Launch Options to LD_PRELOAD=/full/path/to/align-tls-get-addr/BUILD/tls_get_addr.so %command%

Created attachment 1288761 [details]
proposed glibc patch

A proposed patch for glibc.

Thanks for tracking this down.  We will likely use a different approach upstream, involving a compatibility symbol for future glibc versions, and use a sysdeps override for backports.

Upstream patch posted for review: https://sourceware.org/ml/libc-alpha/2017-06/msg00922.html

Thanks Florian! Will be be getting this patch into Fedora 26 now?

FWIW I can't support this as a freeze exception because we're already at significant risk for slipping, but it'd be great to see as a zero-day.

(In reply to Christian Fredrik Kalager Schaller from comment #28)
> Thanks Florian! Will be be getting this patch into Fedora 26 now?

The patch has not been approved upstream.  I'd like to wait for that.  I'll build new Fedora 26 (and earlier) glibc packages once the change is upstream.

(In reply to Elad Alfassa from comment #18)
> I suggest we add a CommonBugs entry which should be something along these
> lines:
> 
> --
> Some Unity3D-based video games might freeze on startup, showing only a black
> screen. To work around this issue, install jemalloc ("dnf install jemalloc")
> and start the game using jemalloc.sh ./[GameExecutable]. When using Steam to
> launch your games, right-click on the game on your game list, click
> properties, then click "set launch options" and write "jemalloc.sh
> %command%" in the text box that will appear. (note: it must literally be
> "%command%", there's no need to manually replace that with the executable
> name)
> --
> 
> This is probably the most we can do from Fedora's side, especially
> considering this is all 3rd party software that is not under our control.

+1 for this. I just did a fresh install for the test day and found this exact problem with Cities: Skylines.

The workaround seems to work without any other problems right now - so it would be good to document this.

There is a Common Bugs entry for this at https://fedoraproject.org/wiki/Common_F26_bugs#Games_using_the_Unity_engine_freeze_with_black_screen

It's a wiki, so feel free to improve.

Have updated the entry for this bug on the wiki. It breaks lots of stuff if you use LD_PRELOAD to launch steam itself - so much better to do it on a title by title basis.

(In reply to Steven Haigh from comment #33)
> Have updated the entry for this bug on the wiki. It breaks lots of stuff if
> you use LD_PRELOAD to launch steam itself - so much better to do it on a
> title by title basis.

Excellent -- thanks!

Florian do you have an estimated ETA for upstream approval? I have not experience with glibc so I don't know if average review time is one week or one year :)

(In reply to Christian Fredrik Kalager Schaller from comment #35)
> Florian do you have an estimated ETA for upstream approval? I have not
> experience with glibc so I don't know if average review time is one week or
> one year :)

Please be patient.  There are several competing patches now.

I note that this hasn't been added as a blocker for F26 - however with the freeze currently in place, would be it suitable to be pushed through the freeze to make the installation media etc?

(In reply to Steven Haigh from comment #37)
> I note that this hasn't been added as a blocker for F26 - however with the
> freeze currently in place, would be it suitable to be pushed through the
> freeze to make the installation media etc?

I'm against that.  The bug clearly is not a blocker, and it does not make sense to risk destabilizing the for this issue.  Nothing in Fedora requires this change.

So more a 0day update?

I know we've got the installation notes - but I'm guessing we'll have quite a few hit it just after install.

If we had another week I might be in favor of a Freeze Exception, but we're hoping to go gold today. If by some chance we end up delayed, then we could revisit. I think a zero-day is adequate -- as Florian says, nothing we ship will fail to work out of the box.

Ok, I'll monitor this and if I see anything either way, I'll try to remember to update the wiki with the latest info to hopefully reduce the number of bugzilla reports on the topic.

I've seen a lot of discussion on reddit etc about this issue as well - so I'm kinda expecting the worst :P

glibc-2.24-9.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-364f4be949

glibc-2.25-7.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-ab2085210e

glibc-2.24-9.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-364f4be949

For the record, tested glibc-2.25-7.fc26 as per:
https://bodhi.fedoraproject.org/updates/FEDORA-2017-ab2085210e

Workaround using jemalloc for Unity based games no longer required.

glibc-2.25-7.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-ab2085210e

glibc-2.25-7.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

glibc-2.24-9.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

I would like to flag a possible issue with Cities: Skylines after these new packages.

When I run the game, it launches well, but when I get round to swapping (its a memory hungry game), the system becomes unresponsive for several minutes at a time.

The first swap was after loading a saved game, then launching kmail in the background. This caused ~500Mb to be swapped to /dev/sda2 on an SSD. The system froze for about 8 minutes - barely being able to move the mouse. The HDD LED was on constantly during the entire time.

When exiting the game, the system hung again for ~2 minutes. I'm going to test with using jemalloc again - just to see if I can replicate the issue using it instead.

For what its worth, I was able to reproduce this - but worse.

Using jemalloc, I had to hit the reset button after ~13 minutes of non-responsiveness.

This may mean this change is not to blame - but something else.

Does anyone have any suggestions on where to start looking?

That doesn't have anything to do with this bug, please file a new one.