Description of problem: When I run mathematica, the program operates mostly normally, but the display is messed up. The background of all the widgets is some combination of stuff from elsewhere on the screen. Version-Release number of selected component (if applicable): xorg-x11-server-Xorg-1.1.1-44.fc6, using the ATI proprietary driver 8.29.6 for firegl v3400, and mathematica 5.1 for linux x86_64 Mathematica seems to use some version of motif, statically linked. It looks almost like the transparent terminal windows some people use, so my first guess is that it's some incompatibility with one of these new extensions. Mathematica 5.1 works fine on i386 under FC5.
Created attachment 137748 [details] A screen shot of mathematica gone haywire
This could well be a composite issue - we now enable that by default in the X server. Could you try running mathematica with the environment variable XLIB_SKIP_ARGB_VISUALS set to 1? E.g. from a command line: $ XLIB_SKIP_ARGB_VISUALS=1 mathematica
That environment variable makes it work just fine.
Unfortunately that means it's a bug in Mathematica. It's getting confused by the presence of the extra Composite visual, which is what gets used for per-pixel alpha. Please report this bug to Wolfram. The environment variable given above is safe to set on all versions of X, it will simply have no effect on older Xlib or when the server doesn't support the Composite extension. They should at a minimum add this to their wrapper script (if any), or better yet fix their code to select only the visuals they really want.