Bug 638477

Summary: Strange sound on mp3 flash website
Product: [Fedora] Fedora Reporter: JCHuynh <jc1da.3011>
Component: glibcAssignee: Andreas Schwab <schwab>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 14CC: 777tahder, a9016009, aaron, alehman, alexander.hunt2005, alkana54, andrei.ilie, aquini, awilliam, azelinka, bill-bugzilla.redhat.com, bloch, bobg+redhat, brentrbrian, cfergeau, charlieb-fedora-bugzilla, chris.stone, christopher.fabritius, collura, cra, davidsen, debaditya, dirleyrls, djuran, dougsland, dr.diesel, drewskiwooskie, eparis, e_redhat, extras-orphan, fanisatt, fedora, fedoraproject, felipe.contreras, freetgm, fumiyas, gansalmon, gawain.lynch, gbanchelli, gsking1, harrison.ian, harsh1492, hdfssk, itamar, jaceks, jakub, jan.public, jcm, jfeeney, jgetsoian, jheiv, jh.redhat, johnparmitage, john.robinson, jonathan, joolli, jstodola, jvillalo, kenyon, kevin, kir, knutjbj, kparal, kraymer, kszysiu, kuznetsovvv, laurent.rineau__fedora, leon, ling.ma, linuxover, lkundrak, ll_vectra, lmacken, lsatenstein, luhe, madhu.chinakonda, mail2benny, marcus, martin, mathieu-acct, maurizio.antillon, m.a.young, mdavis, mglantz, mhasko, m.heiming, mhlavink, milan.kerslager, mishu, mkranz, ml-bz-dale, mmello, neumann, notting, ole, oliver.henshaw, o_ojo, panormitis, patrys, pierre-bugzilla, pinto.elia, pmontepagano, pmrpla, portis24, promac, rhbugs, rian, rich, richmattes, rmj, robatino, rob.d.wills, robin.bowes, robin, rod, roth, samuel-rhbugs, savelov, schlaffi, schwab, sebastien.major, seg, shigorin, shur, siarhei.siamashka, simon.lewis, sitsofe, smconvey, sverrel, swantz051, teva.riou, thierry.vignaud, thomas.gasser, thomas, tmraz, tom, torvalds, tristanmoody, tru, tuju, ugis.fedora, vchelban, vik, vlado_potisk, vonbrand, vvaldez, whizzman, wilo108, yann, yaroslav.sapozhnik, ykopkova, zenczykowski, zero456
Target Milestone: ---Keywords: CommonBugs, Patch, Reopened, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: https://fedoraproject.org/wiki/Common_F14_bugs#flash-64-sound
Fixed In Version: 2.13.90-9 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-07-12 04:00:39 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
Valgrind output
none
Valgrind output
none
replace all memcpy calls with memmove calls in libflashplayer.so
none
binary patch for "flashplayer_square_p2_64bit_linux_092710"
none
Linus' workaround
none
example launcher script
none
Program to fix relocation entries in libflashplayer.so
none
x86_64: workaround for new memcpy behavior
none
memcpy-ssse3: add overlap checks
none
x86_64: fix for new memcpy behavior (try 2) none

Description JCHuynh 2010-09-29 02:35:31 EDT
Description of problem:
Strange sound when playing mp3 on website using flash (using Shockwave Flash 10.2 d161). 
Tested and work fine on Youtube ( no strange sound )

[JC1DA@jc1da-laptop ~]$ dmesg | grep realtek
ALSA sound/pci/hda/patch_realtek.c:1358: realtek: No valid SSID, checking pincfg 0x411111f0 for NID 0x1d
ALSA sound/pci/hda/patch_realtek.c:1441: realtek: Enable default setup for auto mode as fallback

Version-Release number of selected component (if applicable):

How reproducible:
terrible sound on this site
http://mp3.zing.vn/mp3/nghe-bai-hat/Tennessee-Line-Daughtry.IW6WUFWE.html

Steps to Reproduce:
1.
2.
3.
  
Actual results:
terrible sound

Expected results:
good as on Fedora 13

Additional info:
Comment 1 Hans Ulrich Niedermann 2010-09-29 08:55:20 EDT
Try as I might, I cannot see how this report is related to the soundtracker package. Could you elaborate on that?
Comment 2 JCHuynh 2010-09-29 12:07:06 EDT
sorry wrong Component
Comment 3 Bill Nottingham 2010-09-29 12:49:53 EDT
*** Bug 638678 has been marked as a duplicate of this bug. ***
Comment 4 Bill Nottingham 2010-09-29 12:50:19 EDT
Have you confirmed that F13 works the same *with the same version of flash*?
Comment 5 JCHuynh 2010-09-29 13:21:23 EDT
No, F13 works fine with Shockwave Flash 10.2 d161
Comment 6 JCHuynh 2010-09-29 13:35:11 EDT
One more things, I used RhythmBox to play mp3, flac files, it works fine ... no strange sound like on some flash websites(exclude youtube)
Comment 7 Michael Young 2010-10-03 17:27:26 EDT
I am hearing similar symptoms with a different sound card (64-bit flash preview gives jumpy sound for some sites in F14, but the same flash works in F13).
lspci -vs 00:1b.0 gives
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 02)
	Subsystem: Dell Device 022f
	Flags: bus master, fast devsel, latency 0, IRQ 47
	Memory at fe9fc000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [130] Root Complex Link
	Kernel driver in use: HDA Intel
	Kernel modules: snd-hda-intel
Comment 8 Linus Torvalds 2010-10-11 12:09:30 EDT
I see this as well. Sounds like clipping or some really bad sample rate frequency conversion. I also use the preview 64-bit adobe flash, and it doesn't happen for everything (not local mp3 players, for example).

It doesn't seem to happen with most (all?) youtube videos, but happens commonly with other sources. The example given in comment #1 is a good example, and breaks for me too. If I had to guess, I'd think some flash audio is using a different sample rate (or bits per sample), and the conversion is buggered.

F13 worked with the same flash binary.

And yes, I'm using Intel HDA too.  Probably not related, though.

00:1b.0 Audio device: Intel Corporation 5 Series/3400 Series Chipset High Definition Audio (rev 06)
Comment 9 Michal Haško 2010-10-27 04:46:58 EDT
Happens to me as well
Also in youtube 240p quality, not in higher, so I think might be the "bad sample rate frequency conversion" as in comment #8.

Flash 64bit version 10.2 d161

$ lspci -vs 1b
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
	Subsystem: Lenovo Device 20f2
	Flags: bus master, fast devsel, latency 0, IRQ 50
	Memory at fc020000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: HDA Intel
	Kernel modules: snd-hda-intel
Comment 10 Bill Nottingham 2010-10-28 20:36:16 EDT
Turning off power_save in snd-hda-intel seems to fix this in a quick test for me. Not sure what's causing it to be an issue in this combination.
Comment 11 Bill Nottingham 2010-10-28 20:48:17 EDT
... perhaps not. It fixed it until the next stream opened.
Comment 12 Bill Nottingham 2010-10-28 21:29:46 EDT
That being said, if I remove pulseaudio, where flash accesses the sound device directly, the audio still stutters. Assigning to the kernel for the driver.
Comment 13 Michael Young 2010-10-29 19:31:04 EDT
There is a problem with the kernel driver theory. If I boot an F14 install with an F12 based kernel then I still get the broken sound.
Comment 14 Linus Torvalds 2010-10-29 19:47:11 EDT
(In reply to comment #13)
> There is a problem with the kernel driver theory. If I boot an F14 install with
> an F12 based kernel then I still get the broken sound.

Also, I did a "yum upgrade" from F13 to F14, and since I (obviously) always compile my own kernels, I can say that both the kernel and the libfrashplayer.so binary stayed constant over the upgrade - yet the problem did not exist in F13.

So while it may be somehow tied to the kernel or to libflashplayer, it is definitely something in Fedora-14 that triggers the problem. 

On a suggestion from Takashi Iwai I tried to "downgrade" alsa-lib to the F13 version (I say "downgrade", because the version number seems to be the same in F13 and F14), and that didn't make any difference.

So it isn't the kernel, it's not libflashplayer.so, and it doesn't seem to be alsa-lib.  If it's not pulseaudio, then what else is involved in sound generation?
Comment 15 Michael Young 2010-10-30 09:08:53 EDT
The trigger of the problem is the glibc version. If I start with F13 (sounds works) and then upgrade to the F14 glibc and nscd packages the sound starts breaking up. If I downgrade to F13 glibc again the sound works again. However it isn't obvious to me why that should make a difference.
Comment 16 Michael Young 2010-10-30 10:29:02 EDT
The reverse is also true. The sound is good with F13 glibc (2.12.1-4) on F14. I have been hunting through glibc versions. The sound was good with glibc-2.12.90-3 but first started sounding corrupted with glibc-2.12.90-4.
Comment 17 Andreas Schwab 2010-11-02 05:10:16 EDT
valgrind?
Comment 18 Michael Young 2010-11-02 06:32:02 EDT
Created attachment 457145 [details]
Valgrind output

This is with seamonkey running the 64-bit flash plugin preview.
Comment 19 Andreas Schwab 2010-11-02 06:38:40 EDT
I see no valgrind of flash player.
Comment 20 Michael Young 2010-11-02 06:49:33 EDT
What options would you suggest to capture it? I was running valgrind --trace-children=yes --leak-check=full -v seamonkey
Comment 21 Andreas Schwab 2010-11-02 06:55:15 EDT
Please trace the process that creates the sound.
Comment 22 Michael Young 2010-11-02 07:09:29 EDT
Created attachment 457151 [details]
Valgrind output

seamonkey is the process that produces the sound, via the flash plugin. I am attaching a second valgrind trace, removing nspluginwrapper from the process in case that improves the trace.

Actually the sound is better when seamonkey is run under valgrind, so it is possible that the hooks valgrind puts in to do its monitoring is bypassing the problem.
Comment 23 Andreas Schwab 2010-11-02 10:43:50 EDT
Please check that there are no calls to memcpy with overlapping regions.
Comment 24 Michael Young 2010-11-02 10:49:42 EDT
How can I do that?
Comment 25 drewskiwooskie 2010-11-02 11:37:30 EDT
I am also getting the issue, but only with the 64bit version of flash (either the square release or latest stable). If I use the 32bit and wrap it as per http://fedoraproject.org/wiki/Flash , either version of square or release works perfectly. Seems it happens just with the 64bit flash plugin.

I am also on hda-intel but not sure if that matters at all.
lspci -vs 1b
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
	Subsystem: Dell Device 0233
	Flags: bus master, fast devsel, latency 0, IRQ 49
	Memory at f6adc000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: HDA Intel
	Kernel modules: snd-hda-intel
Comment 26 rubberglove 2010-11-03 08:16:28 EDT
I can also confirm (as in comment 25) that I only get this issue with the 64-bit version of the plugin, but using the 32-bit wrapped plugin works fine.

I also (like everyone?) have an intel card:

lspci -vs 1b
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
	Subsystem: ASUSTeK Computer Inc. Device 82fe
	Flags: bus master, fast devsel, latency 0, IRQ 45
	Memory at fe3f4000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: HDA Intel
	Kernel modules: snd-hda-intel
Comment 27 JCHuynh 2010-11-03 10:58:07 EDT
I confirm that it works fine with the 64 bit version of flash ... 32 bit wrapped plugins just fine ...
Comment 28 Wyatt Allyn 2010-11-05 02:09:22 EDT
I can confirm as well. Using a Creative card.

lspci -vs 03.0
04:03.0 Multimedia audio controller: Creative Labs CA0106 Soundblaster
	Subsystem: Creative Labs SB0570 [SB Audigy SE]
	Flags: bus master, medium devsel, latency 64, IRQ 19
	I/O ports at cca0 [size=32]
	Capabilities: <access denied>
	Kernel driver in use: CA0106
	Kernel modules: snd-ca0106
Comment 29 Sitsofe Wheeler 2010-11-06 12:35:00 EDT
Just adding another me too (on Fedora 14). Temporarily turning off selinux and doing

LD_PRELOAD="/dev/shm/glibc-2.12.1-4/lib64/libc.so.6 /dev/shm/glibc-2.12.1-4/lib64/libpthread.so.0" chromium-browser --app=http://news.bbc.co.uk/1/hi/entertainment/8266142.stm

didn't produce the distorted sound (I used chromium so that the LD_PRELOAD environment variable would actually be passed to the flash plugin). Looking at the changelog for glibc-2.12.90-4 shows:

* Fri Jul 02 2010 Andreas Schwab <schwab@redhat.com> - 2.12.90-4
- Update from master
  - Work around kernel rejecting valid absolute timestamps
  - Improve 64bit memcpy/memmove for Atom, Core 2 and Core i7
  - Fix error handling in Linux getlogin*
- Workaround assembler bug sneaking in nopl (#579838)
- Fix scope handling during dl_close
- Fix setxid race handling exiting threads

It would be interesting to know if this is happening on AMD 64 bit machine. It is interesting that previous comments suggest the 32 bit flash does not seem to be demonstrating this problem.

I used chromium to run valgrind on the flash plugin ( chromium-browser --plugin-launcher="valgrind" --app=http://news.bbc.co.uk/1/hi/entertainment/8266142.stm --no-sandbox ). Large amounts of
"Conditional jump or move depends on uninitialised value(s)"
went past but just before it started outputting sound the following came up:

==2100== Conditional jump or move depends on uninitialised value(s)
==2100==    at 0x1B2E9361: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF89F92: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF8A784: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF8F8CA: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF916E0: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF5FF4A: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF61ABD: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF61AD2: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF18AC7: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF18BE4: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF80D43: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AE7126A: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==
==2100== Thread 9:
==2100== Source and destination overlap in memcpy(0x256d7170, 0x256d7570, 1280)
==2100==    at 0x4A06A3A: memcpy (mc_replace_strmem.c:497)
==2100==    by 0x1B122B87: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1B1230DF: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1B1232C5: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF491F0: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1AF4AFDA: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1B06DA69: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1B06EA23: ??? (in /usr/lib64/mozilla/plugins/libflashplayer.so)
==2100==    by 0x1EF26D88: write_data (flashsupport.c:872)
==2100==    by 0x1F14F505: ??? (in /usr/lib64/libpulse.so.0.12.2)
==2100==    by 0x1F390C02: ??? (in /usr/lib64/libpulsecommon-0.9.21.so)
==2100==    by 0x1F391358: pa_pdispatch_run (in /usr/lib64/libpulsecommon-0.9.21.so)
==2100==

While running under valgrind the distortion issue did not seem to occur (but sound was played back with big pauses every few seconds).

On a related note, is there a glibc environment flag that can be set to force a generic memcpy routine to run?
Comment 30 Sitsofe Wheeler 2010-11-06 12:37:53 EDT
(I forgot to say, the machine I'm using is an Intel Core 2 laptop with an Intel Corporation N10/ICH 7 Family High Definition Audio Controller (rev 02))
Comment 31 Linus Torvalds 2010-11-06 12:50:36 EDT
(In reply to comment #29)
> Looking at the changelog for glibc-2.12.90-4 shows:
> 
> * Fri Jul 02 2010 Andreas Schwab <schwab@redhat.com> - 2.12.90-4
> [...]
>   - Improve 64bit memcpy/memmove for Atom, Core 2 and Core i7
> [...]
> I used chromium to run valgrind on the flash plugin [...]
> ==2100== Thread 9:
> ==2100== Source and destination overlap in memcpy(0x256d7170, 0x256d7570, 1280)
> ==2100==    at 0x4A06A3A: memcpy (mc_replace_strmem.c:497)

That does look like a very possible smoking gun. 

Normally, a memcpy that copies _downwards_ (like the one above) will work
perfectly well in practice, because the "natural" way to do memcpy() by making
it just copy things upwards will "just work" even for the overlapping case.

So it would be a bug to use memcpy for overlapping areas, but it would be a
bug that normally would never show up.

But if the new improved 64bit memcpy started copying things backwards, it might
cause trouble with such an overlapping memcpy user.

[ That said - why the heck would you ever do memcpy() backwards? Things like
  automatic prefetchers usually work better for the "natural" patterns, so a
  backwards memcpy is generally a bad thing to do unless you have some active
  reason for it, like doing a memmove() ]

> On a related note, is there a glibc environment flag that can be set to force a
> generic memcpy routine to run?

Indeed, that would be very interesting to see. 

Andreas?
Comment 32 Sitsofe Wheeler 2010-11-08 02:37:55 EST
Linus is right - looking at the memcpy patch comments ( http://article.gmane.org/gmane.comp.lib.glibc.alpha/1527 ) shows it is going backwards. One of the CPU feature tests that must pass before this memcpy is used is called HAS_FAST_COPY_BACKWARD.
Comment 33 Sitsofe Wheeler 2010-11-08 02:39:53 EST
Sorry, the link should have been http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278 .
Comment 34 Andreas Schwab 2010-11-08 04:32:00 EST
Tell adobe.
Comment 35 Julian Sikorski 2010-11-08 05:04:24 EST
https://bugs.adobe.com/jira/browse/FP-5739
Comment 36 Julian Sikorski 2010-11-08 05:05:44 EST
If someone knows how to make Adobe pay more attention than usual, this would be perfect.
Comment 37 Michael Young 2010-11-08 05:33:17 EST
I did notice a forum post http://forums.adobe.com/thread/748698?tstart=0 which it might be worth adding a comment to if you have a suitable account to do so. It might also be explicitly pointing to the memcpy man page which says it shouldn't be used for overlapping regions so that Adobe realize they are doing something broken which just happened to work previously.


MEMCPY(3)                  Linux Programmer's Manual                 MEMCPY(3)

NAME
       memcpy - copy memory area

SYNOPSIS
       #include <string.h>

       void *memcpy(void *dest, const void *src, size_t n);

DESCRIPTION
       The  memcpy()  function  copies  n bytes from memory area src to memory
       area dest.  The memory areas should not overlap.  Use memmove(3) if the
       memory areas do overlap.

....
Comment 38 Linus Torvalds 2010-11-08 10:52:30 EST
So in the kernel we have a pretty strict "no regressions" rule, and that if people depend on interfaces we exported having side effects that weren't intentional, we try to fix things so that they still work unless there is a major reason not to.

So I'm disappointed glibc just closes this as NOTABUG. There's no real reason to do the copy backwards that I can see, so doing it that way is just stupid.

But whatever. You can do a LD_PRELOAD trick to get a sane memcpy(), and it does indeed fix the sound for me. Just do something like this:

    prompt$ cat > mymemcpy.c
    #include <sys/types.h>

    void *memcpy(void *dst, const void *src, size_t size)
    {
	void *orig = dst;
	asm volatile("rep ; movsq"
		:"=D" (dst), "=S" (src)
		:"0" (dst), "1" (src), "c" (size >> 3)
		:"memory");
	asm volatile("rep ; movsb"
		:"=D" (dst), "=S" (src)
		:"0" (dst), "1" (src), "c" (size & 7)
		:"memory");
	return orig;
    }
    ^D
    prompt$ gcc -O2 -c mymemcpy.c
    prompt$ ld -G mymemcpy.o -o mymemcpy.so
    prompt$ LD_PRELOAD mymemcpy.so /opt/google/chrome/google-chrome &

and it does seem to work. Not a lot of testing, but at least TED-talks work for me again (and I tested the Daughtry thing too, although I am not convinced it sounds all that much better without the sound corruption).
Comment 39 Andreas Schwab 2010-11-08 11:10:27 EST
The only stupidity is crap software violating well known rules that have existed forever.
Comment 40 Linus Torvalds 2010-11-08 11:20:39 EST
(In reply to comment #39)
> The only stupidity is crap software violating well known rules that have
> existed forever.

Umm. Bugs happen. That's a fact.

You can call it "crap software" all you like, but the thing is, if memcpy  doesn't warn about overlaps, there's no test coverage, and in that case even well-designed software will have bugs.

Then the question becomes one of "Why break it?"

I have yet to hear the actual _reason_ for making the change to memcpy. We know what the downside is. What's the upside?
Comment 41 Andreas Schwab 2010-11-08 11:38:23 EST
See comment 29.  This is well known.
Comment 42 Linus Torvalds 2010-11-08 11:46:09 EST
Andreas - I know it's a well-known issue. That doesn't change anything at all. Read my comment: bugs happen.

What's the upside of breaking things? That's all I'm asking for.
Comment 43 Jakub Jelinek 2010-11-08 11:56:36 EST
The upside is (ought to be) faster memcpy, which is something that helps a lot of apps.  Recent glibcs use STT_GNU_IFUNC magic to select one out of several different versions of many of the common string/memory ops as well as a few other functions, depending on what CPU is used.
Comment 44 Sitsofe Wheeler 2010-11-08 12:07:26 EST
Jakub:
Is there a way to force the function STT_GNU_IFUNC runs for glibc to select a different version based on an environment variable (I searched the web a few days ago but turned up nothing)?
Comment 45 Jakub Jelinek 2010-11-08 12:18:12 EST
No (except for LD_PRELOAD).
The thing is, the IFUNC decision function can be only very limited, it is called during relocation processing, so it can't rely too much on relocation processing having been performed already.  And it must be very fast and thread-safe, doing getenv there would be very problematic.  It usually just either tests cpuid flags or tests saved cpuid flags and vars computed from that.
Plus, each of the function usually has a different set of decisions, there is no common notion of oldest etc.
Comment 46 Linus Torvalds 2010-11-08 13:13:02 EST
(In reply to comment #43)
> The upside is (ought to be) faster memcpy, which is something that helps a lot
> of apps.  

Hey, I'm a big believer in fast memcpy's, I just don't believe that going
backwards helps performance. 

In the kernel, the optimized x86 memcpy we use is actually a memmove(), because while performance is really important, so is repeatability and avoiding surprises (strictly speaking, we have two: the "rep movs" version for the case where that is supposed to be fast, and the open-coded copy version. The "rep movs" version is forwards-only and doesn't handle overlapping areas).

I dunno. I just tested my stupid "mymemcpy.so" against the glibc memcpy() on the particular kind of memcpy that valgrid reports (16-byte aligned 1280-byte memcpy).

I did both cached (same block over and over) and non-cached (a million blocks in sequence).

For the cached case my stupid LD_PRELOAD version was consistently a bit faster.
 
The noise on the non-cached case was larger, but the glibc memcpy may have been faster. I say "may have been" because it went both ways: I did ten runs, and my LD_PRELOAD one still won 6 out of those 10 runs, but the noise was large enough that I will allow that I'm not going to guarantee anything.

Do I have a point? I bothered to _measure_ the speed, and according to my measurements, glibc wasn't any faster than my trivial version and was likely slower. But I only tested two cases.

Regardless, it boils down to: we know the glibc change resulted in problems for real users. We do _not_ know that it helped anything at all.

And in the end, the big question is simple:

Are you seriously going to do a Fedora-14 release with a known non-working flash player?
Comment 47 Tom Horsley 2010-11-08 14:33:00 EST
Then, of course, there is the wacked out side effects STT_GNU_IFUNC has
when you are running a process in a virtual machine and try to live migrate
it to a slightly different architecture host :-). Glibc is trying to be
too clever for anyone's good.
Comment 48 Linus Torvalds 2010-11-08 14:40:53 EST
(In reply to comment #47)

I doubt that is much of a problem in practice, since it's presumably done once at process startup, so live migration at worst just means that you might have the slightly differently optimized version running. 

And if you depend on actual semantic issues (ie "does this machine support SSE4"), then that just means that you'd better live migrate between machines that are sufficiently similar for that to all work.

So no, I don't think live migration is likely to be a big issue. Not compared to "flash doesn't work right".
Comment 49 Magnus Glantz 2010-11-08 15:19:36 EST
(In reply to comment #38)
>     prompt$ gcc -O2 -c mymemcpy.c
>     prompt$ ld -G mymemcpy.o -o mymemcpy.so
>     prompt$ LD_PRELOAD mymemcpy.so /opt/google/chrome/google-chrome &

For anyone that is having issues following this, change 
prompt$ LD_PRELOAD mymemcpy.so /opt/google/chrome/google-chrome &
to
prompt$ LD_PRELOAD=mymemcpy.so /opt/google/chrome/google-chrome &


A note:
I myself did have issues using this fix, I got 6 instances of this error message:
ERROR: ld.so: object 'mymemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
Comment 50 Magnus Glantz 2010-11-08 15:23:12 EST
(In reply to comment #49)
I had no issues compiling it, no real clues (for me, a non-programmer).
Running 2.6.35.6-48.fc14.x86_64, glibc-2.12.90-18
Comment 51 Teva Riou 2010-11-08 15:31:25 EST
Try something like this:
LD_PRELOAD=/full/path/to/mymemcpy.so /opt/google/chrome/google-chrome &

or for firefox:
LD_PRELOAD=/full/path/to/mymemcpy.so /usr/bin/firefox &
Comment 52 Linus Torvalds 2010-11-08 15:33:08 EST
(In reply to comment #49)
>
> I myself did have issues using this fix, I got 6 instances of this error
> message:
> ERROR: ld.so: object 'mymemcpy.so' from LD_PRELOAD cannot be preloaded:
> ignored.

Try giving it the whole absolute pathname. 

I should really have cut-and-pasted my actual command lines rather than try to
write them out and getting them wrong. My actual command line was really

  LD_PRELOAD=/home/torvalds/mymemcpy.so /opt/google/chrome/google-chrome &

but you'd obviously have to fix that "/home/torvalds" part to match where-ever
you end up installing that .so file.

The nicest alternative might be to just install that mymemcpy.so into the
google chrome directory, and add the LD_PRELOAD to the wrapper shell script
that google chrome already uses for the xdg binaries and the ffmpeg library.

And obviously something similar should work for firefox. I just happen to use
chrome, so I gave the directions (approximate as they were) for the thing I
tried. 

No guarantees. It was a really quick hack.
Comment 53 Magnus Glantz 2010-11-08 15:42:23 EST
(In reply to comment #52)
> Try giving it the whole absolute pathname. 
That was it :-) Works great for me. Thanks Linus!

So, here's how to bypass the bug (without downgrading glibc), using Google Chrome.

Follow the instructions in comment #38 except for the last intruction, replace:
prompt$ LD_PRELOAD mymemcpy.so /opt/google/chrome/google-chrome &
with
prompt$ LD_PRELOAD=$(pwd)/mymemcpy.so /opt/google/chrome/google-chrome &
Comment 54 Magnus Glantz 2010-11-08 15:47:01 EST
Verified that this fix works for Firefox as well.

Just replace
LD_PRELOAD=$(pwd)/mymemcpy.so /opt/google/chrome/google-chrome &
with
LD_PRELOAD=$(pwd)/mymemcpy.so /usr/bin/firefox &

(Make sure you've shutdown all running copies of firefox before doing this)
Comment 55 Magnus Glantz 2010-11-08 16:03:19 EST
I'm sure thousands of people will find their way here, so, here's a quicky. To bypass this issue (which is an issue in Adobe Flash), you may run the below "fix" brought forth in comment #38

1. Cut and paste this into a prompt:
---Cut below---
cat > $HOME/Downloads/linusmemcpy.c <<EOF
#include <sys/types.h>

void *memcpy(void *dst, const void *src, size_t size)
{
void *orig = dst;
asm volatile("rep ; movsq"
:"=D" (dst), "=S" (src)
:"0" (dst), "1" (src), "c" (size >> 3)
:"memory");
asm volatile("rep ; movsb"
:"=D" (dst), "=S" (src)
:"0" (dst), "1" (src), "c" (size & 7)
:"memory");
return orig;
}
EOF
cd $HOME/Downloads
gcc -O2 -c linusmemcpy.c
ld -G linusmemcpy.o -o linusmemcpy.so
---Stop cutting here---

2. Shutdown any running copies of your webbrowser.

3. Until a Adobe has fixed their Flash player, start your webbrowser as below:

For Firefox users:
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/firefox &

For Google Chrome users:
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /opt/google/chrome/google-chrome &
Comment 56 Patrick 2010-11-08 21:52:31 EST
(In reply to comment #38)

Thank you for this neat workaround. Flash is now as happy as can be expected. In spite of the NOTABUG, hopefully the glibc developers or Fedora will come up with a more permanent solution because if we have to wait for Adobe to fix flash, hell in several dimensions will have frozen over multiple times.
Comment 57 drewskiwooskie 2010-11-09 08:30:40 EST
Bypass fixed my problem as well, thanks!
Comment 58 Neil Underwood 2010-11-09 12:48:05 EST
For Firefox users:
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/firefox &

For Google Chrome users:
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /opt/google/chrome/google-chrome &

and for opera users?
Comment 59 Teva Riou 2010-11-09 12:59:00 EST
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/opera &
Comment 60 Dan O'Brien 2010-11-09 13:30:55 EST
LD_PRELOAD works for me an FF.  Thanks, Linus.

Tag.
Comment 61 Neil Underwood 2010-11-09 13:39:30 EST
This fix does not work for me.  


[1037][neil@neil-laptop:~/Downloads]$ ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.                              
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.                                                                    
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.                                                                    
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.                                                                    
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
Comment 62 Bill Nottingham 2010-11-09 13:43:08 EST
If you're running with SELinux, you'd need to set the file context on the shared object correctly.

chcon --reference=/lib64/libc.so.6 <your file>
Comment 63 Neil Underwood 2010-11-09 13:54:51 EST
This still does not work for me after:

chcon --reference=/lib64/libc.so.6 <your file>

What exactly is this code doing?
Comment 64 Teva Riou 2010-11-09 13:59:14 EST
(In reply to comment #61)
> This fix does not work for me.  
> 
> 
> [1037][neil@neil-laptop:~/Downloads]$ ERROR: ld.so: object
> '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded:
> ignored.                              
> ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> cannot be preloaded: ignored.                                                   
> ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> cannot be preloaded: ignored.                                                   
> ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> cannot be preloaded: ignored.                                                   
> ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> cannot be preloaded: ignored.

Make sure linusmemcpy.so is in /home/neil/Downloads, otherwise copy it there or type the right path.

LD_PRELOAD=/full/path/to/linusmymemcpy.so /usr/bin/opera &
Comment 65 Neil Underwood 2010-11-09 14:08:45 EST
(In reply to comment #64)
> (In reply to comment #61)
> > This fix does not work for me.  
> > 
> > 
> > [1037][neil@neil-laptop:~/Downloads]$ ERROR: ld.so: object
> > '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded:
> > ignored.                              
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.                                                   
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.                                                   
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.                                                   
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.
> 
> Make sure linusmemcpy.so is in /home/neil/Downloads, otherwise copy it there or
> type the right path.
> 
> LD_PRELOAD=/full/path/to/linusmymemcpy.so /usr/bin/opera &

Everything is where it should be:

[1102][neil@neil-laptop:~/Downloads]$ locate linusmemcpy.so
/home/neil/Downloads/linusmemcpy.so                                                                                                                                         
[1102][neil@neil-laptop:~/Downloads]$ chcon --reference=/lib64/libc.so.6 /home/neil/Downloads/linusmemcpy.so
[1102][neil@neil-laptop:~/Downloads]$ LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/opera &
[2] 14013
[1]   Done                    LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/opera
[1103][neil@neil-laptop:~/Downloads]$ ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.

Sorry to be a pain.  My system doesn't seem like this.  This does work great for Firefox though.  It's just Opera.
Comment 66 Bill Nottingham 2010-11-09 14:15:25 EST
(In reply to comment #63)
> This still does not work for me after:
> 
> chcon --reference=/lib64/libc.so.6 <your file>

Just to be clear, replace '<your file>' with the path to your shared object (in your case /home/neil/Downloads/linusmemcpy.so ... change as necessary.)

> What exactly is this code doing?

Setting the SELinux context so that it's treated as a shared object with code as opposed to just a normal file.
Comment 67 Neil Underwood 2010-11-09 14:20:07 EST
(In reply to comment #64)
> (In reply to comment #61)
> > This fix does not work for me.  
> > 
> > 
> > [1037][neil@neil-laptop:~/Downloads]$ ERROR: ld.so: object
> > '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded:
> > ignored.                              
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.                                                   
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.                                                   
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.                                                   
> > ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD
> > cannot be preloaded: ignored.
> 
> Make sure linusmemcpy.so is in /home/neil/Downloads, otherwise copy it there or
> type the right path.
> 
> LD_PRELOAD=/full/path/to/linusmymemcpy.so /usr/bin/opera &

Everything is where it should be:

[1102][neil@neil-laptop:~/Downloads]$ locate linusmemcpy.so
/home/neil/Downloads/linusmemcpy.so                                                                                                                                         
[1102][neil@neil-laptop:~/Downloads]$ chcon --reference=/lib64/libc.so.6 /home/neil/Downloads/linusmemcpy.so
[1102][neil@neil-laptop:~/Downloads]$ LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/opera &
[2] 14013
[1]   Done                    LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/opera
[1103][neil@neil-laptop:~/Downloads]$ ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/home/neil/Downloads/linusmemcpy.so' from LD_PRELOAD cannot be preloaded: ignored.

Sorry to be a pain.  My system doesn't seem like this.  This does work great for Firefox though.  It's just Opera.
Comment 68 Neil Underwood 2010-11-09 14:25:38 EST
AAArrrrggg.  Sorry about the double post.  To be perfectly clear on my end, this is the exact command I am issuing:

chcon --reference=/lib64/libc.so.6 ~/Downloads/linusmemcpy.c

the exact location of linusmemcpy.c is /home/neil/Downloads

and the exact command to launch Opera:

LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/opera &

And by "What exactly is this code doing?"  I was referring to the contents of linusmemcpy.c.  I understand the SELinux issue.
Comment 69 Sitsofe Wheeler 2010-11-09 18:45:36 EST
Neil:
file `which opera`

If it's a script which plays with LD_LIBRARY_PATH this may be why you are struggling to get it working.
Comment 70 Neil Underwood 2010-11-09 21:51:42 EST
(In reply to comment #69)
> Neil:
> file `which opera`
> 
> If it's a script which plays with LD_LIBRARY_PATH this may be why you are
> struggling to get it working.

[1843][neil@neil-laptop:~]$ file `which opera`
/usr/bin/opera: POSIX shell script text executable

The contents of said "POSIX shell script" are:

#!/bin/sh
export OPERA_DIR=${OPERA_DIR:-/usr/share/opera}
exec /usr/lib/opera/opera "$@"

This is just a shell script that points to "/usr/lib/opera/opera"?  I'm not quite sure I get the reason behind this.

I appreciate all the assistance.  I'll just have to wait for the official update.  If I really can't handle the sound I'll just use FF for the time being.

*sigh*...why does Opera always have to be so damn difficult?...
Comment 71 Júlíus Þór Bess 2010-11-10 06:09:22 EST
Add:

export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so

to the Opera shell script so it looks like:

#!/bin/sh
export OPERA_DIR=${OPERA_DIR:-/usr/share/opera}
export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so
exec /usr/lib/opera/opera "$@"

That should do the trick.
Comment 72 Neil Underwood 2010-11-10 09:47:39 EST
(In reply to comment #71)
> Add:
> 
> export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so
> 
> to the Opera shell script so it looks like:
> 
> #!/bin/sh
> export OPERA_DIR=${OPERA_DIR:-/usr/share/opera}
> export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so
> exec /usr/lib/opera/opera "$@"
> 
> That should do the trick.

THAT DID IT!  Thank you so much.  I should have thought to try that.  I shall share this wealth of knowledge.
Comment 73 bacole 2010-11-10 14:46:21 EST
So this bug is marked NOTABUG, yet on the adobe forum, it's still asserted that the bug is in glibc (obviously not strictly correct but that's the conclusion there).

I don't see any bug report in the adobe bug database against flash:

https://bugs.adobe.com/jira/secure/IssueNavigator.jspa?header=FP

Seems like a bug should have been officially submitted to adobe before this bug report got moved to NOTABUG.

It's great that there is a workaround, but expecting all fedora 14 users to compile and maintain their own memcpy until adobe realizes what is wrong is not a very good situation.
Comment 74 H.J. Lu 2010-11-10 15:00:40 EST
64bit Fedora 14 is pretty much broken on machines with
SSE4.2. I ran into random crashes with 64bit Fedora 14 on
Intel Core i7. It turns out that 64bit strncasecmp
is broken on machines with SSE 4.2:

https://bugzilla.redhat.com/show_bug.cgi?id=651638
Comment 75 bacole 2010-11-10 15:16:50 EST
Ah, sorry I missed 
https://bugs.adobe.com/jira/browse/FP-5739
somehow didn't show up in my searches at first.
Comment 76 Julian Sikorski 2010-11-10 15:40:23 EST
This bug is mentioned here, in comment 35 to be precise.
Comment 77 Pavel Roskin 2010-11-10 17:41:59 EST
Just for the record, the problem playing music and video on vkontakte.ru (a.k.a. vk.com) is also due to this.  I tested the site extensively in Firefox with mymemcpy.so preloaded, and almost everything was fine.  I was able to get one video to produce the noise, so there must be another, less common problem unrelated to this issue.  However, almost everything would play correctly, whereas without mymemcpy.so, I would be lucky if any soundtrack would play.
Comment 78 Jan Engelhardt 2010-11-10 17:53:03 EST
(In reply to comment #40)
> 
> You can call it "crap software" all you like, but the thing is, if memcpy 
> doesn't warn about overlaps, there's no test coverage

So let's just add an abort() call to memcpy if it is detected that the areas overlap. Doing so is within the C specification and will surely get everybody their test coverage. Performance degradation, I hear? Limit it to -D_FORTIFY_SOURCE then maybe.
Comment 79 Charlie Brady 2010-11-10 18:08:51 EST
(In reply to comment #37)

> DESCRIPTION
>        The  memcpy()  function  copies  n bytes from memory area src to memory
>        area dest.  The memory areas should not overlap.

I read that as saying "should". Not "must". memcpy should copy n bytes from memory area src to memory area dest, even if the addresses overlap.
Comment 80 Matthew Garrett 2010-11-10 18:10:10 EST
Is it possible to use symbol versioning to provide applications built against new glibcs with the faster memcpy, without breaking buggy applications that rely on the old behaviour?
Comment 81 Bojan Smojver 2010-11-10 20:25:03 EST
(In reply to comment #46)
 
> The noise on the non-cached case was larger, but the glibc memcpy may have been
> faster. I say "may have been" because it went both ways: I did ten runs, and my
> LD_PRELOAD one still won 6 out of those 10 runs, but the noise was large enough
> that I will allow that I'm not going to guarantee anything.

Out of curiosity, on which CPU what this?
Comment 82 Christopher Curtis 2010-11-10 20:28:21 EST
(In reply to comment #79)

If you prefer, the POSIX man page for memcpy states: "If copying takes place between objects that overlap, the behavior is undefined." The change history is: "First released in Issue 1. Derived from Issue 1 of the SVID."

http://www.opengroup.org/onlinepubs/009695399/functions/memcpy.html


A quick Google search shows this to be the preferred language.  I'm not sure why it differs in the Linux man pages.  'Should' is semantically meaningless - it is either predictable or it is not.


The BSD man page [1993] corroborates:

http://www.unix.com/man-page/FreeBSD/3/memcpy/

BUGS
     In this implementation memcpy() is implemented using bcopy(3), and there-
     fore the strings may overlap.  On other systems, copying overlapping
     strings may produce surprises.  Programs intended to be portable should
     use memmove(3) when src and dst may overlap.


And here the text from an old version of the GNU C library documentation:

http://www.cs.utah.edu/dept/old/texinfo/glibc-manual-0.02/library_5.html#SEC61

Function: void * memcpy (void *to, const void *from, size_t size)

The memcpy function copies size bytes from the object beginning at from into the object beginning at to. The behavior of this function is undefined if the two arrays to and from overlap; use memmove instead if overlapping is possible.


For others looking back, I also have to note that you omitted the rest of the text from comment #37, which states: "Use memmove(3) if the memory areas do overlap."
Comment 83 Charlie Brady 2010-11-10 21:01:30 EST
> If you prefer, the POSIX man page for memcpy states: "If copying takes
> place between objects that overlap, the behavior is undefined." 

Yes, for POSIX compliance, it is necessary to assume that the behaviour is
undefined. I doubt, though, that Adobe is claiming POSIX conformance or
portability for its code.

I don't think we are talking about whether glibc/linux *may* change the
implementation while still staying POSIX compliant. We should be talking about
whether glibc/linux *should* change the current behaviour while programs exist
which don't follow the fine (but soft) advice of current documentation, and
will fail if that behaviour changes. That's an issue worthy of discussion,
which can't be decided by references to BSD documentation and POSIX standards.
Comment 84 Bojan Smojver 2010-11-10 21:14:32 EST
(In reply to comment #83)
 
> Yes, for POSIX compliance, it is necessary to assume that the behaviour is
> undefined. I doubt, though, that Adobe is claiming POSIX conformance or
> portability for its code.

Forget POSIX. It's been like this for forever. K&R C book, 2nd edition, page 250.
Comment 85 Siarhei Siamashka 2010-11-11 00:21:54 EST
(In reply to comment #46)
> (In reply to comment #43)
> > The upside is (ought to be) faster memcpy, which is something that helps a lot
> > of apps.  
> 
> Hey, I'm a big believer in fast memcpy's, I just don't believe that going
> backwards helps performance. 
> 
> In the kernel, the optimized x86 memcpy we use is actually a memmove(), because
> while performance is really important, so is repeatability and avoiding
> surprises (strictly speaking, we have two: the "rep movs" version for the case
> where that is supposed to be fast, and the open-coded copy version. The "rep
> movs" version is forwards-only and doesn't handle overlapping areas).
> 
> I dunno. I just tested my stupid "mymemcpy.so" against the glibc memcpy() on
> the particular kind of memcpy that valgrid reports (16-byte aligned 1280-byte
> memcpy).
> 
> I did both cached (same block over and over) and non-cached (a million blocks
> in sequence).
> 
> For the cached case my stupid LD_PRELOAD version was consistently a bit faster.

The same Intel developers submitted a similar optimization to libpixman, and provided the following explanation when asked about about this copying backwards part:
http://lists.freedesktop.org/archives/pixman/2010-August/000423.html

I also was not totally convinced that the backwards copy is really the best solution for the problem:
http://lists.freedesktop.org/archives/pixman/2010-September/000465.html
http://lists.freedesktop.org/archives/pixman/2010-September/000469.html
Comment 86 Timur Iskhodzhanov 2010-11-11 03:10:07 EST
(I'm sorry to jump in the middle of the discussion)

When a program is run under Valgrind it replaces memcpy with its own implementation:
http://code.google.com/p/valgrind-variant/source/browse/trunk/valgrind/memcheck/mc_replace_strmem.c#452

The latter is intentionally overlap-safe, so no surprise it 'fixes' the sound for you.
Plus it throws a Valgrind overlap report like those you've seen.
Comment 87 freetg 2010-11-11 19:51:14 EST
That's work for me.Thanks
Comment 88 Pavel Roskin 2010-11-12 00:09:41 EST
I wonder if libflashsupport could be resurrected to deal with this and further breakages in Adobe flash player.  The current libflashplayer code also adds jack support, which is another good thing: 
http://repo.or.cz/w/libflashsupport-jack.git
Comment 89 john getsoian 2010-11-12 08:03:08 EST
(In reply to comment #54)
> Verified that this fix works for Firefox as well.
> 
> Just replace
> LD_PRELOAD=$(pwd)/mymemcpy.so /opt/google/chrome/google-chrome &
> with
> LD_PRELOAD=$(pwd)/mymemcpy.so /usr/bin/firefox &
> 
> (Make sure you've shutdown all running copies of firefox before doing this)

Success on two systems running firefox with this fix. Many thanks!
Comment 90 John Villalovos 2010-11-12 08:47:18 EST
It would be interesting to see if the fix from Comment 19 of:
https://bugzilla.redhat.com/show_bug.cgi?id=651638#c19

Resolves this bug also.  There is an updated glibc there.
Comment 91 Charlie Brady 2010-11-12 08:56:51 EST
> It would be interesting to see if the fix from Comment 19 of:
> https://bugzilla.redhat.com/show_bug.cgi?id=651638#c19
>
> Resolves this bug also.

I don't see 'resolve regressions in misused memcpy' in the changes list:

    Update from master

        * Fix memory leak in fnmatch
        * Support Intel processor model 6 and model 0x2c
        * Fix comparison in sqrtl for IBM long double
        * Fix one exit path in x86-64 SSE4.2 str{,n}casecmp (BZ#12205, #651638)
        * Fix warnings in __bswap_16 (BZ#12194)
        * Use IFUNC on x86-64 memset
        * Power7-optimized mempcpy
        * Handle uneven cache size in 32bit SSE2 memset (BZ#12191)
        * Verify in ttyname that the symlink is valid (BZ#12167)
        * Update Danish translations
        * Fix concurrency problem between dl_open and dl_iterate_phdr
        * Fix x86-64 strchr propagation of search byte into all bytes of SSE

    register (BZ#12159)

        * Fix perturbing in malloc on free (BZ#12140)
        * PPC/A2 optimized memcpy function
        * Add C99 FP_FAST_FMA{,F,L} macros to <math.h>
        * Check that the running kernel is new enough (#649589)
Comment 92 John Villalovos 2010-11-12 09:43:20 EST
(In reply to comment #91)
> > It would be interesting to see if the fix from Comment 19 of:
> > https://bugzilla.redhat.com/show_bug.cgi?id=651638#c19
> >
> > Resolves this bug also.
> 
> I don't see 'resolve regressions in misused memcpy' in the changes list:

That is why I thought it would be interesting to see if it actually does resolve the issue :)
Comment 93 Pavel Roskin 2010-11-12 11:27:16 EST
I've tried glibc-2.12.90-19, as it doesn't fix the problem (quite expectedly).
Comment 94 Ray Strode [halfline] 2010-11-13 12:02:36 EST
Created attachment 460254 [details]
replace all memcpy calls with memmove calls in libflashplayer.so

I wrote this quick and dirty script last night after checking in on this bug report and reading the results. It fixes up the flash plugin to use memmove instead of memcpy across the board.  Caveats:

1) The script takes a long time to run (although I don't really know how long since I went off and did other things while it ran)
2) There may be practical performance implications in using memmove where memcpy is okay not sure. Although, comment 46 suggests it may not be that bad of a hit.
3) It works okay for me in that my audio doesn't clip anymore on the one version of libflashplayer.so that I've tried it on, but I haven't done any significant testing.
Comment 95 Uģis 2010-11-13 13:07:38 EST
(In reply to comment #94)
> Created attachment 460254 [details]
> replace all memcpy calls with memmove calls in libflashplayer.so
> 

Thanks man, you script works fine for me too. Gonna send fixed plugin to my non-geek brother, who is using fedora only becouse its the only distro that recognises his old usb wifi dongle...
Comment 96 Ben Bromley 2010-11-14 08:55:05 EST
(In reply to comment #71)
> Add:
> 
> export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so
> 
> to the Opera shell script so it looks like:
> 
> #!/bin/sh
> export OPERA_DIR=${OPERA_DIR:-/usr/share/opera}
> export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so
> exec /usr/lib/opera/opera "$@"
> 
> That should do the trick.

In a similar vein, if you run your own version of Firefox, you can add export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so to the run-mozilla.sh file.  For me, its in /opt/firefox/.  If you scroll most of the way to the bottom of that file, you will see this:
export MOZILLA_FIVE_HOME LD_LIBRARY_PATH
export SHLIB_PATH LIBPATH LIBRARY_PATH ADDON_PATH DYLD_LIBRARY_PATH

Right below that, I added:
export LD_PRELOAD=$HOME/Downloads/linusmemcpy.so

And it works just fine just calling /opt/firefox/firefox, so you can use a toolbar launcher instead of having to launch it from the command line every time.
Comment 97 Dan O'Brien 2010-11-14 09:16:43 EST
For a CLOSED NOTABUG bug report, seems to be a lot of traffic on it.  Is ANYONE actually fixing this problem either in glibc or in Flash?  This is ridiculuous.
Comment 98 Bill Davidsen 2010-11-14 09:23:00 EST
(In reply to comment #46)
> (In reply to comment #43)
> > The upside is (ought to be) faster memcpy, which is something that helps a lot
> > of apps.  
> 
> Hey, I'm a big believer in fast memcpy's, I just don't believe that going
> backwards helps performance. 
> 

> Do I have a point? I bothered to _measure_ the speed, and according to my
> measurements, glibc wasn't any faster than my trivial version and was likely
> slower. But I only tested two cases.
> 
> Regardless, it boils down to: we know the glibc change resulted in problems for
> real users. We do _not_ know that it helped anything at all.
> 

The justification of the change was to make memcpy faster *on little processors*. So unless you tested on something other than your usual machine, it may not have given you the relevant data. You never seemed like an ATOM kind of guy. But checking for overlap is just competent software design, not rocket science. We did it in the MULTICS era (late 1960s) in assembler. Omitting a "will this work" check in a library seems shoddy.
Comment 99 Ma Ling 2010-11-15 02:17:54 EST
The blow reason is why to use backward copy, it is good on Core2, NHM, Opteron.
glbc memcpy will prefer to backward/forward copy mode according to offset from source and destination.

Optimize memcpy by avoiding memory false dependece

All read operations after allocation stage can run speculatively,
all write operation will run in program order, and if addresses are
different read may run before older write operation, otherwise wait
until write commit. However CPU don't check each address bit,
so read could fail to recognize different address even they
are in different page.For example if rsi is 0xf004, rdi is 0xe008,
in following operation there will generate big performance latency.
1. movq (%rsi),	%rax
2. movq %rax,	(%rdi)
3. movq 8(%rsi), %rax
4. movq %rax,	8(%rdi)

If %rsi and rdi were in really the same meory page, there are TRUE
read-after-write dependence because instruction 2 write 0x008 and
instruction 3 read 0x00c, the two address are overlap partially.
Actually there are in different page and no any issues,
but without checking each address bit CPU could think they are
in the same page, and instruction 3 have to wait for instruction 2
to write data into cache from write buffer, then load data from cache,
the cost time read spent is equal to mfence instruction. We may avoid it by
tuning operation sequence as follow.

1. movq 8(%rsi), %rax
2. movq %rax,	8(%rdi)
3. movq (%rsi),	%rax
4. movq %rax,	(%rdi)

Instruction 3 read 0x004, instruction 2 write address 0x010, no any
dependence.  At last on Core2 we gain 1.83x speedup compared with
original instruction sequence.  In this patch we first handle small
size(less 20bytes), then jump to different copy mode. Based on our
micro-benchmark small bytes from 1 to 127 bytes, we got up to 2X
improvement, and up to 1.5X improvement for 1024 bytes on Corei7. 

Thanks
Ling
Comment 100 Michael Shurtleff 2010-11-16 21:28:07 EST
Since the Atom processor is mentioned, I noticed that this problem didn't occur on my Toshiba netbook with a N455 atom processor, but was intolerable on a laptop with dual core Pentium, both with updated Fedora 14.

I am not much affected by this problem, but I do think if you are serious about offering Linux on the desktop, you have to either make it work with Adobe flash, or state publicly that you don't support it. The average user doesn't care squat about whether the coding follows the rules or arguments over who is responsible if it doesn't work. If it works on Win 7 and doesn't work on Fedora, it is Fedora and Linux that take the crap. Period.
Comment 101 Linus Torvalds 2010-11-16 21:44:45 EST
I'd personally suggest that glibc just alias memcpy() to memmove().

Yes, clearly overlapping memcpy's are invalid code, but equally clearly they do happen. And from a user perspective, what's the advantage of doing the wrong thing when you could just do the right thing? Optimize the hell out of memmove(), by all means.

Of course, it would be even better if the distro also made it easy for developers to see the bad memcpy's, so that they can fix their apps. Even if they'd _work_ fine (due to the memcpy just doing the RightThing(tm)), fixing the app is also the right thing to do, and this would just make Fedora and glibc look good.

Rather than make it look bad in the eyes of users who really don't care _why_ flash doesn't work, they just see it not working right.

There is no advantage to being just difficult and saying "that app does something that it shouldn't do, so who cares?". That's not going to help the _user_, is it?

And what was the point of making a distro again? Was it to teach everybody a lesson, or was it to give the user a nice experience?
Comment 102 Yann Droneaud 2010-11-17 03:00:25 EST
(In reply to comment #101)

> Of course, it would be even better if the distro also made it easy for
> developers to see the bad memcpy's, so that they can fix their apps. Even if
> they'd _work_ fine (due to the memcpy just doing the RightThing(tm)), fixing
> the app is also the right thing to do, and this would just make Fedora and
> glibc look good.
> 

This is probably already done: using -D_FORTIFY_SOURCE=1 enable use of 
__builtin___memcpy_chk()
Comment 104 Magnus Glantz 2010-11-17 03:51:50 EST
From https://bugs.adobe.com/jira/browse/FP-5739
Shu Wang added a comment - 11/16/10 02:30 AM
Thanks very much for all your time to report the issue. Flash Player team is looking into it. Thanks.

I've emailed Shu Wang at Adobe to see if he can give a time estimate on fixing this.
Comment 105 Magnus Glantz 2010-11-17 04:20:51 EST
Shu Wang at Adobe told me that the Flash Player team is actively investigating the issue. They have no time estimate to share yet. I'll update this BZ as I get updated information.
Comment 107 Magnus Glantz 2010-11-17 04:25:34 EST
Here's the bomb:
Adobe says it may take months for them to fix this.
Comment 108 devzero2000 2010-11-17 04:32:22 EST
What is strange for a proprietary product ? :=) It is the norm.
Comment 109 Magnus Glantz 2010-11-17 04:39:18 EST
I request this bug to be re-opened.

Seriously degraded flash, for months, in Fedora seems much to much for at least me. This may cause more than a few users to change to other Linux distributions.

Andreas Schwab, can you roll back this change or make it work with Adobe Flash by some other means?
Comment 110 Andrei ILIE 2010-11-17 04:48:46 EST
Why not use f13 until Adobe releses an updated flash ?
Comment 111 Andrei ILIE 2010-11-17 04:49:55 EST
(In reply to comment #110)
> Why not use f13 until Adobe releses an updated flash ?

...or downgrade glibc
Comment 112 Michael Young 2010-11-17 05:54:08 EST
There is now a FESCo ticket about this issue at https://fedorahosted.org/fesco/ticket/501
Comment 113 Uģis 2010-11-17 05:59:24 EST
(In reply to comment #111)
> (In reply to comment #110)
> > Why not use f13 until Adobe releses an updated flash ?
> 
> ...or downgrade glibc

Or use Ray Strode's script to patch libflashplayer.so ...
Comment 114 Oliver Henshaw 2010-11-17 07:13:38 EST
(In reply to comment #107)
> Here's the bomb:
> Adobe says it may take months for them to fix this.

Could you give a link to this statement? I couldn't see it in the adobe bug, and don't know where else to look.
Comment 115 Magnus Glantz 2010-11-17 08:11:24 EST
(In reply to comment #114)
> (In reply to comment #107)
> > Here's the bomb:
> > Adobe says it may take months for them to fix this.
> 
> Could you give a link to this statement? I couldn't see it in the adobe bug,
> and don't know where else to look.
There is no link, I e-mailed Shu Wang at Adobe.
Comment 116 Yann Droneaud 2010-11-17 08:54:37 EST
(In reply to comment #102)
> (In reply to comment #101)
> 
> > Of course, it would be even better if the distro also made it easy for
> > developers to see the bad memcpy's, so that they can fix their apps.
> 
> This is probably already done: using -D_FORTIFY_SOURCE=1 enable use of 
> __builtin___memcpy_chk()

Nope.

When looking at glibc (and gcc), _FORTIFY_SOURCE only enable checks for
overflow, not overlap.
Comment 117 Magnus Glantz 2010-11-17 09:17:09 EST
(In reply to comment #94)
> Created attachment 460254 [details]
> replace all memcpy calls with memmove calls in libflashplayer.so
> 
> I wrote this quick and dirty script last night after checking in on this bug
> report and reading the results. It fixes up the flash plugin to use memmove
> instead of memcpy across the board.  Caveats:
> 
> 1) The script takes a long time to run (although I don't really know how long
> since I went off and did other things while it ran)
> 2) There may be practical performance implications in using memmove where
> memcpy is okay not sure. Although, comment 46 suggests it may not be that bad
> of a hit.
> 3) It works okay for me in that my audio doesn't clip anymore on the one
> version of libflashplayer.so that I've tried it on, but I haven't done any
> significant testing.
Works excellent for me! The script took 1-2 minutes to run on my laptop.

In short for anyone ending up here, you may try this to bypass the issue:
In a terminal, run:

wget https://bugzilla.redhat.com/attachment.cgi?id=460254 -O ~/Downloads/memcpy-to-memmove.sh
chmod +x ~/Downloads/memcpy-to-memmove.sh
cp /path/to/your/libflashplayer.so /path/to/your/libflashplayer.so.backup
~/Downloads/memcpy-to-memmove.sh /path/to/your/libflashplayer.so
Comment 118 Magnus Glantz 2010-11-17 14:25:04 EST
Good new, I just got an e-mail from Adobe, saying..

..they have a fix
..the fix has been send to QA/QE

They say that they cannot commit to any dates, but that they are taking the issue seriously.

I told them that if they want volunteers trying out their fix, we can help.
Comment 119 Pavel Roskin 2010-11-19 16:50:59 EST
Since there is no ETA from Adobe, I think we should consider a fix for the time being.  I'm just trying to brainstorm the problem.  Possible solutions could be:

1) Revert the glibc change.  While I don't think that a perfectly good change in glibc should be reverted to placate a proprietary plugin, this is not a perfectly good change.  It's essentially a workaround for a defect in some Intel processors.  It may improve benchmarks, but we can and should ask about real life benefits.  While operations on short arrays may become faster in some very specific cases, copying of large and well-aligned arrays is likely be slowed down by going backwards.

2) Add a wrapper to Firefox to use memmove() instead of memcpy().  Users of other browsers will do it for themselves.

3) I don't know if it's feasible, but maybe nspluginwrapper could be changed to intercept the memcpy() call from the plugins.  Maybe another wrapper could be bundled with nspluginwrapper.

4) Another wrapper could be written for flash plugin.  That wrapper should be part of the flash-plugin package, and it should be very strongly recommended by the fedora Flash page (http://fedoraproject.org/wiki/Flash)

5) The flash-plugin package should patch the plugin on install.  rpm verification will fail, but perhaps it's OK.  We could mark the plugin as a "configuration file".

6) The flash-plugin package should patch the plugin at the build time.  This could have legal consequences, but I don't think Adobe would actually enforce them considering that the change is purely to fix a flaw in their product.
Comment 120 Oliver Henshaw 2010-11-19 17:21:57 EST
Isn't nspluginwrapper unnecessary now that firefox has Out-of-Process-Plugins (i.e. plugin-container)? I honestly don't know what utility nspluginwrapper has remainiig to it, but suggestion #3 may amount to suggestion #2.

Re: suggestions #4-6 - isn't the flash-plugin package provided by adobe itself?
Comment 121 Pavel Roskin 2010-11-19 18:02:29 EST
There is an alternative package by Leigh Scott, mentioned on http://fedoraproject.org/wiki/Flash

It already includes a thin wrapper adding support for Athlon64 processors without support for the lahf instruction.  I tried just adding a safe memcpy() implementation to that file, but it didn't work.

I realize that it would be problematic to get ordinary users to use that repository, so integration of the fix with with nspluginwrapper, firefox or glibc would be preferable.
Comment 122 Paulius Zaleckas 2010-11-20 11:33:02 EST
Created attachment 461740 [details]
binary patch for "flashplayer_square_p2_64bit_linux_092710"

use bspatch from bsdiff package to apply this patch.

This patch changes only one byte in libflashplayer.so. I did this manually with hexedit after studying objdump -S output. The idea behind this change is not to replace every memcpy call with memmove one, but to alter dynamic symbol table to point to memmove instead of memcpy.
I hope someone can make a similar script like Ray Strode did, but using this less intrusive method.
Comment 123 Vinny Valdez 2010-11-24 15:39:35 EST
Comment #38 worked for me, thanks Linus!
Comment 124 Vlado Potisk 2010-11-25 11:01:58 EST
I tried the binary patch from comment #122, it works too.
Comment 125 Derrick Ornelas 2010-11-29 18:38:29 EST
I also tried the binary patch from comment #122, and it's been working without any problems so far.  Here's the general command for applying the patch:

    bspatch libflashplayer.so libflashplayer.sonew flash_64.bsdiff
Comment 126 Neil Underwood 2010-11-30 14:37:00 EST
(In reply to comment #122)
> Created attachment 461740 [details]
> binary patch for "flashplayer_square_p2_64bit_linux_092710"
> 
> use bspatch from bsdiff package to apply this patch.
> 
> This patch changes only one byte in libflashplayer.so. I did this manually with
> hexedit after studying objdump -S output. The idea behind this change is not to
> replace every memcpy call with memmove one, but to alter dynamic symbol table
> to point to memmove instead of memcpy.
> I hope someone can make a similar script like Ray Strode did, but using this
> less intrusive method.

This patch does not work for rekonq.  Works fine for me with konqueror, firefox, chromium & google-chrome however.  Rekonq still has the same sound issues.  Anyone know where rekonq looks for plugins?  I've installed the patched version of libflashplayer.so to:

/usr/lib/opera/plugins/libflashplayer.so
/usr/lib64/chromium-browser/plugins/libflashplayer.so
/usr/lib64/flash-plugin/libflashplayer.so
/usr/lib64/mozilla/plugins/libflashplayer.so
/usr/lib64/seamonkey-2.0.10/plugins/libflashplayer.so

And all the corresponding browsers work, except for rekonq.
Comment 127 Robin Bowes 2010-11-30 20:14:46 EST
Well, after enduring crappy flash sound since November 2, I finally stumbled across this bug.

I have implemented the workaround as per comment #55 and it is working for me.

I just want to comment that it is *ridiculous* that this issue has been closed as "NOTABUG" - it quite manifestly *is* a bug.

I'm no great fan of flash but it's an essential part of life on the web these days and I had thought that the Fedora project had finally put its days of broken flash support behind it.

The average punter, looking to evaluate Fedora will *not* care one jot *why* sound is broken in flash audio/video - they will just think "Fedora is crap" and move on to something else that isn't broken.

Please, re-open this issue and get a fix implemented for it ASAP.
Comment 128 Andre Robatino 2010-11-30 20:38:59 EST
(In reply to comment #127)

> I just want to comment that it is *ridiculous* that this issue has been closed
> as "NOTABUG" - it quite manifestly *is* a bug.

In Adobe's software.

> I'm no great fan of flash but it's an essential part of life on the web these
> days and I had thought that the Fedora project had finally put its days of
> broken flash support behind it.

Fedora's flash support is fine. Adobe's software is broken.

> The average punter, looking to evaluate Fedora will *not* care one jot *why*
> sound is broken in flash audio/video - they will just think "Fedora is crap"
> and move on to something else that isn't broken.

That's fine. Fedora Project's main priority is to improve its own software, not necessarily to maximize the number of users (unlike proprietary software, where maximizing the number of paid users is an end in itself). In particular, developers shouldn't waste time creating workarounds for third-party software and as a result causing developers of said software to be even less responsive than they already are, creating the need for more workarounds, ad infinitum.

> Please, re-open this issue and get a fix implemented for it ASAP.

Anything done in Fedora is a workaround, not a fix. Only Adobe can fix its own software.

In any case, it looks like Adobe may have "fixed" the problem in their usual responsive fashion. The latest update at http://labs.adobe.com/downloads/flashplayer10.html shows only 32-bit versions. If they have in fact abandoned 64-bit Flash yet again, it means that users have no choice but to use the 32-bit wrapped plugin which is not affected by this bug (in their software).
Comment 129 Linus Torvalds 2010-11-30 20:50:25 EST
(In reply to comment #128)
> 
> In Adobe's software.
> 
> > I'm no great fan of flash but it's an essential part of life on the web these
> > days and I had thought that the Fedora project had finally put its days of
> > broken flash support behind it.
> 
> Fedora's flash support is fine. Adobe's software is broken.

Quite frankly, I find your attitude to be annoying and downright stupid.

How hard can it be to understand the following simple sentence:

   THE USER DOESN'T CARE.

Pushing the blame around doesn't help anybody. The only thing that helps is Fedora being helpful, not being obstinate.

Also, the fact is, that from a Q&A standpoint, a memcpy() that "just does the right thing" is simply _better_. Quoting standards is just stupid, when there's two simple choices: "it works" or "it doesn't work because bugs happen".

Standards are paper. I use paper to wipe my butt every day. That's how much that paper is worth.

Reality is what matters. When glibc changed memcpy, it created problems. Saying "not my problem" is irresponsible when it hurts users. 

And pointing fingers at Adobe and blaming them for creating bad software is _doubly_ irresponsible if you are then not willing to set a higher standard for your own project.  And "not my problem" is not a higher standard.

So please just fix it.

The easy and technically nice solution is to just say "we'll alias memcpy to memmove - good software should never notice, and it helps bad software and a known problem".
Comment 130 Andre Robatino 2010-11-30 21:58:53 EST
(In reply to comment #129)

> Also, the fact is, that from a Q&A standpoint, a memcpy() that "just does the
> right thing" is simply _better_.

Assuming memcpy() (the existing one) is never more efficient than memmove(). If this is in fact known to _always_ be the case today, then I would agree. Otherwise, it's a bad idea. Optimized functions should continue to be available for those who need them, and it's not like Adobe didn't have a few decades of advance notice on how to code these functions properly. Anyone who finds that memmove() is just as fast in a particular case is free to use it.

> The easy and technically nice solution is to just say "we'll alias memcpy to
> memmove - good software should never notice, and it helps bad software and a
> known problem".

Good software _will_ notice, if it's using memcpy() deliberately, for better performance, and doesn't want it aliased. It's equally easy for Adobe (or anyone else) to just do a global search and replace in their own code, so as not to slow down everyone else's. But as mentioned above, it appears that they've abandoned their own 64-bit plugin again, so the "known problem" is gone - unless by their next version, they still haven't figured out the difference.
Comment 131 Michael Cronenworth 2010-11-30 22:32:40 EST
Andre, no one has abandoned anything. The Adobe website was just poorly updated. The 64-bit plugin was updated[1]. I am running on a *64-bit* Flash plugin version 10.3.162.29. The memcpy() issue is not fixed though. The date stamp is on 2010-11-16, which is around the time the issue was reported, but obviously too old.

As far as Fedora is concerned, I believe all of us here are free to make Fedora what we want. I don't know of any particular Fedora policy (please name one if I am wrong) that prevents any one of us from applying a workaround (be it glibc, be it nspluginwrapper, etc) that helps users *use* Fedora. Being petty and small and telling users to "get lost" makes me disgusted to be considered a Fedora contributor.

P.S. There's a bit much political BS in this bug, which is closed. It may be best to hold further comments until the results of tomorrow's FESCO meeting are known.

[1] http://labs.adobe.com/downloads/flashplayer10_square.html
Comment 132 Linus Torvalds 2010-11-30 23:02:26 EST
(In reply to comment #130)
> 
> Good software _will_ notice, if it's using memcpy() deliberately, for better
> performance, and doesn't want it aliased. 

Bullshit. You clearly don't know what you're talking about.

There are exactly two reasons for memmove() existing in the first place:

 (a) memcpy() hasn't necessarily handled overlapping areas, so people had to make up a new function just because you couldn't _rely_ on memcpy working right for that case.

   IOW, it's not an issue of "memcpy shouldn't handle overlapping areas automatically", but a "you can't rely on it even if it does, so for portability reasons you shouldn't".

 (b) you can make a _simpler_ memcpy() if you assume there are no overlapping areas (which is obviously the reason why memcpy originally didn't). But the "simpler" is really trivial - the traditional difference between memcpy and memmove is literally that memcpy always copied upwards, and memmove() simply just checks whether the destination address is above the source address and decides to copy backwards if so.


And quite frankly, neither excuse is valid these days for not just aliasing the two to the same thing (and make both do memmove).

There is no performance difference. The "is it overlapping" is about two small and fast instructions (no cache issues etc), depending on just how exact you want to do it (ie again, traditionally it's just that single compare and conditional jump - but you can be more precise if you want to by adding just a few more instructions).

In fact, I can pretty much guarantee that if you have a program that notices the one or two cycles of the overlapping check, then you have a program that actually prefers the _old_ glibc memcpy(), the simpler one that worked fine with the flash player too. The traditional one that always moved things upwards.

Because the whole bug was introduced by the change (did you take a look at glibc sources? I did) that made memcpy() _much_ more complicated, and now it does computed indirect jumps based on size etc. So the new-and-improved one is the one that takes a lot more cycles, exactly because it tries to handle the special cases. But then it doesn't handle the _simple_ special case of "is it overlapping?".

In short: there is simply no excuse for the current memcpy() behavior. It doesn't match historical behavior, so it breaks existing applications. And there is certainly no "simplicity" argument: if you want a simpler and more maintainable glibc code base, you just say "let's do memcpy and memmove with the same code, and not have those ugly #ifdef's and duplicate compilations".

And there is no performance argument. None. If the argument is that "memcpy()" should be as simple as possible and have minimal overhead and perform best for the case where a single cycle matters (ie everything cached, nicely aligned arguments etc), then the glibc changes should just be reverted entirely. 

So either re-instate the old traditional behavior ("memcpy is simple") that at least has some _historical_ reason behind it, and that makes flash work again.

Or alternatively, just make memcpy DTRT automatically. The check for overlap is not going to be noticed. And _not_ checking for overlap obviously was noticed.
Comment 133 Andre Robatino 2010-12-01 02:12:07 EST
Assuming your analysis is correct, aliasing memcpy() to memmove() would be preferable, since maintaining the extra code just to save one or two cycles per call isn't worth it IMO. And the bug Summary could be changed to something like "alias memcpy() to memmove()", since improving the GCC code itself is enough reason to make the change, regardless of whether broken applications are more likely to work. (Just wondering about the affect on application size, though - how different is the size of the memcpy() and memmove() code, and how often are they inlined? I'm guessing this is not a real issue, but it's the only other one I can think of.)
Comment 134 Kevin Fenzi 2010-12-01 14:19:26 EST
The latest flash 64bit plugin seems to have fixed this: 

http://download.macromedia.com/pub/labs/flashplayer10/flashplayer10_2_p3_64bit_linux_111710.tar.gz

(At least it fixes it fine here). 

As to reverting this due to technical arguments like https://bugzilla.redhat.com/show_bug.cgi?id=638477#c132 could the glibc maintainers please consider that and reply here? If they wish to close this bug again, thats their right, but one more revisit and reply to the technical arguments would be welcome. 

Thanks.
Comment 135 Michael Young 2010-12-01 14:33:41 EST
(In reply to comment #134)
> The latest flash 64bit plugin seems to have fixed this: 
> 
> http://download.macromedia.com/pub/labs/flashplayer10/flashplayer10_2_p3_64bit_linux_111710.tar.gz
> 
> (At least it fixes it fine here).

It is still broken for me. Maybe some of your workarounds are still in place.
Comment 136 Kevin Fenzi 2010-12-01 14:46:02 EST
Sigh. My bad. It's still broken in that version too. 

I tested a handfull of random youtube videos, but they were all ones that didn't have the bitrate that shows this. ;( 

So, yes, it's still broken. Use the workarounds or 32bit official plugin with nspluginwrapper. 
Sorry for the false alarm.
Comment 137 Christopher Stone 2010-12-03 22:32:58 EST
OH MY GOD!  I just tried Ray Strode's patch and not only does not fix the sound bug, but it makes flash WAY MORE STABLE!  I used to get flash crashing on me all the time whenever I played several videos at once, and now I'm noticing that I no longer see any crashes at all!  Incredible!
Comment 138 Christopher Stone 2010-12-03 22:33:56 EST
oopse that should have read "not only does fix the sound bug" :o
Comment 139 Geoff King 2010-12-04 12:58:24 EST
I just encountered this bug while previewing mp3s on the Amazon mp3 download website and was able to workaround by using #38/#55.  However, I'd like to note that this is type of bug is is especially frustrating since after reading the comments above it smells too political and idealistic.  

Please be reminded that according to the Fedora front page http://fedoraproject.org/en/, it is supposed to be stable and for everyday use.  This update clearly broke some of that stability whatever the cause.  There are many people (my mom and many co-workers) that would not have any clue as to how to fix this even with the workaround posted here.  

I would recommend that a notice be put on the fedora website or help page http://fedoraproject.org/en/get-help that references this bug or provides information as it apparently affects quite a few websites and both google-chrome and firefox.

Regards, Geoff
Comment 140 Dan O'Brien 2010-12-04 13:19:01 EST
Good comments, GS, but from the above discussion, it is apparent that Fedora is not meant for "users" but for "contributers".  Your mom and coworkers would be better off with Ubuntu. If Fedora keeps going in the direction it is,  Ubuntu might end up with a few more followers and Fedora with a few less.  I want a Linux system that works.  I've put up with Fedora's "purity" policies from the beginning, but this one might be the last straw up with which this camel will not put.
Comment 141 john getsoian 2010-12-05 00:23:41 EST
(In reply to comment #137)
> OH MY GOD!  I just tried Ray Strode's patch and not only does not fix the sound
> bug, but it makes flash WAY MORE STABLE!  I used to get flash crashing on me
> all the time whenever I played several videos at once, and now I'm noticing
> that I no longer see any crashes at all!  Incredible!

Indeed. I updates from Linus' patched memcpy preload to Ray Strode's patch and all the the little hesitations and timing stumbles I'd always gotten on some sites disappeared.
Comment 142 Pablo Rodríguez 2010-12-06 15:53:24 EST
I'm an average user that experiences distorted sound at full-screen videos from YouTube (such as http://www.youtube.com/user/jamesbluntmusic#p/u/0/x1yOGhnmYfI) or from Colbert Nation (http://www.colbertnation.com/the-colbert-report-videos/367133/).

Since my processor is 32 bits and not 64 bits, I wonder whether it is the same bug as mine.

I have tried to compile the code from comment 55, but I'm afraid I get the following error:

  $ gcc -O2 -c linusmemcpy.c
  linusmemcpy.c: Assembler messages:
  linusmemcpy.c:6: Error: number of operands mismatch for `movs'

This is all Greek to me. Any idea on how to fix this?

Many thanks for your help,


Pablo
Comment 143 Benny 2010-12-06 17:34:40 EST
Created attachment 465094 [details]
The patched flashplayer plugin 10_2_p3_64bit 111710

Just install this plugin instead of the adobe one and everything should work. I fixed the adobe one it with Ray Strode's script.
Comment 144 Jesse Keating 2010-12-06 17:42:14 EST
(In reply to comment #143)
> Created attachment 465094 [details]
> The patched flashplayer plugin 10_2_p3_64bit 111710
> 
> Just install this plugin instead of the adobe one and everything should work. I
> fixed the adobe one it with Ray Strode's script.

This is likely something we should not be distributing.
Comment 145 sverrel 2010-12-06 19:07:49 EST
Please do - trying to get to the attached pre patched flashplayer #143 - give me permission denied :(
Comment 146 Michael Young 2010-12-06 19:43:38 EST
(In reply to comment #145)
> Please do - trying to get to the attached pre patched flashplayer #143 - give
> me permission denied :(

Posting a patched version of flash player breaks the license http://labs.adobe.com/technologies/eula/flashplayer10.html (in two places I think) which will be why access to that file has been removed.
Comment 147 Benny 2010-12-07 04:28:32 EST
My deepest apologies for breaking the license. It wont happen again... I was just trying to help some people who got mixed up in all the amount of comments here... 

For users who still have problems: 
I "incidentally found" the file here: http://rapidshare.com/files/435418850/flashplayer10_2_p3_64bit_linux_111710.tar.gz
Comment 148 sverrel 2010-12-07 11:25:27 EST
Great find Benny ! Worked like a sharm :) Thank !!
Comment 149 e_redhat 2010-12-08 10:41:13 EST
I've been using RedHat since RedHat used to release a binary desktop, before they created the Fedora Project.  I've used Fedora and Centos for non-profit uses since, including for OpenStandards.net and personal use.  

When I installed the 64-bit Fedora, and had to go through a pain to install 64-bit Flash, I used to, but unhappy with, the process for installing such a critical component.  

When recently installing Ubuntu on a friend's Netbook to "correct" her Windows virus problem, I couldn't believe how easy it was to install Flash.  It was a check box on the normal install process!  I found out non-technical friends were installing Ubuntu and very happy with it.  This was obviously because Ubuntu make it possible for a non-technical person to install Flash.  We all know that if you replace Windows with Ubuntu for a friend, and do not install Flash, they will call you pretty quickly asking why they can't view things on the web.  

I've noticed the sound problem with Fedora, narrowing it down to just the web browsers, both Firefox and Chrome.  Adding the lack of a Boxee build for Fedora, and I've been telling all non-technical friends to install Ubuntu.  I've described Fedora as an unstable experimental version designed only for technical people.  (To be sure, previous incidents with the nVidia driver leading me to have to re-install Fedora after a kernel update caused the loss of video, and paranoia that this can happen again, fed into this conclusion.)

Then, I found and read this thread.  I used Ray's patch to correct my sound problem.  Thanks, Ray!  

I concur 100% with Linus on this one.  The changes to glibc need to be rolled back in an update to Fedora 14 until Adobe corrects their Flash binary!  If you don't do this, there will be a lot more people like me, who, seeing only the surface, will tell everyone to install Ubuntu.  This needs to be done immediately!  People have over 4 GB of ram in new computers, and they need the 64-bit version to use it.  

I'm glad to see there are people like Linus that would like to see Fedora become a mainstream desktop, and not just an experimental distribution for technical people.  I hope those in control will learn from this and not only roll back the glibc change ASAP, but also look into streamlining the Flash install like Ubuntu did so non-technical people can have it when they install Fedora with default settings.
Comment 150 Vlado Potisk 2010-12-08 12:12:32 EST
I'd like to point out that malloc() has its MALLOC_CHECK_ environment variable which activates less efficient implementation designed to be tolerant 
against simple errors.

Memcpy() bugs happen just like malloc() bugs do. So why not a MEMCPY_CHECK_ ?
Comment 151 Benny 2010-12-08 14:54:12 EST
(In reply to comment #149)

Hear hear.
Comment 152 Homme Bitter 2010-12-08 16:21:57 EST
One more supporter for Linus' plea. End users don't care what the cause is, they want it fixed. Sure, Adobe is making themselves look bad by taking way too long to patch this. Even an intermediate fix, just for this problem, can be pushed through bureaucratic mazes in just a few days. However, this is not taking the responsibility away from Fedora to make things "just work".

I, being someone that is heavily into performance tuning, think that winning cycles, no matter how little, in often used functions is a good thing(tm). Therefor, I agree that improving glibc by doing these optimizations should be done.

Several workarounds have been mentioned to either mitigate the problem in glibc, or to provide a workaround somewhere else in Fedora. Neither has been done, in over a month and over 100 comments on this bug. Given the amount of updates I receive daily for FC14, I think this is an issue that should have been dealt with long ago. No way is it possible that nobody with write access to the relevant packages hasn't had any time to deal with this yet.

I would like to urge anyone with any political power withing Fedora to get this problem dealt with immediately. An organization this big and mature shouldn't be having trouble dealing with this sort of thing, fix it, make certain it won't happen again and move on.
Comment 153 Bill Davidsen 2010-12-08 16:30:37 EST
(In reply to comment #149)

> I concur 100% with Linus on this one.  The changes to glibc need to be rolled
> back in an update to Fedora 14 until Adobe corrects their Flash binary!  If you
> don't do this, there will be a lot more people like me, who, seeing only the
> surface, will tell everyone to install Ubuntu.  This needs to be done
> immediately!  People have over 4 GB of ram in new computers, and they need the
> 64-bit version to use it.  

This is a fine urban myth, but only that. The 32 bit "PAE" kernel can address 36 bits of memory space, or 64GB of physical RAM. The only real benefit would be to people who have a single user process limited by 4GB of RAM, and that's a very small number. If you compile your own apps for 64 bit, with the right options and use the right algorithms, you might see about a 5% speedup due to more registers. I have never seen that in any RPM distributed Fedora software, and I would love 5% faster ffmpeg.

I humbly advise that given my lack of drama with 24GB RAM and PAE, and lack of issues running better supported 32 bit apps, that people stay with 32 bit PAE unless they have a specific reason to go 64. I know it's not the "next big thing" but it works and uses all your memory.
Comment 154 Kevin Fenzi 2010-12-08 16:41:16 EST
Pretty please with sugar on top, could people please refrain from adding comments here that are not technical in nature? I have asked the maintainers to revisit this based on the technical arguments. 

I would suggest to those that need flash working now that you use the 32bit flash plugin with nspluginwrapper. See: http://fedoraproject.org/wiki/Flash for information on installing it. As this is in fact the only released and security updated flash plugin adobe ships (the 64bit one is a beta and doesn't promise security releases). 

Adding non technical comments just makes it less likely that maintainers will be able to filter out those and revisit this. Thanks.
Comment 155 hdfssk 2010-12-14 01:01:36 EST
Would someone who's allowed please add the CommonBugs keyword to this bug? It'll save a lot of folks search time once it's listed on the http://fedoraproject.org/wiki/Bugs/Common page.
Comment 156 Adam Williamson 2010-12-14 03:59:04 EST
anyone can add keywords, I think.
Comment 157 Alan Lehman 2010-12-23 21:51:12 EST
In replay to comment #147, this seemed to correct the problem for Youtube/240P, but not for some others, such as:
http://bestofthebaytv.com/view/1275/Salesjobscom

Fedora 14, 64 bit
Comment 158 john getsoian 2010-12-24 11:29:29 EST
RE: comment 157

The site referenced plays OK here (Firefox 3.6.13) using Ray Strode's patch (comment 94)
Comment 159 Jacek Stępniewski 2010-12-24 18:49:58 EST
Just wanted to say, that there's the same problem with Rhythmbox. When I wanted to play BBC Radio 4 podcasts, I got similar distorted sound.
Using mymemcpy from comment #38 also helps.

Jacek
Comment 160 Sergio Monteiro Basto 2010-12-24 20:08:10 EST
Comment 94 also fix sound problem for me
Comment 161 Alan Lehman 2010-12-24 22:35:23 EST
That seemed to work for me as well.
Comment 162 Paulo Roma Cavalcanti 2010-12-25 15:52:19 EST
MPD (Music Player Daemon - http://mpd.wikia.com/wiki/Music_Player_Daemon_Wiki) can be set to output an http stream, which is processed by the totem plugin in firefox.

MPD is also being affected by this bug, and it has nothing to do with the flash-plugin. The fix on comment #38 (Linus Torvalds) works just fine in this case, though. 

I also have an .src.rpm that applies the fix on comment #94 (Ray Strode) during
the build, that is, it brings an unmodified flash-plugin an applies
the script when the rpm is created for 64 bits:

http://orion.lcg.ufrj.br/RPMS/src/flash-plugin-10.3-1.fc14.src.rpm

    File: libflashplayer.so
    Version: 
    Shockwave Flash 10.3 d162
Comment 163 Paulo Roma Cavalcanti 2010-12-28 05:01:00 EST
The real problem is gstreamer.

Just try to play an avi (e.g., with divx codec) using gst123 or totem,
and the sound will be completely distorted.

The fix on comment #38 also solves the problem.
Comment 164 Paulo Roma Cavalcanti 2010-12-29 05:35:13 EST
I tracked down the problem using valgrind,
and the culprit was a libgstflump3dec.so
in my ~/.gstreamer-0.10/plugins/

This plugin was dated from 2007. After deleting it,
gstreamer is working just fine.

However, this plugin has never caused any trouble before, though.
Comment 165 Dan O'Brien 2010-12-29 06:28:29 EST
#164, nice work. You found another "proprietary" binary that used the *bad* memcpy for overlapping memory.
Comment 166 Patrick 2010-12-29 07:55:30 EST
#164: nice find. did you report this to Fluendo?
Comment 167 Paulo Roma Cavalcanti 2010-12-31 04:37:28 EST
Another person has already emailed Fluendo directly.


I also have Linux Torvalds' solution here:

http://orion.lcg.ufrj.br/RPMS/src/memcpy-1.0-1.fc14.src.rpm

It installs a fixed memcpy in /opt/memcpy and provides a script in /usr/bin
that preloads it to a given program:

memcpy-preload.sh   prog_name   prog_arguments

for instance:

memcpy-preload.sh google-chrome

Booth solutions are equivalent in terms of fixing the 64 bit flash-plugin, but this one can be applied to other programs as well.
Comment 168 Pablo Rodríguez 2010-12-31 07:09:34 EST
(In reply to comment #167)
> I also have Linux Torvalds' solution here:
> 
> http://orion.lcg.ufrj.br/RPMS/src/memcpy-1.0-1.fc14.src.rpm

I'm extremely interested in your fix.

As I already reported (comment 142), I experience a similar problem in Flash when played at full screen (Fedora 13 played the videos fine, but Fedora 14 distorts their audio)

I'm on 32bit and I cannot compile it due to the following error:

+ ./make-mymemcpy.sh
mymemcpy.c: Assembler messages:
mymemcpy.c:6: Error: number of operands mismatch for `movs'
mymemcpy.o: could not read symbols: File in wrong format

Is there any way I can compile it under 32bits?

Thanks for your help and happy New Year to everybody,


Pablo
Comment 169 John Robinson 2010-12-31 10:29:45 EST
(In reply to comment #168)
[...]
> I'm on 32bit and I cannot compile it

Pablo, Ray Strode's script in comment #94 should work on 32 bits as well.
Comment 170 Paulo Roma Cavalcanti 2010-12-31 11:09:39 EST
(In reply to comment #168)
> (In reply to comment #167)
> > I also have Linux Torvalds' solution here:
> > 
> > http://orion.lcg.ufrj.br/RPMS/src/memcpy-1.0-1.fc14.src.rpm
> 
> I'm extremely interested in your fix.
> 
> As I already reported (comment 142), I experience a similar problem in Flash
> when played at full screen (Fedora 13 played the videos fine, but Fedora 14
> distorts their audio)
> 
> I'm on 32bit and I cannot compile it due to the following error:
> 
> + ./make-mymemcpy.sh
> mymemcpy.c: Assembler messages:
> mymemcpy.c:6: Error: number of operands mismatch for `movs'
> mymemcpy.o: could not read symbols: File in wrong format
> 
> Is there any way I can compile it under 32bits?
> 
> Thanks for your help and happy New Year to everybody,

The code is written in assembler, and is intended to 64 bits.
That is why you can not compile it.

I can play both videos fine (#142), but only the James Blunt's video goes to full screen here.

However, I think your issue is different. The memcpy problem in flash 64 bits makes the sound unrecognisable, and it is not a simple stuttering or something like that.

As John said in comment #161, Ray Strode's script replaces all memcpy calls for
memove, and can be applied in 32 bits (although my .src.rpm only applies the script for 64 bits).
Comment 171 Hamidou Dia 2011-01-03 05:18:24 EST
*** Bug 663784 has been marked as a duplicate of this bug. ***
Comment 172 Pablo Rodríguez 2011-01-03 15:43:18 EST
(In reply to comment #170)
> The code is written in assembler, and is intended to 64 bits.
> That is why you can not compile it.

Thanks for your replies, Paulo and John.

I had no idea that the code was written in assembler (I'm only a user, not a programmer, so all code is Greek to me).

> However, I think your issue is different. The memcpy problem in flash 64 bits
> makes the sound unrecognisable, and it is not a simple stuttering or something
> like that.

I know that it is different, but it might have the same cause. I already reported it (bug 661385). But since I had no reply, I wanted to try whether any of these fixes could work.
 
> As John said in comment #161, Ray Strode's script replaces all memcpy calls for
> memove, and can be applied in 32 bits (although my .src.rpm only applies the
> script for 64 bits).

I have downloaded the script, but invoking "./fix-flash.sh /usr/bin/firefox" gives me the following warning:

./fix-flash.sh /usr/bin/firefox
objdump: /usr/bin/firefox: File format not recognized
objdump: Section '.plt' mentioned in a -j option, but not found in any input file
Can't find memcpy call in /usr/bin/firefox PLT

Sorry, but again it's Greek to me. How should I use the fix?

Thanks for your help,


Pablo
Comment 173 Dale Stimson 2011-01-03 16:08:04 EST
Re Comment 172

You can't patch /usr/bin/firefox because (under Fedora, at least) it is
a bash script that in turn runs the firefox binary, which currently (Fedora 13)
is /usr/lib/firefox-3.6/firefox  .  If you wish, try patching that:

./fix-flash.sh /usr/lib/firefox-3.6/firefox
Comment 174 Pablo Rodríguez 2011-01-03 16:19:06 EST
(In reply to comment #173)
> Re Comment 172
> 
> You can't patch /usr/bin/firefox because (under Fedora, at least) it is
> a bash script that in turn runs the firefox binary, which currently (Fedora 13)
> is /usr/lib/firefox-3.6/firefox  .  If you wish, try patching that:

Thanks, Dale.

I'm afraid I get another errorsA:

./fix-flash.sh /usr/lib/firefox-3.6/firefox 
./fix-flash.sh: line 11: [: too many arguments
./fix-flash.sh: line 16: 0x080495a8 - 0x08049338
08049478: value too great for base (error token is "08049478")

No idea what might be wrong.

Thanks again,

Pablo
Comment 175 Paulo Roma Cavalcanti 2011-01-03 16:39:53 EST
You did not understand. What should be patched is the flash-plugin (libflashplayer.so), and not firefox.

I changed my .src.rpm to patch even the 32 bit version:

rpmbuild --rebuild flash-plugin-10.3-1.fc14.src.rpm

You should see it downloading the source from adobe and patching it.
After doing that, install the created rpm and restart firefox.

You should check using "about:plugins" in firefox address bar, whether it is really using the new version.
Comment 176 Pablo Rodríguez 2011-01-04 10:04:00 EST
(In reply to comment #175)
> I changed my .src.rpm to patch even the 32 bit version:
> 
> rpmbuild --rebuild flash-plugin-10.3-1.fc14.src.rpm
> 
> You should see it downloading the source from adobe and patching it.
> After doing that, install the created rpm and restart firefox.

Thanks, I rebuilt your source rpm and it downloaded and patched the flash-plugin.

But it didn't work.

Thanks anyway,


Pablo
Comment 177 Alexander Hunt 2011-01-17 23:33:55 EST
Re: comment 175:
This method worked perfectly for me. (F14-x86_64 - Intel MacBook)
 
For other users like myself unfamiliar with rpmbuild (I had to do some googling to figure this part out) install rpmdevtools (inc deps) and chrpath (I could only find 0.13.6.fc12-x86_64, but it worked). 

Thank You to all involved with figuring this one out, and an aside I didn't have to implement Linus' (et.al) patch, although Thank You to all of you as well, that was next...
Br, 
alex
Comment 178 Aaron Luchko 2011-01-18 03:33:01 EST
Paulo's rpm didn't build for me because it was complaining about a missing build id. I uploaded an updated version of the rpm with the ld line changed to "ld -G --build-id mymemcpy.o -o mymemcpy.so" and it's working for me.

http://luchko.ca/~aaron/memcpy-1.1-1.fc14.src.rpm
Comment 179 devzero2000 2011-01-21 08:34:29 EST
Just for fun, it's appropriate to say so it seems to me :=), I set up the GNU build system for mymemcpy, so you can do the usual ./configure make make install. Yes, it is overkill but it is "nice" hope. It is here

https://github.com/yersinia/junkcode/tree/master/linux/mymemcpy

and here the new (make distcheck) tarball

https://github.com/yersinia/junkcode/blob/master/linux/mymemcpy/mymemcpy-1.1.tar.gz

Regards
Comment 180 Michael Convey 2011-01-25 11:44:52 EST
I have no sound in firefox, but do have sound in other applications. 

I tried to cut and paste the code from #38 to my command prompt (as root) but nothing seemed to run. 

I also tried to copy the code from #94 into a file called ray.sh, made it executable, and issued the command "bash ray.sh" as root, but got the following error:

# bash ray.sh
objdump: 'a.out': No such file
objdump: Section '.plt' mentioned in a -j option, but not found in any input file
Can't find memcpy call in  PLT

Please forgive my ignorance, but can someone please provide detailed steps on how to implement these fixes.

Thanks
Comment 181 devzero2000 2011-01-26 08:20:57 EST
If you want something more simple download the tarball referenced on #179 and do

tar -zxvf mymemcpy-1.1.tar.gz

cd mymemcpy-1.1

./configure

make 

make install (ad root or use sudo)

e follow the instruction on the README, changing the application after LD_PRELOAD if necessary.

Regards
Comment 182 Siddhesh Poyarekar 2011-01-26 10:09:45 EST
A slightly modified version of Linus' memcpy routine so that it builds for 32 bit as well:

1. Cut and paste this into a prompt:
---Cut below---
cat > $HOME/Downloads/linusmemcpy.c <<EOF
#include <sys/types.h>

void *memcpy(void *dst, const void *src, size_t size)
{
void *orig = dst;
#if __WORDSIZE == 64
asm volatile("rep ; movsq"
#else
asm volatile("rep ; movsl"
#endif
:"=D" (dst), "=S" (src)
:"0" (dst), "1" (src), "c" (size >> 3)
:"memory");
asm volatile("rep ; movsb"
:"=D" (dst), "=S" (src)
:"0" (dst), "1" (src), "c" (size & 7)
:"memory");
return orig;
}
EOF
cd $HOME/Downloads
gcc -O2 -c linusmemcpy.c
ld -G linusmemcpy.o -o linusmemcpy.so
---Stop cutting here---

2. Shutdown any running copies of your webbrowser.

3. Until a Adobe has fixed their Flash player, start your webbrowser as below:

For Firefox users:
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /usr/bin/firefox &

For Google Chrome users:
LD_PRELOAD=$HOME/Downloads/linusmemcpy.so /opt/google/chrome/google-chrome &
Comment 183 Siddhesh Poyarekar 2011-01-26 10:11:53 EST
*** Bug 661385 has been marked as a duplicate of this bug. ***
Comment 184 Michael Young 2011-01-26 10:43:57 EST
(In reply to comment #182)
> A slightly modified version of Linus' memcpy routine so that it builds for 32
> bit as well:

It builds for 32-bit but it doesn't work for me for 32-bit. I tried a simple test with ls which truncates filenames and ls -al which segfaults!
Comment 185 Maciej Żenczykowski 2011-01-26 19:16:14 EST
Here, this one actually has a chance of working, still, totally untested.

---Cut below---
cat > $HOME/Downloads/linusmemcpy.c <<EOF
#include <sys/types.h>

void *memcpy(void *dst, const void *src, size_t size)
{
void *orig = dst;
#if __WORDSIZE == 64
asm volatile("rep ; movsq" :"=D" (dst), "=S" (src)
: "0" (dst), "1" (src), "c" (size >> 3) : "memory");
asm volatile("rep ; movsb" : "=D" (dst), "=S" (src)
: "0" (dst), "1" (src), "c" (size & 7) : "memory");
#else
asm volatile("rep ; movsl" :"=D" (dst), "=S" (src)
: "0" (dst), "1" (src), "c" (size >> 2) : "memory");
asm volatile("rep ; movsb" : "=D" (dst), "=S" (src)
: "0" (dst), "1" (src), "c" (size & 3) : "memory");
#endif
return orig;
}
EOF
cd $HOME/Downloads
gcc -O2 -c linusmemcpy.c
ld -G linusmemcpy.o -o linusmemcpy.so
---Stop cutting here---
Comment 186 Aaron Luchko 2011-01-28 22:42:27 EST
Just for people who don't want to build the rpm themselves I put up a pre-built version

http://luchko.ca/~aaron/memcpy-1.1-1.fc14.x86_64.rpm

just install and start firefox as

$ memcpy-preload.sh firefox

I've tested this on two machines and it fixes the problem for me.
Comment 187 Alexander Hunt 2011-01-29 01:30:29 EST
RE: Comment 186.
Thanks Aaron, that was great timing. I just did a fresh install of F14-x86_64, so I've just installed your rpm, and it works perfectly for me too. I just had to use the full path since I'm testing the the beta Firefox. The path for my shortcut is: memcpy-preload.sh /usr/share/mozilla/firefox/firefox
Again Thanks and Best Regards
Comment 188 Fanis Attard 2011-01-29 01:44:32 EST
Thanks a lot Aaron. Three machines included mine....
Now I can listen again to the music of hundreds/thousands of sites .....
I installed the memcpy as you said above.
Then I changed the command in the menu editor for the firefox and instead of : firefox %u , I put your command : memcpy-preload.sh firefox .
So I have the same feeling as previously. There is no need to run firefox in the konsole.
Really, I expected an official Fedora update to fix this major problem, even if Adobe has the responsibility for the bug.
Because the flash plugin is a necessary villain but a really small "tool" compared to the Fedora OS, that is a big "factory" with rich equipment and a lot of "deposits" full of tools. 
It is not permitted for such a factory to raise the hands.......
Comment 189 Michael Convey 2011-01-29 03:53:06 EST
RE: Comment 186

Thanks. I downloaded your rpm and installed it as follows:

yum localinstall --nogpgcheck memcpy-1.1-1.fc14.x86_64.rpm

It wouldn't install without the "--nogpgcheck" flag

RE Comment 186 & Comment 187

I copied my firefox shortcut and changed the command to 

memcpy-preload.sh /usr/share/mozilla/firefox/firefox

but it didn't work. So I tried 

memcpy-preload.sh firefox

And it worked! However the sound is kind of quiet even with the speakers turned all the way up. Also, if I exit a youtube video during play, it makes a brief, but very loud hum.
Comment 190 Fanis Attard 2011-02-09 00:47:30 EST
I got the official update of glibc today (Feb 9 2011) and I hoped that the problem with flash plugin might be solved. So I tried the command firefox %u but I was wrong..... The problem remains !!
Thanks again Aaron for your memcpy-preload.sh firefox.
Earth calls Fedora Planet. Is anybody out there ? Over .
Comment 191 Leslie Satenstein 2011-02-14 22:53:42 EST
Up to date as of 2011Feb14, and the problem persists.
Comment 192 Christopher Fabritius 2011-02-16 19:18:28 EST
I noticed the status is now assigned, but the fix (based on my understanding of the discussion above) appears to be a trivial rollback of a change to glibc.

Do you need additional triage information to fix this issue? 
 
Confirming the issue on a current Fedora 14, flash-plugin-10.3.162.29-1.x86_64
Comment 193 Simon Lewis 2011-02-21 06:32:11 EST
Please rollback changes to Glib - this is a servere bug on fedora fc14 x86_64 with intel hardware...

Also, the severity should be marked as "urgent" not medium - this is a real show stopper.
Comment 194 Adam Williamson 2011-02-21 13:05:26 EST
a glitch with an unsupported (upstream) version of an unsupported (by us) third-party plugin does not qualify for any kind of 'urgent' status by any stretch of the imagination, I'm afraid.

running the 64-bit Flash plugin is just generally a bad idea, because they never keep it up to date with the 32-bit one, so it's always vulnerable to several known serious security issues. And 'known serious security issues in Flash' is really not something you want to be exposing your computer to. Just run the 32-bit version, really, you'll be doing yourself a favour.
Comment 195 Dan O'Brien 2011-02-21 15:40:31 EST
I give up waiting for a fix and will just install the 32 bit wrapped one.  This does not make 64 bit Fedora a standout product.  You can sit in the ivory tower all you want, but you end up looking silly.
Comment 196 Alexander Hunt 2011-02-21 17:47:49 EST
From an end-user perspective, and in my humble opinion:
I refuse to install anything that pulls in i686 dependencies to my x86_64 system and for that reason will not install software such as Skype and any of the Google for Linux software, and that includes flash wrappers. If I wanted a mixed up UNIX OS I would just use Snow-Leopard on my MacBook and save myself a ton of work making Linux work as I want my OS to work. 
I understand that having compatibility to the new mini-processors is important to somebody who uses mini-machines, but really, would it be that difficult to make a version of glib for those and another one for the mainstream processors? What I mean is make a note in the release notes of the current version for which mini-processors it is supporting. At the same time, revert the glib changes, re-package it, and make a note in the release notes for which processors it is supporting. Problem solved. If someone makes an error in installation it is easy to revert to the other version.
I have read through every comment in this bug issue several times since it was started and have noticed some interesting things: Redhat Fedora developers defending their product, saying they aren't interested how many users switch to different flavours of linux, blaming a 3rd party vendors product for all the problems, and being somewhat bull-headed about changing what has been done. 
So what happens when some other 3rd party company starts doing what Adobe has done that causes(?) problems with functionality in the OS? There will be another bugzilla report about that one and a long argument about who is to blame, who should fix what and a small group of end-users and end-user developers trying to find a workaround so that the software works as it should. I know for a fact that all software will have bugs; that is a reality of computing. Passing blame and being bull-headed about your position doesn't help anyone, and leaving workarounds up to end-users isn't very professional. 
One last point to Redhat: Yes it is nice of you to provide a free OS to the world and spend lots of money and your paid developers' time working on it, but the fact is, you also get a great number of people testing your cutting edge software for free and reporting back, and working with your employees and other un-paid developers to fix the issues before you port the software over to your paid version, which drastically reduces the amount of support time and time spent fixing glitches for your paying customers which makes you look really good to them, so please don't underestimate the value of Fedora users staying with Fedora as an OS. I have used Redhat products as an OS since there was only one fast way to get it; buy the book with the CD in it (the other option being spending a week on dial-up downloading it), and I don't want to change now or in the future, so please, just a little respect for what you get out of the end-users would be appreciated. Granted, you are a company, but also; all of us are a community with pride in making Fedora the best OS out there. Let us not forget that. BTW, this is not a rant, just my opinion. Thanks for reading.
Comment 197 Julian Sikorski 2011-02-21 17:59:52 EST
Created attachment 480017 [details]
Linus' workaround

Come on guys, please stop whining. This does not add any real value and only makes *working* workarounds harder to find. I personally use the one Linus posted in comment 38 - the cat commands posted in this bug only work if ~/Downloads exists, which is not the case if you have localised XDG folders.
Just grab the file I attached, compile it, and then make a script in your ~/bin that adds it to LD_PRELOAD (I'll attach mine for convenience). Then you can forget that this problem ever existed until Adobe fix their bug.
Comment 198 Julian Sikorski 2011-02-21 18:02:15 EST
Created attachment 480018 [details]
example launcher script
Comment 199 Linus Torvalds 2011-02-21 18:19:22 EST
Don't use my workaround: it was a stupid hack to test the bug, and show that "always copy upwards" works better than the crap that is in glibc now.

A much better workaround is likely to just implement memcpy() as memmove() (you can replace the inline asm by that in my preload example if you want to). Once memcpy() isn't small and trivial any more, that's just the right thing to do. 

The fact that the glibc people don't do that, and that this hasn't been elevated despite clearly being a big usability problem (normal users SHOULD NOT HAVE TO google bugzillas and play with LD_PRELOAD to have a working system), is just sad.

Quite frankly, there is no reason for the current memcpy() mess. There is no _technical_ reason for it, and there is certainly no usability reason for it. Why the Fedora people don't just fix it, I don't understand. It's a shame and a disgrace.

The fact that Adobe does something that isn't technically right is no excuse for having a sub-par crap memcpy() implementation.

And how does one raise the priority for a bug in bugzilla, or get it re-assigned to somebody who cares?
Comment 200 Michael Young 2011-02-21 18:34:04 EST
(In reply to comment #199)
> And how does one raise the priority for a bug in bugzilla, or get it
> re-assigned to somebody who cares?

You can complain to FESCO https://fedoraproject.org/wiki/FESCO though unfortunately this has been tried already, see https://fedorahosted.org/fesco/ticket/501 .
Comment 201 Adam Williamson 2011-02-21 19:09:39 EST
This is a bug reporting system, not a discussion forum.

Linus: this is Fedora Bugzilla, not upstream glibc bugzilla; if this is a significant issue in upstream glibc it'd be best to raise it through the appropriate upstream channels. In talking about this specific report, all I can go on is the practical impact on immediate Fedora issues.
Comment 202 Adam Williamson 2011-02-21 19:10:55 EST
upstream glibc bugzilla: http://sources.redhat.com/bugzilla/
Comment 203 Adam Williamson 2011-02-21 19:16:17 EST
btw, priority field is by Fedora policy reserved to maintainers for the purpose of organizing bugs however they want. It has no intrinsic meaning; there's no policy requiring a certain response time to high priority bugs or anything like that. It doesn't really make any sense to mandate a project-wide policy for the Priority field since developers are entirely free to ignore it if they wish.

I don't think the glibc maintainers actually use the priority field to organize their bugs, so changing the priority field on this bug would achieve exactly nothing.
Comment 204 Robin Bowes 2011-02-21 19:29:07 EST
<sigh> And there's a beautiful example of the disconnect between the Redhat devs and the real world.

Somebody (actually, not just "somebody" but the guy who MADE IT POSSIBLE FOR YOU GUYS TO HAVE A JOB AT ALL) asks the question "And how does one raise the priority for a bug in bugzilla, or get it re-assigned to somebody who cares?" and you respond by talking about a field in a bug reporting system.

C'mon guys, get real. Fix it. Patch it. Whatever. But get your heads out of your arses and do something!

R.
Comment 205 Michael Convey 2011-02-21 19:34:38 EST
I believe the following link is the correct place to report a bug in glibc:

http://cygwin.com/bugzilla/

However, I'm not knowledgeable enough to properly report this bug there. Could someone please post this bug there and then post here to let us know what the bug number is so we can follow.

Thanks.
Comment 206 Adam Williamson 2011-02-21 19:44:37 EST
robin: linus asked how you raise the priority for a bug in Fedora bugzilla. The answer is, you don't, for the reasons I explained: given how the Fedora project is set up, there is no single simple mechanism for doing so, as we (intentionally) don't have any authority which can, in the usual course of events, wield a big stick and say This Bug Is High Priority. your not liking the answer does not make it not the answer.

The Fedora project trusts its glibc maintainers to maintain glibc correctly, in general, so they are trusted to prioritize their workload accordingly. should someone have a problem with this they can, as Michael Young pointed out, raise it with FESCo, but that's hardly a regular bug priority assignment system.
Comment 207 Charlie Brady 2011-02-21 23:13:24 EST
> The Fedora project trusts its glibc maintainers to maintain glibc correctly,

Fools. They can't be trusted - the evidence is before you. You can fix it - for Fedora. The fact that glibc project doesn't is no excuse for you not to.
Comment 208 Homme Bitter 2011-02-22 01:57:20 EST
I'm pulling the "security card". 

By allowing the memcpy/memmv thing to happen, you are introducing a lot of potential security hazards. You can say that the hazard is introduced by Adobe in Flash, but in fact there have already been several related bugs found in Fedora's own repository of packages. It's just a symptom of an underlying problem that is much bigger and deeper. You don't want someone to find code that will break a system remotely, or even do a (remote) privilege elevation because he can overwrite parts of memory with bits he likes to be there.

You can't just say "it's up to the package maintainers to fix it". There's SElinux, random start addresses, non executable stack, all large pieces of code in place, just to mitigate this sort of bug turning nasty. By allowing the code to run unchecked, while there are so many known, possible dangerous examples of bad code around, you are compromising security. These packages have all been put in place just to avoid user space code to be abused. I think that is more than adequate proof that you need to take measures to avoid other persons bugs to cause harm.

I know that at this stage it's academical to say it's a security hazard, but given time, enough typewriters and an adequate supply of monkeys, it will be real. History has proven that much. Even all of the above mentioned frameworks have been broken, and I have no doubt some smart and determined person will find a use for this as well. If this behavior is going to be part of glibc, a check to avoid resulting bugs, and/or mitigation for exploiting this should be in place, just like the above mentioned frameworks do.

I propose reversing the memcpy/memmv change for the time being, as a security measure. After that, make sure that the effects of the bugs in other code depending on it are mitigated or removed, before re-introducing it. Maybe by introducing a warning/error in gcc?
Comment 209 Benny 2011-02-22 02:16:38 EST
Last resort: 
Since the guys at glibc are so stubborn and cannot admit they made a mistake... Cant we repackage the newest glibc without the crappy memcpy()? (At least a (temporary) patch) C'mon guys...
Comment 210 Alexander Hunt 2011-02-22 04:45:36 EST
I agree with Benny and Homme Bitter's last paragraph (+Thanks for the security angle)
According to: Michael Young, Comment 16: 
The sound is good with F13 glibc (2.12.1-4) on F14. I
have been hunting through glibc versions. The sound was good with
glibc-2.12.90-3 but first started sounding corrupted with glibc-2.12.90-4.

So when it was changed and the problems started we know.
Even if Fedora doesn't want to do it, would it be possible for somebody who knows how to do it take the newest version of glibc, fix it, and post it somewhere that's not on a Fedora server? Or, maybe some group of people should branch it and put it on rpmfusion or something...
As an aside,this was an interesting read: 
https://fedorahosted.org/fesco/ticket/501
It would be interesting to know why it was last commented as "fixed".It would probably be more appropriately marked as "won't fix" lol...and there has been no further comment for almost 3 months.
Comment 211 Paulo Roma Cavalcanti 2011-02-22 05:33:41 EST
I agree it is complicated to support a different glibc from the one available upstream. In case of a 3rd party repo providing a parallel glibc, who is supposed to be responsible for updating it in case of new bug fixes? This is an endless work, which nobody will want to be responsible for.

In my opinion, for those not wanting to use the 32 bit version, the best fix is using a flash plugin that replaces memcpys for memmoves. 

This is a no source rpm, which downloads the source from adobe and changes the binary:

http://people.atrpms.net/~pcavalcanti/srpms/flash-plugin-10.3-1.fc14.nosrc.rpm

I am more interested in this issue from a performance point of view than
from its effect on the flash plugin. If the new version is really slower that the previous one, well, then there is no excuse for not reverting the code.
Comment 212 Alexander Hunt 2011-02-22 08:04:32 EST
Hopefully this will be the last I have to say about this issue, except for "Thank You" to the people who help get the problem fixed. 
First: Adobe is working on a fix for the problem (ref: Markus, comment 118). Perhaps if we ask nicely, Markus would be willing to email his contact at Adobe and ask for an ETA on the new Flash version (I believe the last on that was it was going to QA). Please Markus? :)
Second: the adaptation of Linus' (et al) test-code provided by Aaron (in comment 186), in combination with the rework of flash provided by Paulo (now in comment 211, and a Thank you to you) works on every flash enabled site I have visited (neither of them alone work everywhere for me), and from a strictly end-user point of view, they are both easy to implement. Disclaimer; I do not know the security issues, speed factor, or any other implications of using both workarounds together. My system doesn't seem to have any problems with it, but your mileage may vary.
Third; If Adobe is going to take any length of time, my vote would be for someone or a group of people to rework glibc for the time-being (I would do it myself if I had any clue where to start), and hope that there are no bug-fixes required before Adobe puts out a revised version of flash. 
Paulo is correct that it probably makes no sense to branch glibc (although I have to disagree that nobody would maintain it; if that was the case Linux would not be where it is today. Linus' creation and the way he did it proves people will work on something they believe in. Thank you very much Linus, and I'm not bashing you about that Paulo, just disagreeing). Regardless, anyone can easily switch back to the mainstream version of glibc when Adobe fixes flash and it works properly for them. I realize that this doesn't address some of the other issues brought up about the coding quality, security, speed (ie. processor cycles used for different functions), and the g-streamer plugin with the same problem, etc, but I am not knowledgeable enough about those issues to speak about them.
Thanks for reading; I pass the torch and am now going to get some sleep. :)
Comment 213 Callum Lerwick 2011-02-22 16:19:14 EST
Seriously people, this is all about one piece of proprietary software, the 64-bit Flash plugin. A piece of 'preview' software about which Adobe themselves say, "We do not recommended that this release be used on production systems or for any mission-critical work."

http://labs.adobe.com/downloads/flashplayer10_square.html

This is a textbook case of *exactly* what Fedora does not support. Fedora is a distribution with core values and a mission. Fedora is *not* about getting market share at any cost.

If you don't like it, you're simply using the wrong distribution.

The 32-bit flash plugin works fine. Y'all are lucky we tacitly support the 32-bit Flash plugin as much as we do.

Though admittedly we seem to be failing to communicate this point to new users. Looking at http://fedoraproject.org/en/about-fedora and what it links to, is nothing but a lot of fluffy words and abstract concepts, and I can see how no one could possibly understand the concrete practical aspects of it. The closest I can find is at the bottom of http://fedoraproject.org/wiki/Objectives

"The Fedora Project is not interested in having its distribution be a platform for proprietary or patent encumbered components. While we do not purposely make installation of such components more difficult, we also do not allow our schedule or processes to be driven by theirs."

Which boils down to "Workarounds for proprietary software are included purely at the package maintainer's discretion and are most likely going to be rejected".
Comment 214 Homme Bitter 2011-02-22 16:36:46 EST
It's NOT about the flash plugin, it's about a change in how a glibc call is handled. This breaks an unknown amount of packages and introduces erratic an possibly insecure behavior for an unknown number of Fedora's own applications. It just happens to hit the common user base the hardest on the 64 bit flash plugin, but the effect on stability and security of the entire distribution is much bigger than just the plugin.
Comment 215 Callum Lerwick 2011-02-22 16:58:11 EST
This is not a change that Fedora made. This is a change that went through all the normal upstream channels that *any* change to glibc goes through. All users of glibc are effected. And yes, this is one of the most fundamental parts of the entire system, copying memory. It effects almost every GNU/Linux user in the entire world.

If it broke things, it would turn up quickly.

If you think this change is untested, you don't understand how open source works.

So far, the only proven breakage is the 64-bit flash plugin.

In short, provide proof, or its just a lot of talk.

(Disclaimer: I am NOT in any way a glibc maintainer)
Comment 216 Tomas Mraz 2011-02-22 17:05:03 EST
Unfortunately the breakage might be very subtle - even normally unnoticeable in some packages. But even in this case it might be very well exploitable by malicious attacker. Of course you can always dismiss it by saying that the affected packages are buggy in using memcpy incorrectly - sure. But this does not make our (Fedora) users of these packages less prone to the attacks. You cannot simply say that Fedora (non-third party) packages are unaffected because the insecure use of memcpy cannot be surely detected in other way than by a careful code review or runtime checks (with abort).
Comment 217 Jon Masters 2011-02-23 11:58:56 EST
Similarly, just because something was "tested" in the artificial world of developers running purely Open Source components doesn't mean that users (like me) don't want to run a flash plugin, or other non-Open Source software on their systems. There are plenty of things I do for electronic engineering on weekends for which there is no Open Source alternative, so the answer is not always to assume that proprietary stuff is "just wrong".

Jon.
Comment 218 Callum Lerwick 2011-02-23 13:20:18 EST
I never said anything about proprietary stuff being wrong. You're projecting.

You speak as if this change to glibc didn't undergo testing by Intel before they submitted it to glibc. Then go through whatever review and testing the glibc project has. Then go through a full Fedora rawhide/beta/testing cycle. And then a Fedora final release. It has not had any special treatment. It has gone through the same testing any change to glibc gets.

These implications that this change to glibc is in any way untested are just patently absurd.

Lets see the bug reports, or you're just spewing FUD about your own product.

(And one again its being glossed over the fact that the 32-bit flash plugin works fine. This is a problem only in the *explicitly* unsupported by Adobe 64-bit *PREVIEW* flash plugin. No one is stopping anyone from having Flash. If you're truly hung up on a geek fetish for some kind of "64-bit purity", a great variety of workarounds have already been established. Isn't open source great? :)
Comment 219 Simon Lewis 2011-02-24 02:40:27 EST
(In reply to comment #218)

Lets get back to basic facts here...

glibc is a security hazard and should be fixed without any further delay, for example by implementing memcpy() as memmove()...
Comment 220 hdfssk 2011-02-24 16:45:51 EST
It actually wasn't a flash player problem that led me to this bug; many of my mp3s started playing with heavy bubbling-sounding interference right after I upgraded to F14. It took a decent while for me to find this bug, and was a good deal longer before comment #164 (https://bugzilla.redhat.com/show_bug.cgi?id=638477#c164) described the workaround... but it's *not* just one piece of software that had this problem.

I hear Adam's point in comment #201 about this not being a discussion forum - I came here today bcs the latest conversation's blowing up my inbox - but at the same time, *where* should someone discuss this sort of thing? The message I'm taking away from all this is "This isn't a problem; why are you all saying it is? Trust us, we Know about these things. Shush!" Well, it pretty plainly is a problem for some folks; what's served by marginalizing their concerns? ...and what does it say to average Fedora users when the guy who built the Linux kernel can't get *his* concerns heard? It sure doesn't encourage me to participate in the Fedora community, or try to volunteer; it seems like a huge waste of time. I’d like to believe that concerns about security will lead to something being done, but... we’ll see.

As for comment #213's "If you don't like it, you're simply using the wrong distribution" - that's a crap attitude. This discussion's got to be frustrating for all involved, but let's please try to treat each with decency and respect, yes?
Comment 221 Adam Williamson 2011-02-24 17:03:53 EST
you can discuss it on an appropriate mailing list, or forum. we have a lot of them.

Listening to people is not the same as doing exactly what they demand. The Fedora project has long-standing values relating to working upstream and prioritizing software freedom; if you don't share those it really *may* be the case that you're not using the right distribution. There are others which place a higher value on accommodating proprietary software and carrying downstream patches.
Comment 222 Linus Torvalds 2011-02-24 17:24:53 EST
(In reply to comment #221)
> 
> There are others which place a higher value on accommodating proprietary
> software and carrying downstream patches.

Why the hell do people like you continue to think this is about "proprietary software"?

This is about "helping users". It has absolutely _nothing_ to do with proprietary or not, and is about the fact that binary compatibility matters.

It's also about doing a good job.

Breaking existing binaries with a shared library update IS A BAD THING. How hard can that be to understand? If you break binary compatibility, you need to update the library major version number.

And there really isn't any valid technical reason for doing a bad job at memcpy. 

What is annoying is how people have turned this "glibc broke existing binaries" into some kind of "free software vs proprietary" crap. That's not the point. I know from personal involvement in git that the whole "oops, we used memcpy where we _should_ have used memmove" is a common bug. Bugs happen. The binary may even have been really well tested, but it was tested with a library that consistently did things some way.

Now glibc does random things. The direction of the copy will seemingly depend on things like random alignment issue, rather than any repeatable thing, so just access patterns etc make it do different things for no good reason.

I repeat: "for no good reason". There's no REASON for glibc to break things.

So stop the whole "proprietaty software" crap. It's about USERS. And it's about Quality-of-implementation.

In the kernel, we have very strict rules of "we don't break binary compatibility unless we _have_ to". It doesn't matter if it's a result of a bug in user space or not ("you shouldn't have been doing that") we revert patches that break things. Sure, occasionally we have major reasons why we can't see it as that kind of absolute rule (security bugs that simply require visible changes, or major re-architecting of some area that makes it impossible to support some old API), but they really need to be major reasons.

Why? Because _users_ are the only thing that makes software useful. Software isn't useful on its own. You cannot say "this is the right thing to do" unless you take users into account. 

And no, "there is a workaround" is not good enough, unless that workaround is then automatic from the distribution. Most users that hit this issue will never ever see this bugzilla. They will just say "Fedora is buggy".

And they'd be right. 

I'm disappointed. Stop the idiotic parroting of "proprietary app". Think about what users do, and think about all the random binaries people run.
Comment 223 Adam Williamson 2011-02-24 17:37:58 EST
again, discussion forum, me, getting sucked into it. let's stop now. I'm just trying to explain the values in play here and why certain things happen that may appear odd. it's not my decision to take, I'm just trying to keep order in the bug report.

I still haven't seen anyone report this to glibc upstream, btw. that still seems like an obvious next step to me. but what needs to happen next on this report is a Fedora glibc maintainer to explain what's going to happen and change the bug status appropriately.
Comment 224 Sergio Monteiro Basto 2011-02-24 17:43:51 EST
Hi, 
I'm off, because this discussion will not end until Adode fix the problem, since this is not a bug on glibc, this bug should be close. 
See comments #164 and the solution on #94 

You have a forum in Adobe 
http://forums.adobe.com/community/labs/flashplayer10/square64bit?view=discussions&start=0
and https://bugs.adobe.com/jira/browse/FP-5739 
which have many bug reports that Adobe doesn't fix and have at least have 2 months.
Comment 225 Charlie Brady 2011-02-24 20:38:33 EST
> I still haven't seen anyone report this to glibc upstream, btw. that still
> seems like an obvious next step to me. 

I would think Fedora maintainers would be quite capable of doing that. Fix glibc to help your users. Then explain to glibc maintainers exactly why you did so (e.g. you could quote Linus). Stop passing the buck (and stop blaming Adobe, and glibc), and stop treating users as though they don't matter.
Comment 226 Simon Lewis 2011-02-25 02:28:09 EST
glibc allows other programs to overwrite adjacent memory - this is a security hazard and must be fixed in glibc...
Comment 227 Patryk Zawadzki 2011-02-25 03:58:48 EST
Computers in general allow people to overwrite memory, you should all get rid of your computers as they are a security threat. Stop blaming glibc maintainers with the faults of others. Stop blaming Fedora. And, for the love of your chosen deity, stop filling my mail account with junk.

Every reply you post reaches at least 120 people who are not Fedora developers. This is not a popularity contest. This is not democracy. You cannot force the devs to fork glibc and maintain the patches for the rest of their lives. If you want this changed, go file bugs with glibc and convince drepper to either pull that change or make it depend on the target CPU (let people explicitly target machines where the new memcpy does make things faster).
Comment 228 Rich Mattes 2011-03-02 14:46:32 EST
(In reply to comment #223)
> I still haven't seen anyone report this to glibc upstream, btw. that still
> seems like an obvious next step to me. but what needs to happen next on this
> report is a Fedora glibc maintainer to explain what's going to happen and
> change the bug status appropriately.

Looks like it's at:
http://sources.redhat.com/bugzilla/show_bug.cgi?id=12518
Comment 229 Adam Williamson 2011-03-02 14:58:00 EST
guh. our external tracker list sucks giant....lollipops.
Comment 230 Felipe Contreras 2011-03-03 18:20:32 EST
Why not name this "regression in glibc causes memory corruption"? That's actually what is happening, and the solution is to either:

 1) Fix glibc so the old behavior is not broken (mapping memcpy to memmove)
 2) Downgrade glibc until it's fixed upstream

The fact that this regression has only been seen on proprietary code is irrelevant, it might be happening all over the place.
Comment 231 Adam Williamson 2011-03-03 20:15:48 EST
"The fact that this regression has only been seen on proprietary code is
irrelevant, it might be happening all over the place."

So might the Giant Flying Spaghetti Monster. We don't work on the basis of what might, theoretically, be happening. Provide concrete examples or just stop extending this report to no good reason.

I see actual useful discussion between people who know what the hell they're doing happening in the upstream bug; I'm all in favour of simply following whatever is decided there in the downstream package.
Comment 232 Felipe Contreras 2011-03-03 20:30:42 EST
(In reply to comment #231)
> So might the Giant Flying Spaghetti Monster. We don't work on the basis of what
> might, theoretically, be happening. Provide concrete examples or just stop
> extending this report to no good reason.
> 
> I see actual useful discussion between people who know what the hell they're
> doing happening in the upstream bug; I'm all in favour of simply following
> whatever is decided there in the downstream package.

See the title of the upstream bug:
"memcpy acts randomly (and differently) with overlapping areas"

_That_ is what is happening, it's not hypothetical, there's 125 people affected by this issue and following it, and upstream has accepted that indeed it's an issue.

How about we turn the papers; _you_ come up with proof that all the memcpy's happening in all the binaries in all packages of Fedora are correct and unaffected by this change, and thus there's no security issue. Can you really do that?
Comment 233 John Villalovos 2011-03-03 22:07:09 EST
(In reply to comment #232)
> (In reply to comment #231)
> > So might the Giant Flying Spaghetti Monster. We don't work on the basis of what
> > might, theoretically, be happening. Provide concrete examples or just stop
> > extending this report to no good reason.
> > 
> > I see actual useful discussion between people who know what the hell they're
> > doing happening in the upstream bug; I'm all in favour of simply following
> > whatever is decided there in the downstream package.
> 
> See the title of the upstream bug:
> "memcpy acts randomly (and differently) with overlapping areas"
> 
> _That_ is what is happening, it's not hypothetical, there's 125 people affected
> by this issue and following it, and upstream has accepted that indeed it's an
> issue.
> 
> How about we turn the papers; _you_ come up with proof that all the memcpy's
> happening in all the binaries in all packages of Fedora are correct and
> unaffected by this change, and thus there's no security issue. Can you really
> do that?

To go even more off tangent.  That is the kind of logic used to say there is a god.  Since you can't prove there is a god there must be a god.  And god could be whichever diety you like (Flying Spaghetti Monster, Thor, Isis, etc).  Basically your request is realistically impossible.
Comment 234 Milan Kerslager 2011-03-03 23:06:46 EST
As written above this glibc regression hits nearly all packages not just Flash.
So the assumption not to fix glibc because Flash is closed source is plain wrong.
Comment 235 Adam Williamson 2011-03-03 23:58:06 EST
Please, please, please stop arguing pointlessly in circles.
Comment 236 Milan Kerslager 2011-03-04 00:23:42 EST
Yes. So please do not argue to Flash when the problem is in Glibc and do not stand with the position not to fix Glibc because Flash is closed source (this is The Circle). The Flash is not worth to talk about. The problem is a function in Glibc - no matter that almost everybody making mistakes using similar Glibc calls in wrong context (and they are pointlessly implemented with different broken ways) no matter if this is in closed or open source code or product.
Comment 237 Callum Lerwick 2011-03-04 02:53:11 EST
You glibc maintainers have been educated evil! You deny the simultaneous 4-cornered 64-bit Flash Cube! We must not let this flashlessness stand! Your ignorance of the Harmonic Flash is demonic!

64-bit Cubic Flash debunks 32-bit AS WITCHCRAFT!
Comment 238 Felipe Contreras 2011-03-04 11:21:25 EST
(In reply to comment #235)
> Please, please, please stop arguing pointlessly in circles.

From what I can see *you* are the only one doing that... Everybody else has agreed it's a real issue.
Comment 239 Adam Williamson 2011-03-04 20:17:43 EST
stating your argument over and over again is exactly what I mean by 'arguing in circles'.
Comment 240 Deb Mukherjee 2011-03-07 00:18:27 EST
I tried both Magnus's (#55) and Ray Strode's (#94) fix. Both of them fix the issue of choppy sound in flash sites. 

I just tested the amazon music preview site in firefox, after the song played nicely, keeping firefox open, I tried to play some music in clementine, mplayer or just gnome music preview. All of them failed to produce any sound. I needed to close firefox to be able to hear music from hard drive. 

Somehow, the fix in firefox is blocking the soundcard access by other applications. 

Any clue?
Comment 241 Kuznetsov Vyacheslav 2011-03-07 03:44:59 EST
@Deb Mukherjee

I met such behaviour of the audio subsystem when I removed all pulseaudio-related packages :) Please, check if you have pulseaudio in your system.

At this moment the following packages are installed in my one:

pulseaudio-0.9.21-7.fc14.x86_64
pulseaudio-libs-0.9.21-7.fc14.x86_64
pulseaudio-libs-0.9.21-7.fc14.i686
pulseaudio-libs-zeroconf-0.9.21-7.fc14.x86_64
pulseaudio-libs-glib2-0.9.21-7.fc14.x86_64
pulseaudio-libs-glib2-0.9.21-7.fc14.i686
pulseaudio-module-gconf-0.9.21-7.fc14.x86_64
pulseaudio-module-x11-0.9.21-7.fc14.x86_64
pulseaudio-module-zeroconf-0.9.21-7.fc14.x86_64
pulseaudio-utils-0.9.21-7.fc14.x86_64
alsa-plugins-pulseaudio-1.0.22-1.fc13.x86_64

Please, note the issue does not caused by the issue from this report. If you are unable to fix the issue, you can to search the solution on the http://www.fedoraforum.org/ or create the new thread there. I believe the fedora comunity will help you.

Thank you.
Comment 242 Deb Mukherjee 2011-03-07 08:35:11 EST
@ Kuznetsov Vyacheslav

Thanks for your help.

<quote> Please, note the issue does not caused by the issue from this report.<\quote>
I am not sure the two issues are separate. I did not have the sound card blocking thing before trying LD_PRELOAD. I have all pulseaudio modules you have listed- and the problem started after I tried #55 fix.
Comment 243 Brent R Brian 2011-03-13 00:00:30 EST
Two systems, F14, x86=64, all updates applied.  This system has problem, other does not.

snd_hda_codec_realtek  298572  1 
snd_hda_intel           24479  2 
snd_hda_codec           86743  2 snd_hda_codec_realtek,snd_hda_intel
snd_hwdep                6392  1 snd_hda_codec
snd_seq                 53791  0 
snd_seq_device           6191  1 snd_seq
snd_pcm                 80190  2 snd_hda_intel,snd_hda_codec
snd_timer               19892  2 snd_seq,snd_pcm
snd                     63984  12 snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer
soundcore                6576  1 snd
snd_page_alloc           7559  2 snd_hda_intel,snd_pcm
Comment 244 Tom Horsley 2011-03-26 23:20:35 EDT
Created attachment 487982 [details]
Program to fix relocation entries in libflashplayer.so

I got ambitious today for some reason and wrote this silly program that takes
an x86_64 shared lib as an argument and fixes any relocation entries that
point to memcpy so they point to memmove instead. It is much faster than the
script posted earlier in this bugzilla.

Sample results:

readelf -r test.so | fgrep mem
000000bf5968  008f00000007 R_X86_64_JUMP_SLO 0000000000000000 memset + 0
000000bf5ea8  014700000007 R_X86_64_JUMP_SLO 0000000000000000 memchr + 0
000000bf63d8  020400000007 R_X86_64_JUMP_SLO 0000000000000000 memmove + 0
000000bf64e0  020400000007 R_X86_64_JUMP_SLO 0000000000000000 memmove + 0
readelf -r libflashplayer.so | fgrep mem
000000bf5968  008f00000007 R_X86_64_JUMP_SLO 0000000000000000 memset + 0
000000bf5ea8  014700000007 R_X86_64_JUMP_SLO 0000000000000000 memchr + 0
000000bf63d8  020400000007 R_X86_64_JUMP_SLO 0000000000000000 memmove + 0
000000bf64e0  022700000007 R_X86_64_JUMP_SLO 0000000000000000 memcpy + 0

Seems to work for me.
Comment 245 Rian Hunter 2011-03-27 00:22:12 EDT
FWIW Conventional wisdom says Linus's intuition to alias memcpy to memmove is favorable. Page 43, Section 2.6 of "The Practice of Programming" by Kernighan and Pike states:

"The ANSI C standard defines two functions: memcpy , which is fast but might overwrite memory if source and destination overlap; and memove, which might be slower but will always be correct. The burden of choosing correctness over speed should not be placed upon the programmer; there should be only one function."

Torvalds, Kernighan, Pike: 1
glibc developers: 0
Comment 246 Lukas Middendorf 2011-03-27 07:18:17 EDT
Up to now I thought it was a reasonable assumption that memcpy copies forwards (at least for not overlapping memory areas).
I'm currently thinking about the case that you have a memory mapped device file from which you want to copy some data and it has to be forwards. With the new random behavior it will break.

If the assumption was always correct in the past, I don't think it is right to change that now. At least every documentation of memcpy should have a big warning that the direction of the copy is random.
Comment 247 Ricky Burgin 2011-03-27 12:48:19 EDT
I completely agree with Linus here; whilst I'm far from a typical desktop end-user, I feel that something, particularly concerned with multimedia (i.e. having a break/having fun) should be fixed as a high priority, particularly when the popularity of such a thing makes people decide not to use Fedora because it doesn't work out of the box.

If I didn't happen to work with so many RHEL-based systems and also have a curiosity which devours my time, I wouldn't have investigated so thoroughly and may have just switched to a different distro by now.

Fedora 15, with GNOME 3, is clearly showing a consensus amongst those in the Fedora project that they want to improve the user experience, particularly so for the type of users who don't visit bugtrackers; by not somehow fixing the root cause of this problem or working around it until it is fixed, the Fedora project is going back on itself.
Comment 248 Callum Lerwick 2011-03-27 16:08:08 EDT
If you don't want the burden of choosing correctness over speed, don't program in C.

Does anyone read the thread? Flash works fine, its only people insisting on using an unsupported (by Adobe themselves) 64-bit preview version of flash having a problem.
Comment 249 Ricky Burgin 2011-03-27 16:19:29 EDT
You mean the same unsupported 64-bit version that performs anywhere near to acceptable (as opposed to other non-Adobe plugins), despite the issue with glibc?

It does nobody any good to hide behind 'unsupported', 'preview', 'less-than-desirable alternative works fine' when it's clear as to how popular and desired this plugin is to have working. It's fine to use them as an excuse for something breaking, but not to ignore the problem.

Nobody wants a crippled web experience on their 64-bit desktop OS.
Comment 250 Callum Lerwick 2011-03-27 16:39:22 EDT
I never said anything about non-Adobe plugins. You're projecting. Read the thread.

You should be using the 32-bit Adobe Flash plugin, with nspluginwrapper. That is the supported configuration, both by Adobe, and "unofficially" by Fedora. There's a reason nspluginwrapper was developed in the first place.
Comment 251 hdfssk 2011-03-27 18:21:41 EDT
FWIW my problem that led to this bug was not related to the flash plugin, see Comment #220.

Inre Comment #248 & Comment #250: please knock that attitude off, Callum; stuff like "read the thread" and "don't program in C" aren't going to make anything better, though they might lead to another argument (and there've been enough of those already!)
Comment 252 Felipe Contreras 2011-03-27 19:29:00 EDT
(In reply to comment #248)
> If you don't want the burden of choosing correctness over speed, don't program
> in C.

Have you measured the speed difference between memcpy and memmove? No? Then don't assert that one is faster than the other, because they aren't. This has been demonstrated in the thread with numbers.

> Does anyone read the thread? Flash works fine, its only people insisting on
> using an unsupported (by Adobe themselves) 64-bit preview version of flash
> having a problem.

AFAIK it happens on 32-bit too, it's only a matter of time before the "square" code gets into the "non-preview" version of Flash.

Anyway, this bug is _not_ about Flash, it's about glibc. Anything that uses memcpy can be affected silently. How much code is misbehaving thanks to this change? It's impossible to know.

For more examples of issues, see:
http://lwn.net/Articles/414496/

So, not fixing this bug means silently breaking stuff with _zero_ gain.

Now, can we rename this bug to:
"memcpy acts randomly (and differently) with overlapping areas"

Or shall I create a new one and make this one depend on that?
Comment 253 Callum Lerwick 2011-03-27 22:08:09 EDT
(In reply to comment #252)
> (In reply to comment #248)
> > If you don't want the burden of choosing correctness over speed, don't program
> > in C.
> 
> Have you measured the speed difference between memcpy and memmove? No? Then
> don't assert that one is faster than the other, because they aren't. This has
> been demonstrated in the thread with numbers.

Kindly read comment #99.

> > Does anyone read the thread? Flash works fine, its only people insisting on
> > using an unsupported (by Adobe themselves) 64-bit preview version of flash
> > having a problem.
> 
> AFAIK it happens on 32-bit too,

Read the thread. No one has said any such thing. If you have experienced otherwise, please let us know.

> it's only a matter of time before the "square"
> code gets into the "non-preview" version of Flash.

A hypothetical statement that makes a lot of assumptions about Adobe's development process. Do you work for Adobe?

They could also have committed a fix into their source tree months ago and its just a matter of time until they release a fixed version of the 64-bit plugin.

I don't work for Adobe and I don't see you with an @adobe.com email so my hypothetical statements are just as good as yours.

> Anyway, this bug is _not_ about Flash, it's about glibc. Anything that uses
> memcpy can be affected silently. How much code is misbehaving thanks to this
> change? It's impossible to know.

If a tree falls in the woods...

> For more examples of issues, see:
> http://lwn.net/Articles/414496/

Just read it, squashfs fixed their code, problem solved. It also links to this:

http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278

"This patch includes optimized 64bit memcpy/memmove for Atom, Core 2 and
Core i7.  It improves memcpy by up to 3X on Atom, up to 4X on Core 2 and
up to 1X on Core i7.  It also improves memmove by up to 3X on Atom, up to
4X on Core 2 and up to 2X on Core i7."

> So, not fixing this bug means silently breaking stuff with _zero_ gain.

Your "zero gain" premise is contradicted by at least two Intel engineers.

And Linus never said what hardware he was performance testing on, so we don't know that his testing contradicts the Intel engineers statements.

The Intel engineer in comment #99 even gave a credible technical explanation as to why it is faster to copy backwards, that makes sense if you know anything about modern processor designs. Presumably Intel understands how their own CPUs work.

You're going to have to give a lot more than just your word as to why we should disbelieve the Intel engineers.
Comment 254 Linus Torvalds 2011-03-27 22:19:27 EDT
(In reply to comment #253)
> 
> Kindly read comment #99.

Bullshit. Read comment #132. Even if you want to copy backwards, there is absolutely zero reason to not check the overlapping case first, to see that you don't do it wrong. 

Why is that so hard for people to understand?

The _only_ reason to do memcpy() that clobbers overlapping areas is that the code is fast BECAUSE IT IS SIMPLE. So if you were to use "rep movsb" as the memcpy implementation, then you have a good argument why it does the overlapping case wrong - it's so simple as to be brainless.

But once you do what the glibc memcpy does, there is just no excuse any more for it.
Comment 255 Ricky Burgin 2011-03-27 22:28:32 EDT
Precisely, it's a trade-off, not an improvement. The argument for keeping it is as good as saying: "Let's make it do nothing at all - at least it'll be faster!".
Comment 256 Felipe Contreras 2011-03-28 05:55:03 EDT
(In reply to comment #253)
> (In reply to comment #252)
> > (In reply to comment #248)
> > > If you don't want the burden of choosing correctness over speed, don't program
> > > in C.
> > 
> > Have you measured the speed difference between memcpy and memmove? No? Then
> > don't assert that one is faster than the other, because they aren't. This has
> > been demonstrated in the thread with numbers.
> 
> Kindly read comment #99.

Hopefully Linus has answered this one.

> > > Does anyone read the thread? Flash works fine, its only people insisting on
> > > using an unsupported (by Adobe themselves) 64-bit preview version of flash
> > > having a problem.
> > 
> > AFAIK it happens on 32-bit too,
> 
> Read the thread. No one has said any such thing. If you have experienced
> otherwise, please let us know.

I did read the thread, everybody mentioned 64-bit instead of "the square version", but Pablo Rodríguez mentioned he experiences the issue in 32-bit too. Anyway, I tried on my 32-bit laptop, and it doesn't seem to run, so I guess that's a different issue.

> > it's only a matter of time before the "square"
> > code gets into the "non-preview" version of Flash.
> 
> A hypothetical statement that makes a lot of assumptions about Adobe's
> development process. Do you work for Adobe?
> 
> They could also have committed a fix into their source tree months ago and its
> just a matter of time until they release a fixed version of the 64-bit plugin.
> 
> I don't work for Adobe and I don't see you with an @adobe.com email so my
> hypothetical statements are just as good as yours.

The 'square' plug-in is 10.3, and the current one is 10.2. If you don't put code in 10.3 beta that you would use in 10.3 final, what's the point of the beta then? Sure, maybe they fixed that internally, but maybe not.

> > Anyway, this bug is _not_ about Flash, it's about glibc. Anything that uses
> > memcpy can be affected silently. How much code is misbehaving thanks to this
> > change? It's impossible to know.
> 
> If a tree falls in the woods...
> 
> > For more examples of issues, see:
> > http://lwn.net/Articles/414496/
> 
> Just read it, squashfs fixed their code, problem solved.

No, problem not solved. That's _one_ issue. Can you be certain that there are no issues lingering there? How many millions of line of code can be affected by this? Sure, we've found so far a few issues that were obvious, but there could be many more hidden, specially if they are subtle, or tricky to trigger.

> It also links to this:
> 
> http://article.gmane.org/gmane.comp.lib.glibc.alpha/15278
> 
> "This patch includes optimized 64bit memcpy/memmove for Atom, Core 2 and
> Core i7.  It improves memcpy by up to 3X on Atom, up to 4X on Core 2 and
> up to 1X on Core i7.  It also improves memmove by up to 3X on Atom, up to
> 4X on Core 2 and up to 2X on Core i7."

So? memcpy can be improved, and so can memmove. But they are the same. So again, this potential breakage provides _zero_ gain.

> > So, not fixing this bug means silently breaking stuff with _zero_ gain.
> 
> Your "zero gain" premise is contradicted by at least two Intel engineers.
> 
> And Linus never said what hardware he was performance testing on, so we don't
> know that his testing contradicts the Intel engineers statements.

You are confused, don't mix two statements. What Intel guys said was that a memcpy _backwards_ is good on certain CPU's, that has _nothing_ to do with memcpy vs memmove.

> The Intel engineer in comment #99 even gave a credible technical explanation as
> to why it is faster to copy backwards, that makes sense if you know anything
> about modern processor designs. Presumably Intel understands how their own CPUs
> work.

Which has nothing to do with the issue at hand.

Anyway, here's the new bug to properly reflect what's causing this: bug #691336.
Comment 257 Siarhei Siamashka 2011-03-28 11:41:32 EDT
(In reply to comment #254)
> (In reply to comment #253)
> > 
> > Kindly read comment #99.
> 
> Bullshit. Read comment #132. Even if you want to copy backwards, there is
> absolutely zero reason to not check the overlapping case first, to see that you
> don't do it wrong.

I love how you use the words 'absolutely' and 'zero reason' :) Ever heard about http://en.wikipedia.org/wiki/Cargo_cult_programming ?

> Why is that so hard for people to understand?
> 
> The _only_ reason to do memcpy() that clobbers overlapping areas is that the
> code is fast BECAUSE IT IS SIMPLE. So if you were to use "rep movsb" as the
> memcpy implementation, then you have a good argument why it does the
> overlapping case wrong - it's so simple as to be brainless.
> 
> But once you do what the glibc memcpy does, there is just no excuse any more
> for it.

That's just pure bullshit. Making assumptions about how exactly undefined behavior might manifest itself with various C library implementations is not something that you should rely on when developing software. What is so hard to understand here?

GNU/Linux is not the only OS which tries to be POSIX compatible, glibc is not the only C library and gcc is not the only C compiler. What you and your fanboys are doing here is just an aggressive promotion of a trivial memcpy misuse bug into a new de-facto standard. I would be really upset if "it is safe because glibc is known to work this way" becomes a common practice and a valid excuse for not fixing bugs.

There are many bugs being introduced and fixed in various software daily. What makes this particular trivial bug so special that it even deserves an update for the C language standard? Especially considering that there are multiple tools capable of detecting this overlapping memcpy problem and it is almost nonexistent in practice.

This bug also highlights a major weakness of the Flash plugin. For the various security problems not addressed over long periods of time they might have an excuse, maybe the bugs were not so easy to fix. But based on how this trivial memcpy issue is being handled, looks like Adobe just does not have a sane process for releasing updates and security fixes. This is very disturbing.
Comment 258 Adam Williamson 2011-03-28 11:42:22 EDT
Once again: this bug is not the right place to discuss this. Fedora is not where significant changes to glibc behaviour happen. Fedora Bugzilla is not the right place to discuss making them. There is already an upstream bug for this. Fedora will follow the approach that is decided on upstream, when it is decided on upstream. Commenting here is achieving precisely nothing.
Comment 259 Bill Davidsen 2011-03-28 15:09:48 EDT
(In reply to comment #257)

> There are many bugs being introduced and fixed in various software daily. What
> makes this particular trivial bug so special that it even deserves an update
> for the C language standard? Especially considering that there are multiple
> tools capable of detecting this overlapping memcpy problem and it is almost
> nonexistent in practice.
> 
Why are developers fighting this so hard? If upstream introduces new code to the kernel which breaks existing programs, it gets fixed in Fedora. Here old versions of the library work with existing code and not with the update. Why is good to fix kernel bugs and bad to fix library bugs.

I'm sure RHEL will not introduce a change which breaks existing programs, why should Fedora? Put the "standard conforming" library in FC15 and be happy, hopefully it will break GNOME3. But unless the Fedora team intends to rewrite and maintain all of the other Fedora software which is using the wrong move, the place to fix the disfunction is at the library, and not deliberately break programs in the names of pedantic adherence to a standard.

> This bug also highlights a major weakness of the Flash plugin. For the various
> security problems not addressed over long periods of time they might have an
> excuse, maybe the bugs were not so easy to fix. But based on how this trivial
> memcpy issue is being handled, looks like Adobe just does not have a sane
> process for releasing updates and security fixes. This is very disturbing.

I'm less disturbed by slow support from Adobe than calling a bug a feature by Fedora.
Comment 260 Jakub Jelinek 2011-03-28 15:38:14 EDT
Perhaps because it is clearly documented?

ISO C99, 7.21.2.1:
"If copying takes place between objects that overlap, the behavior is undefined."
Ditto ISO C90, 7.11.2.1, Unix 98, POSIX 2003, POSIX 2008.
man 3 memcpy also clearly says
"The memory areas should not overlap.  Use memmove(3) if the memory areas do overlap."
After all that is the only difference between memcpy and memmove.

glibc certainly doesn't guarantee bug compatibility, backward compatibility is only provided for correctly written programs that don't trigger undefined behavior.  I'm not sure why you are rising RHEL here, the upcoming major version of RHEL will certainly provide the upstream memcpy version.
Comment 261 Felipe Contreras 2011-03-28 20:04:35 EDT
(In reply to comment #260)
> Perhaps because it is clearly documented?

So? The purpose of Fedora is not to follow some specification documentation, it is to provide stable, reliable, useful system.

If breaking applications was done in favor of some drastic performance improvements, that might be ok, following API break path that upstream is planning to do.

But there's nothing like that. You are fighting to protect a vacuum. You want to break applications in order to gain _nothing_.

I guess it's more important to make a few developers at RedHat feel good about themselves because they manage to close a bug without moving a finger, and claiming POSIX correctness, than to make the system more reliable.

Congratulations, you have knowingly made the system more unreliable in order to gain nothing.
Comment 262 Michael Shigorin 2011-03-30 07:05:07 EDT
(In reply to comment #253)
> Why is that so hard for people to understand?
Maybe because they failed to grok Postel's Law?

In the meanwhile, folks thank you for the master class and point fingers at fedora developers: http://avva.livejournal.com/2323823.html (in Russian)

2 jj: je svedomi? :( there are too few _correctly_ written programs in the world
Comment 263 Uno 2011-03-31 04:33:42 EDT
(In reply to comment #101)
> I'd personally suggest that glibc just alias memcpy() to memmove().

I think You aren't right :)

You suggest to provide *one* of many undocumented behaviours. What do You think do with other undocumented behaviours?

for example: memcpy can (or could) be used to propagate a pattern:

strcpy(buffer, "abc");
memcpy(buffer + 3, buffer, 100);

it will (or would) fill buffer by repeating "abcabcabc..."

Someone can invent the other undocumented variant. If You try provide all these variants You will have no time to do something in linux kernel.

I think that invalid (adobe's) code must be fixed anyway. memcpy shouldn't be alias to memmove.
Comment 264 Uno 2011-03-31 04:49:22 EDT
> I'm no great fan of flash but it's an essential part of life on the web these
> days and I had thought that the Fedora project had finally put its days of
> broken flash support behind it.

to watch Youtube I use gnash. Last version works well.
Comment 265 Milan Kerslager 2011-03-31 04:56:53 EDT
A programmer should not use side effects of any function.
A glibc should not provide function with performance hit for whatever reason.
It does not matter in what situation has been regression observed.
Glibc has performance regression. This should be fixed.
This has been observed by (broken) Flash (but this hits not only Flash).
Flash had broken code before that. This has been observed by glibc regression.
So Flash should to be fixed too.
Broken design of a function(s) should not leads to broken implementation.
Etc, etc...
Comment 266 Knut J BJuland 2011-04-01 01:13:21 EDT
It is also present in Neverwinter nights.
Comment 267 Brendan Jones 2011-04-06 10:28:38 EDT
*** Bug 680802 has been marked as a duplicate of this bug. ***
Comment 268 Riku Seppala 2011-04-08 11:20:20 EDT
http://sources.redhat.com/bugzilla/show_bug.cgi?id=12518 is now marked as FIXED.
So... ?
Comment 269 Adam Williamson 2011-04-08 11:56:04 EDT
so, that patch should get pulled downstream now.
Comment 270 Felipe Contreras 2011-04-10 21:32:08 EDT
Created attachment 491126 [details]
x86_64: workaround for new memcpy behavior

I have backported the fix to 2.13, and I've launched a koji build:

http://koji.fedoraproject.org/koji/taskinfo?taskID=2992294

Enjoy :)
Comment 271 Felipe Contreras 2011-04-10 21:36:12 EDT
Created attachment 491127 [details]
memcpy-ssse3: add overlap checks

I used Linus' patch as a guide to add overlap checks that would crash improper apps, and tried to boot my Fedora 14 system with it.

I noticed pulseaudio and readahead-collector crash, so they must be doing something wrong.

I say if Fedora cares about the quality of the full system something like this should be done to find all the bad code.
Comment 272 JCHuynh 2011-04-10 22:54:07 EDT
I'm testing testing Fedora 15 ... Problem is fixed now
Comment 273 Jakub Jelinek 2011-04-11 02:16:19 EDT
(In reply to comment #270)
> Created attachment 491126 [details]
> x86_64: workaround for new memcpy behavior
> 
> I have backported the fix to 2.13, and I've launched a koji build:
> 
> http://koji.fedoraproject.org/koji/taskinfo?taskID=2992294

This is ABI incompatible library, so please don't spread this out.
Comment 274 Brent R Brian 2011-04-11 06:22:47 EDT
Will this be made available as an yum update on FC14 ?
Comment 275 Felipe Contreras 2011-04-11 06:45:33 EDT
Created attachment 491198 [details]
x86_64: fix for new memcpy behavior (try 2)

Here's a second try, should achieve the same thing, but ABI compatible.
Comment 276 Felipe Contreras 2011-04-11 07:35:26 EDT
(In reply to comment #275)
> Created attachment 491198 [details]
> x86_64: fix for new memcpy behavior (try 2)
> 
> Here's a second try, should achieve the same thing, but ABI compatible.

Here's the corresponding koji build, now it's ABI compatible.
Comment 278 Felipe Contreras 2011-04-13 05:24:02 EDT
Thanks to Ulrich Drepper I managed to write a simple check to find memcpy overlaps issues system-wide and I created a tracking but to link my findings so far. It would be useful if other people ran this on their systems as well:

Bug #696096.
Comment 279 Simon Lewis 2011-05-13 15:00:49 EDT
Ubuntu is also using the MEMCPY hack - the deb package refers to redhat bug #638477

See http://www.ubuntuupdates.org/packages/show/301005

I guess Ubuntu is wating for redhat to fix glibc.......
Comment 280 Bill McGonigle 2011-05-19 19:25:02 EDT
Adobe didn't fix this in Flash 10.3.  :(  I applied Ray Strode's script to the 10.3 binary and it still appears to work.

2.6.35.13-91.fc14.x86_64
glibc-2.13-1.x86_64
Comment 281 Felipe Contreras 2011-05-22 09:38:35 EDT
(In reply to comment #280)
> Adobe didn't fix this in Flash 10.3.  :(  I applied Ray Strode's script to the
> 10.3 binary and it still appears to work.

Flash 10.3 is ambiguous, 10.3.162 is flash "square" preview 3 for 64-bit systems, 10.3.181.14 is 10.3 final, for 32-bit systems.
Comment 282 Michael H. 2011-05-26 14:44:48 EDT
Thx a bunch for Linus quick patch, saves me from the annoying sound with youtube videos on FC 14...
Comment 283 Brent R Brian 2011-06-02 11:51:08 EDT
Patch works for me as well
Comment 284 Michael Cronenworth 2011-07-11 18:34:39 EDT
Andreas or Jakub, what is the status of this?

I just ran into an overlapping memcpy() in a piece of third-party proprietary software used by my $DAYJOB. It seems this memcpy() change is in RHEL6 as the customers effected are running RHEL6. I can, of course, fix the problem, but other softwares out there may be effected, too. RHEL6 may need to be fixed as other proprietary houses may not be so friendly.
Comment 285 Michael Cronenworth 2011-07-12 14:27:07 EDT
glibc-2.14-3 still exhibits the new, backwards behavior for me. (Fedora 15)

Is this expected?
Comment 286 Matthew Garrett 2011-07-12 14:44:50 EDT
For anything built against new glibc, yes. Old binaries should pick up the old memcpy behaviour.
Comment 287 Felipe Contreras 2011-07-13 18:43:00 EDT
FTR the public beta of Flash 11 seems to have the problem fixed.
Comment 288 Sébastien Major 2011-08-02 12:38:23 EDT
Does the subject can be a concrete one ?
I did not read as carefully as evidently needed but, as entirely user under Linux and after GNU/Linux since 1994, I think we could just get away from this talk and move the debate target.

Super : adobe done the new glibc update.

In my humble opinion, we need all people to raise up Linux and not read comments that disturb the project. If Linus Torvalds said ... We must follow, unless, it's stupid and get us or another in danger ! Completely agree with the facts of he told.

What about have memcpy be exact aliasing to memset and memcpy32, memcpy64, memcpy128, ... be other functions ?

Please, don't bicker yourselves about this one year subject Bug. I think we need you for largely others things.
Comment 289 Kir Kolyshkin 2011-08-16 10:16:18 EDT
Guys,

Thanks for the solution! I was tired of manually adding LD_PRELOAD to firefox/google-chrome start script, so I ended up automating the process using RPM triggers. Now all I (and you) need to do is to install flash-fix RPM and forget about the problem.

Get the stuff from here: http://kir.sacred.ru/flash-fix/
Comment 290 Riku Seppala 2011-08-16 10:25:48 EDT
Adobe has updated flash plugin that fix this. Works for me.

http://labs.adobe.com/downloads/flashplayer11.html
Comment 291 Jeffrey Walton 2011-09-17 15:17:51 EDT
(In reply to comment #152)
> ...
> I would like to urge anyone with any political power withing Fedora to get this
> problem dealt with immediately. An organization this big and mature shouldn't
> be having trouble dealing with this sort of thing, fix it, make certain it
> won't happen again and move on.
Unfortunately, Fedora politcos will probably not be able to get Adobe to move any faster or do a better job with their software. Adobe has a chronic, progressive problems (which, not surprisingly, spills over into security related defects).  There's a reason Apple selectively bans Adobe from their platforms.

"Adobe surpasses Microsoft as favorite hacker’s target" [1]
"Adobe predicted as top 2010 hacker target" [2] 
"Adobe products are legendary for their insecurity." [3]

Adobe security related history: http://www.google.com/#q=adobe+pointer+site:securityfocus.com

[1] http://lastwatchdog.com/adobe-surpasses-microsoft-favorite-hackers-target/
[2] http://www.theregister.co.uk/2009/12/29/security_predictions_2010/
[3] http://techcrunch.com/2011/08/20/revenge-of-the-killer-script-kiddies/
Comment 292 Vik Tara 2011-09-19 05:45:06 EDT
Garbled audio on Fedora 14 with flash also happens when using:
http://demo.bigbluebutton.org/

The workaround in this case has no effect.