Bug 770602

Summary: orc try to wrongly share a temporary directory between users
Product: [Fedora] Fedora Reporter: Yann Droneaud <yann>
Component: orcAssignee: Matthias Clasen <mclasen>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 16CC: ds, fabian.deutsch, mclasen, metherid, michel, nathanael, otte, spoyarek
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: orc-0.4.16-5.fc15 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-19 01:26:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
.xsession-errors
none
strace log with failure
none
strace log with success
none
valgrind log with failure
none
Hardened temporary directory creation patch none

Description Yann Droneaud 2011-12-27 23:05:14 UTC
Just after upgrading to F16 from F14 using preupgrade,
I tried to launch shotwell ... but I'm still waiting for it after a long time.

Shotwell seems to have started gst-plugin-scanner and then, nothing ... but hundreds of gst-plugin-scanner processes :

[...]

       ├─shotwell───gst-plugin-scan

[...]

├─180*[gst-plugin-scan]


During that time, .xsessions-errors is filled with messages like

(tracker-miner-fs:2057): Tracker-WARNING **:   Got extraction DBus error on 'file:///home/ydroneaud/Images/2010/12/26/IMGP0021.JPG': GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus)

(tracker-miner-fs:2057): Tracker-WARNING **:   Got second extraction DBus error on 'file:///home/ydroneaud/Images/2010/08/15/IMGP0238.PEF'. Adding only non-embedded metadata to the SparQL, the error was: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus)

But thoses messages started to be reported before launching shotwell.

Comment 1 Yann Droneaud 2011-12-28 19:59:23 UTC
Today I've tried to leave shotwell time to start ... one full day, and nothing.
The same day I've tried to start rhythmbox which shows the same problem. Same thing for totem.

After more than 950 gst-plugin-scanner processes, my system refused to start one more process, and I have to log as root to issue multiple "killall gst-plugin-scanner commands".

In my .xsession-errors there are those lines:
*** glibc detected *** /usr/libexec/gstreamer-0.10/gst-plugin-scanner: malloc(): memory corruption: 0x0000000001dd6bf0 ***
*** glibc detected *** /usr/libexec/gstreamer-0.10/gst-plugin-scanner: malloc(): memory corruption: 0x0000000001dd6bf0 ***

*** glibc detected *** /usr/libexec/gstreamer-0.10/gst-plugin-scanner: realloc(): invalid next size: 0x0000000000df3f30 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c2d6)[0x7fe7542c42d6]
/lib64/libc.so.6(+0x7f9b2)[0x7fe7542c79b2]
/lib64/libc.so.6(realloc+0xee)[0x7fe7542c920e]
/usr/lib64/liborc-0.4.so.0(orc_compiler_append_code+0x129)[0x7fe74d12d399]
/usr/lib64/liborc-0.4.so.0(orc_x86_emit_prologue+0x36)[0x7fe74d1463c6]
/usr/lib64/liborc-0.4.so.0(orc_compiler_sse_assemble+0x153)[0x7fe74d144cd3]
/usr/lib64/liborc-0.4.so.0(orc_program_compile_full+0x649)[0x7fe74d12e7d9]
/usr/lib64/gstreamer-0.10/libgstvideotestsrc.so(+0x9fbe)[0x7fe73ac87fbe]
/usr/lib64/gstreamer-0.10/libgstvideotestsrc.so(+0x5679)[0x7fe73ac83679]
/usr/lib64/libgstreamer-0.10.so.0(+0x63bd8)[0x7fe755be0bd8]
/usr/lib64/libgstreamer-0.10.so.0(gst_plugin_load_file+0x32b)[0x7fe755be1e6b]
/usr/lib64/libgstreamer-0.10.so.0(+0x67e7a)[0x7fe755be4e7a]
/usr/lib64/libgstreamer-0.10.so.0(_gst_plugin_loader_client_run+0xc1)[0x7fe755be5c31]
/usr/libexec/gstreamer-0.10/gst-plugin-scanner[0x40074f]
/lib64/libc.so.6(__libc_start_main+0xed)[0x7fe75426969d]
/usr/libexec/gstreamer-0.10/gst-plugin-scanner[0x400785]

[...]

And some others:

ERROR: Caught a segmentation fault while loading plugin file:
/usr/lib64/gstreamer-0.10/libgstadder.so

Please either:
- remove it and restart.
- run with --gst-disable-segtrap --gst-disable-registry-fork and debug.

After killing the processes, each application now start and restart fine.

Comment 2 Yann Droneaud 2011-12-28 20:01:57 UTC
Created attachment 549850 [details]
.xsession-errors

.xsession-errors with all the backtraces

Comment 3 Yann Droneaud 2011-12-28 21:04:57 UTC
To reproduce the problem, I have to remove .gstreamer-0.10/registry.x86_64.bin

This allowed me to trace down the problem:

  stat("/tmp/.orc", {st_dev=makedev(253, 5), st_ino=8198, st_mode=S_IFDIR|0700, st_nlink=2, st_uid=501, st_gid=501, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=2011/12/27-22:46:42, st_mtime=2011/12/09-10:58:54, st_ctime=2011/12/26-17:16:08}) = 0
  open("/tmp/.orc/orcexec.6f9xL1", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EACCES (Permission denied)
  stat("/run/user/ydroneaud/.orc", 0x7fff81eb9a40) = -1 ENOENT (No such file or directory)
  open("/dev/tty", O_RDWR|O_NOCTTY|O_NONBLOCK) = 8
  writev(8, [{"*** glibc detected *** ", 23}, {"/usr/libexec/gstreamer-0.10/gst-"..., 46}, {": ", 2}, {"malloc(): memory corruption", 27}, {": 0x", 4}, {"00000000008d3bb0", 16}, {" ***\n", 5}], 7) = 123

after that, orc is failing

In fact /tmp/.orc/ was created by another user on my system. With restricted rights. After removing the directory everything does fine:

  stat("/tmp/.orc", {st_dev=makedev(253, 5), st_ino=131085, st_mode=S_IFDIR|0700, st_nlink=2, st_uid=500, st_gid=500, st_blksize=4096, st_blocks=8, st
_size=4096, st_atime=2011/12/28-21:32:11, st_mtime=2011/12/28-21:32:12, st_ctime=2011/12/28-21:32:12}) = 0
  open("/tmp/.orc/orcexec.DeNMOo", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
  unlink("/tmp/.orc/orcexec.DeNMOo")      = 0
  ftruncate(3, 65536)                     = 0


So there's at least 3 bugs:

- orc uses a common temporary directory wrongly shared between user
- orc does not remove the temporary directory
- orc seems to handle badly the lack of temporary file

Comment 4 Yann Droneaud 2011-12-28 21:07:59 UTC
Created attachment 549859 [details]
strace log with failure

Comment 5 Yann Droneaud 2011-12-28 21:08:33 UTC
Created attachment 549860 [details]
strace log with success

Comment 6 Yann Droneaud 2011-12-28 21:19:37 UTC
Created attachment 549863 [details]
valgrind log with failure

Comment 7 Yann Droneaud 2011-12-28 22:05:48 UTC
This is a Fedora only bug (not upstream) against orc

orc create a temporary directory which is wrongly shared against users.
Others users won't be able to create file under this directory and orc corrupt memory. This is the root cause of gstreamer lockup (and rhythmbox, shotwell and totem).

This commit is the root of the problem:

http://pkgs.fedoraproject.org/gitweb/?p=orc.git;a=commitdiff;h=196f35e3d76981dbd45fbdf1150d72348d5652db

This is related to the bug reported (patch proposed)

https://bugs.freedesktop.org/show_bug.cgi?id=41446


In order to fix the problem, in a multi user system, a directory must be created per user

- either a very temporary directory (using mkdtemp) which will be destroyed as soon it's no more needed
- or a not so temporary directory, using a common pattern, eg appending the username (and uid to be safer), like those used by tracker, orbit, virtual, etc.

Comment 8 Fabian Deutsch 2011-12-28 22:19:16 UTC
Yann,

thanks for your detailed report.
I'd suggest to create the temporary orc dir within the user's home, e.g. ~/.local/cache/orc . 
I'd like to go tis way, because to me this seems more suitable for the original selinux problem.
Is this fine for you?

Comment 9 Yann Droneaud 2011-12-28 23:53:05 UTC
(In reply to comment #8)
> Yann,
> 
> thanks for your detailed report.
> I'd suggest to create the temporary orc dir within the user's home, e.g.
> ~/.local/cache/orc . 
> I'd like to go tis way, because to me this seems more suitable for the original
> selinux problem.
> Is this fine for you?

Why not. It's a safe bet.

Comment 10 Yann Droneaud 2011-12-28 23:56:30 UTC
Created attachment 549877 [details]
Hardened temporary directory creation patch

Here's a patch that replace the previous one.

It addresses the security concerns I have ... but it won't address the selinux globbing problem.

Comment 11 Yann Droneaud 2011-12-29 00:01:01 UTC
(In reply to comment #8)

Using my patch you could try to this first:

   if (tmpdir && orc_code_region_allocate_codemem_dual_map (region,
        tmpdir, ".local/cache/orc", NULL, -1)) return;

But it will fail if .local/cache doesn't exist. You will need to add a wrapper around mkdir() to work like mkdir -p

Comment 12 Yann Droneaud 2011-12-29 00:02:04 UTC
(In reply to comment #11)

>    if (tmpdir && orc_code_region_allocate_codemem_dual_map (region,
>         tmpdir, ".local/cache/orc", NULL, -1)) return;
>

with tmpdir = getenv("HOME");

Comment 13 Yann Droneaud 2011-12-29 00:07:34 UTC
(In reply to comment #12)

Last one for today:

  tmpdir = getenv("XDG_CACHE_HOME"); /* $HOME/.cache */

  if (tmpdir && orc_code_region_allocate_codemem_dual_map (region,
          tmpdir, "orc", NULL, -1)) return;


But according to
http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

XDG_RUNTIME_DIR is a better choice.

Comment 14 Yann Droneaud 2011-12-30 09:01:15 UTC
See also bug #770923

Comment 15 Yann Droneaud 2011-12-30 09:41:04 UTC
(In reply to comment #7)
> 
> This commit is the root of the problem:
> 
> http://pkgs.fedoraproject.org/gitweb/?p=orc.git;a=commitdiff;h=196f35e3d76981dbd45fbdf1150d72348d5652db
> 
> This is related to the bug reported (patch proposed)
> 
> https://bugs.freedesktop.org/show_bug.cgi?id=41446
> 
>

There's a off-by-one error in the patch : the 'dir' allocation is missing a byte to hold NUL byte:

-  dir = malloc (strlen (basedir) + strlen ("/.orc"));
+  dir = malloc (strlen (basedir) + strlen ("/.orc") + 1);

This seems to be the root cause of the memory corruption.
But the memory corruption is more likely to happen if orc_code_region_allocate_codemem_dual_map() is called more than one time, which happen if /tmp/.orc exists and owned by another user.

Note : I also suggest:

-  stat (dir, &stat_p);
-  if (!S_ISDIR (stat_p.st_mode))
+  if (stat (dir, &stat_p) == -1 || !S_ISDIR (stat_p.st_mode))

Comment 16 Yann Droneaud 2011-12-30 09:46:40 UTC
*** Bug 770923 has been marked as a duplicate of this bug. ***

Comment 17 Fabian Deutsch 2012-01-03 10:37:32 UTC
I've updated the patch here
http://pkgs.fedoraproject.org/gitweb/?p=orc.git;a=blob;f=orc-subdir.patch;h=e765f1af48768def66891d82f6af67cb4ffdf3ad;hb=48b8e8aca806b55cef2ff5c5708e1e06f3a5da12

This also introduces a build param, which disables non-user-dependent temp dirs.
Could someone review the patch?

Comment 19 David Schleef 2012-01-07 20:24:30 UTC
Surely there must be a way to detect SELinux at runtime, and if it's enabled, avoid using allocation methods that trigger warnings.

Comment 20 Fabian Deutsch 2012-01-08 16:04:40 UTC
(In reply to comment #19)
> Surely there must be a way to detect SELinux at runtime, and if it's enabled,
> avoid using allocation methods that trigger warnings.

I will see if there is a way to reliable detect selinux at runtime.
Would you also allow linking against some selinux lib if it was necessary?

Comment 21 David Schleef 2012-01-08 18:32:50 UTC
> Would you also allow linking against some selinux lib if it was necessary?

No.

I consider the lack of approved (and documented) method of creating write/execute memory a bug in SELinux.  It should be fixed there.

Comment 22 Fedora Update System 2012-01-08 19:17:55 UTC
orc-0.4.16-5.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/orc-0.4.16-5.fc16

Comment 23 Fedora Update System 2012-01-08 19:53:22 UTC
orc-0.4.16-5.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/orc-0.4.16-5.fc15

Comment 24 Fedora Update System 2012-01-11 06:09:12 UTC
Package orc-0.4.16-5.fc15:
* should fix your issue,
* was pushed to the Fedora 15 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing orc-0.4.16-5.fc15'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-0327/orc-0.4.16-5.fc15
then log in and leave karma (feedback).

Comment 25 Fedora Update System 2012-01-19 01:26:26 UTC
orc-0.4.16-5.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 26 Fedora Update System 2012-01-29 19:56:47 UTC
orc-0.4.16-5.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.