Bug 2203863

Summary: php: Should not use RTLD_DEEPBIND for opening extension modules
Product: [Fedora] Fedora Reporter: Ralf Ertzinger <redhat-bugzilla>
Component: phpAssignee: Remi Collet <fedora>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 38CC: dmalcolm, fedora, fweimer, jakub, jlaw, jorton, jwakely, mcermak, mpolacek, msebor, nickc, sipoyare
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-05-22 10:58:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ralf Ertzinger 2023-05-15 13:39:59 UTC
Note: I've filed this under GCC because I don't really knwo where the problem lies here, but I'm reasonably certain it's somewhere low level.

Reproducer:
$ sudo dnf install uwsgi-plugin-php uwsgi-logger-file  php-gd

$ rpm -q uwsgi-plugin-php uwsgi-logger-file  php-gd php-embedded uwsgi libgomp
uwsgi-plugin-php-2.0.21-7.fc38.x86_64
uwsgi-logger-file-2.0.21-7.fc38.x86_64
php-gd-8.2.5-1.fc38.x86_64
php-embedded-8.2.5-1.fc38.x86_64
uwsgi-2.0.21-7.fc38.x86_64
libgomp-13.1.1-1.fc38.x86_64

$ uwsgi --plugins php,logfile --logger file:/dev/tty --req-logger file:/dev/tty
*** Starting uWSGI 2.0.21 (64bit) on [Mon May 15 15:32:28 2023] ***
compiled with version: 13.0.1 20230401 (Red Hat 13.0.1-0) on 21 April 2023 00:00:00
os: Linux-6.2.14-300.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Mon May  1 00:55:28 UTC 2023
nodename: kendra
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /home/sun
detected binary path: /usr/sbin/uwsgi
your processes number limit is 63057
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
!!! uWSGI process 43334 got Segmentation Fault !!!
*** backtrace of 43334 ***
uwsgi(uwsgi_backtrace+0x4b) [0x556e3a1d258b]
uwsgi(uwsgi_segfault+0x2f) [0x556e3a1d2a5f]
/lib64/libc.so.6(+0x3db70) [0x7f2548cebb70]
/lib64/libgomp.so.1(+0xba4e) [0x7f2547038a4e]
/lib64/ld-linux-x86-64.so.2(+0x517f) [0x7f25496e917f]
/lib64/ld-linux-x86-64.so.2(+0x527d) [0x7f25496e927d]
/lib64/ld-linux-x86-64.so.2(_dl_catch_exception+0x142) [0x7f25496e55c2]
/lib64/ld-linux-x86-64.so.2(+0xbe3c) [0x7f25496efe3c]
/lib64/ld-linux-x86-64.so.2(_dl_catch_exception+0xa3) [0x7f25496e5523]
/lib64/ld-linux-x86-64.so.2(+0xc1b4) [0x7f25496f01b4]
/lib64/libc.so.6(+0x886d4) [0x7f2548d366d4]
/lib64/ld-linux-x86-64.so.2(_dl_catch_exception+0xa3) [0x7f25496e5523]
/lib64/ld-linux-x86-64.so.2(+0x1679) [0x7f25496e5679]
/lib64/libc.so.6(+0x881b3) [0x7f2548d361b3]
/lib64/libc.so.6(dlopen+0x6f) [0x7f2548d3678f]
/lib64/libphp-8.2.so(php_load_shlib+0x18) [0x7f254881f548]
/lib64/libphp-8.2.so(php_load_extension+0x205) [0x7f254881f785]
/lib64/libphp-8.2.so(zend_llist_apply+0x27) [0x7f25488e2637]
/lib64/libphp-8.2.so(+0x28dfc5) [0x7f254888dfc5]
/lib64/libphp-8.2.so(php_module_startup+0x836) [0x7f25488866e6]
/usr/lib64/uwsgi/php_plugin.so(+0x5b2b) [0x7f25496d6b2b]
/usr/lib64/uwsgi/php_plugin.so(+0x5eae) [0x7f25496d6eae]
uwsgi(uwsgi_start+0x5b7) [0x556e3a1d3637]
uwsgi(uwsgi_setup+0x17ac) [0x556e3a1d5f8c]
uwsgi(main+0xd) [0x556e3a178ded]
/lib64/libc.so.6(+0x27b4a) [0x7f2548cd5b4a]
/lib64/libc.so.6(__libc_start_main+0x8b) [0x7f2548cd5c0b]
uwsgi(_start+0x25) [0x556e3a178e25]
*** end of backtrace ***

This blows up in libgomp, in this code:

```
 2057       set to the initial global values.  */
 2058    add_initial_icv_to_list (GOMP_DEVICE_NUM_FOR_NO_SUFFIX, 0, NULL);
 2059    none = gomp_get_initial_icv_item (GOMP_DEVICE_NUM_FOR_NO_SUFFIX);
 2060    initialize_icvs (&none->icvs);
 2061
*2062    for (env = environ; *env != 0; env++)
 2063      {
 2064        if (!startswith (*env, "OMP_"))
 2065      continue;
 2066
```

`env` (not `*env`) turns out to be 0x0, at the very start of the `for` loop, the loop never runs. I don't even know how that's possible.

I've, so far, been only able to reproduce this when running php under uwsgi, running the php CLI interpreter works fine, even though it also loads the php-gd plugin, which subsequently loads libgomp.

The only obvious difference I can see right now is that uwsgi adds some variables to the environment before it loads php (via `setenv`). This might be some relocation gone wrong?

I have core dump files, if required, but the issue is easily reprodu

Reproducible: Always

Comment 1 Jakub Jelinek 2023-05-15 13:44:58 UTC
If environ is NULL, it is IMNSHO a bug in whatever program changed that pointer to NULL.

Comment 2 Florian Weimer 2023-05-15 13:48:09 UTC
There's also a glibc bug in this area, when dlopen is called from a preinit function. Not sure if this applies in this case.

Setting a hardware watchpoint on the environ variable should be illuminating.

Comment 3 Ralf Ertzinger 2023-05-15 13:59:54 UTC
But `environ` is not NULL, that's the strange thing. `environ` points to a, as far as I can tell, completely valid environment structure.

```
>>> print environ
$1 = (char **) 0x55555564ca70
>>> print env
$2 = (char **) 0x0
>>> print *(environ)
$3 = 0x555555648e60 "SHELL=/bin/bash"
```

This is the disassembly around the error:
```
   0x00007ffff5909a13 <+99>:    mov    rax,QWORD PTR [rip+0x4559e]        # 0x7ffff594efb8
   0x00007ffff5909a1a <+106>:   pxor   xmm0,xmm0
   0x00007ffff5909a1e <+110>:   mov    WORD PTR [rbp+0x54],0x100
   0x00007ffff5909a24 <+116>:   lea    r14,[rbp+0x8]
   0x00007ffff5909a28 <+120>:   movups XMMWORD PTR [rbp+0x8],xmm0
   0x00007ffff5909a2c <+124>:   movdqa xmm1,XMMWORD PTR [rip+0x363bc]        # 0x7ffff593fdf0
   0x00007ffff5909a34 <+132>:   lea    r13,[rip+0x358d4]        # 0x7ffff593f30f
   0x00007ffff5909a3b <+139>:   mov    r15,QWORD PTR [rax]
   0x00007ffff5909a3e <+142>:   mov    QWORD PTR [rbp+0x48],0xffffffffffffffff
   0x00007ffff5909a46 <+150>:   mov    BYTE PTR [rbp+0x56],0x0
   0x00007ffff5909a4a <+154>:   movups XMMWORD PTR [rbp+0x28],xmm0
=> 0x00007ffff5909a4e <+158>:   mov    r12,QWORD PTR [r15]
```

`[r15]` blows up, because `r15` is 0. This gets loaded from `[rax]` four lines above, and `rax` in turn was loaded from a `rip` relative position at the start. This looks like the assembly expects `[rip+0x4559e]` to be the memory location where `environ` is, but that's not where it is?

I'm not really useful with gdb, I admit.

Comment 4 Florian Weimer 2023-05-15 14:04:41 UTC
It's due to use of RTLD_DEEPBIND in php:

#  define DL_LOAD(libname)                      dlopen(libname, PHP_RTLD_MODE | RTLD_GLOBAL | RTLD_DEEPBIND)

RTLD_DEEPBIND changes the binding of environ in libc.so.6 from the main executable (present there because of a copy relocation, and correctly initialized) to the previously dormant symbol in libc.so.6 (which is uninitialized). GDB doesn't really handle this rebinding, which is why environ shows up as not NULL.

Ideally, php would stop using RTLD_DEEPBIND. If this is not possible, you can build the main program (uswgi in this case) as PIC, not PIE, which avoids copy relocations.

Comment 5 Ralf Ertzinger 2023-05-15 14:15:24 UTC
Is there another can of worms that I'd open with the PIE/PIC change? Is that a change that's required for uwsgi as a whole (binary and dynamically loadable modules), or just for the php module?

Comment 6 Florian Weimer 2023-05-15 14:19:34 UTC
(In reply to Ralf Ertzinger from comment #5)
> Is there another can of worms that I'd open with the PIE/PIC change? Is that
> a change that's required for uwsgi as a whole (binary and dynamically
> loadable modules), or just for the php module?

Just the main program (uwsgi), the modules are already PIC.

Comment 7 Ralf Ertzinger 2023-05-15 14:27:57 UTC
Thanks, I'll try that.

Comment 8 Joe Orton 2023-05-15 14:42:37 UTC
I was tagged into an upstream user hitting problems with RTLD_DEEPBIND recently. https://github.com/php/php-src/issues/10670

It may be time to turn this into a config option and flip the default from On to Off.

Comment 9 Ralf Ertzinger 2023-05-15 18:57:18 UTC
I've rebuilt uwsgi with

%undefine _hardened_build

which should disable PIE according to https://docs.fedoraproject.org/en-US/packaging-guidelines/#_pie

And the executable no longer says "LSB pie executable"

/usr/sbin/uwsgi: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=160cd2740fd9e07a5a72f574f62d2043a8fa075d, for GNU/Linux 3.2.0, stripped

Unfortunately it still blows up in the same way.

Comment 10 Florian Weimer 2023-05-15 19:43:28 UTC
(In reply to Ralf Ertzinger from comment #9)
> I've rebuilt uwsgi with
> 
> %undefine _hardened_build
> 
> which should disable PIE according to
> https://docs.fedoraproject.org/en-US/packaging-guidelines/#_pie
> 
> And the executable no longer says "LSB pie executable"
> 
> /usr/sbin/uwsgi: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
> dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2,
> BuildID[sha1]=160cd2740fd9e07a5a72f574f62d2043a8fa075d, for GNU/Linux 3.2.0,
> stripped
> 
> Unfortunately it still blows up in the same way.

You need to build as PIC. Try adding -fPIC to CFLAGS.

Comment 11 Ralf Ertzinger 2023-05-17 20:19:22 UTC
I've rebuilt the main binary (and the php plugin for good measure) with -fPIC (both in CFLAGS and LDFLAGS), and it still crashes the same way.

Build logs: https://download.copr.fedorainfracloud.org/results/ertzing/scratch_x64/fedora-38-x86_64/05929259-uwsgi/build.log.gz

Comment 12 Ralf Ertzinger 2023-09-12 14:53:59 UTC
what are the chances of having php buiit without RTLD_DEEPBIND for Fedora? I've managed to build uwsgi so this isn't an acute issue any more, but this requires disabling `_hardened_build` in the .spec file, which according to https://docs.fedoraproject.org/en-US/packaging-guidelines/#_pie I probably shouldn't be doing.

Comment 13 Aoife Moloney 2024-05-22 10:58:21 UTC
Fedora Linux 38 entered end-of-life (EOL) status on 2024-05-21.

Fedora Linux 38 is no longer maintained, which means that it
will not receive any further security or bug fix updates. As a result we
are closing this bug.

If you can reproduce this bug against a currently maintained version of Fedora Linux
please feel free to reopen this bug against that version. Note that the version
field may be hidden. Click the "Show advanced fields" button if you do not see
the version field.

If you are unable to reopen this bug, please file a new report against an
active release.

Thank you for reporting this bug and we are sorry it could not be fixed.