Bug 1362244

Summary: selinux strips LD_LIBRARY_PATH from 'Environment=LD_LIBRARY_PATH=/usr/local/lib' in [Service] section
Product: [Fedora] Fedora Reporter: David Kaspar // Dee'Kej <deekej>
Component: selinux-policyAssignee: Lukas Vrabec <lvrabec>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: rawhideCC: ads.kuknus, alexfilan0921, berrange, collura, dmitriy.poterukha, dtardon, dwalsh, fradisel, glendagoodman310, helendam.gm, johannbg, john.obaterspok, jwakely, katherinewallitonejohn, ljglmail, lnykryn, lvrabec, msekleta, muadda, plautrba, raymondbarry78, s, thosew.erethedayswheniwas, vmojzis, wonrya6660, zbyszek, zetisonapi, zpytela
Target Milestone: ---Keywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-20 14:32:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Kaspar // Dee'Kej 2016-08-01 16:26:10 UTC
Description of problem:
=======================
After an upgrade from F23->F24, systemd started to ignore 'Environment' setup in [Service] section of a custom service:
---------------
[Unit]
Description=Razer device state daemon
Requires=local-fs.target
After=local-fs.target

[Service]
Type=forking
Environment=LD_LIBRARY_PATH=/usr/local/lib
ExecStart=/usr/local/bin/razerd -f -B -P /run/razerd/razerd.pid

[Install]
Alias=multi-user.target.wants/razerd.service


When I try to launch the service with 'sudo systemctl start razerd.service', it fails. Output of 'systemctl status razerd.service':
----------------
● razerd.service - Razer device state daemon
   Loaded: loaded (/usr/lib/systemd/system/razerd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2016-08-01 18:07:41 CEST; 41s ago
  Process: 26264 ExecStart=/usr/local/bin/razerd -f -B -P /run/razerd/razerd.pid (code=exited, status=127)

systemd[1]: Starting Razer device state daemon...
razerd[26264]: /usr/local/bin/razerd: error while loading shared libraries: librazer.so.1: cannot open shared object file: No such file or directory
systemd[1]: razerd.service: Control process exited, code=exited status=127
systemd[1]: Failed to start Razer device state daemon.
systemd[1]: razerd.service: Unit entered failed state.
systemd[1]: razerd.service: Failed with result 'exit-code'.

[I have removed the timestamps and localhost name.]

The *.so object exists:
-----------------------
dkaspar@Normandy-SR4 /usr/local/lib $ ll
lrwxrwxrwx. 1 root root   13 Mar 17 17:57 librazer.so -> librazer.so.1*
-rwxr-xr-x. 1 root root 126K Aug  1 17:26 librazer.so.1*

Version-Release number of selected component (if applicable):
=========================
systemd-229-9.fc24.x86_64

How reproducible:
=================
N/A

Steps to Reproduce:
===================
1. Download the Razer Drivers repository from: https://github.com/mbuesch/razer
2. Install dependencies: python3 libusb cmake (python3-pyside is not necessary)
3. Add the line below into razerd.service.template (section [Service]):

  Environment=LD_LIBRARY_PATH=/usr/local/lib

4. Run 'cmake .' and 'make'
5. Install the service by running 'sudo make install'
6. Start the service by running 'sudo systemctl start razerd.service'

Actual results:
===============
Job for razerd.service failed because the control process exited with error code. See "systemctl status razerd.service" and "journalctl -xe" for details.

Expected results:
=================
Service is started without error.

Additional info:
================
I'm able to borrow you my secondary Razer Mouse for testing purposes, I'm located in Red Hat Brno office.

Comment 1 Zbigniew Jędrzejewski-Szmek 2016-08-01 17:27:55 UTC
I think that's unlikely that LD_LIBRARY_PATH is not set properly or ignored. I made a local test with /bin/sleep, and lsof shows that it in fact uses a library from LD_LIBRARY_PATH as expected.

Please test that your binary works fine without systemd, for example by doing "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/razerd" from a terminal.

Comment 2 David Kaspar // Dee'Kej 2016-08-02 10:55:05 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #1)
> Please test that your binary works fine without systemd, for example by
> doing "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/razerd" from a terminal.

dkaspar@Normandy-SR4 /usr/local/bin $ sudo LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/razerd
Razer device service daemon
^CTerminating razerd.

... So when I specify the LD_LIBRARY_PATH manually, as you suggested, it runs OK.

For a testing purposes, I've then replaced /usr/local/bin/razerd with a shell script to dump the output of 'env' when it is launched, here's the result:

dkaspar@Normandy-SR4 /tmp $ cat razerd-env
LC_PAPER=cs_CZ.UTF-8
LC_ADDRESS=cs_CZ.UTF-8
LC_MONETARY=cs_CZ.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_TELEPHONE=cs_CZ.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
LC_IDENTIFICATION=en_US.UTF-8
PWD=/
LANG=en_US.UTF-8
LC_MEASUREMENT=cs_CZ.UTF-8
SHLVL=1
LC_TIME=en_GB.UTF-8
LC_NAME=cs_CZ.UTF-8
_=/usr/bin/env

... The path does not contain /usr/local/lib, nor the LD_LIBRARY_PATH is set in the environment when the /usr/local/bin/razerd is launched via 'sudo systemctl start razerd.service'.

Comment 3 Zbigniew Jędrzejewski-Szmek 2016-08-02 11:46:51 UTC
I works here:
razerd[15666]: Razer device service daemon
razerd[15666]: librazer: No config file /etc/razer.conf present. Ignoring.

I would guess that you have some old copy of the service lying around or something similar. Can you check that:
- "systemctl cat razerd" shows you the expected contents
- "systemctl show -p Environment razerd" has the expected value

Comment 4 David Kaspar // Dee'Kej 2016-08-02 13:59:22 UTC
dkaspar@Normandy-SR4 ~ $ systemctl cat razerd
# /usr/lib/systemd/system/razerd.service
[Unit]
Description=Razer device state daemon
Requires=local-fs.target
After=local-fs.target

[Service]
Type=forking
Environment=LD_LIBRARY_PATH=/usr/local/lib
ExecStart=/usr/local/bin/razerd -f -B -P /run/razerd/razerd.pid

[Install]
Alias=multi-user.target.wants/razerd.service

dkaspar@Normandy-SR4 ~ $ systemctl show -p Environment razerd
Environment=LD_LIBRARY_PATH=/usr/local/lib

... When I make the symlink from /usr/lib/librazer.so.1 to /usr/local/lib/librazer.so.1 and run 'sudo ldconfig', then the service is able to start correctly. But not without it... :-/

Comment 5 John Obaterspok 2017-07-18 11:54:53 UTC
It's happening for me on Fedora 26 as well.

The below is not working (results in .so not found)
[Service]
Type=notify
Environment=LD_LIBRARY_PATH=/usr/local/lib/
ExecStart=/usr/local/bin/myapp

Where as if I do like this it *is* working
[Service]
Type=notify
ExecStart=/bin/sh -c 'LD_LIBRARY_PATH=/usr/local/lib "/usr/local/bin/myapp"'

So something has changed since Fedora 23/24 where LD_LIBRARY_PATH did indeed work. I also tried to setenforce 0 before starting the service but that didnt help.

I have 3 different /usr/local/ services that worked before and are failing now.

Comment 6 John Obaterspok 2017-07-18 12:03:16 UTC
Neither setting it with systemcl set-environment makes it work

# systemctl show-environment
LANG=sv_SE.UTF-8
LD_LIBRARY_PATH=/usr/local/lib
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin


Adding /usr/local/lib to /etc/ld.so.conf makes it work

Comment 7 Zbigniew Jędrzejewski-Szmek 2017-07-18 14:11:59 UTC
Ha, maybe it's SELinux! Are y'all running in "enforcing" mode? If yes, can you check whether it works in "permissive"?

Comment 8 John Obaterspok 2017-07-18 18:13:39 UTC
Comment 5 said "I also tried to setenforce 0 before starting the service but that didnt help."

Shouldn't that be enough to temporarily disable SELinux?

Comment 9 John Obaterspok 2017-07-18 20:54:53 UTC
Well, I just tried to disable selinux in /etc/selinux/config and reboot
=> It works

Then I flipped selinux back to enforcing and after a reboot
=> It does not work

I noticed these thread around selinux & ld_library_path:
http://selinux.tycho.nsa.narkive.com/yQnAv3QF/selinux-policy-regarding-ld-library-path
https://bugzilla.redhat.com/show_bug.cgi?id=1382706

Comment 10 Zbigniew Jędrzejewski-Szmek 2017-07-18 21:17:10 UTC
Ah, right. I missed that.

See also the linked bug and https://lists.fedoraproject.org/archives/list/selinux@lists.fedoraproject.org/message/EQABLLVU4CS5H6AT56D7D5SQAXIZFZBH/.

I tried to debug this, and it *seems* selinux related, but I cannot pin down the exact rules. I installed fresh F26 server, and httpd, and added 
# /etc/systemd/system/httpd.service.d/override.conf
[Service]
ExecStartPre=/usr/bin/env
Environment=LD_LIBRARY_PATH=/usr/local/lib MY_LIBRARY_PATH=/foobar

When selinux is in enforcing mode, /usr/bin/env reports
Jul 18 17:07:11 localhost.localdomain env[1779]: LANG=C
Jul 18 17:07:11 localhost.localdomain env[1779]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Jul 18 17:07:11 localhost.localdomain env[1779]: INVOCATION_ID=c44bf72e001c4e1f885724c55c85d474
Jul 18 17:07:11 localhost.localdomain env[1779]: JOURNAL_STREAM=9:35997
Jul 18 17:07:11 localhost.localdomain env[1779]: MY_LIBRARY_PATH=/foobar

when selinux is in permissive mode,
Jul 18 17:06:55 localhost.localdomain env[1722]: LANG=C
Jul 18 17:06:55 localhost.localdomain env[1722]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Jul 18 17:06:55 localhost.localdomain env[1722]: INVOCATION_ID=57a11b7f01bf4b64817b709af8302c6a
Jul 18 17:06:55 localhost.localdomain env[1722]: JOURNAL_STREAM=9:35672
Jul 18 17:06:55 localhost.localdomain env[1722]: LD_LIBRARY_PATH=/usr/local/lib
Jul 18 17:06:55 localhost.localdomain env[1722]: MY_LIBRARY_PATH=/foobar

But httpd which is started after that always has
$ sudo cat /proc/1780/environ|tr '\0' '\n'
LANG=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
NOTIFY_SOCKET=/run/systemd/notify
INVOCATION_ID=c44bf72e001c4e1f885724c55c85d474
JOURNAL_STREAM=9:36075
LD_LIBRARY_PATH=/usr/local/lib
MY_LIBRARY_PATH=/foobar

So either there's something seriously wrong with systemd (e.g. memory corruption in the code to set variables), or SELinux has some very inconsistent rules. Maybe somebody who understands selinux better can shed some light.

Comment 11 John Obaterspok 2017-07-19 06:00:28 UTC
Just to be on the safe side, did you notice my follow up comment 10 where I said disabling SELinux makes it work? (I wasn't sure when you said "and it *seems* selinux related")

Comment 12 Zbigniew Jędrzejewski-Szmek 2017-07-19 16:41:31 UTC
Yes, I'm pretty certain that it's SELinux following it's policy.

Nevertheless, I think that this SELinux behaviour makes no sense and should be stripped away. Not just for this case, but as a general approach: filtering one special variable when the caller completely controls the full environment and arguments of the callee is just not useful. The only thing this can do is result in random breakage like this bug. @dwalsh: can we just nuke the whole concept from selinux policy please?

Comment 13 Daniel Walsh 2017-07-24 13:45:54 UTC
Could you collect the AVC messages when this happens. If you are not seeing any, please try to disable the SELinux rules

# semodule -DB

Cause the behaviour
Collect the AVCs

# ausearch -m avc -ts recent

Turn back on dontautit rules

# semodule -B

Comment 14 Fedora End Of Life 2017-07-25 22:11:35 UTC
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 15 John Obaterspok 2017-07-28 20:28:25 UTC
# ausearch -m avc -ts recent
----
time->Fri Jul 28 22:23:16 2017
type=AVC msg=audit(1501273396.643:2823): avc:  denied  { siginh } for  pid=7810 comm="sonoscontrol" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=0
----
time->Fri Jul 28 22:23:16 2017
type=AVC msg=audit(1501273396.644:2824): avc:  denied  { noatsecure } for  pid=7810 comm="sonoscontrol" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=0
----
time->Fri Jul 28 22:23:16 2017
type=AVC msg=audit(1501273396.883:2828): avc:  denied  { siginh } for  pid=7812 comm="sonoscontrol" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=0
----
time->Fri Jul 28 22:23:16 2017
type=AVC msg=audit(1501273396.884:2829): avc:  denied  { noatsecure } for  pid=7812 comm="sonoscontrol" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=0
----
time->Fri Jul 28 22:23:17 2017
type=AVC msg=audit(1501273397.134:2833): avc:  denied  { siginh } for  pid=7813 comm="sonoscontrol" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=0
----
time->Fri Jul 28 22:23:17 2017
type=AVC msg=audit(1501273397.134:2834): avc:  denied  { noatsecure } for  pid=7813 comm="sonoscontrol" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=0
----
time->Fri Jul 28 22:23:20 2017
type=AVC msg=audit(1501273400.449:2839): avc:  denied  { siginh } for  pid=7815 comm="setroubleshootd" scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tclass=process permissive=0
----
time->Fri Jul 28 22:23:20 2017
type=AVC msg=audit(1501273400.450:2840): avc:  denied  { noatsecure } for  pid=7815 comm="setroubleshootd" scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tclass=process permissive=0
----
time->Fri Jul 28 22:23:20 2017
type=AVC msg=audit(1501273400.448:2838): avc:  denied  { rlimitinh } for  pid=7815 comm="setroubleshootd" scontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tcontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tclass=process permissive=0


sonoscontrol is a service of my own

Comment 16 John Obaterspok 2019-10-04 19:08:19 UTC
I just verified this is working fine in both F30/Centos8

Comment 17 Zdenek Pytela 2020-01-20 14:32:13 UTC
John,

Thank you for confirming. Closing this bz per your previous comment,

Comment 29 zetiano 2023-07-01 01:14:41 UTC Comment hidden (spam)