NB: This is most likely not an issue in java directly, but rather in e.g. selinux-policy. Posting it here first so that the java team is aware of it and also so that it gets assigned to the appropriate team if it's not an SELinux issue. Description of problem: If a systemd service file uses /usr/bin/java in its ExecStart, java will fail with: /usr/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory Version-Release number of selected component (if applicable): # rpm -q systemd java-1.8.0-openjdk systemd-229-8.fc24.x86_64 java-1.8.0-openjdk-1.8.0.92-5.b14.fc24.x86_64 # uname -r 4.5.5-300.fc24.x86_64 How reproducible: Always Steps to Reproduce: # cat > /run/systemd/system/foobar.service << EOF [Service] ExecStart=/usr/bin/java -version EOF # systemctl daemon-reload # systemctl start foobar # systemctl status foobar ● foobar.service Loaded: loaded (/run/systemd/system/foobar.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2016-07-20 19:08:07 UTC; 3s ago Process: 1515 ExecStart=/usr/bin/java -version (code=exited, status=127) Main PID: 1515 (code=exited, status=127) Jul 20 19:08:07 jlebon-tmp.localdomain systemd[1]: Started foobar.service. Jul 20 19:08:07 jlebon-tmp.localdomain java[1515]: /usr/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory Jul 20 19:08:07 jlebon-tmp.localdomain systemd[1]: foobar.service: Main process exited, code=exited, status=127/n/a Jul 20 19:08:07 jlebon-tmp.localdomain systemd[1]: foobar.service: Unit entered failed state. Jul 20 19:08:07 jlebon-tmp.localdomain systemd[1]: foobar.service: Failed with result 'exit-code'. Additional info: This is all conjecture from my own research and discussions with folks on #java: the root of the issue seems to be that SELinux is setting the AT_SECURE entry in the auxv causing the $ORIGIN var in the java rpath to go unsubstituted (which is normally how libjli.so is located). Disabling SELinux does make the issue go away.
Can confirm that this is a AT_SECURE issue. I compiled this program and ran it using systemd: #include <sys/auxv.h> #include <stdio.h> int main(int argc, char *argv[]) { unsigned long secure = getauxval(AT_SECURE); printf("AT_SECURE: %lu\n", secure); return 0; } # systemctl status foobar -l ● foobar.service Loaded: loaded (/run/systemd/system/foobar.service; static; vendor preset: disabled) Active: inactive (dead) Jul 21 18:34:24 trotts systemd[1]: Started foobar.service. Jul 21 18:34:24 trotts getauxval[11021]: AT_SECURE: 1 And this AT_SECURE is definitely impacting RPATH lookup. Running java -version as root, without systemd shows: # LD_DEBUG=all /usr/bin/java -version 2>&1 | grep 'file=libjli.so ...; needed by /usr/bin/java' -A4 11310: file=libjli.so [0]; needed by /usr/bin/java [0] 11310: find library=libjli.so [0]; searching 11310: search path=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.92-5.b14.fc24.x86_64/jre/bin/../lib/amd64/jli:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.92-5.b14.fc24.x86_64/jre/bin/../lib/amd64 (RPATH from file /usr/bin/java) 11310: trying file=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.92-5.b14.fc24.x86_64/jre/bin/../lib/amd64/jli/libjli.so 11310: But running it via systemd shows: Jul 21 18:37:41 trotts java[11110]: 11110: file=libjli.so [0]; needed by /usr/bin/java [0] Jul 21 18:37:41 trotts java[11110]: 11110: find library=libjli.so [0]; searching Jul 21 18:37:41 trotts java[11110]: 11110: search path=tls/x86_64:tls:x86_64: (RPATH from file /usr/bin/java) Jul 21 18:37:41 trotts java[11110]: 11110: trying file=tls/x86_64/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=tls/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=x86_64/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: search cache=/etc/ld.so.cache Jul 21 18:37:41 trotts java[11110]: 11110: search path=/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_ Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/lib64/tls/x86_64/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/lib64/tls/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/lib64/x86_64/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/lib64/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/usr/lib64/tls/x86_64/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/usr/lib64/tls/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/usr/lib64/x86_64/libjli.so Jul 21 18:37:41 trotts java[11110]: 11110: trying file=/usr/lib64/libjli.so And this fails because the libjli can only be located via RPATH.
I can also confirm that setting SELinux to permissive "fixes" the issue: # setenforce 0 # systemctl start foobar # systemctl status foobar ● foobar.service Loaded: loaded (/run/systemd/system/foobar.service; static; vendor preset: disabled) Active: inactive (dead) Jul 22 11:52:14 trotts systemd[1]: Started foobar.service. Jul 22 11:52:15 trotts java[3772]: openjdk version "1.8.0_92" Jul 22 11:52:15 trotts java[3772]: OpenJDK Runtime Environment (build 1.8.0_92-b14) Jul 22 11:52:15 trotts java[3772]: OpenJDK 64-Bit Server VM (build 25.92-b14, mixed mode) Assigning to SELinux to get some feedback from SELinux experts. Please feel free to reassign this back to java-1.8.0-openjdk if you think this can/should be fixed on the openjdk-side.
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
*** Bug 1388533 has been marked as a duplicate of this bug. ***
Another workaround is to use /bin/bash -c to run java. Avoids the use of permissive mode.
What AVC's are you seeing? ausearch -m avc -ts recent
Hey Dan, The AVC's affecting this don't show up usually as they are in the DONTAUDIT rules... Take a look at the bug that we marked a dupe of this: https://bugzilla.redhat.com/show_bug.cgi?id=1388533#c3 Specifically java uses rpath in it to locate some of the libraries but under systemd there is selinux confinement, unlike when run direct by a user... the AVC was: time->Tue Oct 25 19:07:27 2016 type=AVC msg=audit(1477418847.318:291): avc: denied { noatsecure } for pid=2074 comm="java" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=process permissive=1 Omair/Decaux ... on my system since the only java is the packaged one I symlinked /usr/lib64/libjli.so to /usr/lib/jvm/java-openjdk/lib/amd64/jli/libjli.so so it'd be within the normal path lookup and then it'd survive upgrades. Since it's in the normal path then the noatsecure being blocked doesn't affect it.
Looks reasonable to add this.
I am also having the exact same problem. Currently working on using the bash -c workaround, but that can only be a workaround. $ sudo systemctl daemon-reload $ sudo systemctl start some.service $ sudo systemctl status some.service ● some.service - Some server Loaded: loaded (/usr/local/lib/systemd/system/some.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2016-11-22 16:32:58 EST; 6s ago Process: 1247 ExecStop=/usr/bin/kill -s SIGTERM $MAINPID (code=exited, status=1/FAILURE) Process: 1245 ExecStart=/usr/bin/java -version (code=exited, status=127) Main PID: 1245 (code=exited, status=127) Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx systemd[1]: Started Some server. Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx systemd[1245]: some.service: Executing: /usr/bin/java -version Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx java[1245]: /usr/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx systemd[1]: some.service: Main process exited, code=exited, status=127/n/a Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx systemd[1247]: some.service: Executing: /usr/bin/kill -s SIGTERM Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx systemd[1]: some.service: Control process exited, code=exited status=1 Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx systemd[1]: some.service: Unit entered failed state. Nov 22 16:32:58 fedora-xxx-xxx-xx-xxxxxxxx systemd[1]: some.service: Failed with result 'exit-code'.
Any update? This is preventing unifi controller service from starting on F24.
from my experience this issue was caused by $ORIGIN being wiped when running java as a system process. For me the solution was to install "java-service-wrapper". In the systemd unit file have: ExecStart=/usr/sbin/java-service-wrapper /etc/java/minecraft-wrapper.conf and in the wrapper conf file; for the example of the minecraft server I run: wrapper.java.classpath.1=/usr/lib64/java-service-wrapper/wrapper.jar wrapper.java.classpath.2=server.jar wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp wrapper.app.parameter.1=net.minecraft.server.MinecraftServer There are several other configuration options for this file, but these can be researched via https://wrapper.tanukisoftware.com/doc/english/properties.html I hope this helps.
Thanks for the tip but that's a plain ugly workaround :) I may be grasping at straws but is this the issue? # semanage fcontext --list | grep java /emul/ia32-linux/usr(/.*)?/java/.*\.jar regular file system_u:object_r:lib_t:s0 /emul/ia32-linux/usr(/.*)?/java/.*\.jsa regular file system_u:object_r:lib_t:s0 /emul/ia32-linux/usr(/.*)?/java/.+\.so(\.[^/]*)* regular file system_u:object_r:lib_t:s0 /opt/(.*/)?java/.+\.jar regular file system_u:object_r:lib_t:s0 /opt/ibm/java.*/jre/.+\.jar regular file system_u:object_r:lib_t:s0 /opt/ibm/java.*/jre/.+\.so(\.[^/]*)* regular file system_u:object_r:textrel_shlib_t:s0 /opt/ibm/java.*/jre/bin/.+\.so(\.[^/]*)* regular file system_u:object_r:textrel_shlib_t:s0 /usr/(.*/)?java/.+\.jar regular file system_u:object_r:lib_t:s0 /usr/(.*/)?java/.+\.jsa regular file system_u:object_r:lib_t:s0 /usr/(.*/)?java/.+\.so(\.[^/]*)* regular file system_u:object_r:textrel_shlib_t:s0 /usr/lib/jvm/java(.*/)bin(/.*) all files system_u:object_r:bin_t:s0 /usr/lib/jvm/java(.*/)bin(/.*)?/.*\.so regular file system_u:object_r:textrel_shlib_t:s0 /usr/lib/libjavascriptcoregtk[^/]*\.so.* regular file system_u:object_r:textrel_shlib_t:s0 Specifically: /usr/lib/jvm/java(.*/)bin(/.*) all files system_u:object_r:bin_t:s0 /usr/lib/jvm/java(.*/)bin(/.*)?/.*\.so regular file system_u:object_r:textrel_shlib_t:s0 Do we need equivalent context set for: /usr/lib/jvm/java(.*/)lib(/.*) /usr/lib/jvm/java(.*/)lib(/.*)?/.*\.so ???
Never mind, I better understand the problem now... This should be fixed in the java package using alternatives... My VERY UGLY workaround which doesn't require java wrappers: Problem: $ORIGIN doesn't get interpreted... # chrpath /usr/bin/java /usr/bin/java: RPATH=$ORIGIN/../lib/amd64/jli:$ORIGIN/../lib/amd64 Solution: Add the right directory to ld.so.conf.d Problem #2, the correct location depends on which java you're running Solution #2: Set it on the fly: [Service] Type=simple User=unifi PermissionsStartOnly=true WorkingDirectory=/var/lib/unifi ExecStartPre=/usr/bin/bash -c "echo $(dirname $(readlink -f /usr/bin/java))/../lib/amd64/jli > /etc/ld.so.conf.d/java.conf" ExecStartPre=/usr/bin/bash -c "echo $(dirname $(readlink -f /usr/bin/java))/../lib/amd64 >> /etc/ld.so.conf.d/java.conf" ExecStartPre=/usr/sbin/ldconfig ExecStart=/usr/bin/java -jar /usr/share/unifi/lib/ace.jar start ExecStop=/usr/bin/java -jar /usr/share/unifi/lib/ace.jar stop
(In reply to Richard Shaw from comment #13) > Never mind, I better understand the problem now... This should be fixed in > the java package using alternatives... It's not just Java which has this problem. systemd needs a way to run system services without an AT_SECURE transition even if the SELinux policy triggers one, otherwise many libraries will disable further functionality controlled by environment variables.
Well my "solution" broke other instance if java so I had to give up on that one. I've also tried creating a copy of the java binary and changing the RPATH to remove $ORIGIN and make it an absolute path but then it fails to find other libraries... REALLY NEED a practical solution here, the wrapper method is overly complicated and quite frankly ridiculous.
Easy but bad solution, allow init_t to run in permissive mode. At least you don't have to run the whole system in permissive mode... Enable: # semanage permissive -a init_t Disable: # semanage permissive -d init_t
(In reply to Richard Shaw from comment #15) > Well my "solution" broke other instance if java so I had to give up on that > one. I've also tried creating a copy of the java binary and changing the > RPATH to remove $ORIGIN and make it an absolute path but then it fails to > find other libraries... REALLY NEED a practical solution here, the wrapper > method is overly complicated and quite frankly ridiculous. Richard, does the workaround in comment #7 (creating a /usr/lib64/libjli.so symlink) work for you? It worked for me (running Jenkins from a systemd unit)... Or is there some serious downside to working around it in that way?
Doesn't look like it would work for 32-bit systems or a system based package install... Need something that will work from a packaging point of view.
(In reply to Richard Fearn from comment #17) > (In reply to Richard Shaw from comment #15) > > Well my "solution" broke other instance if java so I had to give up on that > > one. I've also tried creating a copy of the java binary and changing the > > RPATH to remove $ORIGIN and make it an absolute path but then it fails to > > find other libraries... REALLY NEED a practical solution here, the wrapper > > method is overly complicated and quite frankly ridiculous. > > Richard, does the workaround in comment #7 (creating a /usr/lib64/libjli.so > symlink) work for you? It worked for me (running Jenkins from a systemd > unit)... > > Or is there some serious downside to working around it in that way? Hi, I just ran into this issue too, while trying to run apache tika server. Workaround from comment #7 works for me, on Fedora 25.
This also affects openfire: https://www.igniterealtime.org/projects/openfire/ Running the launcher script ExecStart=/opt/openfire/bin/openfire.sh works. Using /usr/bin/java directly: ExecStart=/usr/binjava -server .... fails with the libjli.so issue. This is a really obscure issue especially because there were no audit2allow mesages. audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=openfire comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' java[19612]: /usr/lib/jvm/jre-openjdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
This also affects openfire: https://www.igniterealtime.org/projects/openfire/ Running the launcher script ExecStart=/opt/openfire/bin/openfire.sh works. Using /usr/bin/java directly: ExecStart=/usr/binjava -server .... fails with the libjli.so issue. This is a really obscure issue especially because there were no audit2allow messages. audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=openfire comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' java[19612]: /usr/lib/jvm/jre-openjdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
No reason this doesn't affect F26 but I've only verified it's still a problem on my F25 desktop...
(In reply to Richard Shaw from comment #24) > No reason this doesn't affect F26 but I've only verified it's still a > problem on my F25 desktop... Does affect F26 (with systemd-233-6 and java-1.8.0-openjdk-1.8.0.141-1.b16).
I am seeing this issues as well
selinux-policy-3.13.1-260.14.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-d312739a4e
I'm afraid 3.13.1-260.14 doesn't fix it. I just installed it and rebooted, but running the UniFi controller gets /usr/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory Running in permissive mode is fine.
I gave up and just wrapped it in a bash script so $ORIGIN is available... $ cat unifi.sh #!/bin/bash /usr/bin/java -jar /usr/share/unifi/lib/ace.jar $1 You can use my RPM package if you like... https://community.ubnt.com/t5/UniFi-Wireless/Unofficial-RHEL-CentOS-UniFi-Controller-rpm-packages/td-p/1744595 My packages (the 2nd post) are more FHS compliant. Thias (the 1st post) just dumps everything in /opt.
selinux-policy-3.13.1-260.14.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Does this mean that running jar's in a shell as a workaround should no longer be necessary?
This fix does not seem to work for me. I have updated to: selinux-policy-targeted-3.13.1-260.16.fc26.noarch selinux-policy-3.13.1-260.16.fc26.noarch Restarted. The above test still fails: Nov 21 21:44:03 systemd[1]: Started foobar.service. Nov 21 21:44:03 java[17778]: /usr/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory Nov 21 21:44:03 systemd[1]: foobar.service: Main process exited, code=exited, status=127/n/a The '/bin/bash -c' workaround still applies.
(In reply to Richard Shaw from comment #13) > Problem: $ORIGIN doesn't get interpreted... > # chrpath /usr/bin/java > /usr/bin/java: RPATH=$ORIGIN/../lib/amd64/jli:$ORIGIN/../lib/amd64 I just wanted to note on this issue, as glibc maintainer, that an RPATH rooted in a trusted directory e.g. /usr/lib64 or /usr/lib, will be allowed for a setuid/AT_SECURE binary (following a few more additional rules, like $ORIGIN is the first and only DST in the sequence). This allows AT_SECURE binaries to use libraries from non-standard trusted library paths relative to their install.