adding /usr/lib/jvm/java-openjdk/lib/amd64/jli to the ldconfig path of course allows the resolution of the library ...

(In reply to James Hogarth from comment #0)
> Description of problem:
> Java application running directly from /usr/bin/java fails with: 
> error while loading shared libraries: libjli.so: cannot open shared object
> file: No such file or directory 

Does `/usr/bin/java -version` work?

> 1. /usr/bin/java -jar /path/to/jar (I'm using the subsonic media server
> where I observe the failure)

Can you provide link to this jar and the exact command you used? Can you also run this command with the environment variable LD_DEBUG=libs set?

> 2. ldd /usr/bin/java

This is kind of silly. ldd won't traverse the symlink to find the real file. If `/usr/bin/java -version` works, then libjli is being found. You can confirm that yourself using `LD_DEBUG=libs /usr/bin/java -version`. The output should say something about looking for libjli.so in /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b16.fc24.x86_64/jre/bin/../lib/amd64/jli/libjli.so

> ldd to be able to resolve the library when /usr/bin/java is called rather
> than only with a direct call to where alternatives points to

This isn't how ldd works. Try `ldd $(readlink -f /usr/bin/java)` instead to make it follow the link.

(In reply to Omair Majid from comment #2)
> (In reply to James Hogarth from comment #0)
> > Description of problem:
> > Java application running directly from /usr/bin/java fails with: 
> > error while loading shared libraries: libjli.so: cannot open shared object
> > file: No such file or directory 
> 
> Does `/usr/bin/java -version` work?
> 

yes

> > 1. /usr/bin/java -jar /path/to/jar (I'm using the subsonic media server
> > where I observe the failure)
> 
> Can you provide link to this jar and the exact command you used? Can you
> also run this command with the environment variable LD_DEBUG=libs set?
> 

the jar from installing this rpm http://www.subsonic.org/pages/download.jsp

> > 2. ldd /usr/bin/java
> 
> This is kind of silly. ldd won't traverse the symlink to find the real file.
> If `/usr/bin/java -version` works, then libjli is being found. You can
> confirm that yourself using `LD_DEBUG=libs /usr/bin/java -version`. The
> output should say something about looking for libjli.so in
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b16.fc24.x86_64/jre/bin/../lib/
> amd64/jli/libjli.so
> 
> > ldd to be able to resolve the library when /usr/bin/java is called rather
> > than only with a direct call to where alternatives points to
> 
> This isn't how ldd works. Try `ldd $(readlink -f /usr/bin/java)` instead to
> make it follow the link.

that's what I had thought but then the evidence in troubleshooting was confusing ... 

the error in the logs is:

[root@server ~]# systemctl status subsonic
â subsonic.service - Subsonic music server
   Loaded: loaded (/etc/systemd/system/subsonic.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2016-10-25 19:15:23 BST; 1s ago
  Process: 2573 ExecStart=/usr/bin/java $SUBSONIC_MAX_MEMORY $SUBSONIC_HOME $SUBSONIC_HOST $SUBSONIC_PORT $SUBSONIC_CONTEXT_PATH $SUBSONIC_DEFAULT_MUSIC_FOLDER $SUBSONIC_DEFAULT_PODCAST_FOLDER $SUBSONIC_DEFAULT_PLAYLIST_FOLDER -Djava.awt
 Main PID: 2573 (code=exited, status=127)

Oct 25 19:15:23 server.purley.hogarthuk.local systemd[1]: Started Subsonic music server.
Oct 25 19:15:23 server.purley.hogarthuk.local subsonic[2573]: /usr/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
Oct 25 19:15:23 server.purley.hogarthuk.local systemd[1]: subsonic.service: Main process exited, code=exited, status=127/n/a
Oct 25 19:15:23 server.purley.hogarthuk.local systemd[1]: subsonic.service: Unit entered failed state.
Oct 25 19:15:23 server.purley.hogarthuk.local systemd[1]: subsonic.service: Failed with result 'exit-code'.

adding libjli.so to the ldconfig path or symlinking it into the existing LD path lets it work which is why I started going down that root ...

the systemd unit i'm using is:

[Unit]
Description=Subsonic music server

[Service]
EnvironmentFile=/etc/sysconfig/subsonic
User=subsonic
Group=subsonic
WorkingDirectory=/usr/share/subsonic
SyslogIdentifier=subsonic
ExecStart=/usr/bin/java $SUBSONIC_MAX_MEMORY $SUBSONIC_HOME $SUBSONIC_HOST $SUBSONIC_PORT $SUBSONIC_CONTEXT_PATH $SUBSONIC_DEFAULT_MUSIC_FOLDER $SUBSONIC_DEFAULT_PODCAST_FOLDER $SUBSONIC_DEFAULT_PLAYLIST_FOLDER -Djava.awt.headless=true -

[Install]
WantedBy=multi-user.target


[root@server ~]# cat /etc/sysconfig/subsonic
SUBSONIC_HOME="-Dsubsonic.home=/var/subsonic"
SUBSONIC_HOST="-Dsubsonic.host=0.0.0.0"
SUBSONIC_PORT="-Dsubsonic.port=4040"
SUBSONIC_CONTEXT_PATH="-Dsubsonic.contextPath=/subsonic"
SUBSONIC_MAX_MEMORY="-Xmx1024m"
SUBSONIC_DEFAULT_MUSIC_FOLDER="-Dsubsonic.defaultMusicFolder=/var/music"
SUBSONIC_DEFAULT_PODCAST_FOLDER="-Dsubsonic.defaultPodcastFolder=/var/music/Podcast"
SUBSONIC_DEFAULT_PLAYLIST_FOLDER="-Dsubsonic.defaultPlaylistFolder=/var/playlists"

_____________________________

further troubleshooting though adds to some confusion ...

if I pop selinux onto permissive it works but there's no avc's immediately apparent that would look like they should have an effect

I can't recall exactly when this broke but it was before the last update as I bandaided it with a symlink last time due to lack of time for the bug

The symlink from my bandaid is:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.102-1.b14.fc24.x86_64/jre/lib/amd64/jli/libjli.so

So it might have been around then... popping the symlink in again (this time using the java-openjdk symlink to avoid the path changing in future) does let the service work

disabling dontaudit rules there is one suspect though ... the java process in the init_t context was denied noatsecure and creating a quick module to allow init_t process { noatsecure } then allows it to work ...

http://blog.siphos.be/2011/04/selinux-and-noatsecure-or-why-portage-complains-about-ld_preload-and-libsandbox-so/
http://seedit.sourceforge.net/doc/access_vectors/

is the java executable doing anything that environment sanitization during a transition might break the ability to find libjli if not on the standard library path?

I suspect this will affect other jars started from systemd directly rather than through a script for some sort of web service as well

This is a duplicate of bug 1358476.

Basically, the issue boils down to the JVM using RPATHs containing $ORIGIN. The dynamic linker sees the AT_SECURE flag (set by SELinux), and stops using $ORIGIN paths to look up the dependencies. That means the dynamic linker can not find necessary libraries (inlcuding libjli).

Unfortunately, this affects all applications trying to use Java and running under SELinux

*** This bug has been marked as a duplicate of bug 1358476 ***

Ah great that does make sense and explain things... Of course we generally limit rpath stuff but presumably this is to allow multiple jres to coexist?

(In reply to James Hogarth from comment #5)
> Ah great that does make sense and explain things... Of course we generally
> limit rpath stuff but presumably this is to allow multiple jres to coexist?

Upstream produces a JDK that's location independent, so it can be copied (or moved) to /usr/local/jdk, /opt/jdk or ~/local/jdk and it would still work. It uses $ORIGIN to let the dynamic linker know where to look for JDK-internal libraries, without needing the users to modify global configuration or hardcoding the RPATHs.

We just follow what upstream does, here.

Hmm this might be a time when it's sensible as a distribution to diverge from upstream, especially considering the guidelines around rpath as well ...

https://fedoraproject.org/wiki/Packaging:Guidelines#Beware_of_Rpath

This would effectively solve the referenced issue this was marked as a dupe of as well rather than potentially compromising security by allowing init_t process { noatsecure; } ... AT_SECURE is enabled for a reason after all.

Super easy reproducer.
And yes, this also breaks 'java -versions' And no, I don't know how to deal with it in a mock environment. :( 1.5 years on and we still have no resolution?

.....

# Create mock chroot environment and chroot into it...
mock -r fedora-27-x86_64 --init --dnf-cmd install java-1.8.0-openjdk-headless java-1.8.0-openjdk-devel maven java-1.8.0-openjdk-openjfx-devel
sudo chroot /var/lib/mock/fedora-27-x86_64/root/
su -l mockbuild

...here it shows up as screwy...
[mockbuild@localhost ~]$ java -version
java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

[mockbuild@localhost ~]$ ldd /usr/bin/java
linux-vdso.so.1 (0x00007ffd581bb000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9efbca7000)
libz.so.1 => /lib64/libz.so.1 (0x00007f9efba90000)
libjli.so => not found
libdl.so.2 => /lib64/libdl.so.2 (0x00007f9efb88c000)
libc.so.6 => /lib64/libc.so.6 (0x00007f9efb4d6000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9efc0c7000)

[mockbuild@localhost ~]$ ll /usr/bin/java
lrwxrwxrwx. 1 root root 22 Apr 19 10:16 /usr/bin/java -> /etc/alternatives/java

[mockbuild@localhost ~]$ ll /etc/alternatives/java
lrwxrwxrwx. 1 root root 72 Apr 19 10:16 /etc/alternatives/java -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.162-3.b12.fc27.x86_64/jre/bin/java

[mockbuild@localhost ~]$ ldd /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.162-3.b12.fc27.x86_64/jre/bin/java
linux-vdso.so.1 (0x00007ffd81469000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f359f6b2000)
libz.so.1 => /lib64/libz.so.1 (0x00007f359f49b000)
libjli.so => /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.162-3.b12.fc27.x86_64/jre/bin/../lib/amd64/jli/libjli.so (0x00007f359f28c000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f359f088000)
libc.so.6 => /lib64/libc.so.6 (0x00007f359ecd2000)
/lib64/ld-linux-x86-64.so.2 (0x00007f359fad2000)

..................

This breaks things. I assume this breaks java in COPR as well, though I haven't tried yet. I am open to any ideas how to work around this until something more permanent is figured out.

I should add to my last comment...
F27 host and F27 mock environment (all fully updated).
I tried the same while disabling SELinux as well... same result.