Bug 1794083

This bug is adversely impacting AuristorFS and OpenAFS end users that attempt to access arbitrary AFS volume roots via the 

  /afs/.:mount/<cell>:<volume>/

path.

The fix was first included in glibc 2.19.   rhel7 shipped with glibc 2.17.   If the upstream fix was not backported to rhel7, it is required there as well.

The first fix merged upstream for glibc bugzilla 10253 was 

  https://sourceware.org/git/?p=glibc.git;a=commit;h=b75891075bece24be9fd85618f18af4a2daf7f1c

  author	Ondřej Bílka <neleai>	
  Mon, 18 Nov 2013 18:56:57 +0000 (19:56 +0100)
  commit	b75891075bece24be9fd85618f18af4a2daf7f1c
  Fix breaking of RPATH when $ORIGIN contains colons. Fixes bug 10253

  We first expanded origin and then split string by colons. This
  misbehaves when $ORIGIN contain colon so we first split string, then
  expand $ORIGIN.

This change was reverted by

  https://sourceware.org/git/?p=glibc.git;a=commit;h=246a7c1bc2cc021ff4deebc76164bcbcc5a4aa9c
  
  author	Ondřej Bílka <neleai>	
  Mon, 18 Nov 2013 22:42:58 +0000 (23:42 +0100)
  commit	246a7c1bc2cc021ff4deebc76164bcbcc5a4aa9c

  Revert b75891075bece24be9fd85618f18af4a2daf7f1c

A new fix was merged to address the issue a few weeks later:

  https://sourceware.org/git/?p=glibc.git;a=commit;h=2a939a7e6d81f109d49306bc2e10b4ac9ceed8f9

  author	Ondřej Bílka <neleai>	
  Mon, 2 Dec 2013 13:53:14 +0000 (14:53 +0100)
  commit	2a939a7e6d81f109d49306bc2e10b4ac9ceed8f9

  Return fixed version of  breaking of RPATH when $ORIGIN contains colons

This last fix appears to be present in the current rhel7 glibc sources.  Please double-check.

(In reply to Jeffrey Altman from comment #7)
> This bug is adversely impacting AuristorFS and OpenAFS end users that
> attempt to access arbitrary AFS volume roots via the 
> 
>   /afs/.:mount/<cell>:<volume>/
> 
> path.

Thanks. The choice of a separator character which is not in the POSIX safe set for pathnames is rather unfortunate. Obviously, such paths are incompatible with path lists in PATH, LD_LIBRARY_PATH, LD_PRELOAD, Java's -classpath argument etc. This is a fundamental issue that cannot be fixed.

I assume that in other contexts, the ':' does not leak into pathnames because symbolic links are used? And that does not work with $ORIGIN because the pathname has to be resolved for security reasons.

Can you use bind mounts instead of symbolic links to provide the expected pathnames to users?

(In reply to Florian Weimer from comment #11)
> The choice of a separator character which is not in the POSIX safe
> set for pathnames is rather unfortunate.

The selection of colon is unfortunate.  It was selected because its a character
cannot be present in an AFS cell or volume name and it has been used as the 
separator between cell and volume names in the AFS mount point target string
since the late 1980s.  The format of the mount point target string is

  <mp-type>[<cell>:]<volume>

where the <mp-type> is either '%' or '#' and the existence of the <cell> and
separator is optional.  The lack of a cell indicates that the target volume 
is located within in the same cell as the volume the mount point is located.

The colon is of course the CIFS/NTFS separator for Alternate Data Streams.

The /afs directory in modern AuriStorFS/AFS cache managers is a dynamically
generated directory whose directory entries consist of cell names for AFS
normal mount point paths where RW volumes are preferred to RW volumes and 
dot prefixed cell names for AFS rw mount points which always access RW volumes.

 /afs/example.net/
 /afs/.example.net/

Whether a particular directory entry exists is determined either by the existence
of a DNS SRV or DNS AFSDB record for the cell or an entry in a local configuration
file.

In hindsight a better choice would have been 

  /afs/.@mount/<cell>/<volume>/

but that is decades of water under the bridge at this point.

> Obviously, such paths are
> incompatible with path lists in PATH, LD_LIBRARY_PATH, LD_PRELOAD, Java's
> -classpath argument etc. This is a fundamental issue that cannot be fixed.

Understood.  

> I assume that in other contexts, the ':' does not leak into pathnames
> because symbolic links are used? And that does not work with $ORIGIN because
> the pathname has to be resolved for security reasons.
> 
> Can you use bind mounts instead of symbolic links to provide the expected
> pathnames to users?

I have forwarded the question to our mutual customer.

(In reply to Florian Weimer from comment #11)
> Can you use bind mounts instead of symbolic links to provide the expected
> pathnames to users?

The mutual customer replies that bind mounts would work except the process that builds the tree does not execute as root and therefore cannot create bind mounts.  Therefore, symlinks must be used.

It appears the actual goal here is to run some Java application directly off AFS. I do not think this will work with any released JDK because of this code in Hotspot, in hotspot/src/os/linux/vm/os_linux.cpp:

bool os::dll_build_name(char* buffer, size_t buflen,
                        const char* pname, const char* fname) {
  bool retval = false;
  // Copied from libhpi
  const size_t pnamelen = pname ? strlen(pname) : 0;

  // Return error on buffer overflow.
  if (pnamelen + strlen(fname) + 10 > (size_t) buflen) {
    return retval;
  }

  if (pnamelen == 0) {
    snprintf(buffer, buflen, "lib%s.so", fname);
    retval = true;
  } else if (strchr(pname, *os::path_separator()) != NULL) {
    int n;
    char** pelements = split_path(pname, &n);
    if (pelements == NULL) {
      return false;
    }


This is called with a pname value like this:

$3 = 0x7ffff0002470 "/afs/.:mount/cell:volume/java-1.8.0-openjdk-1.8.0.242.b08-1.static.jdk.openjdkportable.x86_64/jre/
lib/amd64"

Which means that we are trying to split the AFS path at the ':' path separator at this point, which results in unusable paths. The path in pname has been subject to symlink resolution, so running the JDK through symbolic links will not avoid the ':' here.

Even if we fix *that* (and the customer is willing to run a patched JDK, not the one they have certified for their use so far), we still don't know what will happen next. Some other path initialization in the JDK could fail. Or the actual Java application that tries to run chokes on the ':' character in those paths.

Therefore, if bind mounts are not an option, the customer has to copy the JDK (and presumably the application) to local storage before running it.

Given that fixing the glibc bug will not fix the actual customer issue and that Red Hat Enterprise Linux is currently in Maintenance Support 2 phase <https://access.redhat.com/support/policy/updates/errata/>, I'm closing this as WONTFIX.

(In reply to Florian Weimer from comment #14)
> It appears the actual goal here is to run some Java application directly off
> AFS. I do not think this will work with any released JDK because of this
> code in Hotspot, ...

The customer reports that the Java applications launch successfully on RHEL 7 and not on RHEL 6.
The following output is from RHEL 7.  All identifying information has been replaced given the public setting.

  [root@host-043 hotspot]# readlink /gfs/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2
  /afs/gfs/depot_1/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2/1
  [root@host-043 hotspot]# /gfs/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2/bin/java -version
  java version "1.8.0_121"
  Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
  Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
  [root@host-043 hotspot]# fs lv /afs/gfs/depot_1/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2/1
  File /afs/gfs/depot_1/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2/1 (543151106.1.1) contained in volume 543151106
  Volume status for vid = 543151106 named v_12342342.readonly
  Current disk quota is unlimited
  Current blocks used are 194720
  The partition has 638006376 blocks available out of 830688768
  
  [root@host-043 hotspot]# ls -l /afs/gfs
  lrwxr-xr-x. 1 root root 21 Jan  1  1970 /afs/gfs -> afs-lhr.example.com
  [root@host-043 hotspot]# rm -f /gfs/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2
  [root@host-043 hotspot]# ln -s '/afs/.:mount/afs-lhr.example.com:v_12342342.readonly' /gfs/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2
  [root@host-043 hotspot]# readlink /gfs/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2
  /afs/.:mount/afs-lhr.example.com:v_12342342.readonly
  [root@host-043 hotspot]# /gfs/mw/java/64bit/oracle/hotspot/jre-1.8.0_121_b13_2/bin/java -version
  java version "1.8.0_121"
  Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
  Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

whereas the same operations fail on RHEL 6 which doesn't include the glibc fix.

However, this demonstration only shows that the JRE can be loaded.  It does not demonstrate loading from /gfs either Java classes or JNI shared libraries.

(In reply to Jeffrey Altman from comment #16)
> However, this demonstration only shows that the JRE can be loaded.  It does
> not demonstrate loading from /gfs either Java classes or JNI shared
> libraries.

As pointed out by the customer, since the PATH, LD_LIBRARY_PATH, LD_PRELOAD, and Java's -classpath argument only include the symlinks there is no colon in the path components that are parsed by the referenced hotspot code.

The glibc fix is sufficient and also addresses the security issue for AT_SECURE programs.

AuriStorFS v0.193 and beyond supports the syntax

  /afs/.@mount/<cell>/<volume>/

The prior syntax 

  /afs/.:mount/<cell>/<volume>/

is also supported for backward compatibility.

(In reply to Jeffrey Altman from comment #18)
> AuriStorFS v0.193 and beyond supports the syntax
> 
>   /afs/.@mount/<cell>/<volume>/
> 
> The prior syntax 
> 
>   /afs/.:mount/<cell>/<volume>/
> 
> is also supported for backward compatibility.

Very nice! Thanks.