Bah, I thought
# Enable the SCL for all bash scripts.
ENV BASH_ENV=/opt/app-root/etc/scl_enable \
    ENV=/opt/app-root/etc/scl_enable \
    PROMPT_COMMAND=". /opt/app-root/etc/scl_enable"
was supposed to handle all these cases.  Not sure yet why it is ignored.

Note that devtoolset-4-perftools-docker behaves the same.

(In reply to Marek Polacek from comment #2)
> Note that devtoolset-4-perftools-docker behaves the same.

Yes, that is true, and since it has its own bugzilla component, I cloned this bug, to keep track.

If I run
$ docker run -i -t 95d79b670a3a sh
then
sh-4.2$ env
lists all envvars as when running bash -c 'env'.  Perhaps some difference between (non-)interactive shell.

Another thing to consider:
$ docker run -i -t 95d79b670a3a /bin/bash -i -c env
(does not contain the envvars).

This is probably expected given the documentation here http://wiki.bash-hackers.org/scripting/bashbehaviour.

And yet another option is --posix.  At this point I'm not even sure this classifies as a bug.

I wonder how other RHSCL images make this work... So far, the only idea I got was to modify Dockerfile, and add another ENV X_SCLS=... there, which is obviously not what for example rhscl/ruby-22-rhel7 does :/ /usr/bin/python is the first binary to be run in the container, yet it has all necessary variables set, I'm not sure this happens when bash is not involved...

.qa.[root@ibm-x3630m4-01 ~]# docker run --rm rhscl/ruby-22-rhel7 python -c 'import os, pprint; pprint.pprint(dict(os.environ), width=1)'
{'CPATH': '/opt/rh/v8314/root/usr/include',
 'HOME': '/opt/app-root/src',
 'HOSTNAME': 'be4eeb0ce1b1',
 'LD_LIBRARY_PATH': '/opt/rh/v8314/root/usr/lib64:/opt/rh/nodejs010/root/usr/lib64:/opt/rh/rh-ruby22/root/usr/lib64',
 'LIBRARY_PATH': '/opt/rh/v8314/root/usr/lib64',
 'MANPATH': '/opt/rh/v8314/root/usr/share/man:/opt/rh/nodejs010/root/usr/share/man:/opt/rh/rh-ruby22/root/usr/share/man:',
 'PATH': '/opt/rh/v8314/root/usr/bin:/opt/rh/nodejs010/root/usr/bin:/opt/rh/rh-ruby22/root/usr/bin:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
 'PKGM': 'yum',
 'PKG_CONFIG_PATH': '/opt/rh/v8314/root/usr/lib64/pkgconfig:/opt/rh/rh-ruby22/root/usr/lib64/pkgconfig',
 'PWD': '/opt/app-root/src',
 'PYTHONPATH': '/opt/rh/v8314/root/usr/lib/python2.7/site-packages:/opt/rh/nodejs010/root/usr/lib/python2.7/site-packages',
 'RUBY_VERSION': '2.2',
 'SHLVL': '0',
 'STI_SCRIPTS_PATH': '/usr/libexec/s2i',
 'STI_SCRIPTS_URL': 'image:///usr/libexec/s2i',
 'XDG_DATA_DIRS': '/opt/rh/rh-ruby22/root/usr/share',
 'X_SCLS': 'v8314 nodejs010 rh-ruby22 ',
 'container': 'docker'}
.qa.[root@ibm-x3630m4-01 ~]#

Year, or e.g. nginx16-docker has this set as well.  I can see that X_SCLS is being set in scl_source but I don't see where it is executed.  ltrace nor strace doesn't reveal anything.

I'm at my wits' end, I'm afraid.  I guess we'll have to ask someone more knowledgeable in these matter than me.  Milos, can you think of someone?

I think I'm getting close, all glory to strace!

There's a /usr/bin/container-entrypoint executable in RHSCL images, and images are told to use it as an entry point, via ENTRYPOINT command in Dockerfile [1]. The container-entrypoint is very simple:

bash-4.2$ cat /usr/bin/container-entrypoint 
#!/bin/bash
exec "$@"
bash-4.2$ 

According to Dockerfile reference, "Command line arguments to docker run <image> will be appended after all elements in an exec form ENTRYPOINT, ..." which is pretty much what I observed using strace:

# docker run --rm rhscl/ruby-22-rhel7 python -c 'import os, pprint; pprint.pprint(dict(os.environ), width=1)'

9450  execve("/proc/self/exe", ["native"], ["_LIBCONTAINER_INITPIPE=3", "_LIBCONTAINER_INITTYPE=standard"]) = 0
9450  syscall_318(0xc2080d5baf, 0x1, 0x1, 0, 0, 0) = -1 (errno 38)
28370 --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
9450  execve("/usr/bin/container-entrypoint", ["container-entrypoint", "python", "-c", "import os, pprint; pprint.pprint(dict(os.environ), width=1)"], ["PATH=/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "HOSTNAME=12d692c075e9", "container=docker", "PKGM=yum", "STI_SCRIPTS_URL=image:///usr/libexec/s2i", "STI_SCRIPTS_PATH=/usr/libexec/s2i", "HOME=/opt/app-root/src", "BASH_ENV=/opt/app-root/etc/scl_enable", "ENV=/opt/app-root/etc/scl_enable", "PROMPT_COMMAND=. /opt/app-root/etc/scl_enable", "RUBY_VERSION=2.2"] <unfinished ...>

And "python -c ..." runs in a freshly created bash session, with every scl_enable executed... RHSCL images inherit this from their parent image, rhscl/s2i-base-rhel7, according to `docker inspect`.

This issue affects also the most simple usage of the DTS container I can imagine, for example using DTS gcc to compile sources on the host:

.qa.[root@ibm-x3630m4-01 ~]# ls -al /tmp/foo
total 8
drwxrwxrwx.  2 root root   16 Apr  5 03:11 .
drwxrwxrwt. 15 root root 4096 Apr  5 03:09 ..
-rw-r--r--.  1 root root   92 Apr  5 03:08 t.c
.qa.[root@ibm-x3630m4-01 ~]# cat /tmp/foo/t.c
#include <stdio.h>

int main(int argc, char **argv) { printf("Hello, world\n"); return 0; }
.qa.[root@ibm-x3630m4-01 ~]# 
.qa.[root@ibm-x3630m4-01 ~]# docker run --rm -v /tmp/foo:/tmp/foo:z rhscl/devtoolset-4-toolchain-rhel7 gcc /tmp/foo/t.c -o /tmp/foo/t
.qa.[root@ibm-x3630m4-01 ~]# 
.qa.[root@ibm-x3630m4-01 ~]# /tmp/foo/t
Hello, world
.qa.[root@ibm-x3630m4-01 ~]# 

Docker starts containerized gcc directly, and gcc runs then with the "incomplete" set of environmental variables:

10845 execve("/opt/rh/devtoolset-4/root/usr/bin//gcc", ["gcc", "/tmp/foo/t.c", "-o", "/tmp/foo/t"], ["PATH=/opt/app-root/src/bin:/opt/app-root/bin:/opt/rh/devtoolset-4/root/usr/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "HOSTNAME=18891fe89bb7", "container=docker", "PKGM=yum", "HOME=/opt/app-root/src", "BASH_ENV=/opt/app-root/etc/scl_enable", "ENV=/opt/app-root/etc/scl_enable", "PROMPT_COMMAND=. /opt/app-root/etc/scl_enable"] <unfinished ...>


I guess adding the "container-entrypoint"-like (or even the same...) into DTS images - with ENTRYPOINT ["/usr/bin/container-entrypoint"] (using the "exec" form of the directive) in Dockerfile - would work, and so far I cannot think of anything that would be broken by this, but hey, I'm no bash expert. Or docker, anyway, so it looks like a viable solution to me :)


[1] https://docs.docker.com/engine/reference/builder/#entrypoint

Thanks a lot!  That seems to do the trick.  I've built devtoolset-4-toolchain-docker-4-9.4 and I see:

$  docker run -i -t 36261601ece3 env
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
MANPATH=/opt/rh/devtoolset-4/root/usr/share/man:
HOSTNAME=13db464b736d
TERM=xterm
PERL5LIB=/opt/rh/devtoolset-4/root//usr/lib64/perl5/vendor_perl:/opt/rh/devtoolset-4/root/usr/lib/perl5:/opt/rh/devtoolset-4/root//usr/share/perl5/vendor_perl
X_SCLS=devtoolset-4 
JAVACONFDIRS=/opt/rh/devtoolset-4/root/etc/java:/etc/java
PCP_DIR=/opt/rh/devtoolset-4/root
LD_LIBRARY_PATH=/opt/rh/devtoolset-4/root/usr/lib64:/opt/rh/devtoolset-4/root/usr/lib
XDG_CONFIG_DIRS=/opt/rh/devtoolset-4/root/etc/xdg:/etc/xdg
PATH=/opt/rh/devtoolset-4/root/usr/bin:/opt/app-root/src/bin:/opt/app-root/bin:/opt/rh/devtoolset-4/root/usr/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PKGM=yum
PWD=/opt/app-root/src
SHLVL=0
HOME=/opt/app-root/src
PYTHONPATH=/opt/rh/devtoolset-4/root/usr/lib64/python2.7/site-packages:/opt/rh/devtoolset-4/root/usr/lib/python2.7/site-packages
XDG_DATA_DIRS=/opt/rh/devtoolset-4/root/usr/share:/usr/local/share:/usr/share
INFOPATH=/opt/rh/devtoolset-4/root/usr/share/info
container=docker

Also works with:
bash -i -c env
bash -c env
bash --posix -c env
sh -i -c env
sh -c env

Hopefully fixed then.  The patch was:

diff --git a/Dockerfile b/Dockerfile
index 40da41a..2b2c0d5 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -32,6 +32,13 @@ USER 1001
 
 WORKDIR ${HOME}
 
+# Use entrypoint so path is correctly adjusted already at the time the command
+# is searching, so something like docker run IMG gcc runs binary from SCL.
+ADD contrib/bin/container-entrypoint /usr/bin/container-entrypoint
+
+# Install the usage script with base image usage informations
+ADD contrib/bin/usage /usr/local/bin/usage
+
 ADD contrib/etc/scl_enable /opt/app-root/etc/scl_enable
 
 # Enable the SCL for all bash scripts.
@@ -40,4 +47,5 @@ ENV BASH_ENV=/opt/app-root/etc/scl_enable \
     PROMPT_COMMAND=". /opt/app-root/etc/scl_enable"
 
 # Set the default CMD to print the usage of the language image
-CMD /opt/app-root/bin/usage
+ENTRYPOINT ["container-entrypoint"]
+CMD ["usage"]
diff --git a/contrib/bin/container-entrypoint b/contrib/bin/container-entrypoint
new file mode 100755
index 0000000..16c17b9
--- /dev/null
+++ b/contrib/bin/container-entrypoint
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+set -eu
+cmd="$1"; shift
+exec $cmd "$@"
+

Awesome, I ran few tests, the one that revealed this issue is passing on your last image, no regressions in sanity tests, and I can easily compile C code using containerized DTS gcc. More tests will follow, however it looks good!

Verified for build rhscl/devtoolset-4-toolchain-rhel7:4-9.5.

08:45:01 ### TEST 1 - docker run + python script
08:45:06 [ PASS ] Environment is set correctly.
08:45:06 
08:45:06 ### TEST 2 - docker run + bash script (with -ti flags)
08:45:11 [ PASS ] Environment is set correctly.
08:45:11 
08:45:11 ### TEST 3 - docker run + "bash -c"
08:45:16 [ PASS ] Environment is set correctly.
08:45:16 
08:45:16 ### TEST 4 - docker run + bash (with simulated tty and with -ti flags)
08:45:20 [ PASS ] Environment is set correctly.
08:45:20 
08:45:20 ### TEST 5 - docker run + bash -l (with simulated tty and with -ti flags)
08:45:26 [ PASS ] Environment is set correctly.
08:45:26 
08:45:26 ### TEST 6 - docker exec + bash script (with -ti flags)
08:45:26 [ PASS ] Environment is set correctly.
08:45:26 
08:45:26 ### TEST 7 - docker exec + "bash -c"
08:45:27 [ PASS ] Environment is set correctly.
08:45:27 
08:45:27 ### TEST 8 - docker exec + bash (with simulated tty and with -ti flags)
08:45:27 [ PASS ] Environment is set correctly.
08:45:27 
08:45:27 ### TEST 9 - docker exec + bash -l (with simulated tty and with -ti flags)
08:45:28 [ PASS ] Environment is set correctly.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1181