Bug 1386725

Created attachment 1212165 [details]
data dir as seen inside cnode container

Created attachment 1212166 [details]
hawkular stdout log

It looks like the cassandra and hs process are not running as the root user.  I've had it on my todo list for a while to fix the cassandra container to not need root to access the data dir, I created bz1387423 to fix this, and hopefully should be ready in the next DR.

Are you starting the containers via kubernetes, or just with regular command line?

I started the 2 containers with `docker run` but the environment is RHEL Atomic 7.  The team has been testing in RHEL7.  You're correct that HS and Cassandra container should be able to run as as non-root (or as arbitrary user per OpenShift image guidelines) unless we have a good reason to run as root.

When starting with 'docker run' did you use the --user option, or does Atomic have some config which automatically uses a different user?  The current Cassandra image should be able to run as non-root, but you have to set the volume (/opt/apache-cassandra/data) permissions outside the container.

Something like this:

chown myuser:myuser /tmp/mydatadir
docker run -d -v /tmp/mydatadir:/opt/apache-cassandra/data:Z --user $(id -u myuser) jboss/casssandra

I didn't use --user option.  I can confirm that hawkular-casssandra container runs fine if I specify the -v option

I have changed the cassandra and hawkular containers in bz1387423 so that neither of them runs as root by default.  I'll update the quickstart doc with some info related to the volume configuration.

Still failing if -v flag is omitted.

# docker run -d --name hawkular-cassandra --user root -e MAX_HEAPSIZE=1024M -e CASSANDRA_START_RPC=true brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss/cassandra:latest

Output:

# docker logs hawkular-cassandra
WARNING: Container started without specifiying user,  this configuration is not supported for production
chown: changing ownership of '/opt/apache-cassandra/data': Permission denied

# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.1

This is probably due to selinux policy on the host filesystem.
Can you try running the container with the selinux label turned off?
# docker run --security-opt label:disable ...

If this is the cause of the permissions issue, then we'll have to decide whether we want to support running the container as root, and how it should be configured.  Currently when you run the container as root, it tries to create a VOLUME in a default location.  We could remove this VOLUME config from the Dockerfile, but this could hurt disk I/O performance and will cause the data to be lost when the container is destroyed.  Another option is to just require the user to configure a volume and run the container with a specific userid.

Some additional info here:
http://www.projectatomic.io/blog/2016/03/dwalsh_selinux_containers/
http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/

Also, if you run "docker inspect" on the container, what is the location where the volume is being created?  It should be visible under "Mounts:, Source:".

Cassandra container can start after disabling SELinux for Docker.

Important discovery: I was on RHEL Atomic 7.1 where Docker is older

# docker version
Client version: 1.4.1-dev
Client API version: 1.17
Go version (client): go1.3.1
Git commit (client): d26b358/1.4.1
OS/Arch (client): linux/amd64
Server version: 1.4.1-dev
Server API version: 1.17
Go version (server): go1.3.1
Git commit (server): d26b358/1.4.1


Switched to 7.2 the startup problem went away without the need to disable SELinux

-bash-4.2# cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.2
-bash-4.2# docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Package Version: docker-1.8.2-8.el7.x86_64
 Go version:   go1.4.2
 Git commit:   a01dc02/1.8.2
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Package Version: 
 Go version:   go1.4.2
 Git commit:   a01dc02/1.8.2
 Built:        
 OS/Arch:      linux/amd64

Great!  Is there anything left to do on this issue?  Sounds like we just need to document the different options for the user.

Perhaps state the minimum Docker version requirement in your Getting Started page?

Ok, I added a note to the quickstart wiki page to use RHEL 7.2 or Atomic Host 7.2 and Docker 1.8 or higher.

Created attachment 1217066 [details]
Getting started page