Bug 1376572

Summary: Need a proper way to read configuration file
Product: Red Hat Enterprise Linux 7 Reporter: Qian Cai <qcai>
Component: flannel-containerAssignee: Giuseppe Scrivano <gscrivan>
Status: CLOSED NOTABUG QA Contact: Martin Jenner <mjenner>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: atomic-bugs, avagarwa, giuseppe, jchaloup, yruseva
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Release Note
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-16 18:13:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Qian Cai 2016-09-15 18:44:15 UTC
Description of problem:
Unlike flannel rpm that it will read configuration from /etc/sysconfig/flanneld, flannel-docker hard-coded the environment variables to run the flanneld binary.

# cat /var/lib/containers/atomic/flannel/config.json
...
                "args": [
                        "/usr/bin/flanneld-run.sh"
                "env": [
                        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                        "FLANNELD_ETCD_PREFIX=/atomic.io/network",
                        "FLANNELD_ETCD_ENDPOINTS=http://127.0.0.1:2379",
                        "TERM=xterm",
                        "NAME=flannel"
                ],
...
# cat /var/lib/containers/atomic/flannel/rootfs/usr/bin/flanneld-run.sh 
#!/bin/bash

# Create flannel.conf for docker service
echo "[Service]" > /etc/systemd/system/docker.service.d/flannel.conf
echo "EnvironmentFile=-/run/flannel/docker" >> /etc/systemd/system/docker.service.d/flannel.conf

/usr/bin/flanneld-docker-env.sh &

exec /usr/bin/flanneld

# cat /var/lib/containers/atomic/flannel/rootfs/usr/bin/flanneld-docker-env.sh 
#!/bin/bash

while true
do
  [ -f /run/flannel/subnet.env ] && break
  sleep 2
done

/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker

Looks at the etcd rpm's unit file,
# cat /usr/lib/systemd/system/flanneld.service
...
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS
...

Personally, I think there are two fundamental issues. One is that a configuration file needs to be made available for users to edit like flannel rpm. The other is that flannel-docker needs to use the environment variables to start the etcd binary.

Version-Release number of selected component (if applicable):
# flanneld --version
0.5.5

How reproducible:
always

Comment 1 Giuseppe Scrivano 2016-09-16 18:13:57 UTC
system containers have a different mechanism to pass configuration and we need this mechanism instead of hardcoding configuration file paths because it 1) makes difficult have more instances, 2) will not work with user containers, that we want to support later.

You can set two values at installation time (in the example with the default values):

atomic install --system --set FLANNELD_ETCD_PREFIX=/atomic.io/network \
      --set FLANNELD_ETCD_ENDPOINTS=http://127.0.0.1:2379 flannel

I have a PR to support the etcd configuration in the same way:

https://github.com/aveshagarwal/etcd-container/pull/6


So closing it as not a bug.

Comment 2 Qian Cai 2016-09-16 18:29:29 UTC
One drawback I can think of is that users have no template of provided configuration options they can refer to anymore compared to using a configuration file.

Another one is that there seems no easy to way to show the current running configuration options apart from digging from the command line history.

Comment 3 Giuseppe Scrivano 2016-09-16 18:32:18 UTC
we just lack that part to show the settings, there are some improvements to do in this area.  Internally, this information is already stored in the /var/lib/containers/atomic/$CONTAINER_NAME/info file.

Comment 4 Qian Cai 2016-09-16 18:37:25 UTC
We may also need a proper documentation to minimize the surprise.

Comment 5 Avesh Agarwal 2016-09-16 18:40:15 UTC
Giuseppe,

The container file system for system containers with runc is very wide open on the host that every container file is accessible inside /var/lib/containers/atomic/etcd.0.

So what if root on the host modified the following file to add new env files: 

/var/lib/containers/atomic/etcd.0/config.json 

and then restarts the container with systemd?

Comment 6 Giuseppe Scrivano 2016-09-16 18:49:02 UTC
yes it is possible to modify /var/lib/containers/atomic/etcd.0/config.json and restart the service, this will work (or just modify it before starting the service).

But I think we should not suggest this, because for example this file will be deleted when the container is updated as the new image might have a different /exports/config.json.template that we want to use.

Comment 7 Avesh Agarwal 2016-09-16 18:55:32 UTC
Giuseppe,

Have you checked with Mrunal or Dan Walsh what is the recommended way of configuring containers with runc at run time?

If config.json is the way to configure containers with runc at run time, then there are 2 ways I think to handle this, for example:

1) atomic could save config.json inside /var/lib/containers/atomic/etcd.0/ like rpm does 

2) atomic could put this file like inside /etc and with a symlink inside /var/lib/containers/atomic/etcd.0/ and then save this file at every update like rpm does.

Comment 8 Avesh Agarwal 2016-09-16 18:58:30 UTC
One thing I am not sure if /var/lib/containers/atomic/etcd.0/ is writable on atomic host or not. Then only the 2nd option above would be workable I think.

Comment 9 Giuseppe Scrivano 2016-09-16 20:16:28 UTC
/var is writable in an Atomic Host.

nothing prevents us to have a configuration file for etcd under /etc/ that can be bind mount into the container (and have a different file for each instance if needed), as:

{
  "source": "/etc/etcd/${NAME}.conf",
   "destination": "/etcd/etcd.conf",
    "type": "bind",
     "options": [
      "rw",
       "rbind",
        "rprivate"
    ]
}

but I would really like that we don't do this.

The etcd image as it is now, is usable as an user container on Fedora and we should aim to make images working in both modes.

There is still much to do, but to get an idea of it, you can install bwrap-oci on Fedora 24, and try (as non privileged user):

$ atomic install --user gscrivano/etcd

and the same image will work through bwrap-oci and bubblewrap.  If we bind mount configuration files from /etc, then a regular user won't be able to do that.