Bug 1414880

(In reply to Michael Nguyen from comment #0)
> Description of problem:
> Unable to initialize docker swarm in RHEL Atomic Host 7.3.2 (although it
> could technically be run against earlier versions swapping over to
> docker-latest).  I was following the directions in this KB article.
> 
> https://access.redhat.com/solutions/2740471
> 
> Version-Release number of selected component (if applicable):
> docker-1.12.5-14.el7.x86_64
> 
> How reproducible:
> Always
> 
> Steps to Reproduce:
> On RHEL Atomic Host 7.3.2, run:
> 
> docker swarm init --advertised-addr 127.0.0.1
> 
> Actual results:
> Error response from daemon: --live-restore daemon configuration is
> incompatible with swarm mode

seems pretty clear that there's a "--live-restore" flag set in /etc/sysconfig/docker which is incompatible with swarm mode. Could you pull it off and re-try?

> 
> Expected results:
> Swarm initialized
> 
> Additional info:
> # rpm -qa docker
> docker-1.12.5-14.el7.x86_64
> # docker info
> Containers: 0
>  Running: 0
>  Paused: 0
>  Stopped: 0
> Images: 0
> Server Version: 1.12.5
> Storage Driver: devicemapper
>  Pool Name: atomicos-docker--pool
>  Pool Blocksize: 524.3 kB
>  Base Device Size: 10.74 GB
>  Backing Filesystem: xfs
>  Data file: 
>  Metadata file: 
>  Data Space Used: 20.45 MB
>  Data Space Total: 7.151 GB
>  Data Space Available: 7.131 GB
>  Metadata Space Used: 40.96 kB
>  Metadata Space Total: 25.17 MB
>  Metadata Space Available: 25.12 MB
>  Thin Pool Minimum Free Space: 715.1 MB
>  Udev Sync Supported: true
>  Deferred Removal Enabled: true
>  Deferred Deletion Enabled: false
>  Deferred Deleted Device Count: 0
>  Library Version: 1.02.135-RHEL7 (2016-11-16)
> Logging Driver: journald
> Cgroup Driver: systemd
> Plugins:
>  Volume: local
>  Network: null host bridge overlay
>  Authorization: rhel-push-plugin
> Swarm: inactive
> Runtimes: docker-runc runc
> Default Runtime: docker-runc
> Security Options: seccomp selinux
> Kernel Version: 3.10.0-514.6.1.el7.x86_64
> Operating System: Red Hat Enterprise Linux Atomic Host 7.3
> OSType: linux
> Architecture: x86_64
> Number of Docker Hooks: 2
> CPUs: 2
> Total Memory: 1.796 GiB
> Name: rhelah.localdomain
> ID: OVEA:QEUX:HNBL:FHHM:DNGF:VLE4:I2UG:J5SH:34PB:HYNV:URQX:5XK5
> Docker Root Dir: /var/lib/docker
> Debug Mode (client): false
> Debug Mode (server): false
> Registry: https://registry.access.redhat.com/v1/
> Insecure Registries:
>  127.0.0.0/8
> Registries: registry.access.redhat.com (secure), docker.io (secure)
> # docker swarm init --advertise-addr 127.0.0.1
> Error response from daemon: --live-restore daemon configuration is
> incompatible with swarm mode
> # systemctl status docker -l
> ● docker.service - Docker Application Container Engine
>    Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor
> preset: disabled)
>   Drop-In: /usr/lib/systemd/system/docker.service.d
>            └─flannel.conf
>    Active: active (running) since Thu 2017-01-19 16:16:45 UTC; 4min 15s ago
>      Docs: http://docs.docker.com
>  Main PID: 4413 (dockerd-current)
>    Memory: 54.1M
>    CGroup: /system.slice/docker.service
>            ├─4413 /usr/bin/dockerd-current --add-runtime
> docker-runc=/usr/libexec/docker/docker-runc-current
> --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin
> --exec-opt native.cgroupdriver=systemd
> --userland-proxy-path=/usr/libexec/docker/docker-proxy-current
> --selinux-enabled --log-driver=journald --signature-verification=false
> --storage-driver devicemapper --storage-opt dm.fs=xfs --storage-opt
> dm.thinpooldev=/dev/mapper/atomicos-docker--pool --storage-opt
> dm.use_deferred_removal=true --add-registry registry.access.redhat.com
>            └─4520 /usr/bin/docker-containerd-current -l
> unix:///var/run/docker/libcontainerd/docker-containerd.sock --shim
> docker-containerd-shim --metrics-interval=0 --start-timeout 2m --state-dir
> /var/run/docker/libcontainerd/containerd --runtime docker-runc
> --runtime-args --systemd-cgroup=true
> 
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.478145464Z" level=info msg="API listen on
> /var/run/docker.sock"
> Jan 19 16:16:45 rhelah.localdomain systemd[1]: Started Docker Application
> Container Engine.
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.340699161Z" level=info msg="{Action=init,
> Username=cloud-user, LoginUID=1000, PID=11238}"
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.351436323Z" level=error msg="Error initializing
> swarm: --live-restore daemon configuration is incompatible with swarm mode"
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.351461000Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.351470566Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.307264075Z" level=info msg="{Action=init,
> Username=cloud-user, LoginUID=1000, PID=12000}"
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.308206582Z" level=error msg="Error initializing
> swarm: --live-restore daemon configuration is incompatible with swarm mode"
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.308228403Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.308236386Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> # journalctl -b -r -u docker --no-pager
> -- Logs begin at Thu 2017-01-19 16:16:36 UTC, end at Thu 2017-01-19 16:19:59
> UTC. --
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.308236386Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.308228403Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.308206582Z" level=error msg="Error initializing
> swarm: --live-restore daemon configuration is incompatible with swarm mode"
> Jan 19 16:19:02 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:19:02.307264075Z" level=info msg="{Action=init,
> Username=cloud-user, LoginUID=1000, PID=12000}"
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.351470566Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.351461000Z" level=error msg="Handler for POST
> /v1.24/swarm/init returned error: --live-restore daemon configuration is
> incompatible with swarm mode"
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.351436323Z" level=error msg="Error initializing
> swarm: --live-restore daemon configuration is incompatible with swarm mode"
> Jan 19 16:17:03 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:17:03.340699161Z" level=info msg="{Action=init,
> Username=cloud-user, LoginUID=1000, PID=11238}"
> Jan 19 16:16:45 rhelah.localdomain systemd[1]: Started Docker Application
> Container Engine.
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.478145464Z" level=info msg="API listen on
> /var/run/docker.sock"
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.473236198Z" level=info msg="Docker daemon"
> commit="047e51b/1.12.5" graphdriver=devicemapper version=1.12.5
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.473221740Z" level=info msg="Daemon has completed
> initialization"
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.473106641Z" level=info msg="Loading containers:
> done."
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.339160739Z" level=info msg="Default bridge
> (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip
> can be used to set a preferred IP address"
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.160841708Z" level=info msg="Firewalld running:
> false"
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.126877478Z" level=info msg="Loading containers:
> start."
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.126140685Z" level=info msg="Graph migration to
> content-addressability took 0.00 seconds"
> Jan 19 16:16:45 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:45.037040805Z" level=info msg="devmapper:
> Successfully created filesystem xfs on device docker-253:0-6304485-base"
> Jan 19 16:16:44 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:44.874159813Z" level=info msg="devmapper: Creating
> filesystem xfs on device docker-253:0-6304485-base"
> Jan 19 16:16:43 rhelah.localdomain dockerd-current[4413]:
> time="2017-01-19T16:16:43.798628652Z" level=info msg="libcontainerd: new
> containerd process, pid: 4520"
> Jan 19 16:16:43 rhelah.localdomain systemd[1]: Starting Docker Application
> Container Engine...

Just about to say the same thing, but Antonio beet me to it.

I don't see the flag in my /etc/sysconfig/docker

# cat /etc/sysconfig/docker
# /etc/sysconfig/docker

# Modify these options if you want to change the way the docker daemon runs
OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
if [ -z "${DOCKER_CERT_PATH}" ]; then
    DOCKER_CERT_PATH=/etc/docker
fi

# If you want to add your own registry to be used for docker search and docker
# pull use the ADD_REGISTRY option to list a set of registries, each prepended
# with --add-registry flag. The first registry added will be the first registry
# searched.
ADD_REGISTRY='--add-registry registry.access.redhat.com'

# If you want to block registries from being used, uncomment the BLOCK_REGISTRY
# option and give it a set of registries, each prepended with --block-registry
# flag. For example adding docker.io will stop users from downloading images
# from docker.io
# BLOCK_REGISTRY='--block-registry'

# If you have a registry secured with https but do not have proper certs
# distributed, you can tell docker to not look for full authorization by
# adding the registry to the INSECURE_REGISTRY line and uncommenting it.
# INSECURE_REGISTRY='--insecure-registry'

# On an SELinux system, if you remove the --selinux-enabled option, you
# also need to turn on the docker_transition_unconfined boolean.
# setsebool -P docker_transition_unconfined 1

# Location used for temporary files, such as those created by
# docker load and build operations. Default is /var/lib/docker/tmp
# Can be overriden by setting the following environment variable.
# DOCKER_TMPDIR=/var/tmp

# Controls the /etc/cron.daily/docker-logrotate cron job status.
# To disable, uncomment the line below.
# LOGROTATE=false
#

# docker-latest daemon can be used by starting the docker-latest unitfile.
# To use docker-latest client, uncomment below lines
#DOCKERBINARY=/usr/bin/docker-latest
#DOCKER_CONTAINERD_BINARY=/usr/bin/docker-containerd-latest
#DOCKER_CONTAINERD_SHIM_BINARY=/usr/bin/docker-containerd-shim-latest

I found the live-restore in /etc/docker/daemon.json.  When I set it to false and restart the daemon I am able to initialize docker swarm. 

The host I am running on has a fresh boot of the released RHEL Atomic Host 7.3.2 qcow and this is the default configuration.


# cat /etc/docker/daemon.json 
{
    "live-restore": true
}

# vi /etc/docker/daemon.json 
# sudo systemctl restart docker
# docker swarm init --advertise-addr 127.0.0.1
Swarm initialized: current node (0r77yz7mkwyzbwsrjbcxqoxo3) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token SWMTKN-1-0jtcn92153o8u6eqlilw2b22irnl2ck9mx4m8jadcy72h77u50-15dncpytiepukuvsujlt96o4g \
    127.0.0.1:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

(In reply to Michael Nguyen from comment #4)
> I found the live-restore in /etc/docker/daemon.json.  When I set it to false
> and restart the daemon I am able to initialize docker swarm. 

Not sure why that flag has been added in RHELAH honestly

rpm -qf /etc/docker/daemon.json

# rpm -qf /etc/docker/daemon.json
docker-common-1.12.5-14.el7.x86_64

In the docker official docs page ,it points out that:The live restore option is not compatible with Docker Engine swarm mode.
see:
https://docs.docker.com/engine/admin/live-restore/#live-restore-upon-restart

So you must disable live-restore for this to work.  BTW We should not have live-restore enabled on RHEL, this is a bug and should be fixed in our next release.