Description of problem: In a swarm mode cluster using docker 1.13 in fedora atomic 26, it's impossible to create a service with a secret mounted in its containers. The service is created but the containers are crashing. Version-Release number of selected component (if applicable): docker-1.13.1-5.git5be1549.fc26.x86_64 How reproducible: 100% Steps to Reproduce: 1. Boot a F26AH with [1] 2. Login to the host and switch to the root user 3. # docker swarm init Swarm initialized: current node (yxoss27y1h5d87mrtgfij4q3d) is now a manager. To add a worker to this swarm, run the following command: docker swarm join \ --token SWMTKN-1-0sg2bj4onjvyaygcmcsftvx5z48f2r0oq6abg4izaflch9d8r8-dva3kq1zw7vrzjg8sav3e5mcs \ xxx.xxx.xxx.xxx:2377 To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. 4. # docker service ls ID NAME MODE REPLICAS IMAGE 5. # echo "This is a secret" | docker secret create my_secret_data - 2ab8kotd9e9q9wcpe387tryez 6. # docker service create --name="httpd" --secret="my_secret_data" docker.io/httpd ri4wtno7wu93yfczzb93l6s8s Actual results: # docker service ps httpd --no-trunc ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS p6jrn0kybq8vjx69avb3xdcxc httpd.1 docker.io/httpd:latest@sha256:fc9b21c3faf2e1aa4cbe91d60df40a0d30ff151d8a5f5228d77fe5e0a18fa3c2 strigazi-fa26-swarm-mode-04.cern.ch Ready Ready 2 seconds ago yx2mquqmna77u8uxo7exgxc6m \_ httpd.1 docker.io/httpd:latest@sha256:fc9b21c3faf2e1aa4cbe91d60df40a0d30ff151d8a5f5228d77fe5e0a18fa3c2 strigazi-fa26-swarm-mode-04.cern.ch Shutdown Failed 2 seconds ago "starting container failed: RemoveSecretsPath failed: remove /var/lib/docker/containers/115332b56220016b062dcdd69f128775736f86605219d3081af848d5b4ca6a15/secrets/my_secret_data: read-only file system" Expected results: The service running, something like: # docker service ps redis ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS bkna6bpn8r1a redis.1 redis:alpine ip-172-31-46-109 Running Running 8 seconds ago Additional info: Follow [2] using a F26AH like [1]. [1] https://s3.amazonaws.com/fedora-atomic-s3/Fedora-26-20170331.n.0/Fedora-Atomic-26-20170331.n.0.x86_64.qcow2 [2] https://docs.docker.com/engine/swarm/secrets/#simple-example-get-started-with-secrets
rewrote the secret patch to allow this workflow (patch set is at https://github.com/runcom/docker/tree/secrets-rewrite) Not sure how we're going with this. The commits I did aren't in the main "docker-1.13.1 tree" as I think we need full docker testing (autotest also) to run with my branches before merging my patches. By any change, can you build docker from https://github.com/runcom/docker/tree/secrets-rewrite and test this out? it's working to my testing.
PR opened https://github.com/projectatomic/docker/pull/235
I used this [1] image, upgraded to 26 and installed these packages: docker-1.13.1-9.gitdcc0a71.fc27.x86_64.rpm docker-common-1.13.1-9.gitdcc0a71.fc27.x86_64.rpm docker-rhel-push-plugin-1.13.1-9.gitdcc0a71.fc27.x86_64.rpm Now the container is running: # docker service ps redis ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS whvth13frrhv redis.1 docker.io/redis:alpine strigazi-fedora-cloud-base-25-20170411-01.cern.ch Running Running 2 minutes ago BUT I can't access the secret: # echo "This is a secret" | docker secret create my_secret_data - jq9szkiqdg1kh3rip836mry36 # docker service create --name="redis" --secret="my_secret_data" docker.io/redis:alpine vfhequwmelqg1ljzueh109jcx # docker exec $(docker ps --filter name=redis -q) ls -l /run/secrets total 0 ls: can't open '/run/secrets': Permission denied [1] https://kojipkgs.fedoraproject.org/compose/Fedora-Cloud-25-20170411.0/compose/CloudImages/x86_64/images/Fedora-Cloud-Base-25-20170411.0.x86_64.qcow2
That's caused by SELinux, I'm trying to fix the patch
Confirmed, with selinux in permissive, it works.
alright, fixed the selinux issue in the latest iteration of https://github.com/runcom/docker/commits/secrets-rewrite I'll have a scratch build for you to test!
Scratch build at https://koji.fedoraproject.org/koji/taskinfo?taskID=18935284
working, thanks
Are we getting this fix in F26 ?
(In reply to Spyros Trigazis from comment #12) > Are we getting this fix in F26 ? yeah, possibly.
updated scratch builds: rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=18952865 f26: https://koji.fedoraproject.org/koji/taskinfo?taskID=18952872
Any chance you can re-test with new rpms?
you can pull down this qcow which has the new docker installed. http://artifacts.ci.centos.org/sig-atomic/expires-7-days/20170413-scratch-docker/0bef1375-6bb2-47f6-807b-cfba18eacbaa.body.qcow2
Antonio, so this was just swarm created the directory with a bad label?
(In reply to Antonio Murdaca from comment #15) > Any chance you can re-test with new rpms? Tested both builds They work fine, thanks! docker swarm init echo "This is a secret" | docker secret create my_secret_data - docker service create --name="httpd" --secret="my_secret_data" docker.io/httpd # wait for service to start docker exec $(docker ps --filter name=httpd -q) cat /run/secrets/my_secret_data (In reply to Dusty Mabe from comment #16) > you can pull down this qcow which has the new docker installed. > http://artifacts.ci.centos.org/sig-atomic/expires-7-days/20170413-scratch- > docker/0bef1375-6bb2-47f6-807b-cfba18eacbaa.body.qcow2 Downloading. It is really slow.
(In reply to Dusty Mabe from comment #16) > you can pull down this qcow which has the new docker installed. > http://artifacts.ci.centos.org/sig-atomic/expires-7-days/20170413-scratch- > docker/0bef1375-6bb2-47f6-807b-cfba18eacbaa.body.qcow2 Works great! Thanks! SELinux enforcing :)
docker-1.13.1-7.git14cc629.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e66179a210
docker-1.13.1-7.git14cc629.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e66179a210
docker-1.13.1-7.git14cc629.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
PR still pending! https://github.com/projectatomic/docker/pull/235
did this make it into a build? can we attach it to a proper bodhi update?
docker-1.13.1-13.git51eb16e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-8a6d53a5d6
docker-latest-1.13-30.git51eb16e.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-e8d134ef02
docker-1.13.1-13.git51eb16e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-8a6d53a5d6
docker-latest-1.13-30.git51eb16e.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-e8d134ef02
docker-1.13.1-13.git51eb16e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
docker-latest-1.13-30.git51eb16e.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
hey strigazi can you test this in latest F26AH and make sure it is fixed for you? https://kojipkgs.fedoraproject.org/compose/branched/Fedora-26-20170611.n.1/compose/CloudImages/x86_64/images/
Tested, works fine! Thanks!