Bug 1401260

Antonio do we have https://github.com/projectatomic/docker/pull/198/commits/ba713bad6c70d02adf450faacf2830d757d2f75f in this package?

Dan, we don't afaict from https://github.com/projectatomic/docker/blob/docker-1.12.3/container/container_unix.go

Has that patch ever been proposed upstream? Should we? Should we keep carrying it?

Yes this patch has been merged upstream in docker-1.13 I believe.

https://github.com/docker/docker/issues/28936

Ack, I'll backport it and rebuild Docker in Fedora. Thanks.

I think it has been backported already in upstream docker (1.12.4) 2 days ago:

https://github.com/docker/docker/pull/29050

Yup, it has, though 1.12.4 isn't released yet upstream. The 1.12.4 branch there is still missing some backports. I think we can still backport this to 1.12.3 and then update to 1.12.4 as soon as it's GA upstream.

(In reply to Antonio Murdaca from comment #7)
> Yup, it has, though 1.12.4 isn't released yet upstream. The 1.12.4 branch
> there is still missing some backports. I think we can still backport this to
> 1.12.3 and then update to 1.12.4 as soon as it's GA upstream.

Great, Looking forward to testing the new update as soon as it hits the updates-testing repository.

Thanks.

Great, I'll build and push an updated just after https://bodhi.fedoraproject.org/updates/FEDORA-2016-33e5756dfc lands into stable (in order not to delay that update anymore...)

BTW, I backported the fix to our docker-1.12.3 branch. Moving this to modified.

docker-1.12.3-13.git0423d89.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-de032c73d6

docker-1.12.3-13.git0423d89.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-de032c73d6

docker-1.12.3-15.git0423d89.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-37c2c59240

I don't know, I think the problem still persists after update.

I thought at first that it worked but it was me that I've set SELinux to permissive. The directory of the volume still gets the same context as before.

$ sudo ls -lZ /var/lib/docker/volumes/
total 236
drwxr-xr-x. 3 root root system_u:object_r:container_var_lib_t:s0   4096 Dec 10 20:16 dbstore
-rw-------. 1 root root system_u:object_r:container_var_lib_t:s0 262144 Dec 10 20:16 metadata.db

$ rpm -qa | grep docker
python3-docker-py-1.10.6-1.fc25.noarch
docker-v1.10-migrator-1.12.3-13.git0423d89.fc25.x86_64
python-dockerpty-0.4.1-3.fc25.noarch
python2-docker-pycreds-0.2.1-2.fc25.noarch
docker-1.12.3-13.git0423d89.fc25.x86_64
python3-docker-pycreds-0.2.1-2.fc25.noarch
docker-forward-journald-1.9.1-9.gitee06d03.fc23.x86_64
docker-vim-1.12.3-13.git0423d89.fc25.x86_64
python-docker-py-1.10.6-1.fc25.noarch
docker-common-1.12.3-13.git0423d89.fc25.x86_64
docker-zsh-completion-1.12.3-13.git0423d89.fc25.x86_64
python-dockerfile-parse-0.0.5-5.fc25.noarch
docker-compose-1.8.1-1.fc25.noarch
devassistant-dap-docker-0.11-3.fc24.noarch

What do you think?

Anass could you try with docker-1.12.3-15.git0423d89.fc25 (which is updating the selinux policy as well)

Oh, I didn't notice the 13 to 15 thing. I thought it was the git commit that really matters :)

I'm trying to find a way to install it now as it's clearly not yet available in the testing repository.

Downloaded the latest build from Koji and upgraded the packages:
$ koji download-build docker-1.12.3-15.git0423d89.fc25 --arch=x86_64
$ sudo dnf upgrade *.rpm
$ rpm -qa | grep -E '^(docker|container)'
docker-forward-journald-1.9.1-9.gitee06d03.fc23.x86_64
docker-1.12.3-15.git0423d89.fc25.x86_64
docker-zsh-completion-1.12.3-15.git0423d89.fc25.x86_64
container-selinux-1.12.3-15.git0423d89.fc25.x86_64
docker-common-1.12.3-15.git0423d89.fc25.x86_64
docker-compose-1.8.1-1.fc25.noarch
docker-vim-1.12.3-15.git0423d89.fc25.x86_64
docker-v1.10-migrator-1.12.3-15.git0423d89.fc25.x86_64

Still have the same issue:
$ docker run -v dbstore:/var/lib/postgresql/data -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo --name db postgres
chmod: changing permissions of ‘/var/lib/postgresql/data’: Permission denied

$ ausearch -c 'chmod' --raw
type=AVC msg=audit(1481411519.689:1730): avc:  denied  { setattr } for  pid=23813 comm="chmod" name="_data" dev="sda2" ino=1234711 scontext=system_u:system_r:container_t:s0:c102,c533 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0

# ls -lZ /var/lib/docker/volumes/
total 236
drwxr-xr-x. 3 root root system_u:object_r:container_var_lib_t:s0   4096 Dec 11 01:11 dbstore

---

Do I need to do anything after the update (I already restarted the docker after upgrade just in case, though I know it already happens in the %post stage if the service is up and running)?!!

docker-1.12.3-15.git0423d89.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-37c2c59240

Attempt to create a new container with  a new volume, the fixing of the label is only going to happen on initial container creation, I believe.

(In reply to Daniel Walsh from comment #18)
> Attempt to create a new container with  a new volume, the fixing of the
> label is only going to happen on initial container creation, I believe.

Every time I execute the command, I make sure that the container had been deleted and the named volume too (and all the dangling volumes).

I guess I don't follow.

docker-1.12.4-2.git1b5971a.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-bb5ee53c0a

docker-1.12.4-5.git1b5971a.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-2a18b9e056

docker-1.12.4-2.git1b5971a.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-bb5ee53c0a

docker-1.12.4-6.git1b5971a.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-44ed3dd527

I've just upgraded to:

container-selinux-1.12.4-6.git1b5971a.fc25.x86_64
docker-forward-journald-1.9.1-9.gitee06d03.fc23.x86_64
docker-vim-1.12.4-6.git1b5971a.fc25.x86_64
docker-common-1.12.4-6.git1b5971a.fc25.x86_64
docker-1.12.4-6.git1b5971a.fc25.x86_64
docker-v1.10-migrator-1.12.4-6.git1b5971a.fc25.x86_64
docker-compose-1.8.1-1.fc25.noarch
docker-zsh-completion-1.12.4-6.git1b5971a.fc25.x86_64

using `koji download-build` then `dnf update *.rpms`.

same issue still standing:

$ docker run -v dbstore:/var/lib/postgresql/data -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo --name db postgres
chmod: changing permissions of ‘/var/lib/postgresql/data’: Permission denied

$ sudo ausearch -c 'chmod' --raw
type=AVC msg=audit(1478201991.682:3567): avc:  denied  { setattr } for  pid=12471 comm="chmod" name="_data" dev="sda2" ino=1341114 scontext=system_u:system_r:container_t:s0:c58,c892 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1478202023.965:3587): avc:  denied  { setattr } for  pid=15251 comm="chmod" name="_data" dev="sda2" ino=1341114 scontext=system_u:system_r:container_t:s0:c58,c892 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1478202040.078:3603): avc:  denied  { setattr } for  pid=16695 comm="chmod" name="_data" dev="sda2" ino=1341114 scontext=system_u:system_r:container_t:s0:c58,c892 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1481393198.087:1261): avc:  denied  { setattr } for  pid=30179 comm="chmod" name="_data" dev="sda2" ino=1234814 scontext=system_u:system_r:container_t:s0:c423,c554 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1481393362.782:1327): avc:  denied  { setattr } for  pid=9805 comm="chmod" name="_data" dev="sda2" ino=1234711 scontext=system_u:system_r:container_t:s0:c358,c997 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1481393771.328:1450): avc:  denied  { setattr } for  pid=31288 comm="chmod" name="_data" dev="sda2" ino=1234711 scontext=system_u:system_r:container_t:s0:c118,c774 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1481411461.731:1682): avc:  denied  { setattr } for  pid=18884 comm="chmod" name="_data" dev="sda2" ino=1234711 scontext=system_u:system_r:container_t:s0:c106,c126 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1481411519.689:1730): avc:  denied  { setattr } for  pid=23813 comm="chmod" name="_data" dev="sda2" ino=1234711 scontext=system_u:system_r:container_t:s0:c102,c533 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
type=AVC msg=audit(1481465703.692:2072): avc:  denied  { setattr } for  pid=26075 comm="chmod" name="_data" dev="sda2" ino=1234928 scontext=system_u:system_r:container_t:s0:c149,c958 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=1

$ sudo ls -ldZ /var/lib/docker/volumes/dbstore
drwxr-xr-x. 3 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 14 20:33 /var/lib/docker/volumes/dbstore

$ sudo ls -lZ /var/lib/docker/volumes/dbstore
total 4
drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 14 20:33 _data


I started to doubt that I'm doing something wrong!

Can you destroy the dbstore volume and start from scratch?  The problem is the dbstore was already created with the prior container, so it does not get created, I believe.

(In reply to Daniel Walsh from comment #25)
> Can you destroy the dbstore volume and start from scratch?  The problem is
> the dbstore was already created with the prior container, so it does not get
> created, I believe.

I did forget to mention that I did after upgrade:

$ sudo systemctl restart docker
$ docker stop db # though, it wasn't running because of the issue
$ docker rm -f db
$ docker volume rm dbstore

then I executed the previous commands in comment 24.

but I think my next move will be removing the whole /var/lib/docker directory and start over again from scratch.

No that should not be necessary.

 I have reproduced, I will look into it.

(In reply to Daniel Walsh from comment #28)
>  I have reproduced, I will look into it.

Good.

Just to let you know, I removed the /var/lib/docker directory completely and still had the same issue.

Right the patch that I thought fixed this issue only effects bind mounted volumes not volumes created on the host file system.

docker-1.12.4-2.git1b5971a.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

I believe it's still open. It hasn't been solved yet.

1.12.4-2 does not fix this bug

Yeah sorry for this, some bodhi weirdness going on.

https://github.com/projectatomic/docker/pull/216
Is in projectatomic to fix this issue.

Upstream

https://github.com/docker/docker/pull/29428

I'll rebuild and update Docker in Fedora with that fix so ppl can test it out.

docker-1.12.4-6.git1b5971a.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-44ed3dd527

(In reply to Anass Ahmed from comment #29)
> (In reply to Daniel Walsh from comment #28)
> >  I have reproduced, I will look into it.
> 
> Good.
> 
> Just to let you know, I removed the /var/lib/docker directory completely and
> still had the same issue.

docker run -v dbstore:/var/lib/postgresql/data -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo --name db postgres

I cannot reproduce the error you're having though. How did you create "dbstore"? docker volume create --name dbstore? or any other command?

docker run -v dbstore:/var/lib/postgresql/data -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo --name db postgres

Auto creates the dbstore in /var/lib/docker/volumes/dbstore

No need for the volumes command

My test does

# docker volume rm dbstore; docker run -ti -v dbstore /var/data fedora ls -lZd /var/data
# docker volume rm dbstore; docker run -ti -v dbstore /var/data:z fedora ls -lZd /var/data
# docker volume rm dbstore; docker run -ti -v dbstore /var/data:Z fedora ls -lZd /var/data

First two should look like container_file_t:s0, last one should look like 
container_file_t:s0:c1,c2

(In reply to Daniel Walsh from comment #39)
> docker run -v dbstore:/var/lib/postgresql/data -e POSTGRES_USER=odoo -e
> POSTGRES_PASSWORD=odoo --name db postgres
> 
> Auto creates the dbstore in /var/lib/docker/volumes/dbstore
> 
> No need for the volumes command
> 
> My test does
> 
> # docker volume rm dbstore; docker run -ti -v dbstore /var/data fedora ls
> -lZd /var/data
> # docker volume rm dbstore; docker run -ti -v dbstore /var/data:z fedora ls
> -lZd /var/data
> # docker volume rm dbstore; docker run -ti -v dbstore /var/data:Z fedora ls
> -lZd /var/data
> 
> First two should look like container_file_t:s0, last one should look like 
> container_file_t:s0:c1,c2

testing with docker-1.12.4-2.git1b5971a.fc25.x86_64 which landed just today in stable:

$ docker run -ti -v dbstore:/var/data fedora ls -lZd /var/data
drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15 15:02 /var/data
$ docker run -ti -v dbstore:/var/data:z fedora ls -lZd /var/data
drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15 15:02 /var/data
$ docker run -ti -v dbstore:/var/data:Z fedora ls -lZd /var/data
drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15 15:03 /var/data

I cannot reproduce this failure. I'm testing with overlayfs, should I use devicemapper?

Those are all failures.  If your process tried to write to /var/data you will get permission denied.

The correct labels should look like

system_u:object_r:container_file_t:s0

And

system_u:object_r:container_file_t:s0:c1,c2

container_var_lib_t is the default label for content in /var/lib/docker.   We want to block confined containers from this type.

(In reply to Antonio Murdaca from comment #40)
> 
> $ docker run -ti -v dbstore:/var/data fedora ls -lZd /var/data
> drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15
> 15:02 /var/data
> $ docker run -ti -v dbstore:/var/data:z fedora ls -lZd /var/data
> drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15
> 15:02 /var/data
> $ docker run -ti -v dbstore:/var/data:Z fedora ls -lZd /var/data
> drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15
> 15:03 /var/data
> 
> I cannot reproduce this failure. I'm testing with overlayfs, should I use
> devicemapper?

You also need to `docker volume rm dbstore` with every create, because the first time the volume is created will take a specific context (as stated above).

Another test would be:
# docker volume rm dbstore; docker run -ti -v dbstore /var/data fedora ls -lZd /var/data && touch /var/data/test
# docker volume rm dbstore; docker run -ti -v dbstore /var/data:z fedora ls -lZd /var/data && touch /var/data/test
# docker volume rm dbstore; docker run -ti -v dbstore /var/data:Z fedora ls -lZd /var/data && touch /var/data/test

to see the issue in action.

Use --rm on each container, otherwise the 
docker volume rm dbstore 
will fail.

(In reply to Daniel Walsh from comment #41)
> Those are all failures.  If your process tried to write to /var/data you
> will get permission denied.

alright, the problem is that even with the wrong labels and selinux enforcing, my container _can_ write there (I cannot reproduce the original op issue indeed). May be related to something else though...

> 
> The correct labels should look like
> 
> system_u:object_r:container_file_t:s0
> 
> And
> 
> system_u:object_r:container_file_t:s0:c1,c2
> 
> container_var_lib_t is the default label for content in /var/lib/docker.  
> We want to block confined containers from this type.

alright, btw, I'm still getting the same labels even with your patch on projectatomic/docker:

$ docker run -ti -v dbstore:/var/data fedora ls -lZd /var/data
drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15 15:19 /var/data
$ docker run -ti -v dbstore:/var/data:z fedora ls -lZd /var/data
drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15 15:19 /var/data
$ docker run -ti -v dbstore:/var/data:Z fedora ls -lZd /var/data
drwxr-xr-x. 2 root root system_u:object_r:container_var_lib_t:s0 4096 Dec 15 15:19 /var/data

alright, I cannot reproduce either, I'm going to build docker for F25 and do an update for you guys to test.

Antonio run this command.

 docker run -ti -v dbstore:/var/data:Z fedora cat /proc/self/attr/current
To see if your container is actually running as spc_t, or container_t?

testing with docker 1.12.4-6:

$ docker volume ls
DRIVER              VOLUME NAME
local               dbstore
$ docker rm db
db
$ docker volume rm dbstore
dbstore
$ docker run -v dbstore:/var/lib/postgresql/data:Z -e POSTGRES_USER=odoo -e POSTGRES_PASSWORD=odoo --name db postgres
chmod: changing permissions of ‘/var/lib/postgresql/data’: Permission denied
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS               NAMES
7fe45f23ddd8        postgres            "/docker-entrypoint.s"   30 seconds ago      Exited (1) 28 seconds ago                       db
$ docker volume ls
DRIVER              VOLUME NAME
local               dbstore

This time, I added :Z to the volume mount. The driver for the volumes is local.

Right if the label is not changing, then this version of docker either does not have my patch or my patch is broken.

Alright, I can reproduce it now, I was missing `--selinux-enabled` in docker. Now I can see the bug and Dan's patch fixes it.

Thanks guys. Going to rebuild soon(ish).

docker-1.12.4-7.gita7cae3f.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f1fa11634

docker-1.12.4-7.gita7cae3f.fc25 fixed it for me. It works with and without :Z option at the end of the volume.

with :Z, only the current (or the recent) container can access the named volume. without :Z or with :z it becomes shared volume (can be accessed from multiple containers).

Thanks for your effort.
Waiting for the new package to hit stable to update my servers.

Please update karma.

(In reply to Daniel Walsh from comment #52)
> Please update karma.

I did soon after I tested it, but posted my comment on the wrong bug (bug 1405131) then re-posted it here.

docker-1.12.4-7.gita7cae3f.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-0f1fa11634

docker-1.12.4-7.gita7cae3f.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.