Bug 1256299

Both bugs seem to have the same root cause, can you verify that it still happens if you include this in your docker file:

 RUN touch /var/lib/yum/rpmdb-indexes/*

Verified, still happens when I include touch /var/lib/yum/rpmdb-indexes/. I removed all cached docker images for good measure.

# docker build -t pyther/baseimg:7.0 baseimg/
Sending build context to Docker daemon 8.192 kB
Sending build context to Docker daemon 
Step 0 : FROM centos:7
 ---> 7322fbe74aa5
Step 1 : MAINTAINER matthew
 ---> Running in fa5ca6cc7b68
 ---> b2c5918fccd6
Removing intermediate container fa5ca6cc7b68
Step 2 : RUN touch /var/lib/rpm/*
 ---> Running in 61e8f48b21a7
 ---> 8966e2a4c98a
Removing intermediate container 61e8f48b21a7
Step 3 : RUN touch /var/lib/yum/rpmdb-indexes/*
 ---> Running in ffdb48b4834c
 ---> 4b2e3b594ee7
Removing intermediate container ffdb48b4834c
Step 4 : RUN sed -i '/\[main\]/ a proxy=http://skynet.rnd.lan:3128' /etc/yum.conf
 ---> Running in 618be562822e
 ---> c9083c4ea10a
Removing intermediate container 618be562822e
Step 5 : ADD CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo
 ---> c038ba7385f8
Removing intermediate container 219246a98d6d
Step 6 : RUN yum clean all
 ---> Running in b1987db0cd1e
Loaded plugins: fastestmirror
Cleaning repos: base extras updates
Cleaning up everything
lstat /var/lib/docker/overlay/b1987db0cd1e2ed4d6f554321d2136b3c761881fd59815ab390cab0d921ee026/merged/var/lib/yum/rpmdb-indexes/conflicts: no such file or directory

Thanks, I'm going to replicate & investigate this now.

As a side note, running `yum clean all` as a single instruction doesn't possibly do what you want it to do:

   1) The files will still remain in the lower layer
   2) New extra layer with *only* whiteouts is created

So in the end the resulting image is actually larger.

The best way to run `yum clean all` is as a part of another yum command which actually makes use of the caches. E.g:

  RUN yum install -y some pkgs\
   && yum builddep -y pkg\
   && yum clean all

Docker essentially diffs the file system before/after executing each instruction, so by cleaning the cache as part of the same instruction that created it the files will never be seen.

Are you seeing this problem

https://github.com/shaded-enmity/yum-overlayfs

Dan, I'm not entirely sure what problem you are referring to.

I tried the plugin you linked to. This does not solve the problem. It is important to note I did have to patch the plugin. In the CentOS7 container, 'overlay / overlay' is the second line in mtab, not the first. The should_touch() function expects 'overlay / overlay' to be the first line in /etc/mtab.

# diff -rNup baseimg/overlayfs/overlayfs.py.orig baseimg/overlayfs/overlayfs.py
--- baseimg/overlayfs/overlayfs.py.orig	2015-08-24 12:24:00.067498108 -0400
+++ baseimg/overlayfs/overlayfs.py	2015-08-24 12:39:10.807999806 -0400
@@ -65,8 +65,9 @@ def should_touch():
         if not path.exists(mtab):
                 return False
         with open(mtab, 'r') as f:
-                line = f.readline()
-                return line.startswith('overlay / overlay')
+                for line in f.readlines():
+			if line.startswith('overlay / overlay'):
+				return True
         return False
 
 def prereposetup_hook(conduit):


Below you can see the patched plugin running, "ovl: Copying up (14) files from OverlayFS lower layer" and the lstat error.



# docker build -t pyther/baseimg:7.0 baseimg/
Sending build context to Docker daemon 16.38 kB
Sending build context to Docker daemon 
Step 0 : FROM centos:7
 ---> 7322fbe74aa5
Step 1 : MAINTAINER matthew
 ---> Using cache
 ---> b2c5918fccd6
Step 2 : ADD overlayfs/overlayfs.conf /etc/yum/pluginconf.d/overlayfs.conf
 ---> Using cache
 ---> 7d559fce281a
Step 3 : ADD overlayfs/overlayfs.py /usr/lib/yum-plugins/overlayfs.py
 ---> 2144032f1a54
Removing intermediate container bfb7e72edd80
Step 4 : RUN touch /var/lib/rpm/*
 ---> Running in 51da09c0ed86
 ---> 01d1f71c69f8
Removing intermediate container 51da09c0ed86
Step 5 : RUN touch /var/lib/yum/rpmdb-indexes/*
 ---> Running in ac782d3313cc
 ---> b31d17df80d7
Removing intermediate container ac782d3313cc
Step 6 : RUN sed -i '/\[main\]/ a proxy=http://skynet.rnd.lan:3128' /etc/yum.conf
 ---> Running in b059a9cc6800
 ---> e80d48ae0160
Removing intermediate container b059a9cc6800
Step 7 : ADD CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo
 ---> 2a5062c08f30
Removing intermediate container 98985bab424e
Step 8 : RUN yum -v -d3 -y install PyYAML  && yum clean all
 ---> Running in 1e5f61e7471a
Loading "fastestmirror" plugin
Loading "overlayfs" plugin
Config time: 0.008
Yum version: 3.4.3
rpmdb time: 0.000
Setting up Package Sacks
ovl: Copying up (14) files from OverlayFS lower layer
Determining fastest mirrors
 * base: www.gtlib.gatech.edu
 * extras: www.gtlib.gatech.edu
 * updates: www.gtlib.gatech.edu
pkgsack time: 8.044
Obs Init time: 0.054
Resolving Dependencies
--> Running transaction check
---> Package PyYAML.x86_64 0:3.10-11.el7 will be installed
Checking deps for PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('python(abi)', 'EQ', ('0', '2.7', None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('rtld(GNU_HASH)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libc.so.6(GLIBC_2.14)(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libyaml-0.so.2()(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libpython2.7.so.1.0()(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libpthread.so.0()(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
PyYAML-3.10-11.el7.x86_64 requires: libyaml-0.so.2()(64bit)
--> Processing Dependency: libyaml-0.so.2()(64bit) for package: PyYAML-3.10-11.el7.x86_64
Searching pkgSack for dep: libyaml-0.so.2()(64bit)
Building updates object
up:simple updates time: 0.002
up:obs time: 0.002
up:condense time: 0.000
updates time: 0.057
TSINFO: Marking libyaml-0.1.4-11.el7_0.x86_64 as install for PyYAML-3.10-11.el7.x86_64
--> Running transaction check
---> Package libyaml.x86_64 0:0.1.4-11.el7_0 will be installed
Checking deps for libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('rtld(GNU_HASH)', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('libc.so.6(GLIBC_2.14)(64bit)', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('/sbin/ldconfig', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('/sbin/ldconfig', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
--> Finished Dependency Resolution
Dependency Process ending
Depsolve time: 0.093

Dependencies Resolved

================================================================================
 Package          Arch            Version                   Repository     Size
================================================================================
Installing:
 PyYAML           x86_64          3.10-11.el7               base          153 k
Installing for dependencies:
 libyaml          x86_64          0.1.4-11.el7_0            base           55 k

Transaction Summary
================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 208 k
Installed size: 760 k
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/libyaml-0.1.4-11.el7_0.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for libyaml-0.1.4-11.el7_0.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              570 kB/s | 208 kB  00:00     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-1.1503.el7.centos.2.8.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Member: libyaml.x86_64 0:0.1.4-11.el7_0 - u
Adding Package libyaml-0.1.4-11.el7_0.x86_64 in mode u
Member: PyYAML.x86_64 0:3.10-11.el7 - u
Adding Package PyYAML-3.10-11.el7.x86_64 in mode u
Running transaction check
Transaction check time: 0.006
Running transaction test
Transaction test succeeded
Transaction test time: 0.005
Running transaction
  Installing : libyaml-0.1.4-11.el7_0.x86_64                                1/2 
  Installing : PyYAML-3.10-11.el7.x86_64                                    2/2 
  Verifying  : libyaml-0.1.4-11.el7_0.x86_64                                1/2 
  Verifying  : PyYAML-3.10-11.el7.x86_64                                    2/2 
VerifyTransaction time: 0.322
Transaction time: 1.123

Installed:
  PyYAML.x86_64 0:3.10-11.el7                                                   

Dependency Installed:
  libyaml.x86_64 0:0.1.4-11.el7_0                                               

Complete!
Loaded plugins: fastestmirror, overlayfs
Cleaning repos: base extras updates
Cleaning up everything
Cleaning up list of fastest mirrors
lstat /var/lib/docker/overlay/1e5f61e7471a50af489cb5122e1c53a596ac0ede94e1f017c905a86e55aca66d/merged/var/lib/yum/rpmdb-indexes/conflicts: no such file or directory

I was able to reproduce this on a CentOS 7 host, it works fine under Fedora though. I'll investigate and report back tomorrow.

Thanks for the bit about the plugin, I'll update the sources to search entire mtab file (which I was hesitant to do due to possible false positives).

This does not seem to be related to YUM, after some peeking I figured that the message from `lstat` isn't actually coming from the container, but rather from Docker runtime crashing hard. Notice how the output is actually truncated, since if you inject a faulty command you'll get this:

   The command `/bin/sh -c false` returned a non-zero code: 1

Note that this error manifests only during build, if you try:

   docker run centos:7 yum clean all

It should work just fine.

Can you also provide us with the output of `docker info` ?

Your observation that this only happens during build has been my experience as well.

# docker run centos:7 yum clean all; echo $?
Loaded plugins: fastestmirror
Cleaning repos: base extras updates
Cleaning up everything
0

# docker run centos:7 yum -y -q install PyYAML; echo $?
warning: /var/cache/yum/x86_64/7/base/packages/libyaml-0.1.4-11.el7_0.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for libyaml-0.1.4-11.el7_0.x86_64.rpm is not installed
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-1.1503.el7.centos.2.8.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
0


# docker info
Containers: 27
Images: 44
Storage Driver: overlay
 Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.11.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 11.38 GiB
Name: nas.rnd.lan
ID: 6R5W:IPYH:U4IQ:XQCK:EY2T:RAUS:JBGG:ZED3:XXRS:AGNT:73IZ:WL4K

I've tried installing Docker 1.8.1 from CBS and there it works ok.
http://cbs.centos.org/koji/buildinfo?buildID=3706

I'd suggest waiting for 1.8.1 to hit CentOS base as the 1.7 release was particularly problematic.

Fixed in docker-1.8.1

Using this build http://cbs.centos.org/koji/buildinfo?buildID=3714 which resolves a dependency issue for docker-selinux I get further, but the docker build still fails.

ApplyLayer exit status 1 stdout:  stderr: link / /usr/lib64/python2.7/site-packages/yaml/__init__.pyc: operation not permitted

I set selinux to permissive to ensure it isn't getting in the way.

# cat /sys/fs/selinux/enforce 
0

# uname -a
Linux nas.rnd.lan 3.10.0-229.11.1.el7.x86_64 #1 SMP Thu Aug 6 01:06:18 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# docker info
Containers: 3
Images: 10
Storage Driver: overlay
 Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.11.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 4
Total Memory: 11.38 GiB
Name: nas.rnd.lan
ID: 6R5W:IPYH:U4IQ:XQCK:EY2T:RAUS:JBGG:ZED3:XXRS:AGNT:73IZ:WL4K
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

# rpm -qa | grep docker
docker-1.8.1-4.git32b8b25.el7.x86_64
docker-selinux-1.8.1-4.git32b8b25.el7.x86_64

# ps -ef | grep docker
root     14065     1  1 16:41 ?        00:00:14 /usr/bin/docker -d --selinux-enabled=false --storage-driver overlay



# docker build -t pyther/baseimg:7.0 baseimg/
Sending build context to Docker daemon 16.38 kB
Step 0 : FROM centos:7
 ---> 7322fbe74aa5
Step 1 : MAINTAINER matthew
 ---> Running in e6f414164e0f
 ---> 27fded0d45e9
Removing intermediate container e6f414164e0f
Step 2 : ADD overlayfs/overlayfs.conf /etc/yum/pluginconf.d/overlayfs.conf
 ---> 43b11499aa73
Removing intermediate container a484a2ac71c7
Step 3 : ADD overlayfs/overlayfs.py /usr/lib/yum-plugins/overlayfs.py
 ---> 4a52f6b844bc
Removing intermediate container 6278873fde16
Step 4 : RUN touch /var/lib/rpm/*
 ---> Running in f2ae710fee01
 ---> a1583fa50164
Removing intermediate container f2ae710fee01
Step 5 : RUN touch /var/lib/yum/rpmdb-indexes/*
 ---> Running in ca4db9343289
 ---> 145ff9d1418f
Removing intermediate container ca4db9343289
Step 6 : RUN sed -i '/\[main\]/ a proxy=http://skynet.rnd.lan:3128' /etc/yum.conf
 ---> Running in d07eaff08c52
 ---> b74dafb0e285
Removing intermediate container d07eaff08c52
Step 7 : ADD CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo
 ---> 771848ceaae4
Removing intermediate container 86a2a202687e
Step 8 : RUN yum -v -d3 -y install PyYAML  && yum clean all
 ---> Running in 9e051be0852e
Loading "fastestmirror" plugin
Loading "overlayfs" plugin
Config time: 0.007
Yum version: 3.4.3
rpmdb time: 0.000
Setting up Package Sacks# cat /sys/fs/selinux/enforce 
0
ovl: Copying up (14) files from OverlayFS lower layer
Determining fastest mirrors
 * base: mirrors.kernel.org
 * extras: mirrors.kernel.org
 * updates: mirrors.kernel.org
pkgsack time: 9.984
Obs Init time: 0.054
Resolving Dependencies
--> Running transaction check
---> Package PyYAML.x86_64 0:3.10-11.el7 will be installed
Checking deps for PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('python(abi)', 'EQ', ('0', '2.7', None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('rtld(GNU_HASH)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libc.so.6(GLIBC_2.14)(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libyaml-0.so.2()(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libpython2.7.so.1.0()(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
looking for ('libpthread.so.0()(64bit)', None, (None, None, None)) as a requirement of PyYAML.x86_64 0:3.10-11.el7 - u
PyYAML-3.10-11.el7.x86_64 requires: libyaml-0.so.2()(64bit)
--> Processing Dependency: libyaml-0.so.2()(64bit) for package: PyYAML-3.10-11.el7.x86_64
Searching pkgSack for dep: libyaml-0.so.2()(64bit)
Building updates object
up:simple updates time: 0.002
up:obs time: 0.002
up:condense time: 0.000
updates time: 0.059
TSINFO: Marking libyaml-0.1.4-11.el7_0.x86_64 as install for PyYAML-3.10-11.el7.x86_64
--> Running transaction check
---> Package libyaml.x86_64 0:0.1.4-11.el7_0 will be installed
Checking deps for libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('rtld(GNU_HASH)', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('libc.so.6(GLIBC_2.14)(64bit)', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('/sbin/ldconfig', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
looking for ('/sbin/ldconfig', None, (None, None, None)) as a requirement of libyaml.x86_64 0:0.1.4-11.el7_0 - u
--> Finished Dependency Resolution
Dependency Process ending
Depsolve time: 0.096

Dependencies Resolved

================================================================================
 Package          Arch            Version                   Repository     Size
================================================================================
Installing:
 PyYAML           x86_64          3.10-11.el7               base          153 k
Installing for dependencies:
 libyaml          x86_64          0.1.4-11.el7_0            base           55 k

Transaction Summary
================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 208 k
Installed size: 760 k
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/libyaml-0.1.4-11.el7_0.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for libyaml-0.1.4-11.el7_0.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              562 kB/s | 208 kB  00:00     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-1.1503.el7.centos.2.8.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Member: libyaml.x86_64 0:0.1.4-11.el7_0 - u
Adding Package libyaml-0.1.4-11.el7_0.x86_64 in mode u
Member: PyYAML.x86_64 0:3.10-11.el7 - u
Adding Package PyYAML-3.10-11.el7.x86_64 in mode u
Running transaction check
Transaction check time: 0.006
Running transaction test
Transaction test succeeded
Transaction test time: 0.005
Running transaction
  Installing : libyaml-0.1.4-11.el7_0.x86_64                                1/2 
  Installing : PyYAML-3.10-11.el7.x86_64                                    2/2 
  Verifying  : libyaml-0.1.4-11.el7_0.x86_64                                1/2 
  Verifying  : PyYAML-3.10-11.el7.x86_64                                    2/2 
VerifyTransaction time: 0.285
Transaction time: 1.651

Installed:
  PyYAML.x86_64 0:3.10-11.el7                                                   

Dependency Installed:
  libyaml.x86_64 0:0.1.4-11.el7_0                                               

Complete!
Loaded plugins: fastestmirror, overlayfs
Cleaning repos: base extras updates
Cleaning up everything
Cleaning up list of fastest mirrors
ApplyLayer exit status 1 stdout:  stderr: link / /usr/lib64/python2.7/site-packages/yaml/__init__.pyc: operation not permitted

Could be something being blocked by a missing capability.  Try to do the same yum command within a container.  If it blows up the same way then try to do it using docker run --privileged.  

If this works then you are being blocked by some other security mechanism other then SELinux.

I would figure capabilities.  But not sure which. 

sys_admin would be a likely candidate.

I should have mentioned the yum install works fine within the container.

# docker run centos:7 yum -y -q install PyYAML; echo $?
warning: /var/cache/yum/x86_64/7/base/packages/libyaml-0.1.4-11.el7_0.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for libyaml-0.1.4-11.el7_0.x86_64.rpm is not installed
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-1.1503.el7.centos.2.8.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
0

I have no idea.  Does dmesg say anything when the docker build fails.

Must be some problem in Overlayfs,  

docker build and docker run should run with the same lockdown of permissions.

This is another hard crash within Docker runtime (the filesystem diff engine, UnpackLayer).
I'm going to try different backing FS than xfs.

I wonder *why* this manifests only during `docker build`, making the debugging particularly painful.

Thinking about it, it doesn't happen during `docker run` because the layer is not committed. If we commit the layer via `docker commit`, it blows:

 $ docker run -d centos:7 bash -c "yum install -y PyYAML ; bash"
 d7e22192a8f2a461beacca47de675be68b8cca597e245b2bb697c425ea4bf637

 $ docker commit d7e22192a8f2a461beacca47de675be68b8cca597e245b2bb697c425ea4bf637 test:test
 Error response from daemon: ApplyLayer exit status 1 stdout:  stderr: link / /usr/lib64/python2.7/site-packages/yaml/__init__.pyc: operation not permitted

Confirmed #16.

This *does not* happen if the backing filesystem is ext4:

# docker info
Containers: 0
Images: 4
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 1.797 GiB
Name: localhost.localdomain
ID: JDT4:MPHY:HRFV:VOCD:HRRP:VYRY:LX2J:5TZD:E33Y:NBI5:NAMQ:D3TM

# docker build -t a .
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon 
Step 0 : FROM centos:7
 ---> 7322fbe74aa5
Step 1 : RUN yum install -y PyYAML && yum clean all
 ---> Running in 2f91117135ab
Loaded plugins: fastestmirror, overlayfs
Determining fastest mirrors
 * base: mirror.karneval.cz
 * extras: mirror.karneval.cz
 * updates: mirror.karneval.cz
Resolving Dependencies
--> Running transaction check
---> Package PyYAML.x86_64 0:3.10-11.el7 will be installed
--> Processing Dependency: libyaml-0.so.2()(64bit) for package: PyYAML-3.10-11.el7.x86_64
--> Running transaction check
---> Package libyaml.x86_64 0:0.1.4-11.el7_0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package          Arch            Version                   Repository     Size
================================================================================
Installing:
 PyYAML           x86_64          3.10-11.el7               base          153 k
Installing for dependencies:
 libyaml          x86_64          0.1.4-11.el7_0            base           55 k

Transaction Summary
================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 208 k
Installed size: 760 k
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/libyaml-0.1.4-11.el7_0.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for libyaml-0.1.4-11.el7_0.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              826 kB/s | 208 kB  00:00     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-1.1503.el7.centos.2.8.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libyaml-0.1.4-11.el7_0.x86_64                                1/2 
  Installing : PyYAML-3.10-11.el7.x86_64                                    2/2 
  Verifying  : libyaml-0.1.4-11.el7_0.x86_64                                1/2 
  Verifying  : PyYAML-3.10-11.el7.x86_64                                    2/2 

Installed:
  PyYAML.x86_64 0:3.10-11.el7                                                   

Dependency Installed:
  libyaml.x86_64 0:0.1.4-11.el7_0                                               

Complete!
Loaded plugins: fastestmirror
Cleaning repos: base extras updates
Cleaning up everything
Cleaning up list of fastest mirrors
 ---> 4c600eeea766
Removing intermediate container 2f91117135ab
Successfully built 4c600eeea766

Does this reproduce on Fedora?  I ask because all of the overlay-related xfs fixes that we did are upstream, and backported to RHEL7.2.

So could you try Fedora, perhaps?

(In reply to Jeremy Eder from comment #19)
> Does this reproduce on Fedora?  I ask because all of the overlay-related xfs
> fixes that we did are upstream, and backported to RHEL7.2.
> 
> So could you try Fedora, perhaps?

I suppose this is the problem since we're testing w/ CentOS 7.0.

I'm going to try against Fedora 22.

I can confirm that this also happens on Fedora 22, here's a minimal reproducer:

  docker commit $(docker run -d fedora:22 bash -c "rm /bin/ls;bash") test:latest

And yes, the problematic part is *file removal*, if you just add bunch of files to the layer it works fine.

# docker info
Containers: 5
Images: 6
Storage Driver: overlay
 Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.6-200.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 2
Total Memory: 1.954 GiB
Name: localhost.localdomain
ID: YCC6:QHC3:3IAF:YU3J:ZUT7:CU2K:FJAD:SPHT:6ZSE:FX47:44EX:XCOQ

I was able to reproduce the issue on Fedora 22 with Docker 1.7.1, but no with Docker 1.8.1.

I used 'docker-1.8.1-2.git32b8b25.fc23.src.rpm' from http://koji.fedoraproject.org/koji/buildinfo?buildID=679708, removing Requires(post): policycoreutils-python-utils' from the spec. To ensure a clean environment, I used mock to build the package 'mock --rebuild /tmp/docker-1.8.1-2.git32b8b25.fc22.src.rpm'

Here are my Test Results

== Stock Fedora 22 ==

# uname -a
Linux tux.rnd.lan 4.1.6-200.fc22.x86_64 #1 SMP Mon Aug 17 19:54:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | grep docker
docker-1.7.1-8.gitb6416b7.fc22.x86_64
docker-selinux-1.7.1-8.gitb6416b7.fc22.x86_64
devassistant-dap-docker-0.11-1.fc22.noarch

EXT4
----
# docker info
Containers: 0
Images: 0
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.6-200.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 8
Total Memory: 23.5 GiB
Name: tux.rnd.lan
ID: GKEC:LS34:7GVI:2SN2:VKR4:2JCQ:G43L:F6QP:MW6B:6TPP:BRJ6:H3O4

# docker commit $(docker run -d fedora:22 bash -c "rm /bin/ls;bash") test:latest
f90feb51f0bfc2c74d747c8f224852a5853daa300715c5b4dec5f70b13fdfc81

XFS
---
# docker info
Containers: 1
Images: 2
Storage Driver: overlay
 Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.6-200.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 8
Total Memory: 23.5 GiB
Name: tux.rnd.lan
ID: GKEC:LS34:7GVI:2SN2:VKR4:2JCQ:G43L:F6QP:MW6B:6TPP:BRJ6:H3O4

# docker commit $(docker run -d fedora:22 bash -c "rm /bin/ls;bash") test:latest
Error response from daemon: lstat /var/lib/docker/overlay/b649cb2eddcdaf5dfbb4579fd77a07d6d9e872b24cbf1e7ec43e14467992514d/merged/usr/bin/ls: no such file or directory

== Docker 1.8.1 on Fedora 22 ==
# docker info
Containers: 0
Images: 0
Storage Driver: overlay
 Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.6-200.fc22.x86_64
Operating System: Fedora 22 (Twenty Two)
CPUs: 8
Total Memory: 23.5 GiB
Name: tux.rnd.lan
ID: GKEC:LS34:7GVI:2SN2:VKR4:2JCQ:G43L:F6QP:MW6B:6TPP:BRJ6:H3O4

# docker commit $(docker run -d fedora:22 bash -c "rm /bin/ls;bash") test:latest
c7a8ad2fc9ece5dfefd5239619202c1d84e6846219a2f371a4f42166dce7f654

Confirmed, Fedora 22 + Docker 1.8.1 works fine.

Pavel, I don't see your docker version listed in this bug.  Since you've got a quick repro env setup; we would also want to know if RHEL7.2 + Docker 1.8.1 worked or not?

I think I already claim that: if use overlay on xfs, the mkfs.xfs must use ftype=1 option. you can refer my email for this:

"overlayfs known issues"

* based on xfs, ftype flag must be 1

(In reply to XuWang from comment #25)
> I think I already claim that: if use overlay on xfs, the mkfs.xfs must use
> ftype=1 option. you can refer my email for this:
> 
> "overlayfs known issues"
> 
> * based on xfs, ftype flag must be 1

Well, this is *very* important, notice how the resulting image is almost 4x the base size:

# cat Dockerfile
FROM fedora:22
RUN dnf install -y aide
RUN dnf clean all -y
RUN dnf install -y vim
RUN dnf clean all -y
RUN dnf install -y groff
RUN dnf clean all -y

# docker build -t test .
(snip)
Installed size: 517 k
Installed size: 63 M
Installed size: 3.8 M

# docker images | grep ^test
test                latest              e6da5dbee0e5        38 seconds ago      663.9 MB

# docker images | grep fedora
docker.io/fedora    22                  ded7cd95e059        3 months ago         186.5 MB

This happens for me on latest F23 Vagrant box as well:

...
ApplyLayer exit status 1 stdout:  stderr: link / /usr/lib64/python2.7/site-packages/yaml/__init__.pyc: operation not permitted

So Fedora is also affected.

Btw I was running:

sudo docker build --tag=fedora-django Fedora-Dockerfiles/Django

I do see the same issue (different file) as Josef on my F23, but I don't use overlayfs:

$ docker info                                                              
Containers: 6
Images: 24
Storage Driver: devicemapper
 Pool Name: docker-253:0-3670592-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 908.9 MB
 Data Space Total: 107.4 GB
 Data Space Available: 100.1 GB
 Metadata Space Used: 1.896 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107 (2015-09-05)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.2.3-300.fc23.x86_64
Operating System: Fedora 23 (Twenty Three)
CPUs: 4
Total Memory: 11.44 GiB
Name: mistress
ID: GSDO:A5AC:373A:I6IA:75BH:WGYH:VXCF:E24K:NIY2:LS47:BF7C:Y7MG

Upgrading to docker-1.8.2-9.gitbdb52b6.fc23.x86_64.rpm and docker-selinux-1.8.2-9.gitbdb52b6.fc23.x86_64.rpm from updates-testing fixed the issue.

Please update karma