Bug 1443274

Summary: Using secrets in a build config invalidates cache
Product: OpenShift Container Platform Reporter: Brennan Vincello <bvincell>
Component: BuildAssignee: Ben Parees <bparees>
Status: CLOSED CANTFIX QA Contact: Wenjing Zheng <wzheng>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.4.0CC: aos-bugs, bparees, bvincell, dyan, lucastetreault, shea.stewart
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-03 18:30:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Brennan Vincello 2017-04-19 01:43:44 UTC
Description of problem:

Using a secret (as described here https://docs.openshift.com/enterprise/3.2/dev_guide/builds.html#using-secrets) invalidates the docker build cache and causes additional performance / resource issues. The secret appears as a unique file with each build, even though the file has not changed. 

Version-Release number of selected component (if applicable): OCP 3.4

How reproducible: Very

Steps to Reproduce:

https://docs.openshift.com/enterprise/3.2/dev_guide/builds.html#using-secrets-in-the-buildconfig
$ oc secrets new secret-npmrc .npmrc=~/.npmrc
$ oc new-build openshift/nodejs-010-centos7~https://github.com/openshift/nodejs-ex.git --build-secret secret-npmrc
(repeat)

Actual results:

Build cache invalidated.

Expected results:

Cache validated, efficiently load secret and leverage layer caching for builds. 

Additional info:

The following output shows two independent runs with no change in the source file: 

Step 6 : RUN stat .npmrc
 ---> Running in e2e71d83ef0e
  File: '.npmrc'
  Size: 170       	Blocks: 8          IO Block: 4096   regular file
Device: fd37h/64823d	Inode: 4402288     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-04-05 01:05:10.000000000 +0000
Modify: 2017-04-05 01:05:10.000000000 +0000
Change: 2017-04-05 01:05:38.677806593 +0000
 Birth: -
 ---> 613afe4d8d70
Removing intermediate container e2e71d83ef0e
Step 7 : RUN md5sum .npmrc
 ---> Running in a18c9a056cad
b9252eabffea5384b938639e13fa19f8  .npmrc

Step 6 : RUN stat .npmrc
 ---> Running in fe84e33c80ff
  File: '.npmrc'
  Size: 170       	Blocks: 8          IO Block: 4096   regular file
Device: fd2dh/64813d	Inode: 4407408     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-04-05 01:00:58.000000000 +0000
Modify: 2017-04-05 01:00:58.000000000 +0000
Change: 2017-04-05 01:01:17.910914143 +0000
 Birth: -
 ---> 6139cf84c73d
Removing intermediate container fe84e33c80ff
Step 7 : RUN md5sum .npmrc
 ---> Running in b8c417611d88
b9252eabffea5384b938639e13fa19f8  .npmrc
 ---> f86ec764d64d
Removing intermediate container b8c417611d88

Comment 1 Ben Parees 2017-04-19 12:56:45 UTC
The new-build command you're running should be producing an s2i build, not a docker-type build, so i'm not even sure why you're getting a docker build here.  (And s2i builds don't care about layer caching because there is only one layer).

Can you provide the full output of your new-build invocation?

In any case when you are doing a docker build, we must copy the secret file into your buildcontext directory each time (the same as we must do w/ your git source code) since the buildcontext directory is not persisted between runs, so i'm not surprised that docker is going to treat those as new/invalidated files for caching purposes and short of getting to the point where builds can be run based on persistent volumes, there's no simple fix i'm afraid.

Comment 3 Ben Parees 2017-05-09 17:48:26 UTC
ok, I see you're getting a docker type build because your repo contains a dockerfile.  But in anycase my comment above still applies, there's really no way for us to fix this, since the mounted secret file is always going to be considered newer than the cached docker layer.

Comment 4 Lucas Tetreault 2017-06-13 16:08:18 UTC
Hey Ben, as of Docker 1.8 timestamps don't matter for cache invalidation: https://github.com/moby/moby/pull/12031

When I mount an secret .npmrc as described in the issue I get the following files in my docker image. 

/app $ ls -al                                                                                                                                                                                         
total 2092                                                                                                                                                                                            
drwxrwxr-x   11 911      root          4096 Jun 13 15:57 .                                                                                                                                            
drwxr-xr-x   20 root     root           243 Jun 13 00:21 ..                                                                                                                                           
drwxrwxr-x    2 911      root            20 Jun 13 00:12 ..6986_13_06_00_12_52.761028215                                                                                                              
lrwxrwxrwx    1 911      root            31 Jun 13 00:12 ..data -> ..6986_13_06_00_12_52.761028215                                                                                                                                                                                                                                     
lrwxrwxrwx    1 911      root            13 Jun 13 00:12 .npmrc -> ..data/.npmrc 

.npmrc is a symlink to ..data/.nprmc which in turn is a symlink to ..6986_13_06_00_12_52.761028215/.npmrc. 

6986_13_06_00_12_52.761028215 appears to be a timestamp and changes every time we build the image. This means that metadata for .npmrc is changing and the cache invalidation should happen. 

In conclusion, the cache invalidation is caused by how the secret is mounted and this should be fixed by redhat. This is not a problem with docker.

Comment 5 Ben Parees 2017-06-13 16:16:09 UTC
Thanks for the details Lucas.  The mechanism by which secrets are mounted into the pod is defined by kubernetes(w/ the symlink structure) and it's unlikely to be changed, i believe it's done that way to allow/ensure atomic updates to the mounted secret content.

However the way the docker build strategy works on openshift, that file you found is first copied into a working directory which becomes the context for the docker build.  So i'm not sure what metadata you're referring to, but ultimately the docker build is going to be run in a directory containing a file named ".npmrc"  on every build.  If docker is ignoring the file timestamp, then I don't know what else about that file would be invalidating the cache.