Description of problem: With the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1368024 installed the quick start apps behave as expected. After the initial build, the applications can be built in different namespaces and only the changed layers will be uploaded to the registry. However, some applications still upload all layers when building across namespaces. In our testing of different OpenShift-provided applications we've found that the EAP samples will push all layers when being built across namespaces. If an EAP app is only built in a single namespace, it shows the desired behavior, only changed layers are pushed. Version-Release number of selected component (if applicable): 3.3.0.27 How reproducible: always Steps to Reproduce: 0. Make sure Docker with the fix for bz 1368024 is installed - docker-common-1.10.3-46.el7.11.x86_64 or higher 1. Create two projects, A and B. oc new-app --template eap64-basic-s2i 2. In project A oc start-build eap-app several times. Use oc logs --timestamps bc/eap-app to examine the logs. After a warmup build or two, each build will only push two layers and the push should take about 5 seconds 3. Repeat step 2 in project B until the builds repeatedly push only 2 layers 4. Switch back to project A and build it - it will push all layers. Examine the docker image on the builder node with docker history <id> to see that all layers are not changed. 5. Repeat the experiment with nodejs-example or cakephp-example and you will see that they do not exhibit this behavior. They continue to only push fewer layers when switching namespaces. Actual results: Switching between building the EAP app in two different namespaces results in a push of all layers, not just unchanged layers. Expected results: Only changed layers are pushed, similar to what is seen for other applications. Additional info: Let me know what additional information is required.
I'll try to reproduce. I have no idea why the described behavior happens just for eap64-basic-s2i, and not for the others. Thanks for the report Mike!
So the problem here is that registry.access.redhat.com/jboss-eap-6/eap64-openshift was pushed by some older docker version. The uploaded blobs stored there have different digests than those uploaded with docker 1.10 to the internal registry. Take a look at docker's internal metadata: $ cat v2metadata-by-diffid/sha256/821a26b9542a1eb5c575a9f5095cb5c652a5f05b14b17d77e661ae1df7003a2b | json_reformat [ { "Digest": "sha256:821a26b9542a1eb5c575a9f5095cb5c652a5f05b14b17d77e661ae1df7003a2b", "SourceRepository": "registry.access.redhat.com/jboss-eap-6/eap64-openshift" }, { "Digest": "sha256:42ee9072ed9116e9ea4b4a4c5d67db98237f9bb3582a31a5bb45586f2388075e", "SourceRepository": "172.30.241.183:5000/eap1/eap-app" }, { "Digest": "sha256:42ee9072ed9116e9ea4b4a4c5d67db98237f9bb3582a31a5bb45586f2388075e", "SourceRepository": "172.30.241.183:5000/eap2/eap-app" } ] The `sha256:821a26b9542a1eb5c575a9f5095cb5c652a5f05b14b17d77e661ae1df7003a2b` is a digest of blob pulled from registry.access.redhat.com/jboss-eap-6/eap64-openshift. The same blob uploaded to internal registry using docker 1.10.3 has different digest `sha256:42ee9072ed9116e9ea4b4a4c5d67db98237f9bb3582a31a5bb45586f2388075e`. The difference is caused either because of changes in tar-split library or perhaps because of different building host environment. The version of docker used to build the image is a main factor. What happens during a blob upload is that daemon checks for the presence of a digest in the first entry of above json in the internal registry. And obviously fails because the digest exists only in registry.access.redhat.com/jboss-eap-6/eap64-openshift. So my previous Docker patch is flawed. I'm working on enhanced version that I'll submit upstream.
Here's an upstream PR: https://github.com/docker/docker/pull/26564
Upstream PR has been merged. It is being back-ported to our 1.10 and 1.12 releases: - https://github.com/projectatomic/docker/pull/202 - https://github.com/projectatomic/docker/pull/203
Merged in docker-1.12.2 branch.
*** Bug 1347022 has been marked as a duplicate of this bug. ***
Can we close this one?
Is there a red hat docker build in our channels with the fix included?
extras-rhel-7.3-candidate or brew build https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=520605
Mike, is there a chance we can get this verified by using the the brew build docker?
RHEL 7.3 (and the corresponding extras) only has docker-1.10.3-57.el7 so I don't think it right to close this, since our users are still experiencing it.
Verified the issue is fixed using docker 1.12.3-4 from the extras repo on mirror.openshift.com
docker-1.12.3 should be released by the time that openshift 3.4 is released. Marking this as part of the 3.4 release and moving it through that errata process.
I accidentally switched this to VERIFIED. Moving back to ON_QA.
Hi Mike, Could you help check if this bug has been fixed? If it has been fixed,help verify it,thanks!
Verified on Docker 1.12.3-8 and OCP 3.4.0.33
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0066