RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1417242 - Some builds fail with error: AuthZPlugin.AuthZReq: Error response from daemon: layer does not exist
Summary: Some builds fail with error: AuthZPlugin.AuthZReq: Error response from daemon...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Daniel Walsh
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-27 16:31 UTC by Vikas Laad
Modified: 2019-03-06 01:46 UTC (History)
8 users (show)

Fixed In Version: docker-1.12.6-28.git1398f24.el7_3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-26 14:31:53 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1325 0 normal SHIPPED_LIVE docker bug fix and enhancement update 2017-05-26 18:14:24 UTC

Description Vikas Laad 2017-01-27 16:31:13 UTC 
Description of problem:
Building concurrently EAP project fails to push images with following error

root@ip-172-31-18-61: ~/svt/openshift_performance/ci/scripts # oc logs -n proj30      eap-app-6-build 
Cloning "https://github.com/jboss-openshift/openshift-quickstarts" ...
	Commit:	caec20220374804b2cb3d3622a754f9091af7c57 (Fixed bug: summary is displayed instead of description (cherry picked from com..)
	Author:	Marko Luksa <marko.luksa>
	Date:	Tue Feb 9 22:55:09 2016 +0100
Pulling image "registry.access.redhat.com/jboss-eap-6/eap64-openshift@sha256:2527fe6337f48bde11ffb43bbbd000172fe5ab26aeca93d590e296d2db2f6ed3" ...
Pulling image "registry.access.redhat.com/jboss-eap-6/eap64-openshift@sha256:2527fe6337f48bde11ffb43bbbd000172fe5ab26aeca93d590e296d2db2f6ed3" ...
Copying all war artifacts from /home/jboss/source/. directory into /opt/eap/standalone/deployments for later deployment...
Copying all ear artifacts from /home/jboss/source/. directory into /opt/eap/standalone/deployments for later deployment...
Copying all rar artifacts from /home/jboss/source/. directory into /opt/eap/standalone/deployments for later deployment...
Copying all jar artifacts from /home/jboss/source/. directory into /opt/eap/standalone/deployments for later deployment...
Copying all war artifacts from /home/jboss/source/deployments directory into /opt/eap/standalone/deployments for later deployment...
Copying all ear artifacts from /home/jboss/source/deployments directory into /opt/eap/standalone/deployments for later deployment...
Copying all rar artifacts from /home/jboss/source/deployments directory into /opt/eap/standalone/deployments for later deployment...
Copying all jar artifacts from /home/jboss/source/deployments directory into /opt/eap/standalone/deployments for later deployment...

Pushing image 172.25.247.149:5000/proj30/eap-app:latest ...
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount
Registry server Password: <<non-empty>>
error: build error: Failed to push image: API error (500): {"message":"plugin rhel-push-plugin failed with error: AuthZPlugin.AuthZReq: Error response from daemon: layer does not exist"}

Version-Release number of selected component (if applicable):
openshift v3.5.0.9+e84be2b
kubernetes v1.5.2+43a9be4
etcd 3.1.0

How reproducible:
around 10 builds failed from 280 builds

Steps to Reproduce:
1. Create OCP cluster in aws with 2 worker, 1 infra and 1 master nodes (m4.xlarge)
2. build EAP quickstart app concurrently in the steps (5,10,20,30,40,50)
3. Error starts happening when the concurrent build reaches 20

Actual results:
Push error in the build logs, build errors out

Expected results:
Should not be build error

Additional info:
Comment 1 Vikas Laad 2017-01-27 17:24:32 UTC 
Also failed in another environment with cakephp app, in this env only 1 build is running at a time.

E, [2017-01-26 20:23:54#2929] ERROR -- :  stdout: build "cakephp-mysql-example-3" started[#]Cloning "https://github.com/openshift/cakephp-ex.git" ...[#]        Commit: fb2aabb3c104df7f590b3ccb041c28885ca213c2 (Merge pull request #57 from coreydaley/update_templates_to_use_secrets)[#]    Author: Ben Parees <bparees.github.com>[#]        Date:   Wed Jan 18 13:19:54 2017 -0500[#]---> Installing application source...[#][#]Pushing image 172.25.40.166:5000/cakephp-mysql-u-100-33-150-54/cakephp-mysql-example:latest ...[#]Registry server Address: [#]Registry server User Name: serviceaccount[#]Registry server Email: serviceaccount[#]Registry server Password: <<non-empty>>[#]error: build error: Failed to push image: API error (500): {"message":"plugin rhel-push-plugin failed with error: AuthZPlugin.AuthZReq: Error response from daemon: layer does not exist"}[#]
Comment 2 Ben Parees 2017-01-30 15:25:39 UTC 
This error is coming from the image registry, moving to that component.
Comment 3 Antonio Murdaca 2017-02-06 17:31:49 UTC 
The error is coming from docker & rhel-push-plugin
Comment 4 Antonio Murdaca 2017-02-06 17:43:16 UTC 
Lokesh, could you rebuild rhel-push-plugin in RHEL from latest master to test this out again?
Comment 5 Michal Fojtik 2017-02-06 18:01:08 UTC 
(moving to docker).
Comment 9 Daniel Walsh 2017-04-13 18:40:56 UTC 
What versions of docker are you seeing this with?
Comment 10 Vikas Laad 2017-04-13 18:44:35 UTC 
I have seen this with docker 1.12.6
Comment 11 Antonio Murdaca 2017-05-12 09:28:44 UTC 
This should be fixed with latest rhel-push-plugin, Lokesh can you rebuild?
Comment 14 Vikas Laad 2017-05-18 14:41:09 UTC 
Verified on following docker version

root@ip-172-31-44-158: ~ # docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-28.git1398f24.el7.x86_64
 Go version:      go1.7.4
 Git commit:      1398f24/1.12.6
 Built:           Wed May 17 01:16:44 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-28.git1398f24.el7.x86_64
 Go version:      go1.7.4
 Git commit:      1398f24/1.12.6
 Built:           Wed May 17 01:16:44 2017
 OS/Arch:         linux/amd64

root@ip-172-31-44-158: ~ # openshift version
openshift v3.6.75
kubernetes v1.6.1+5115d708d7
etcd 3.1.0


Completed 3000 builds, did not see this error.
Comment 16 errata-xmlrpc 2017-05-26 14:31:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1325
Note You need to log in before you can comment on or make changes to this bug.