RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1311758 - unable to pull rhel6 base image - failed to register layer: ApplyLayer exit status 1
Summary: unable to pull rhel6 base image - failed to register layer: ApplyLayer exit s...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 7.2
Assignee: Daniel Walsh
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1485394
TreeView+ depends on / blocked
 
Reported: 2016-02-24 22:22 UTC by Micah Abbott
Modified: 2020-07-16 08:42 UTC (History)
26 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-30 15:14:17 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Micah Abbott 2016-02-24 22:22:13 UTC 
Description of problem:

When trying to pull the rhel6 base image, it fails with the error:

Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument


Version-Release number of selected component (if applicable):
# rpm -q docker
docker-1.10.2-4.git0f5ac89.fc23.x86_64

(From http://koji.fedoraproject.org/koji/buildinfo?buildID=738485)

How reproducible:
100% - I was able to reproduce this on two separate F23 VMs


Steps to Reproduce:
1.  Install docker 1.10
2.  docker pull registry.access.redhat.com/rhel6


Actual results:
# docker pull registry.access.redhat.com/rhel6:latest                                                                                                                                        
Trying to pull repository registry.access.redhat.com/rhel6 ... 
Pulling repository registry.access.redhat.com/rhel6
c631ca2f5641: Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument 
Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument


Expected results:
Image should be pulled successfully


Additional info:
# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: devicemapper
 Pool Name: docker--group-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 20.45 MB
 Data Space Total: 8.577 GB
 Data Space Available: 8.557 GB
 Metadata Space Used: 45.06 kB
 Metadata Space Total: 25.17 MB
 Metadata Space Available: 25.12 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.109 (2015-09-22)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.3.5-300.fc23.x86_64
Operating System: Fedora 23 (Cloud Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 0
CPUs: 2
Total Memory: 1.954 GiB
Name: rhel-atomic-7.2-test
ID: 4RMM:ZTGP:3TDQ:B7KK:7UY5:NZAK:M3AV:WUUN:OULH:KW2I:AFVG:45UC
Registries: docker.io (secure)

# lvs
  LV          VG           Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool docker-group twi-a-t--- 7.99g             0.24   0.18                            


Other images can be pulled successfully:

# docker pull busybox
Using default tag: latest
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
f810322bba2c: Pull complete 
a3ed95caeb02: Pull complete 
Digest: sha256:97473e34e311e6c1b3f61f2a721d038d1e5eef17d98d1353a513007cf46ca6bd
Status: Downloaded newer image for docker.io/busybox:latest

# docker pull debian
Using default tag: latest
Trying to pull repository docker.io/library/debian ... 
latest: Pulling from docker.io/library/debian
7268d8f794c4: Pull complete 
a3ed95caeb02: Pull complete 
Digest: sha256:60ea1fc595f0da2afd6fc1d39efb7a00b8d7138a30932673b1adce0500107c5f
Status: Downloaded newer image for docker.io/debian:latest

# docker pull registry.access.redhat.com/rhel7/rsyslog
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7/rsyslog ... 
Pulling repository registry.access.redhat.com/rhel7/rsyslog
61163f44884c: Pull complete 
18c92348de36: Already exists 
Status: Downloaded newer image for registry.access.redhat.com/rhel7/rsyslog:latest
registry.access.redhat.com/rhel7/rsyslog: this image was pulled from a legacy registry.  Important: This registry version will not be supported in future versions of docker.
Comment 1 Daniel Walsh 2016-02-25 15:07:32 UTC 
Can you pull this with docker-1.9?
Comment 2 Antonio Murdaca 2016-02-25 15:14:32 UTC 
my f23 boxes don't suffer this bug

Operating System: Fedora 23 (Cloud Edition)

is there anything different - storage wise - probably in a f23 cloud edition?
Comment 3 Micah Abbott 2016-02-25 15:37:56 UTC 
@dwalsh - I'll go back and try with 1.9.

@runcom - I added a virtual disk to my VM and created a VG from it to be used by d-s-s.  Since both VMs were running on my system, perhaps it is something on my workstation.  I'll try reproducing in a different environment, i.e. bare metal or OpenStack.
Comment 4 Micah Abbott 2016-02-25 22:30:31 UTC 
I was able to reproduce this on an OpenStack VM running Fedora Cloud 23.

This was done using both 1.10 and 1.9.  And I tested using loopback and thin-pool as backing storage.

Additionally, I was able to reproduce this with the same combinations of version + backing storage using Fedora Server 23 on bare metal.

The failure seems to occur when the image is getting extracted to disk right around the 22.68 MB mark.

1.9.1 + loopback
=================
# docker info
Containers: 0
Images: 1
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: docker-252:1-262312-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 295.3 MB
 Data Space Total: 107.4 GB
 Data Space Available: 41.08 GB
 Metadata Space Used: 716.8 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107 (2015-09-05)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.2.3-300.fc23.x86_64
Operating System: Fedora 23 (Cloud Edition)
CPUs: 2
Total Memory: 3.86 GiB
Name: micah-fedora-cloud-23-vm0.novalocal
ID: 7LWQ:OBKJ:PDHJ:P67U:TTFN:3UY4:76VZ:EJPH:EEET:ITSU:O4XL:4ZXZ

# docker pull registry.access.redhat.com/rhel6
Using default tag: latest
c631ca2f5641: Downloading 22.68 MB
failed
Error pulling image (latest) from registry.access.redhat.com/rhel6, ApplyLayer exit status 1 stdout:  stderr: invalid argument



1.9.1 + thin-pool
===================
# docker info
Containers: 0
Images: 0
Server Version: 1.9.1
Storage Driver: devicemapper
 Pool Name: fedora-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 62.39 MB
 Data Space Total: 8.577 GB
 Data Space Available: 8.515 GB
 Metadata Space Used: 45.06 kB
 Metadata Space Total: 25.17 MB
 Metadata Space Available: 25.12 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107 (2015-09-05)
Execution Driver: native-0.2
Logging Driver: journald
Kernel Version: 4.2.3-300.fc23.x86_64
Operating System: Fedora 23 (Cloud Edition)
CPUs: 2
Total Memory: 3.86 GiB
Name: micah-fedora-cloud-23-vm0.novalocal
ID: 7LWQ:OBKJ:PDHJ:P67U:TTFN:3UY4:76VZ:EJPH:EEET:ITSU:O4XL:4ZXZ

# docker pull registry.access.redhat.com/rhel6 
Using default tag: latest
c631ca2f5641: Downloading 22.68 MB
failed
Error pulling image (latest) from registry.access.redhat.com/rhel6, ApplyLayer exit status 1 stdout:  stderr: invalid argument



1.10 + loopback
================
# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: devicemapper
 Pool Name: docker-252:1-262323-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 41.35 GB
 Metadata Space Used: 581.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107 (2015-09-05)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.2.3-300.fc23.x86_64
Operating System: Fedora 23 (Cloud Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 0
CPUs: 2
Total Memory: 3.86 GiB
Name: micah-fedora-cloud-23-vm0.novalocal
ID: 7LWQ:OBKJ:PDHJ:P67U:TTFN:3UY4:76VZ:EJPH:EEET:ITSU:O4XL:4ZXZ
Registries: docker.io (secure)

# docker pull registry.access.redhat.com/rhel6
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel6 ... 
Pulling repository registry.access.redhat.com/rhel6
c631ca2f5641: Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument 
Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument



1.10 + thin-pool
=================
# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: devicemapper
 Pool Name: fedora-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 20.45 MB
 Data Space Total: 8.577 GB
 Data Space Available: 8.557 GB
 Metadata Space Used: 45.06 kB
 Metadata Space Total: 25.17 MB
 Metadata Space Available: 25.12 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107 (2015-09-05)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.2.3-300.fc23.x86_64
Operating System: Fedora 23 (Cloud Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 0
CPUs: 2
Total Memory: 3.86 GiB
Name: micah-fedora-cloud-23-vm0.novalocal
ID: 7LWQ:OBKJ:PDHJ:P67U:TTFN:3UY4:76VZ:EJPH:EEET:ITSU:O4XL:4ZXZ
Registries: docker.io (secure)

# docker pull registry.access.redhat.com/rhel6
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel6 ... 
Pulling repository registry.access.redhat.com/rhel6
c631ca2f5641: Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument 
Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument
Comment 5 Daniel Walsh 2016-02-26 13:46:38 UTC 
I am getting the same issue on rawhide with latest docker.

docker pull registry.access.redhat.com/rhel6:latestTrying to pull repository registry.access.redhat.com/rhel6 ... 
Pulling repository registry.access.redhat.com/rhel6
c631ca2f5641: Error pulling image (latest) from registry.access.redhat.com/rhel6c631ca2f5641: Error pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid arError pulling image (latest) from registry.access.redhat.com/rhel6, failed to register layer: ApplyLayer exit status 1 stdout:  stderr: invalid argument

Moving this to rhel
Comment 7 Frantisek Kluknavsky 2016-02-26 14:24:23 UTC 
According to https://errata.devel.redhat.com/advisory/22738 pull from registry.access.stage.redhat.com was working a week ago. Right now it fails the same way as registry.access.redhat.com:

docker pull registry.access.stage.redhat.com/rhel6
Using default tag: latest
c631ca2f5641: Downloading 22.69 MB
failed
Error pulling image (latest) from registry.access.stage.redhat.com/rhel6, ApplyLayer exit status 1 stdout:  stderr: invalid argument

Might this suggest that the image was built correctly but later some piece of infrastructure broke?
Comment 8 Frantisek Kluknavsky 2016-02-26 15:58:24 UTC 
I am not sure where to report this. Lets start with the next link in the chain and ask a release engineer.
Comment 9 Micah Abbott 2016-03-08 15:12:06 UTC 
When I tried reproducing this again using an F23 Cloud system, I noticed I was getting AVC denials now.


# date
Tue Mar  8 15:09:00 UTC 2016

# docker pull registry.access.redhat.com/rhel6
Using default tag: latest
31b925c88737: Downloading 12.65 MB
failed
Error pulling image (latest) from registry.access.redhat.com/rhel6, ApplyLayer exit status 1 stdout:  stderr: invalid argument

# journalctl --since 15:09
-- Logs begin at Fri 2016-02-26 20:22:41 UTC, end at Tue 2016-03-08 15:09:08 UTC. --
Mar 08 15:09:02 rhel-atomic-7.2-test docker[980]: time="2016-03-08T15:09:02.902958443Z" level=info msg="{Action=create, Username=cloud-user, LoginUID=1000, PID=1141}"
Mar 08 15:09:03 rhel-atomic-7.2-test docker[980]: time="2016-03-08T15:09:03.225377323Z" level=warning msg="Error getting v2 registry: endpoint does not support v2 API"
Mar 08 15:09:06 rhel-atomic-7.2-test kernel: XFS (dm-3): Mounting V5 Filesystem
Mar 08 15:09:06 rhel-atomic-7.2-test kernel: XFS (dm-3): Ending clean mount
Mar 08 15:09:07 rhel-atomic-7.2-test audit[1164]: AVC avc:  denied  { mac_admin } for  pid=1164 comm="exe" capability=33  scontext=system_u:system_r:docker_t:s0 tcontext=system_u:system_r:docker_t:s0 tclass=capa
Mar 08 15:09:07 rhel-atomic-7.2-test audit: SELINUX_ERR op=setxattr invalid_context="system_u:object_r:shutdown_exec_t:s0"
Mar 08 15:09:08 rhel-atomic-7.2-test kernel: XFS (dm-3): Unmounting Filesystem

# journalctl --since 15:09 | grep denied
Mar 08 15:09:07 rhel-atomic-7.2-test audit[1164]: AVC avc:  denied  { mac_admin } for  pid=1164 comm="exe" capability=33  scontext=system_u:system_r:docker_t:s0 tcontext=system_u:system_r:docker_t:s0 tclass=capability2 permissive=0


# rpm -qa | grep docker
docker-selinux-1.9.1-6.git6ec29ef.fc23.x86_64
docker-1.9.1-6.git6ec29ef.fc23.x86_64
python-docker-py-1.5.0-1.fc23.noarch
Comment 10 Daniel Walsh 2016-03-08 15:27:26 UTC 
I am seeing the same ting. It looks like the docker tar ball includes XAttrs that docker is trying to set "shutdown_exec_t" the docker image should not contain any SELinux labels.
Comment 11 Jan Hutař 2016-04-01 18:53:20 UTC 
With SELinux temporarily set to permissive, I see this:

$ sudo docker pull registry.access.redhat.com/rhel6.7:latest
[sudo] password for pok: 
c77d113b1842: Download complete 
Status: Downloaded newer image for registry.access.redhat.com/rhel6.7:latest
registry.access.redhat.com/rhel6.7: this image was pulled from a legacy registry.  Important: This registry version will not be supported in future versions of docker.
$ rpm -qa | grep docker
python-dockerfile-parse-0.0.5-1.fc23.noarch
docker-1.9.1-6.git6ec29ef.fc23.x86_64
python-docker-registry-core-2.0.3-2.fc23.noarch
docker-selinux-1.9.1-6.git6ec29ef.fc23.x86_64
docker-registry-0.9.1-2.fc23.noarch

Is there some old docker-registry version on registry.access.redhat.com?
Comment 13 Antonio Murdaca 2016-06-20 12:09:25 UTC 
(In reply to Jan Hutař from comment #11)
> With SELinux temporarily set to permissive, I see this:
> 
> $ sudo docker pull registry.access.redhat.com/rhel6.7:latest
> [sudo] password for pok: 
> c77d113b1842: Download complete 
> Status: Downloaded newer image for registry.access.redhat.com/rhel6.7:latest
> registry.access.redhat.com/rhel6.7: this image was pulled from a legacy
> registry.  Important: This registry version will not be supported in future
> versions of docker.
> $ rpm -qa | grep docker
> python-dockerfile-parse-0.0.5-1.fc23.noarch
> docker-1.9.1-6.git6ec29ef.fc23.x86_64
> python-docker-registry-core-2.0.3-2.fc23.noarch
> docker-selinux-1.9.1-6.git6ec29ef.fc23.x86_64
> docker-registry-0.9.1-2.fc23.noarch
> 
> Is there some old docker-registry version on registry.access.redhat.com?

no worry, that's just a warning from docker saying that our registry it's V1
Comment 14 Lubos Kocman 2017-01-04 16:01:22 UTC 
Hello team,

we don't own registry itself, we just push data in it. I was also told that this issue with v1 is already fixed (confirmed with Stanislav Graf).

So marking as current release.

Lubos
Comment 23 Daniel Walsh 2017-06-30 15:14:17 UTC 
The SELinux issues are fixed in the current release.
Note You need to log in before you can comment on or make changes to this bug.