Bug 1346930

Summary: atomic pull from RH registry.access.redhat.com gives fatal error
Product: Red Hat Enterprise Linux 7 Reporter: David Darrah/Red Hat QE <ddarrah>
Component: skopeoAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Jenner <mjenner>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.2CC: ajia, bbaude, bbreard, dwalsh, gscrivan, imcleod, mitr
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-13 12:15:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1387217    
Bug Blocks:    

Comment 1 David Darrah/Red Hat QE 2016-06-15 16:37:18 UTC
probably same issue found in https://bugzilla.redhat.com/show_bug.cgi?id=1346930

Comment 2 Alex Jia 2016-06-15 16:40:55 UTC
I think it should be the same issue to bug 1346619.

Comment 3 Alex Jia 2016-06-15 16:50:21 UTC
David, BTW, atomic update should be work for this.

Comment 5 Giuseppe Scrivano 2016-06-22 13:54:08 UTC
at the moment "atomic pull" supports pull only to OSTree and it uses Skopeo for retrieving the images from a v2 registry.  "atomic update" uses "docker pull" internally and should be used to fetch a regular docker image.

Comment 6 Daniel Walsh 2016-08-26 19:03:30 UTC
We need to merge these together, users are not going to know the difference.

atomic pull should allow a user to pull to multiple back ends, docker,ostree atomic-storage.

This is what I see with atomic 1.12-devel

 atomic pull registry.access.redhat.com/rhel7.2

Unable to interact with a V1 registry.

Comment 8 Alex Jia 2016-09-18 03:47:04 UTC
(In reply to Giuseppe Scrivano from comment #5)
> at the moment "atomic pull" supports pull only to OSTree and it uses Skopeo
> for retrieving the images from a v2 registry.  "atomic update" uses "docker
> pull" internally and should be used to fetch a regular docker image.


[cloud-user@atomic-host-001 atomic]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.2

[cloud-user@atomic-host-001 atomic]$ rpm -q atomic docker
atomic-1.10.5-7.el7.x86_64
docker-1.10.3-46.el7.14.x86_64

1. atomic pull

1.1 if it exists registry.access.redhat.com/rhel7 image on the local

[cloud-user@atomic-host-001 atomic]$ sudo atomic pull registry.access.redhat.com/rhel7.2
Missing layer 99dd41655d8a45c2fb74f9eeb73e327b3ad4796f0ff0d602c575e32e9804baed
Missing layer 30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9

NOTE: there is no error "Unable to interact with a V1 registry." is raised now, is registry.access.redhat.com V2 registry? 


1.2 if it doesn't exist registry.access.redhat.com/rhel7 image on the local

[cloud-user@atomic-host-001 atomic]$ sudo docker rmi registry.access.redhat.com/rhel7:latest
Untagged: registry.access.redhat.com/rhel7:latest
Deleted: sha256:4a6b6e1a17d70b7f67787aaee800c1fdb4b145dd3f7ae48959f5d41286eadb0b

[cloud-user@atomic-host-001 atomic]$ sudo atomic pull registry.access.redhat.com/rhel7.2
[cloud-user@atomic-host-001 atomic]$ echo $?
0

NOTE: there is no rhel7 image is displayed by docker images, but it indeed exists in atomic images, and the image can't be removed by docker rmi, so we need to wait for new RHELAH build w/ atomic images delete feature.

[cloud-user@atomic-host-001 atomic]$ sudo atomic images | grep rhel7.2
  rhel7.2                                                          latest   4d0c93462fc8   2016-09-18 03:29

Is it a design the image pulled by atomic can't be seen docker images?


1.3 atomic pull external image

[cloud-user@atomic-host-001 atomic]$ sudo atomic pull ubuntu
Missing layer 24e0251a0e2ccc2fedbdf51dd44f851622f504c7ddfad56ce0f1c63e1101cb20
Missing layer 8296858250fed454169c737c0706958c8a4a130e0bfdca3cf869fa767a19b4f1
Missing layer c19118ca682dcd394eeed77409fbac93d9fa94c0bcff633344dc0a71ead74a66
Missing layer 82659f8f1b7628b94f8919ece229321a16ec0b7139cf289e010b7c064f603516
Missing layer 952132ac251a8df1f831b354a0b9a4cc7cd460b9c332ed664b4c205db6f22c29
g-io-error-quark: Unsupported file type for path "dev/agpgart" (0)n

NOTE: there is no ubuntu image is pulled in atomic/docker images, and got an error g-io-error-quark: Unsupported file type for path "dev/agpgart" (0)n.

Comment 10 Brent Baude 2016-09-24 13:20:29 UTC
Alex,

Is this still the case?  I believe in the latest atomic builds, we have set the default storage to docker.

I am seeing a problem with my current build however.

sudo ./atomic pull registry.access.redhat.com/rhel7.2
Image registry.access.redhat.com/rhel7.2 is being pulled to docker ...
FATA[0003] Error writing blob: Can not stream blob sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 with unknown size to docker-daemon: 

Alex, can you confirm this?

Adding Miloslav on as well.

Comment 11 Alex Jia 2016-09-24 14:43:56 UTC
(In reply to Brent Baude from comment #10)
> Alex,
> 
> Is this still the case?  I believe in the latest atomic builds, we have set
> the default storage to docker.
> 
> I am seeing a problem with my current build however.
> 
> sudo ./atomic pull registry.access.redhat.com/rhel7.2
> Image registry.access.redhat.com/rhel7.2 is being pulled to docker ...
> FATA[0003] Error writing blob: Can not stream blob
> sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 with
> unknown size to docker-daemon: 
> 
> Alex, can you confirm this?
> 
> Adding Miloslav on as well.


Yes, the same issue to me.


1. upstream

successfully pull ubuntu, but failed to pull registry.access.redhat.com/rhel7.2

$ git rev-parse HEAD
b5b95b9efc488d2e724eab7695e4fa7e94b9d156

[cloud-user@atomic-host-001 ~]$ rpm -q docker skopeo oci-systemd-hook
docker-1.10.3-55.el7.x86_64
skopeo-0.1.14-0.6.el7.x86_64
oci-systemd-hook-0.1.4-6.git337078c.el7.x86_64

[cloud-user@atomic-host-001 atomic]$ sudo ./atomic --debug pull registry.access.redhat.com/rhel7.2
Image registry.access.redhat.com/rhel7.2 is being pulled to docker ...
Executing: /usr/bin/skopeo --debug copy --remove-signatures docker://registry.access.redhat.com/rhel7.2 docker-daemon:registry.access.redhat.com/rhel7.2:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.access.redhat.com/rhel7.2:latest 
DEBU[0000] IsRunningImageAllowed for image docker:registry.access.redhat.com/rhel7.2:latest 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0001] Ping https://registry.access.redhat.com/v2/ err <nil> 
DEBU[0001] Ping https://registry.access.redhat.com/v2/ status 200 
DEBU[0001] GET https://registry.access.redhat.com/v2/rhel7.2/manifests/latest 
DEBU[0002] Downloading rhel7.2/blobs/sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 
DEBU[0002] GET https://registry.access.redhat.com/v2/rhel7.2/blobs/sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 
DEBU[0004] Detected compression format gzip             
DEBU[0004] Using original blob without modification     
DEBU[0004] docker-daemon: Closing tar stream to abort loading 
FATA[0004] Error writing blob: Can not stream blob sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 with unknown size to docker-daemon: 

Traceback (most recent call last):
  File "./atomic", line 186, in <module>
    sys.exit(_func())
  File "/var/home/cloud-user/atomic/Atomic/pull.py", line 56, in pull_image
    handler()
  File "/var/home/cloud-user/atomic/Atomic/pull.py", line 44, in pull_docker_image
    skopeo_copy("docker://{}".format(self.args.image), image, debug=self.args.debug, insecure=insecure)
  File "/var/home/cloud-user/atomic/Atomic/util.py", line 361, in skopeo_copy
    return check_call(cmd)
  File "/var/home/cloud-user/atomic/Atomic/util.py", line 130, in check_call
    return subprocess.check_call(cmd, env=env, stdin=stdin, stderr=stderr, stdout=stdout, close_fds=True)
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/bin/skopeo', '--debug', 'copy', '--remove-signatures', 'docker://registry.access.redhat.com/rhel7.2', 'docker-daemon:registry.access.redhat.com/rhel7.2:latest']' returned non-zero exit status 1


[cloud-user@atomic-host-001 atomic]$ sudo ./atomic --debug pull ubuntu
Image ubuntu is being pulled to docker ...
Executing: /usr/bin/skopeo --debug copy --remove-signatures docker://ubuntu docker-daemon:docker.io/library/ubuntu:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for docker.io/library/ubuntu:latest 
DEBU[0000] IsRunningImageAllowed for image docker:docker.io/library/ubuntu:latest 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0001] Ping https://registry-1.docker.io/v2/ err <nil> 
DEBU[0001] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0002] GET https://registry-1.docker.io/v2/library/ubuntu/manifests/latest 
DEBU[0003] Downloading library/ubuntu/blobs/sha256:45bc58500fa3d3c0d67233d4a7798134b46b486af1389ca87000c543f46c3d24 
DEBU[0004] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:45bc58500fa3d3c0d67233d4a7798134b46b486af1389ca87000c543f46c3d24 
DEBU[0005] No compression detected                      
DEBU[0005] Using original blob without modification     
DEBU[0005] Sending as tar file sha256:45bc58500fa3d3c0d67233d4a7798134b46b486af1389ca87000c543f46c3d24 
DEBU[0005] Downloading library/ubuntu/blobs/sha256:ff1f1f1de8626c4e34cb2cec216028cdf6fa5e735bb45aa7fa31475b5642aa22 
DEBU[0007] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:ff1f1f1de8626c4e34cb2cec216028cdf6fa5e735bb45aa7fa31475b5642aa22 
DEBU[0009] Detected compression format gzip             
DEBU[0009] Using original blob without modification     
DEBU[0009] Sending as tar file sha256:ff1f1f1de8626c4e34cb2cec216028cdf6fa5e735bb45aa7fa31475b5642aa22 
DEBU[0016] Downloading library/ubuntu/blobs/sha256:0c7b035e2a1aaecca607158b51b0513f8576f1ef0e4a3bcaef69d6072cba1072 
DEBU[0017] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:0c7b035e2a1aaecca607158b51b0513f8576f1ef0e4a3bcaef69d6072cba1072 
DEBU[0019] Detected compression format gzip             
DEBU[0019] Using original blob without modification     
DEBU[0019] Sending as tar file sha256:0c7b035e2a1aaecca607158b51b0513f8576f1ef0e4a3bcaef69d6072cba1072 
DEBU[0019] Downloading library/ubuntu/blobs/sha256:ac8ee255ff413234d75d0686fae93ca9390544bbb64a374ed277dcc1f15be4dc 
DEBU[0020] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:ac8ee255ff413234d75d0686fae93ca9390544bbb64a374ed277dcc1f15be4dc 
DEBU[0021] Detected compression format gzip             
DEBU[0021] Using original blob without modification     
DEBU[0021] Sending as tar file sha256:ac8ee255ff413234d75d0686fae93ca9390544bbb64a374ed277dcc1f15be4dc 
DEBU[0021] Downloading library/ubuntu/blobs/sha256:bf3d47be55f87e433152ca4725e1243bf7eec2bf7a5d4aaf91ed06e12a1395e7 
DEBU[0022] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:bf3d47be55f87e433152ca4725e1243bf7eec2bf7a5d4aaf91ed06e12a1395e7 
DEBU[0023] Detected compression format gzip             
DEBU[0023] Using original blob without modification     
DEBU[0023] Sending as tar file sha256:bf3d47be55f87e433152ca4725e1243bf7eec2bf7a5d4aaf91ed06e12a1395e7 
DEBU[0023] Downloading library/ubuntu/blobs/sha256:22a909724a97cb7f0226eb51558bae55a543e3c990349ee80cb436cc839ef475 
DEBU[0025] GET https://registry-1.docker.io/v2/library/ubuntu/blobs/sha256:22a909724a97cb7f0226eb51558bae55a543e3c990349ee80cb436cc839ef475 
DEBU[0026] Detected compression format gzip             
DEBU[0026] Using original blob without modification     
DEBU[0026] Sending as tar file sha256:22a909724a97cb7f0226eb51558bae55a543e3c990349ee80cb436cc839ef475 
DEBU[0026] Sending as tar file manifest.json            
DEBU[0026] docker-daemon: Closing tar stream            
DEBU[0026] docker-daemon: Waiting for status            
DEBU[0045] docker-daemon: sending done, status <nil>    
[cloud-user@atomic-host-001 atomic]$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
docker.io/ubuntu    latest              45bc58500fa3        4 days ago          126.8 MB



2. downstream

failed to pull ubuntu and registry.access.redhat.com/rhel7.2

[cloud-user@atomic-host-001 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux Atomic Host release 7.3

[cloud-user@atomic-host-001 ~]$ atomic host status
State: idle
Deployments:
● rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3 (2016-09-23 18:03:35)
        Commit: c1f924894e0601fbe6ac8f04409a686f0f31705aaf648fd096b1fdc08f13258d
        OSName: rhel-atomic-host
      Unlocked: development

  rhel-atomic-host-ostree:rhel-atomic-host/7/x86_64/standard
       Version: 7.3 (2016-09-20 14:44:34)
        Commit: 561537dc289445087debf17eb398f6190fb0674930fa04789d0844463b64c5e5
        OSName: rhel-atomic-host

[cloud-user@atomic-host-001 ~]$ rpm -q atomic docker skopeo oci-systemd-hook
atomic-1.12.3-2.el7.x86_64
docker-1.10.3-55.el7.x86_64
skopeo-0.1.14-0.6.el7.x86_64
oci-systemd-hook-0.1.4-6.git337078c.el7.x86_64

[cloud-user@atomic-host-001 ~]$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

[cloud-user@atomic-host-001 ~]$ sudo atomic --debug pull registry.access.redhat.com/rhel7.2
Image registry.access.redhat.com/rhel7.2 is being pulled to docker ...
Executing: skopeo --debug copy --remove-signatures docker://registry.access.redhat.com/rhel7.2 docker-daemon:registry.access.redhat.com/rhel7.2:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.access.redhat.com/rhel7.2:latest 
DEBU[0000] IsRunningImageAllowed for image docker:registry.access.redhat.com/rhel7.2:latest 
DEBU[0000]  Using default policy section                
DEBU[0000]  Requirement 0: allowed                      
DEBU[0000] Overall: allowed                             
DEBU[0001] Ping https://registry.access.redhat.com/v2/ err <nil> 
DEBU[0001] Ping https://registry.access.redhat.com/v2/ status 200 
DEBU[0001] GET https://registry.access.redhat.com/v2/rhel7.2/manifests/latest 
DEBU[0002] Downloading rhel7.2/blobs/sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 
DEBU[0002] GET https://registry.access.redhat.com/v2/rhel7.2/blobs/sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 
DEBU[0004] Detected compression format gzip             
DEBU[0004] Using original blob without modification     
DEBU[0004] docker-daemon: Closing tar stream to abort loading 
FATA[0004] Error writing blob: Can not stream blob sha256:30cf2e26a24f2a8426cbe8444f8af2ecb7023bd468b05c1b6fd0b2797b0f9ff9 with unknown size to docker-daemon: 

Traceback (most recent call last):
  File "/bin/atomic", line 186, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/pull.py", line 54, in pull_image
    handler()
  File "/usr/lib/python2.7/site-packages/Atomic/pull.py", line 42, in pull_docker_image
    skopeo_copy("docker://{}".format(self.args.image), image, debug=self.args.debug, insecure=insecure)
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 361, in skopeo_copy
    return check_call(cmd)
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 130, in check_call
    return subprocess.check_call(cmd, env=env, stdin=stdin, stderr=stderr, stdout=stdout, close_fds=True)
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['skopeo', '--debug', 'copy', '--remove-signatures', 'docker://registry.access.redhat.com/rhel7.2', 'docker-daemon:registry.access.redhat.com/rhel7.2:latest']' returned non-zero exit status 1

[cloud-user@atomic-host-001 ~]$ sudo atomic --debug pull ubuntu
Image ubuntu is being pulled to docker ...
[Errno -2] Name or service not known
Traceback (most recent call last):
  File "/bin/atomic", line 186, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/pull.py", line 54, in pull_image
    handler()
  File "/usr/lib/python2.7/site-packages/Atomic/pull.py", line 30, in pull_docker_image
    insecure = True if is_insecure_registry(self.d.info()['RegistryConfig'], strip_port(registry)) else False
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 680, in is_insecure_registry
    for r in get_ips_from_host(registry):
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 674, in get_ips_from_host
    return list(set([x[4][0] for x in socket.getaddrinfo(_host, None)]))
gaierror: [Errno -2] Name or service not known

Comment 12 Miloslav Trmač 2016-09-26 13:27:04 UTC
Yeah, this is an unimplemented case in skopeo. I’ll fix this, a skopeo respin will be needed I assume.

Comment 13 Brent Baude 2016-09-26 13:31:25 UTC
Assigning to Miloslav

Comment 19 Daniel Walsh 2016-10-18 15:15:36 UTC
runcom any update on this?

Comment 24 Daniel Walsh 2017-03-12 11:54:42 UTC
Has this bug been fixed?

Comment 25 Alex Jia 2017-03-13 07:02:27 UTC
(In reply to Daniel Walsh from comment #24)
> Has this bug been fixed?

It works well for me now.

[root@atomic-host-test cloud-user]# rpm -q atomic skopeo
atomic-1.15.4-2.el7.x86_64
skopeo-0.1.18-1.el7.x86_64