Bug 1389561

Summary: Skopeo and atomic not using same sha256 for image signing/verification
Product: Red Hat Enterprise Linux 7 Reporter: David Darrah/Red Hat QE <ddarrah>
Component: atomicAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED CURRENTRELEASE QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.3CC: bbaude, ddarrah, mitr, pthomas
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-05 14:30:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description David Darrah/Red Hat QE 2016-10-27 21:32:40 UTC
Description of problem:
When signing a remote image with atomic (e.g. on docker.io), a different sha256 hash is generated than when pulling and verifying,

Version-Release number of selected component (if applicable):
atomic 1.13.1
skopeo 0.1.17

How reproducible:
100%

Steps to Reproduce:
Setup:
local http server running in a container mounting /var/lib/atomic/sigstore into htdocs


[root@rhel-73 registries.d]# atomic --debug sign --sign-by ddarrah docker.io/library/busybox:latest
Namespace(_class=<class 'Atomic.sign.Sign'>, assumeyes=False, debug=True, func='sign', gnupghome='/root/.gnupg', images=['docker.io/library/busybox:latest'], sign_by='ddarrah', signature_path=None)
[
    {
        "search": true,
        "hostname": "registry-1.docker.io",
        "name": "docker.io",
        "secure": true
    },
    {
        "search": true,
        "hostname": "localhost:5000",
        "name": "localhost:5000",
        "secure": false
    },
    {
        "search": true,
        "hostname": "registry.access.stage.redhat.com",
        "name": "registry.access.stage.redhat.com",
        "secure": false
    },
    {
        "search": true,
        "hostname": "registry.access.redhat.com",
        "name": "registry.access.redhat.com",
        "secure": true
    },
    {
        "search": true,
        "hostname": "192.168.124.215:5000",
        "name": "192.168.124.215:5000",
        "secure": false
    },
    {
        "search": true,
        "hostname": "atomic-registry.usersys.redhat.com:5000",
        "name": "atomic-registry.usersys.redhat.com:5000",
        "secure": false
    }
]
Created: /var/lib/atomic/sigstore/docker.io/library/busybox:latest@sha256:74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b/signature-6
[root@rhel-73 registries.d]# curl http://192.168.124.215/sigstore/docker.io/library/busybox:latest@sha256:74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /sigstore/docker.io/library/busybox:latest@sha256:74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b</title>
 </head>
 <body>
<h1>Index of /sigstore/docker.io/library/busybox:latest@sha256:74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b</h1>
<ul><li><a href="/sigstore/docker.io/library/"> Parent Directory</a></li>
<li><a href="signature-1"> signature-1</a></li>
<li><a href="signature-2"> signature-2</a></li>
<li><a href="signature-3"> signature-3</a></li>
<li><a href="signature-4"> signature-4</a></li>
<li><a href="signature-5"> signature-5</a></li>
<li><a href="signature-6"> signature-6</a></li>
</ul>
</body></html>
[root@rhel-73 registries.d]# cat docker.io.yaml 
docker:
  docker.io:
    sigstore: http://192.168.124.215/sigstore/
    sigstore-staging: file:///var/lib/atomic/sigstore
[root@rhel-73 registries.d]# cd ..
[root@rhel-73 containers]# cat policy.json 
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ], 
    "transports": {
        "docker": {
            "docker.io": [
                {
                    "keyType": "GPGKeys", 
                    "type": "signedBy", 
                    "keyPath": "/etc/containers/ddarrah.key"
                }
            ]
        }, 
        "atomic": {
            "192.168.124.215:5000": [
                {
                    "keyType": "GPGKeys", 
                    "type": "signedBy", 
                    "keyPath": "/etc/containers/ddarrah.key"
                }
            ]
        }, 
        "dir": {}
    }
}[root@rhel-73 containers]#atomic --debug pull docker.io/library/busybox
Image docker.io/library/busybox is being pulled to docker ...
+++++++++++++++++++++++++++++++++++++++++++docker.io
+++++++++++++++++++++++++++++docker.io
+++++++++++++++++++++++++++++localhost
+++++++++++++++++++++++++++++registry.access.stage.redhat.com
+++++++++++++++++++++++++++++atomic-registry.usersys.redhat.com
Pulling docker.io/library/busybox:latest ...
Executing: /usr/bin/skopeo --debug copy --remove-signatures docker://docker.io/library/busybox:latest docker-daemon:docker.io/library/busybox:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "docker" namespace docker.io          
DEBU[0000]   Using http://192.168.124.215/sigstore/     
DEBU[0000] IsRunningImageAllowed for image docker:docker.io/library/busybox:latest 
DEBU[0000]  Using transport "docker" specific policy section docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ err <nil> 
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://registry-1.docker.io/v2/library/busybox/manifests/latest 
DEBU[0000] GET http://192.168.124.215/sigstore//docker.io/library/busybox@sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912/signature-1 
DEBU[0000] Requirement 0: denied, done                  
DEBU[0000] docker-daemon: Closing tar stream to abort loading 
FATA[0000] Source image rejected: A signature was required, but no signature exists 

Traceback (most recent call last):
  File "/bin/atomic", line 186, in <module>
    sys.exit(_func())
  File "/usr/lib/python2.7/site-packages/Atomic/pull.py", line 65, in pull_image
    handler()
  File "/usr/lib/python2.7/site-packages/Atomic/pull.py", line 53, in pull_docker_image
    policy_filename=self.policy_filename)
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 405, in skopeo_copy
    return check_call(cmd)
  File "/usr/lib/python2.7/site-packages/Atomic/util.py", line 170, in check_call
    return subprocess.check_call(cmd, env=env, stdin=stdin, stderr=stderr, stdout=stdout, close_fds=True)
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['/usr/bin/skopeo', '--debug', 'copy', '--remove-signatures', 'docker://docker.io/library/busybox:latest', 'docker-daemon:docker.io/library/busybox:latest']' returned non-zero exit status 1
[root@rhel-73 containers]#

Actual results:
Pull fails signature check.

Expected results:
Signed image should pull successfully

Additional info:

Comment 2 Brent Baude 2016-10-27 22:09:13 UTC
I think there are actually two problems.  Firstly, atomic is generating the wrong digest for the manifest and skopeo omits the tag from the the location it looks for.  My understanding was the tag is required.  If not, then they are both in atomic.

The fix for the incorrect digest is -> https://github.com/projectatomic/atomic/pull/737

Comment 3 David Darrah/Red Hat QE 2016-10-27 22:29:33 UTC
With the changes from comment2, I am still seeing a different digest from atomic than skopeo

atomic sign produces
Created: /var/lib/atomic/sigstore/docker.io/library/busybox:latest@sha256:1166b2969d8c3916f290c82773934337bbac373866c177a2a0bd3e2b313b1501/signature-1

skopeo inspect produces
"Digest": "sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912",

Comment 4 Brent Baude 2016-10-28 01:47:34 UTC
@David,

Mine is correct.  Example:

[bbaude@localhost atomic (digest)]$ sudo ./atomic sign docker.io/library/busybox:latest
Created: /var/lib/atomic/sigstore/docker.io/library/busybox:latest@sha256:29f5d56d12684887bdfa50dcd29fc31eea4aaf4ad3bec43daf19026a7ce69912/signature-3
[bbaude@localhost atomic (digest)]$ git log | head -n 3
commit 6665617ceed6c54b36dec9a203190fcd61db8907
Author: Brent Baude <bbaude>
Date:   Thu Oct 27 17:06:23 2016 -0500


Can you attach the log?

Comment 5 David Darrah/Red Hat QE 2016-11-15 17:41:57 UTC
fixed in atomic atomic-1.13.8-1.el7.x86_64.rpm