Bug 2030532

+++ This bug was initially created as a clone of Bug #2010497 +++

Description of problem (please be detailed as possible and provide log
snippests):

OSUS was deployed and is running properly. After OSUS builds the graph, on querying the graph, The release image digest for OCP version is not accurate to the real image digest.   

Version of all relevant components (if applicable):
OSUS 4.6

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Since the graph returns wrong release image digest, we cannot use the updater service to upgrade any of the OCP Clusters

Is there any workaround available to the best of your knowledge?
Nope

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

1

Can this issue reproducible?
No

Can this issue reproduce from the UI?
No

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Mirror openshift release images to a private jfrog artifactory registry
2. Install OSUS operator and point to artifactory
3. Query the graph to check the payload


Actual results:
release image digest for OCP release version should be the same as found in quay or in production OSUS service

For eg, for OCP-4.7.9. Expected release image should be sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458

Expected results:

payload for OCP release version is a random string
For Ocp-4.7.9 OSUS release image digest found is sha256:d5639f376e94842e8ade89b9743d4ba6104db38562b8816f48b551a9e7503384


Additional info:

--- Additional comment from Harshil on 2021-10-04 19:51:43 UTC ---



--- Additional comment from W. Trevor King on 2021-10-05 22:05:04 UTC ---

Cincinnati uses dkregistry to interact with image registries.  To scrape releases there are two steps:

1. Paginated ${REGISTRY}/v2/${NAMESPACE}/tags/list calls, to get a list of tags [1,2].  For example:

    $ curl -s https://quay.io/v2/openshift-release-dev/ocp-release/tags/list | jq . | head -n5
    {
      "name": "openshift-release-dev/ocp-release",
      "tags": [
        "4.0.0-2",
        "4.0.0-3",

2. For each tag, ${REGISTRY}/v2/${NAMESPACE}/manifests/${TAG} [3,4,5,6] with application/vnd.docker.distribution.manifest.v2+json as the preferred media type [7,8].  The digest comes from a docker-content-digest header [9].  For example:

    $ curl -iH Accept:application/vnd.docker.distribution.manifest.v2+json https://quay.io/v2/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64
    HTTP/2 200 
    ...
    content-type: application/vnd.docker.distribution.manifest.v2+json
    content-length: 1582
    docker-content-digest: sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458
    ...

    {
       "schemaVersion": 2,
       "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    ...

  You can confirm that digest locally:

    $ curl -sH Accept:application/vnd.docker.distribution.manifest.v2+json https://quay.io/v2/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64 | sha256sum 
    dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458  -

So to debug, take the troublesome tag, try to fetch the manifest, check the hash of the response, confirm that it matches the digest docker-content-digest was claiming, and, if it does match, diff that manifest JSON vs. the canonical, signed Quay manifest for that release.

[1]: https://github.com/openshift/cincinnati/blob/915cdf1440c3d7801a5abe71cecdadb2c5901bff/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/registry/mod.rs#L254
[2]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/v2/tags.rs#L26
[3]: https://github.com/openshift/cincinnati/blob/915cdf1440c3d7801a5abe71cecdadb2c5901bff/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/registry/mod.rs#L273
[4]: https://github.com/openshift/cincinnati/blob/915cdf1440c3d7801a5abe71cecdadb2c5901bff/cincinnati/src/plugins/internal/graph_builder/release_scrape_dockerv2/registry/mod.rs#L454
[5]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/v2/manifest/mod.rs#L35-L42
[6]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/v2/manifest/mod.rs#L99
[7]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/v2/manifest/mod.rs#L262-L263
[8]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/mediatypes.rs#L26-L29
[9]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/v2/manifest/mod.rs#L56

--- Additional comment from Harshil on 2021-10-08 15:23:59 UTC ---

Here is the debug I ran against the registry. The digests match. From comparing with quay the manifest layers are the same except the header which contains artifactory information instead of quay.

Artifactory output:

$ curl -u<redacted-username> -iH Accept:application/vnd.docker.distribution.manifest.v2+json https://<redacted-private-reg>/v2/<redacted-reg-namespace>/manifests/4.7.9-x86_64
Enter host password for user '<redacted-username>':
HTTP/1.1 200 OK
Server: Artifactory/6.23.21
X-Artifactory-Id: <redacted>
X-Artifactory-Node-Id: <redacted>
Last-Modified: Fri, 07 May 2021 16:13:12 GMT
ETag: <redacted>
X-Checksum-Sha1: 19b6e4cd6dc0e670291cc3631206f0e374bbb094
X-Checksum-Sha256: dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458
X-Checksum-Md5: 2309d78d13b5dcc0c6ca5bffbce3b943
Accept-Ranges: bytes
X-Artifactory-Filename: manifest.json
Content-Disposition: attachment; filename="manifest.json"
Cache-Control: no-store
Docker-Distribution-Api-Version: registry/2.0
Docker-Content-Digest: sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 1582
Date: Fri, 08 Oct 2021 14:59:03 GMT
Set-Cookie: bac_persist=2154334123.37151.0000; path=/; Httponly; Secure

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1716,
      "digest": "sha256:f695757e19f48da83a3e5617bc36ad60dea04738de40534284534532e4a45865"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 75513266,
         "digest": "sha256:13897c84ca5715a68feafcce9acf779f35806f42d1fcd37e8a2a5706c075252d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1790,
         "digest": "sha256:64607cc74f9cbe0e12f167547df0cf661de5a8b1fb4ebe930a43b9f621ca457f"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 6907843,
         "digest": "sha256:b71d7392cc674aa31313364ee2df6495f46575f461ab215deb4e70178dc0dc37"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 10988272,
         "digest": "sha256:a4ee19afd3e9f8e69d645eeda9d6694a76057d873cf574d2ad0760c5f6a54967"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 21016581,
         "digest": "sha256:f80139d0b060fec070c4ce1dab75a4793d0cc3efe52a33db5fd4386a2014375c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 494465,
         "digest": "sha256:c2881dff292ab3159d42a346cdd76fbcb89d0361b8a35915d9254e2dc7f8a9c9"
      }
   ]
}
$ curl  -u<redacted-username> -sH Accept:application/vnd.docker.distribution.manifest.v2+json  https://<redacted-private-reg>/v2/<redacted-reg-namespace>/manifests/4.7.9-x86_64 | sha256sum
Enter host password for user '<redacted-username>':
dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458  -

QUAY Output:

HTTP/1.1 200 OK
Server: nginx/1.12.1
Date: Fri, 08 Oct 2021 15:16:30 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 1582
Connection: close
Docker-Content-Digest: sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458
X-Frame-Options: DENY
Strict-Transport-Security: max-age=63072000; preload

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1716,
      "digest": "sha256:f695757e19f48da83a3e5617bc36ad60dea04738de40534284534532e4a45865"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 75513266,
         "digest": "sha256:13897c84ca5715a68feafcce9acf779f35806f42d1fcd37e8a2a5706c075252d"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1790,
         "digest": "sha256:64607cc74f9cbe0e12f167547df0cf661de5a8b1fb4ebe930a43b9f621ca457f"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 6907843,
         "digest": "sha256:b71d7392cc674aa31313364ee2df6495f46575f461ab215deb4e70178dc0dc37"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 10988272,
         "digest": "sha256:a4ee19afd3e9f8e69d645eeda9d6694a76057d873cf574d2ad0760c5f6a54967"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 21016581,
         "digest": "sha256:f80139d0b060fec070c4ce1dab75a4793d0cc3efe52a33db5fd4386a2014375c"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 494465,
         "digest": "sha256:c2881dff292ab3159d42a346cdd76fbcb89d0361b8a35915d9254e2dc7f8a9c9"
      }
   ]



However, when I am going through OSUS to get the release digest for 4.7.9 tag, The digest does not match:
$ curl --header Accept:application/json https://sample-policy-engine-route-openshift-update-service.apps.<redacted-domain>/api/upgrades_info/v1/graph?channel=stable-4.7 -k | jq
    {
      "version": "4.7.9",
      "payload": "<redacted-private-reg>/<redacted-reg-namespace>@sha256:d5639f376e94842e8ade89b9743d4ba6104db38562b8816f48b551a9e7503384",
      "metadata": {
        "description": "",
        "io.openshift.upgrades.graph.previous.remove_regex": "4\\.6\\..*",
        "io.openshift.upgrades.graph.release.channels": "candidate-4.7,fast-4.7,stable-4.7,candidate-4.8,fast-4.8",
        "io.openshift.upgrades.graph.release.manifestref": "sha256:d5639f376e94842e8ade89b9743d4ba6104db38562b8816f48b551a9e7503384",
        "url": "https://access.redhat.com/errata/RHBA-2021:1365"
      }
    },

--- Additional comment from W. Trevor King on 2021-10-08 21:02:38 UTC ---

(In reply to Harshil from comment #3)
> From comparing with quay the manifest layers are the same except the header which contains artifactory information instead of quay.

That shouldn't affect the hash, which is just over the body content.

> $ curl -u<redacted-username> -iH
> Accept:application/vnd.docker.distribution.manifest.v2+json
> https://<redacted-private-reg>/v2/<redacted-reg-namespace>/manifests/4.7.9-
> x86_64
> ...
> Docker-Content-Digest:
> sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458

So it's claiming the correct digest.  Good.

> $ curl  -u<redacted-username> -sH
> Accept:application/vnd.docker.distribution.manifest.v2+json 
> https://<redacted-private-reg>/v2/<redacted-reg-namespace>/manifests/4.7.9-
> x86_64 | sha256sum
> Enter host password for user '<redacted-username>':
> dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458  -

And it's not lying when it makes that claim.  Also good.

> $ curl --header Accept:application/json
> https://sample-policy-engine-route-openshift-update-service.apps.<redacted-
> domain>/api/upgrades_info/v1/graph?channel=stable-4.7 -k | jq
>     {
>       "version": "4.7.9",
>       "payload":
> "<redacted-private-reg>/<redacted-reg-namespace>@sha256:
> d5639f376e94842e8ade89b9743d4ba6104db38562b8816f48b551a9e7503384",

Let's try building out the dkregistry/Cincinnati accept header in more detail, to see if we can reproduce the d563.  What do you get from:

  $ curl -u<redacted-username> -iH 'Accept: application/vnd.docker.distribution.manifest.v2+json; q=0.5,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4' https://<redacted-private-reg>/v2/<redacted-reg-namespace>/manifests/4.7.9-x86_64

Checking Quay:

  $ curl -iH 'Accept: application/vnd.docker.distribution.manifest.v2+json; q=0.5,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4' https://quay.io/v2/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64
  HTTP/2 200 
  server: nginx/1.12.1
  date: Fri, 08 Oct 2021 21:01:33 GMT
  content-type: application/vnd.docker.distribution.manifest.v2+json
  content-length: 1582
  docker-content-digest: sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458
  ...
  $ curl -sH 'Accept: application/vnd.docker.distribution.manifest.v2+json; q=0.5,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4' https://quay.io/v2/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64 | sha256sum 
  dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458  -

[1]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/mediatypes.rs#L19
[2]: https://github.com/camallo/dkregistry-rs/blob/854d0da53bef5dd85b5e901123e85d43af97c74e/src/v2/manifest/mod.rs#L259-L282

--- Additional comment from Harshil on 2021-10-11 18:34:08 UTC ---

Yup got the d563 sha. What does this mean?

$ curl -u<redacted-username> -iH 'Accept: application/vnd.docker.distribution.manifest.v2+json; q=0.5,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4' https://<redacted-private-reg>>/v2/<redacted-reg-namespace>/manifests/4.7.9-x86_64
HTTP/1.1 200 OK
Server: Artifactory/6.23.21
X-Artifactory-Id: b040e7c3bbfcc986efb9748af0e8adb59e0c7428
X-Artifactory-Node-Id: <redacted-artifactory-node>
Docker-Distribution-Api-Version: registry/2.0
Docker-Content-Digest: sha256:d5639f376e94842e8ade89b9743d4ba6104db38562b8816f48b551a9e7503384
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Transfer-Encoding: chunked
Date: Mon, 11 Oct 2021 18:24:42 GMT
Set-Cookie: bac_persist=2154334123.37151.0000; path=/; Httponly; Secure

{
  "schemaVersion" : 1,
  "name" : "<redacted-reg-namespace>",
  "tag" : "4.7.9-x86_64",
  "architecture" : "amd64",
  "fsLayers" : [ {
    "blobSum" : "sha256:c2881dff292ab3159d42a346cdd76fbcb89d0361b8a35915d9254e2dc7f8a9c9"
  }, {
    "blobSum" : "sha256:f80139d0b060fec070c4ce1dab75a4793d0cc3efe52a33db5fd4386a2014375c"
  }, {
    "blobSum" : "sha256:a4ee19afd3e9f8e69d645eeda9d6694a76057d873cf574d2ad0760c5f6a54967"
  }, {
    "blobSum" : "sha256:b71d7392cc674aa31313364ee2df6495f46575f461ab215deb4e70178dc0dc37"
  }, {
    "blobSum" : "sha256:64607cc74f9cbe0e12f167547df0cf661de5a8b1fb4ebe930a43b9f621ca457f"
  }, {
    "blobSum" : "sha256:13897c84ca5715a68feafcce9acf779f35806f42d1fcd37e8a2a5706c075252d"
  } ],
  "history" : [ {
    "v1Compatibility" : "{\"parent\":\"372ee24e20bb9f8ed7c3062cf24ddf7361b6ff0a930770abb2d53c948e69ca62\",\"size\":114922217,\"os\":\"linux\",\"created\":\"2021-04-28T08:35:07Z\",\"container_config\":{},\"id\":\"38b70dd70a87222dfc4c6b552179f4db138ac54071a25274a86332e6444a977d\",\"docker_version\":\"1.13.1\",\"config\":{\"Hostname\":\"8f676bfe19a0\",\"Env\":[\"__doozer=merge\",\"BUILD_RELEASE=202104250659.p0\",\"BUILD_VERSION=v4.7.0\",\"OS_GIT_MAJOR=4\",\"OS_GIT_MINOR=7\",\"OS_GIT_PATCH=0\",\"OS_GIT_TREE_STATE=clean\",\"OS_GIT_VERSION=4.7.0-202104250659.p0-1db8867\",\"SOURCE_GIT_TREE_STATE=clean\",\"OS_GIT_COMMIT=1db8867\",\"SOURCE_DATE_EPOCH=1618050033\",\"SOURCE_GIT_COMMIT=1db886781f4e1f18965e60d89da80ec1d677228b\",\"SOURCE_GIT_TAG=v1.0.0-456-g1db8867\",\"SOURCE_GIT_URL=https://github.com/openshift/cluster-version-operator\",\"GODEBUG=x509ignoreCN=0,madvdontneed=1\",\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\"],\"Entrypoint\":[\"/usr/bin/cluster-version-operator\"],\"Labels\":{\"io.openshift.release\":\"4.7.9\",\"io.openshift.release.base-image-digest\":\"sha256:794475eaef2f086b6b618ac12fa5f215430cd21cb9c3134d5fc3da2dae47ac87\"}},\"architecture\":\"amd64\"}"
  }, {
    "v1Compatibility" : "{\"id\":\"372ee24e20bb9f8ed7c3062cf24ddf7361b6ff0a930770abb2d53c948e69ca62\",\"parent\":\"b2c3476de6748648b4360f9a12868076621a194f71af7bfa590d3a4cf8be4dee\",\"comment\":null,\"created\":\"2021-04-28T08:35:07Z\",\"throwaway\":false,\"container_config\":{\"Cmd\":[null]}}"
  }, {
    "v1Compatibility" : "{\"id\":\"b2c3476de6748648b4360f9a12868076621a194f71af7bfa590d3a4cf8be4dee\",\"parent\":\"405cda2f4d063ece70a01cb499a5a7082b9d2f66263d9b24ad0a078290c86779\",\"comment\":null,\"created\":\"2021-04-28T08:35:07Z\",\"throwaway\":false,\"container_config\":{\"Cmd\":[null]}}"
  }, {
    "v1Compatibility" : "{\"id\":\"405cda2f4d063ece70a01cb499a5a7082b9d2f66263d9b24ad0a078290c86779\",\"parent\":\"2bf0a4aa6e1f052a18de26e5f489c1cdf7b589099a6eaa8841a98af290aa1dea\",\"comment\":null,\"created\":\"2021-04-28T08:35:07Z\",\"throwaway\":false,\"container_config\":{\"Cmd\":[null]}}"
  }, {
    "v1Compatibility" : "{\"id\":\"2bf0a4aa6e1f052a18de26e5f489c1cdf7b589099a6eaa8841a98af290aa1dea\",\"parent\":\"b6d0d0b73a63bcdbcd92b688372bc9be77f370b741519f46a2e6c6441f6310a2\",\"comment\":null,\"created\":\"2021-04-28T08:35:07Z\",\"throwaway\":false,\"container_config\":{\"Cmd\":[null]}}"
  }, {
    "v1Compatibility" : "{\"id\":\"b6d0d0b73a63bcdbcd92b688372bc9be77f370b741519f46a2e6c6441f6310a2\",\"parent\":\"\",\"comment\":\"Release image for OpenShift\",\"created\":\"2021-04-28T08:35:07Z\",\"throwaway\":false,\"container_config\":{\"Cmd\":[null]}}"
  } ],
  "signatures" : [ {
    "header" : {
      "jwk" : {
        "crv" : "P-256",
        "kty" : "EC",
        "x" : "Re0xaSvngBU1LrbgVBMi_PceN9AhSiAHE1Nl8wKd5lA",
        "y" : "t6H4FQ3Y-YVhjvnGatTZeVbZFvR2qczou0mURZAxGmY"
      },
      "alg" : "ES256"
    },
    "signature" : "_TowVYRHIJS7BlcdBadIyz0ywDvB66_WhNWjCL6eLB4mCT2t7IT4NmKOQ0XMe-MCFmXzGzinu6Eg2I2IXgvcSw",
    "protected" : "eyJmb3JtYXRMZW5ndGgiOjM0NDksImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAyMS0xMC0xMVQyOjI0OjQzLTA0MDAifQ"
  } ]
}

--- Additional comment from Harshil on 2021-10-11 18:38:55 UTC ---

I see SchemaVersion: 1 for dkregistry?


{
  "schemaVersion" : 1,
  "name" : "<redacted-reg-namespace>",
  "tag" : "4.7.9-x86_64",
  "architecture" : "amd64",
  "fsLayers" : [ {
    "blobSum" : "sha256:c2881dff292ab3159d42a346cdd76fbcb89d0361b8a35915d9254e2dc7f8a9c9"
  }, {
    "blobSum" : "sha256:f80139d0b060fec070c4ce1dab75a4793d0cc3efe52a33db5fd4386a2014375c"
  }, {
    "blobSum" : "sha256:a4ee19afd3e9f8e69d645eeda9d6694a76057d873cf574d2ad0760c5f6a54967"
  }, {
    "blobSum" : "sha256:b71d7392cc674aa31313364ee2df6495f46575f461ab215deb4e70178dc0dc37"
  }, {
    "blobSum" : "sha256:64607cc74f9cbe0e12f167547df0cf661de5a8b1fb4ebe930a43b9f621ca457f"
  }, {
    "blobSum" : "sha256:13897c84ca5715a68feafcce9acf779f35806f42d1fcd37e8a2a5706c075252d"
  } ],

--- Additional comment from W. Trevor King on 2021-10-11 22:53:47 UTC ---

It means Artifactory 6.23.21 is not HTTP Accept header spec [1] and its quality values [2].  Cincinnati is setting:

  application/vnd.docker.distribution.manifest.v2+json; q=0.5,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4

And [2] says:

  The weight is normalized to a real number in the range 0 through 1...

So that's "I prefer v2, but send v1 if it is the best available after an 80% (0.4/0.5 * 100) markdown in quality".  The manifest uploaded to Artifactory was v2, so Artifactory should be able to return that v2 content without degradation.  But perhaps there is some Artifactory config knob that says "we are twice as good at serving v1 than we are at serving v2" which convinces the registry that v1 would still work out better for the caller (small client-side quality hit) than v1 (large server-side quality hit)?

Also, seems like d563... should work, except for the fact that there won't be a Red Hat signature on the manifest, because the source of that v1 manifest is Artifactory internally converting from the canonical-for-this-image v2 manifest.

[1]: https://datatracker.ietf.org/doc/html/rfc7231#section-5.3.2
[2]: https://datatracker.ietf.org/doc/html/rfc7231#section-5.3.1

--- Additional comment from Harshil on 2021-10-13 16:32:09 UTC ---

Right but OSUS will not work without the signature right? So we need this to work the way we expect it to be.

So next step would be to see if there is some setting in artifactory that we do to force and send responses only on v2 despite it not being the best available after a markdown in quality? 

Is my understanding correct here?

--- Additional comment from W. Trevor King on 2021-10-14 02:13:09 UTC ---

> Right but OSUS will not work without the signature right?

The update-service doesn't care about signatures.  But yeah, the cluster consuming the update service will be sad about the missing signature for the pullspec the update service recommends, and you'd need to force through that lack-of-sig, which is risky.

> So next step would be to see if there is some setting in artifactory...

You shouldn't have to force it, but yeah, digging into why Artifactory prefers v1 despite the client saying it prefers v2 is the next step.

--- Additional comment from W. Trevor King on 2021-10-20 16:46:33 UTC ---



--- Additional comment from W. Trevor King on 2021-10-21 21:40:22 UTC ---

Harshil was kind enough to set up a test Artifactory instance, and here are some test results.

Reminder of the canonical digest:

  $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.7.9-x86_64 | grep Digest
  Digest:         sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458

Requests as Cincinnati makes them today, showing Artifactory returning v1 content (with an altered digest that has no matching Red Hat signature), despite us saying we prefer v2:

  $ curl -u ... -isH 'Accept: application/vnd.docker.distribution.manifest.v2+json; q=0.5,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4' https://...jfrog.io/v2/default-docker-loc
al/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64 | grep 'Docker-Content-Digest\|Content-Type'
  Enter host password for user '...':
  Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
  Docker-Content-Digest: sha256:36bb12d1e74b626a1b3212c1ad162f4a56a129b46181c54d3f4cd56f0ac117dd
  X-Jfrog-Version: Artifactory/7.26.3 72603900

Dropping q from our v2 entry, but keeping it for v1:

  $ curl -u ... -isH 'Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4' https://...jfrog.io/v2/default-docker-local/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64 | grep 'Docker-Content-Digest\|Content-Type'
  Enter host password for user '...':
  Content-Type: application/vnd.docker.distribution.manifest.v2+json
  Docker-Content-Digest: sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458

So that would work.  And it's a sane approach for dkregistry, because it also says "I prefer v2, but will accept v1 with a markdown in quality".  Which is what we're already saying, but Artifactory is failing to understand our current phrasing.

Confirming that Artifactory is not doing some clever attempt at quality markdown calculation, here's me with a semantically equivalent ask that explicitly sets the default q [1]:

  $ curl -u ... -isH 'Accept: application/vnd.docker.distribution.manifest.v2+json; q=1,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.4' https://...jfrog.io/v2/default-docker-local
/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64 | grep 'Docker-Content-Digest\|Content-Type'
  Enter host password for user '...':
  Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
  Docker-Content-Digest: sha256:36bb12d1e74b626a1b3212c1ad162f4a56a129b46181c54d3f4cd56f0ac117dd

And to really pin it down, here's a completely valid request for "I love v2", where Artifactory says "huh?  Here's some v1":

  $ curl -u ... -isH 'Accept: application/vnd.docker.distribution.manifest.v2+json; q=1' https://...jfrog.io/v2/default-docker-local/openshift-release-dev/ocp-release/manifests/4.7.9-x86_64 | grep
 'Docker-Content-Digest\|Content-Type'
  Enter host password for user '...':
  Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
  Docker-Content-Digest: sha256:36bb12d1e74b626a1b3212c1ad162f4a56a129b46181c54d3f4cd56f0ac117dd

I hunted around briefly, but could not find Artifactory source code.  But my take based on the above testing is:

* Artifactory/7.26.3 72603900 supports ',' splitting between multiple Accept value entries, which is good and compliant with the spec [2].
* Artifactory/7.26.3 72603900 does not support the weight ('q') [1], or possibly any of the accept-params allowed by the spec [2].

Looks like Artifactory is up to 7.27 now [3]; perhaps they have fixed their Accept parser in more recent versions.  If not, it's probably worth having someone with an Artifactory subscription file a ticket with them so they're aware of the issue and can prioritize addressing it.

And in the meantime, we can probably work around Artifactory's limited Accept parser by having dkregistry adjust the requests to use implicit quality for the v2 media type.

[1]: https://datatracker.ietf.org/doc/html/rfc7231#section-5.3.1
[2]: https://datatracker.ietf.org/doc/html/rfc7231#section-5.3.2
[3]: https://www.jfrog.com/confluence/display/JFROG/Artifactory+Release+Notes#ArtifactoryReleaseNotes-Artifactory7.27

--- Additional comment from W. Trevor King on 2021-10-21 21:46:03 UTC ---

Also, just to be extra safe, here's confirmation that the approach I expect to take in dkregistry will work:

  $ curl -u ... -isH 'Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+prettyjws; q=0.8' https://...jfrog.io/v2/default-docker-local/ope
nshift-release-dev/ocp-release/manifests/4.7.9-x86_64 | grep 'Docker-Content-Digest\|Content-Type'
  Enter host password for user '...':
  Content-Type: application/vnd.docker.distribution.manifest.v2+json
  Docker-Content-Digest: sha256:dd75546170e65d7d17130de10a6ffeb425f960399640632cbc8426b9da338458

That's semantically equivalent to our current 0.5 vs. 0.4 weights, but leaving the weight on our favorite implicit to avoid confusing Artifactory's limited, weight-averse parser.

--- Additional comment from Vadim Rutkovsky on 2021-12-01 12:25:58 UTC ---

Created https://github.com/camallo/dkregistry-rs/pull/212 to support custom accept headers in dk_registry. In Cincinnati we might want to redefine that to use q=0.8 or even expose this setting as a plugin setting