Thanks for reaching out!

I fear there is nothing Podman can do.  As mentioned in https://access.redhat.com/articles/4301321, it's up to the individual users to find the tags they want to pull.

Podman has no way to know which tag is the last one. A registry can only list all tags for a given repository/image but this data does not include the date or order. We could hypothetically inspect each of these images one by one and compare the time stamps when they have been created but these time stamps are not a reliable source either.  Oftentimes time stamps are not set at all or wrong.

Note: the displayed error is returned by the registry.

I concur with Valentin about this NOT being an issue with Podman.  However, I am going to assign this to the Documentation team as I believe there are touch-ups to be made in the document noted in the description to make the tags requirement more clear in that particular example.

Gabriela, do you know if the "Documentation" component is the right component to assign this to?

Satyajit,

In case it's helpful, this Podman command will show you the available tags in the repository, if you're using Podman v3.0 or later:

# podman search --list-tags registry.redhat.io/amq7/amq-streams-bridge-rhel7
NAME                                              TAG
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.6.2-1
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.4.1
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.5.0-2.1594897724
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.4.0
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.3.0-10.1579648804
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.5.0-2
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.6.0-2
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.6.0-1
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.5.0
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.6.2
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.6.0
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.3.0
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.5.0-2.1604501257
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.3.0-12
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.4.1-2
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.5.0-2.1594640710
registry.redhat.io/amq7/amq-streams-bridge-rhel7  1.4.0-8
registry.redhat.io/amq7/amq-streams-bridge-rhel7  latest

I don't believe RHEL documentation is the right place for this.  The issue is related to the AMQ product/container images, though I'm not convinced that there is a doc issue here at all.  Both skopeo and podman report that the registry.redhat.io/amq7/amq-streams-bridge-rhel7 image repository has a "latest" tag.  The reported issue here is actually that RH Satellite tries to sync registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest and this fails, because, as we already know, there is no image tagged latest.  Do we know if podman/skopeo assumes that "latest" exists, or maybe the issue is that the registry reports that "latest" exists?  

Here's what skopeo is doing:

# skopeo inspect --debug --raw docker://registry.redhat.io/amq7/amq-streams-bridge-rhel7
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Trying to access "registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest" 
DEBU[0000] Returning credentials from /run/user/0/containers/auth.json 
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io 
DEBU[0000]  cert: /etc/docker/certs.d/registry.redhat.io/4652001432504135603.cert 
DEBU[0000]  key: /etc/docker/certs.d/registry.redhat.io/4652001432504135603.key 
DEBU[0000] GET https://registry.redhat.io/v2/           
DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 
DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=rhn-support-dornelas&scope=repository%3Aamq7%2Famq-streams-bridge-rhel7%3Apull&service=docker-registry 
DEBU[0001] GET https://registry.redhat.io/v2/amq7/amq-streams-bridge-rhel7/manifests/latest 
DEBU[0002] Content-Type from manifest GET is "application/json" 
DEBU[0002] Accessing "registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest" failed: Error reading manifest latest in registry.redhat.io/amq7/amq-streams-bridge-rhel7: unsupported: This repository does not use the "latest" tag to track the most recent image and must be pulled with an explicit version or image reference. For more information, see: https://access.redhat.com/articles/4301321 
FATA[0002] Error parsing image name "docker://registry.redhat.io/amq7/amq-streams-bridge-rhel7": Error reading manifest latest in registry.redhat.io/amq7/amq-streams-bridge-rhel7: unsupported: This repository does not use the "latest" tag to track the most recent image and must be pulled with an explicit version or image reference. For more information, see: https://access.redhat.com/articles/4301321 


From the output I believe it assumes that "latest" exists, as I don't see it ask for the list of tags.  I've tested it, and this causes 'skopeo sync --src docker --dest dir registry.redhat.io/amq7/amq-streams-bridge-rhel7 /tmp/test' to eventually fail to sync the repo. 

Grabbing the tags list manually:

# TOKEN=$(curl -sL -u rhn-support-dornelas "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull" | jq -r '.access_token')
Enter host password for user 'rhn-support-dornelas':

# curl -sL -H "Authorization: Bearer $TOKEN"  "https://registry.redhat.io/v2/amq7/amq-streams-bridge-rhel7/tags/list" | jq .
{
  "name": "amq7/amq-streams-bridge-rhel7",
  "tags": [
    "1.6.2-1",
    "1.4.1",
    "1.5.0-2.1594897724",
    "1.4.0",
    "1.3.0-10.1579648804",
    "1.5.0-2",
    "1.6.0-2",
    "1.6.0-1",
    "1.5.0",
    "1.6.2",
    "1.6.0",
    "1.3.0",
    "1.5.0-2.1604501257",
    "1.3.0-12",
    "1.4.1-2",
    "1.5.0-2.1594640710",
    "1.4.0-8",
    "latest"
  ]
}


So, the registry is reporting that "latest" exists in the repo.  

@sadas can you tell us whether you're trying to sync the entire registry.redhat.io/amq7/amq-streams-bridge-rhel7 repository or if you're just trying to get the registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest image specifically?

Derrick,

I think think this is a documentation issue, I'm not sure if it's a RHEL doc or elsewhere.  The description for this BZ says: 

"

Followed the steps as mentioned in the documentation link.
~~~~~
https://catalog.redhat.com/software/containers/amq7/amq-streams-bridge-rhel7/5e272044bed8bd66f81e6866?container-tabs=gti
~~~~~
Using podman login
~~~~~~~~~~~~~~~~~~
# podman login registry.redhat.io
Username: rhn-support-xxxxx
Password: 
Login Succeeded!

# podman pull registry.redhat.io/amq7/amq-streams-bridge-rhel7
"



The document in the description doesn't mention that a tag is required to pull the image and that "latest" is not a viable tag.  Yet the document that Valentin points to in his comment (https://catalog.redhat.com/software/containers/amq7/amq-streams-bridge-rhel7/5e272044bed8bd66f81e6866?container-tabs=gti) says that a tag is required 

"
For applicable Multi-Stream repositories, customers attempting to pull the latest container image either by using the :latest tag, or by not specifying a tag, will receive the following error response from the container registry:

unsupported: This repository does not use the "latest" tag to track the most recent image and must be pulled with
an explicit version or image reference. For more information, see: https://access.redhat.com/articles/4301321
"


Skopeo and Podman are both spitting out the information that's being returned from the registry and the two pieces of documentation show different requirements.  Perhaps the registry should allow the latest tag to be returned (I think so), but regardless we've two separate documents that say different things.

At least that's my nickels worth.

Hello there, 

Sorry for the late response. 
So, do we need to document it (add some information to the Container Catalog)?

Gabi,

I think the doc change that is needed is in this link:  https://catalog.redhat.com/software/containers/amq7/amq-streams-bridge-rhel7/5e272044bed8bd66f81e6866?container-tabs=gti

and where it shows example pulls like "# podman pull registry.redhat.io/amq7/amq-streams-bridge-rhel7", it needs to have a tag added, and that tag can not be ":latest".  Perhaps use: ':1.6.2-1'.  So the line would change to "# podman pull registry.redhat.io/amq7/amq-streams-bridge-rhel7:1.6.2-1".  It may or may not make sense to add a note similar to the one in https://catalog.redhat.com/software/containers/amq7/amq-streams-bridge-rhel7/5e272044bed8bd66f81e6866?container-tabs=gti that shows a tag must be specified and it can't be "latest".

My $.02.

(In reply to Derrick Ornelas from comment #3)
> I don't believe RHEL documentation is the right place for this.  The issue
> is related to the AMQ product/container images, though I'm not convinced
> that there is a doc issue here at all.  Both skopeo and podman report that
> the registry.redhat.io/amq7/amq-streams-bridge-rhel7 image repository has a
> "latest" tag.  The reported issue here is actually that RH Satellite tries
> to sync registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest and this
> fails, because, as we already know, there is no image tagged latest.  Do we
> know if podman/skopeo assumes that "latest" exists, or maybe the issue is
> that the registry reports that "latest" exists?  
> 
> Here's what skopeo is doing:
> 
> # skopeo inspect --debug --raw
> docker://registry.redhat.io/amq7/amq-streams-bridge-rhel7
> DEBU[0000] Loading registries configuration
> "/etc/containers/registries.conf" 
> DEBU[0000] Trying to access
> "registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest" 
> DEBU[0000] Returning credentials from /run/user/0/containers/auth.json 
> DEBU[0000] Using registries.d directory /etc/containers/registries.d for
> sigstore configuration 
> DEBU[0000]  Using "default-docker" configuration        
> DEBU[0000]  No signature storage configuration found for
> registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest 
> DEBU[0000] Looking for TLS certificates and private keys in
> /etc/docker/certs.d/registry.redhat.io 
> DEBU[0000]  cert:
> /etc/docker/certs.d/registry.redhat.io/4652001432504135603.cert 
> DEBU[0000]  key:
> /etc/docker/certs.d/registry.redhat.io/4652001432504135603.key 
> DEBU[0000] GET https://registry.redhat.io/v2/           
> DEBU[0000] Ping https://registry.redhat.io/v2/ status 401 
> DEBU[0000] GET
> https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/
> auth?account=rhn-support-dornelas&scope=repository%3Aamq7%2Famq-streams-
> bridge-rhel7%3Apull&service=docker-registry 
> DEBU[0001] GET
> https://registry.redhat.io/v2/amq7/amq-streams-bridge-rhel7/manifests/latest 
> DEBU[0002] Content-Type from manifest GET is "application/json" 
> DEBU[0002] Accessing
> "registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest" failed: Error
> reading manifest latest in registry.redhat.io/amq7/amq-streams-bridge-rhel7:
> unsupported: This repository does not use the "latest" tag to track the most
> recent image and must be pulled with an explicit version or image reference.
> For more information, see: https://access.redhat.com/articles/4301321 
> FATA[0002] Error parsing image name
> "docker://registry.redhat.io/amq7/amq-streams-bridge-rhel7": Error reading
> manifest latest in registry.redhat.io/amq7/amq-streams-bridge-rhel7:
> unsupported: This repository does not use the "latest" tag to track the most
> recent image and must be pulled with an explicit version or image reference.
> For more information, see: https://access.redhat.com/articles/4301321 
> 
> 
> From the output I believe it assumes that "latest" exists, as I don't see it
> ask for the list of tags.  I've tested it, and this causes 'skopeo sync
> --src docker --dest dir registry.redhat.io/amq7/amq-streams-bridge-rhel7
> /tmp/test' to eventually fail to sync the repo. 
> 
> Grabbing the tags list manually:
> 
> # TOKEN=$(curl -sL -u rhn-support-dornelas
> "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/
> auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull" | jq
> -r '.access_token')
> Enter host password for user 'rhn-support-dornelas':
> 
> # curl -sL -H "Authorization: Bearer $TOKEN" 
> "https://registry.redhat.io/v2/amq7/amq-streams-bridge-rhel7/tags/list" | jq
> .
> {
>   "name": "amq7/amq-streams-bridge-rhel7",
>   "tags": [
>     "1.6.2-1",
>     "1.4.1",
>     "1.5.0-2.1594897724",
>     "1.4.0",
>     "1.3.0-10.1579648804",
>     "1.5.0-2",
>     "1.6.0-2",
>     "1.6.0-1",
>     "1.5.0",
>     "1.6.2",
>     "1.6.0",
>     "1.3.0",
>     "1.5.0-2.1604501257",
>     "1.3.0-12",
>     "1.4.1-2",
>     "1.5.0-2.1594640710",
>     "1.4.0-8",
>     "latest"
>   ]
> }
> 
> 
> So, the registry is reporting that "latest" exists in the repo.  
> 
> @sadas can you tell us whether you're trying to sync the entire
> registry.redhat.io/amq7/amq-streams-bridge-rhel7 repository or if you're
> just trying to get the
> registry.redhat.io/amq7/amq-streams-bridge-rhel7:latest image specifically?

===============================================================================
@Derrick, I was trying to sync the entire repository and the sync was failing to fetch the latest image. Moreover, in satellite, we don't specify the tags, we just configure the repository and sync the content. While syncing the content from the repository( amq-streams-bridge-rhel7), the pulp was trying to get the latest image and failing with the below warning.

~~~~~~~~~~~~~~
403 Client Error: 'Forbidden' for url: https://registry.redhat.io/v2/amq7/amq-streams-bridge-rhel7/manifests/latest   ====> It's pointing to the latest tag.
~~~~~~~~~~~~~~

@Satyajit Das this repo has been fixed on the registry amq7/amq-streams-bridge-rhel7 and now Satellite sync should succeed.

dornelas the problem in the sync failure was in the fact that latest was advertised as available in the tags/list endpoint. During mirror of the repo, Satellite iterates through this list and expects to be able to fetch those tags.

Hello Satyajit,

As mentioned by Ina in comment #8, the issue is fixed for the amq7/amq-streams-bridge-rhel7 repo and we are able to sync it successfully. Can you confirm it with the customer? 

Regards,
Ashish

Hello,

On behalf of Satyajit, I can confirm that the issue with "amq7/amq-streams-bridge-rhel7" has been fixed as I can sync it on satellite and Now neither satellite tries to grab the "latest" tag for this repo nor I can see the latest tag for this repo , which is good.

$ curl -sL -H "Authorization: Bearer $TOKEN"  "https://registry.redhat.io/v2/amq7/amq-streams-bridge-rhel7/tags/list" | jq .
{
  "name": "amq7/amq-streams-bridge-rhel7",
  "tags": [
    "1.6.2-1",
    "1.4.1",
    "1.5.0-2.1594897724",
    "1.4.0",
    "1.3.0-10.1579648804",
    "1.5.0-2",
    "1.6.0-2",
    "1.6.0-1",
    "1.5.0",
    "1.6.2",
    "1.6.0",
    "1.3.0",
    "1.5.0-2.1604501257",
    "1.3.0-12",
    "1.4.1-2",
    "1.5.0-2.1594640710",
    "1.4.0-8"
  ]
}




But CU had reported exact same issue with two more repos i.e. amq-streams-rhel7-operator and amq-streams-kafka-26-rhel7. If I try to sync them, they will fail exactly while trying to sync latest tagged image.

Error from satellite,
~~
403 Client Error: 'Forbidden' for url: https://registry.redhat.io/v2/amq7/amq-streams-rhel7-operator/manifests/latest
~~
403 Client Error: 'Forbidden' for url: https://registry.redhat.io/v2/amq7/amq-streams-kafka-26-rhel7/manifests/latest
~~

And curl shows the latest tag listed in the list of tags as well. So these repos needs to be fixed in the exact same way as "amq-streams-bridge-rhel7" was fixed. 


Moreover, It will be great if someone one can take a look into all amq-streams-* repos and see if any one else needs the same fix or not.




Thanks & Regards,

Sayan Das

(In reply to Ashish Humbe from comment #10)
> Hello Satyajit,
> 
> As mentioned by Ina in comment #8, the issue is fixed for the
> amq7/amq-streams-bridge-rhel7 repo and we are able to sync it successfully.
> Can you confirm it with the customer? 
> 
> Regards,
> Ashish
============================
@Ashish, sayan has tested and updated the case.

(In reply to Ashish Humbe from comment #10)
> Hello Satyajit,
> 
> As mentioned by Ina in comment #8, the issue is fixed for the
> amq7/amq-streams-bridge-rhel7 repo and we are able to sync it successfully.
> Can you confirm it with the customer? 
> 
> Regards,
> Ashish
============================
@Ashish, sayan has tested and updated the case.

@sadas Yes if you agree we can close this case. 

I have created Jira for fixing the issue for below-listed repos : 

amq7/amq-streams-kafka-24-rhel7
amq7/amq-streams-kafka-25-rhel7
amq7/amq-streams-kafka-26-rhel7
 

Jira link:   https://projects.engineering.redhat.com/browse/SPMM-5337

(In reply to Ashish Humbe from comment #14)
> @sadas Yes if you agree we can close this case. 
> 
> I have created Jira for fixing the issue for below-listed repos : 
> 
> amq7/amq-streams-kafka-24-rhel7
> amq7/amq-streams-kafka-25-rhel7
> amq7/amq-streams-kafka-26-rhel7
>  
> 
> Jira link:   https://projects.engineering.redhat.com/browse/SPMM-5337

========================================================================
@Ashish, as you have opened jira, I guess we are good to close this one.

Closing based on comment #15. Thank you all.