Description of problem: I have a problem pulling images from reposiories: [paul@xxxx docker]$ podman pull registry.fedoraproject.org/f32/httpd Trying to pull registry.fedoraproject.org/f32/httpd... manifest unknown: manifest unknown Error: error pulling image "registry.fedoraproject.org/f32/httpd": unable to pull registry.fedoraproject.org/f32/httpd: unable to pull image: Error initializing source docker://registry.fedoraproject.org/f32/httpd:latest: Error reading manifest latest in registry.fedoraproject.org/f32/httpd: manifest unknown: manifest unknown The same problem happens when running as root. I have tried this on several systems, including a new gnome Boxes installation of Fedora 32 on which I had only installed podman. Using the above for f30 or f31 gives the same result but f29 works like a dream. I also have myown repository. If I push images created by docker all is well. If I push podman built images the same issue is seen. Version-Release number of selected component (if applicable): podman version 1.9.1 How reproducible: Totally. Happens everytime for me on 3 different systems and a VM. Steps to Reproduce: 1. podman pull registry.fedoraproject.org/f32/httpd 2. 3. Actual results: pull fails Expected results: pull image from repository Additional info: You can use old versions: podman pull registry.fedoraproject.org/f29/httpd No problems pushing/pulling images built by docker.
Thanks for your report: - Why do you expect f32/httpd to exist? It is not listed on https://registry.fedoraproject.org , and there is no f32 branch on https://src.fedoraproject.org/container/httpd/branches?branchname=master . As far as I can tell, the image just does not exist and that’s that. - If the image really should exist, please attach the output of (podman --log-level=debug pull …). - If the Fedora image does not exist, but you can reproduce this with some other images you have created, please attach the debug log as described above, for _both_ the push and pull process.
Sorry my example was poor. I am having problems pulling any container built on fedora 32 from my own repository (using the package docker-distribution) and some other repositories. I was attempting to be clever and give an example that would easily work for you. I have just checked the list on the link you gave and indeed httpd is not on f32. The same page says that mysql should be there if I use podman pull registry.fedoraproject.org/f32/mysql. The result of this command is: [paul@ewing ~]$ podman pull registry.fedoraproject.org/f32/mysql Trying to pull registry.fedoraproject.org/f32/mysql... manifest unknown: manifest unknown Error: error pulling image "registry.fedoraproject.org/f32/mysql": unable to pull registry.fedoraproject.org/f32/mysql: unable to pull image: Error initializing source docker://registry.fedoraproject.org/f32/mysql:latest: Error reading manifest latest in registry.fedoraproject.org/f32/mysql: manifest unknown: manifest unknown [paul@ewing ~]$ I could add --log-level-debug but it does not appear to give any extra information to my eyes. I can run built images without issue but after a push and a pull I get the above manifest error. Note however that images built with older software appears to work, certainly the httpd example from f29 works, containers I have built on fedora 30/31 also work although these could have been built and pushed with docker, docker hub and docker in general appear to work. It is as though podman does not understand the container format.
(In reply to Paul Finnigan from comment #2) > [paul@ewing ~]$ podman pull registry.fedoraproject.org/f32/mysql Sadly, (docker pull) for that image fails similarly for me. So this still seems to just be a missing image. At least (podman pull registry.fedoraproject.org/fedora:32) works for me. > I can run built images without issue but after a push and a pull I get the > above manifest error. Precise reproducer, with debug logs, and identification of the server (is that a Sonatype Nexus by any chance?) please. I imagine if podman did not pull images at all, there would be many reports.
Created attachment 1687786 [details] output from first build
Created attachment 1687787 [details] output from fist push
Created attachment 1687788 [details] output from run
Created attachment 1687789 [details] output from fist pull
Created attachment 1687790 [details] output from second build
Created attachment 1687792 [details] output from second push
Created attachment 1687794 [details] Output of second pull
Created attachment 1687796 [details] output of third push
Created attachment 1687797 [details] output of third pull
Created attachment 1687798 [details] Dockerfile1
Created attachment 1687799 [details] Dockerfile2
Created attachment 1687800 [details] podman info
Created attachment 1687801 [details] registry.yml
I have tried this on three installations of Fedora 32 Workstation, actually upgraded from Fedora 31 in all cases. All systems had docker installed with podman running along side in Fedora 31. When I upgraded I decided to remove docker and use just podman. Installed packages are podman, podman-docker and podman-plugins. On the two systems I have docker-distribution installed, including the one I the results are from. You may be able to look at the third pull output and tell me what I am doing wrong. What I have done I used the hello-world container from docker, as it is very simple for no other reason, I get he same results on any container I build as long as there is a change. First using Dockerfile1: [First build] podman build --log-level=debug -t lenovo:5000/hello . [First push] podman push --log-level=debug -t lenovo:5000/hello [First run] podman run --log-level=debug lenovo:5000/hello [First pull] podman pull --log-level=debug -t lenovo:5000/hello All is good to this point. I suspect that the blobs used are just those from the Docker Repositry. Then using Dockerfile2 (to make sure there were some changes) I did: [Second push] podman build --log-level=debug -t lenovo:5000/hello . [Second pull] podman push -t lenovo:5000/hello [Third push] podman pull -t lenovo:5000/hello [Third pull] podman push --log-level=debug -t lenovo:5000/hello [9] podman pull --log-level=debug -t lenovo:5000/hello The second and third pulls fail. Sorry but in the second push/pull I forgot the debug statements. The third push shows that the blobs exist on the registry. I have also provided the output from a 'podman info' and my registry config. Is there anything else that will help you?
Looks like your registry does not understand OCI Images? Try podman build --format docker --log-level=debug -t lenovo:5000/hello . And then push it to the registry. You should be able to pull the image then. That should proove that the problem is on the registry side.
You are correct. The image pulls fine once it is built with the --format docker switch. I will have to look at the docker-distribution package. One for tomorrow for me now. Thanks for your help. I guess the best thing is to close this problem as user error and I will look at the registry a little more closely before raising another problem I should be able to fix. Thanks again.
Thanks for all the data, reproduced. (A simpler reproducer: > skopeo copy --debug --dest-tls-verify=false --format oci docker://hello-world docker://localhost:5000/bar > skopeo copy --debug --src-tls-verify=false docker://localhost:5000/bar dir:new ) This looks like missing https://github.com/docker/distribution/commit/60d9c5dfad8c0d44fd5ee0482f6de7b09a15fb02 from https://github.com/docker/distribution/pull/2076 , which was added in docker/distribution 2.7.1. Sadly the Fedora package uses an older version and only a preliminary version of that upstream work (along with other missing features, like accurately reporting the error cause). Running the docker://distribution:2 container instead, the OCI-formatted images can be pulled fine. Reassigning to docker-distribution.
This message is a reminder that Fedora 32 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '32'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 32 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.