+++ This bug was initially created as a clone of Bug #1241950 +++
Description of problem:
I have an Image with name "fabric8/fabric8-gogs-find-projects:1.4.1" which I successfully build on the Docker host. Then I try to call the following URL for
accessing the Remote API in order to push this image to docker-hub:
Push-URL: : http://127.0.0.1:2375/v1.15/images/fabric8%2Ffabric8-gogs-find-projects/push?tag=1.4.1&force=1
Note that I use the (undocument) query param force to allow to force to push to the default registry,
What I get is the following error:
[INFO] DOCKER> ... The push refers to a repository [docker.io/fabric8/fabric8-gogs-find-projects] (len: 1)
[ERROR] DOCKER> Error pushing to registry: Tag does not exist: 1.4.1
So it really seems to try to push docker.io/fabric8/fabric8-gogs-find-projects and not fabric8/fabric8-gogs-find-projects:1.4.1.
When I name the image "docker.io/fabric8/fabric8-gogs-find-projects:1.4.1" it works perfectly.
Version-Release number of selected component (if applicable):
docker-selinux-1.6.2-14.el7.x86_64
How reproducible:
Try to build an image with 'docker build -t user/project:1.0 .' and then 'docker push' this to your docker hub account. It should give the same error. (I actually don't have direct access to the machine since this happens on a Jenkins slave for which I don't have karma)
--- Additional comment from Jan Kurik on 2015-07-15 09:18:48 EDT ---
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.
(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)
More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23
--- Additional comment from Michal Minar on 2015-07-16 03:21:52 EDT ---
I couldn't reproduce the issue with mentioned docker version. Do you run it on Fedora or RHEL? The combination of Fedora:rawhide and el7 RPM is kind of strange. I did a test on RHEL7 system but I don't think it matters that much. More important is an RPM package version. You list only docker-selinux. Can you please ensure that installed docker RPM is of the same version?
Also it seems strange to me that you use 1.15 api version in you push URL. This version belongs to pre v1.4.0.
Can you please also specify command line options passed to Docker daemon?
--- Additional comment from Jimmi Dyson on 2015-07-16 03:46:51 EDT ---
The Jenkins slave Roland mentions is running Red Hat Enterprise Linux Server release 7.1 (Maipo) with docker-1.6.2-14.el7.x86_64 installed.
--- Additional comment from Jimmi Dyson on 2015-07-16 03:47:40 EDT ---
The Docker daemon running is:
/usr/bin/docker -d --selinux-enabled -H tcp://127.0.0.1:2375 -H unix://var/run/docker.sock --confirm-def-push=false --add-registry index.docker.io
--- Additional comment from Roland Huss on 2015-07-16 03:56:02 EDT ---
Yes, API 1.15 is quite old. It is used from within the docker-maven-plugin and is intentionally kept at the lowest feature set required in order to support older Docker installations, too. I don't think the problem is related to the API version, btw.
I'll try to make a better test case (which is not so easy because as I said I don't have SSH access to the Jenkins slave and can only operate over the Jenkins groovy script console).
--- Additional comment from Roland Huss on 2015-07-16 05:40:42 EDT ---
I tried to reproduced it via the command line and couldn't reproduce it, too (i.e. it works as expected).
I will try to reproduce it via calling the remote API soon.
--- Additional comment from Roland Huss on 2015-08-18 16:25:51 EDT ---
Here are my current findings with 1.6.2.fc21 and 1.6.2.el7 :
When using these two posts
POST /v1.17/build?t=jolokia%2Fdocker-jolokia-demo%3A0.13.4-SNAPSHOT&rm=1
POST /v1.17/images/jolokia%2Fdocker-jolokia-demo%3A0.13.4-SNAPSHOT/tag?repo=jolokia%2Fdocker-jolokia-demo&force=1&tag=latest
the result is the following images:
jolokia/docker-jolokia-demo 0.13.4-SNAPSHOT 435440f7f612 28 minutes ago 3.094 MB
docker.io/jolokia/docker-jolokia-demo latest 435440f7f612 28 minutes ago 3.094 MB
Please note the asymmetric naming /wr to the registry.
Now, when doing a
POST /v1.17/images/jolokia%2Fdocker-jolokia-demo/push?force=1&tag=0.13.4-SNAPSHOT
I get the following error response
The push refers to a repository [docker.io/jolokia/docker-jolokia-demo]
Error pushing to registry: Tag does not exist: 0.13.4-SNAPSHOT
Here also some strange translation between no registry and "docker.io" happens. Personally I wouldn't expect any automatic addition of any registry at all (especially the registry is part of a name which is often used as an id)
--- Additional comment from Michal Minar on 2015-08-19 04:02:36 EDT ---
Again trying with just Docker client. I've tried 1.6.2-1.el7, 1.6.2-14.el7, 1.7.1-111 and current upstream with same results:
1. docker -D build -t miminar/my-tmp-img:1-2-3-4 .
2. docker tag miminar/my-tmp-img:1-2-3-4 miminar/my-tmp-img:latest
Generating following requests:
1. POST /v1.18/build?cgroupparent=&cpusetcpus=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=miminar%2Fmy-tmp-img%3A1-2-3-4
2. POST /v1.18/images/miminar/my-tmp-img:1-2-3-4/tag?repo=miminar%2Fmy-tmp-img&tag=latest
Resulting in these images:
miminar/my-tmp-img 1-2-3-4 f3222ef38bd8 8 minutes ago 258.1 MB
miminar/my-tmp-img latest f3222ef38bd8 8 minutes ago 258.1 MB
The second `POST` looks like a cuplrit in your case, which is quite strange because I don't see a way, how different http headers could affect a result. Does you local image store contain any other jolokia/docker-jolokia-demo or docker.io/jolokia/docker-jolokia-demo before making those requests?
On the other hand, I'm able to reproduce your push issue. I'm working on a fix now.
--- Additional comment from Roland Huss on 2015-08-19 04:14:40 EDT ---
(In reply to Michal Minar from comment #8)
> The second `POST` looks like a cuplrit in your case, which is quite strange
> because I don't see a way, how different http headers could affect a result.
> Does you local image store contain any other jolokia/docker-jolokia-demo or
> docker.io/jolokia/docker-jolokia-demo before making those requests?
I removed all images before the test (you can infer also from the timestamp that they are results from the same experiment). Maybe the URL encoding could be an issue here ? (my request used encoded '/' in the name, you example used it literally as part of the path (but that could be also due of the way how the logging was done)
> On the other hand, I'm able to reproduce your push issue. I'm working on a
> fix now.
Thanks a lot !
--- Additional comment from Roland Huss on 2015-08-19 04:31:05 EDT ---
Indeed when I try your client commands on my environment I get the following images:
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
docker.io/miminar/my-tmp-img latest 20d5c5e57ec4 About a minute ago 3.089 MB
miminar/my-tmp-img 1-2-3-4 20d5c5e57ec4 About a minute ago 3.089 MB
(I reused my buildir here). So the same issue when calling it over the API (so URL looks fine).
The environment is
[root@vagrant vagrant]# uname -a
Linux vagrant.f8 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Name : docker
Arch : x86_64
Version : 1.6.2
Release : 4.gitc3ca5bb.el7
Size : 21 M
Repo : installed
From repo : /docker-1.6.2-4.gitc3ca5bb.el7.x86_64
Summary : Automates deployment of containerized applications
URL : http://www.docker.com
License : ASL 2.0
Description : Docker is an open-source engine that automates the deployment of any
: application as a lightweight, portable, self-sufficient container that will
: run virtually anywhere.
:
: Docker containers can encapsulate any payload, and will run consistently on
: and between virtually any server. The same container that a developer builds
: and tests on a laptop will run at scale, in production*, on VMs, bare-metal
: servers, OpenStack clusters, public instances, or combinations of the above.
It is started with
OPTIONS='--selinux-enabled -H unix://var/run/docker.sock -H tcp://0.0.0.0:2375 --insecure-registry=172.0.0.0/8 --log-level=warn'
DOCKER_CERT_PATH=/etc/docker
--- Additional comment from Roland Huss on 2015-08-19 04:37:01 EDT ---
Finally here are the API calls from the Docker daemon logs
POST /v1.18/build?cpusetcpus=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=miminar%2Fmy-tmp-img%3A1-2-3-4
POST /v1.18/images/miminar/my-tmp-img:1-2-3-4/tag?repo=miminar%2Fmy-tmp-img&tag=latest
for
docker -D build -t miminar/my-tmp-img:1-2-3-4 .
docker tag miminar/my-tmp-img:1-2-3-4 miminar/my-tmp-img:latest
resulting in (complete output):
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
miminar/my-tmp-img 1-2-3-4 e2c77d369d4d 2 minutes ago 3.089 MB
docker.io/miminar/my-tmp-img latest e2c77d369d4d 2 minutes ago 3.089 MB
jolokia/docker-jolokia-demo 0.13.4-SNAPSHOT 08dc1a7faada About an hour ago 3.089 MB
docker.io/jolokia/docker-jolokia-demo latest 08dc1a7faada About an hour ago 3.089 MB
docker.io/openshift/origin-deployer v1.0.4 2478f82cbd24 2 weeks ago 412.2 MB
docker.io/openshift/origin-docker-registry v1.0.4 a5c39c44a38c 2 weeks ago 279.2 MB
docker.io/openshift/origin-haproxy-router v1.0.4 d2685ed64138 2 weeks ago 343.7 MB
docker.io/openshift/origin-pod v1.0.4 696923ac5cc4 2 weeks ago 1.105 MB
docker.io/fabric8/kibana-config 2.2.16 a88d8bed5945 3 weeks ago 172.2 MB
docker.io/fabric8/elasticsearch-logstash-template 2.2.16 8dbff797b823 3 weeks ago 172.2 MB
docker.io/fabric8/fabric8-console v1.0.25 18eb4b7ddc30 4 weeks ago 188.4 MB
docker.io/openshift/origin-haproxy-router v1.0.0 6bc54f04a99f 7 weeks ago 389.2 MB
docker.io/fabric8/kibana4 4.1.0 86ae7a32a40f 7 weeks ago 210.1 MB
docker.io/fabric8/elasticsearch-k8s 1.5.2 9f98887039cd 7 weeks ago 524.4 MB
docker.io/consol/tomcat-7.0 latest e88c09539e11 10 weeks ago 601.2 MB
docker.io/fabric8/fluentd-kubernetes 1.0 191d6c962e08 10 weeks ago 203.8 MB
docker.io/busybox latest 8c2e06607696 4 months ago 2.43 MB
--- Additional comment from Roland Huss on 2015-08-19 04:44:02 EDT ---
Just found out also, when using a base image like 'FROM busybox' this base image gets pulled and stored as `docker.io/busybox:latest`
--- Additional comment from Roland Huss on 2015-08-19 05:07:19 EDT ---
Another interesting observation (sorry when I'm flooding you, simply ignore this if this info is not helpful):
The push only fails if there is an image with "docker.io" prefix but different tag.
E.g. consider you have only
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
jolokia/docker-jolokia-demo 0.13.4-SNAPSHOT 08dc1a7faada About an hour ago 3.089 MB
then a
docker push jolokia/docker-jolokia-demo:0.13.4-SNAPSHOT
works.
BUT when you have
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
docker.io/jolokia/docker-jolokia-demo latest 08dc1a7faada About an hour ago 3.089 MB
jolokia/docker-jolokia-demo 0.13.4-SNAPSHOT 08dc1a7faada About an hour ago 3.089 MB
then
docker push jolokia/docker-jolokia-demo:0.13.4-SNAPSHOT
leads to
The push refers to a repository [docker.io/jolokia/docker-jolokia-demo] (len: 1)
FATA[0000] Error pushing to registry: Tag does not exist: 0.13.4-SNAPSHOT
Seems to be some fallback in the daemon but only if there is not yet an image with the full name (so the fallback works only on image name level, not on tag level)
--- Additional comment from Michal Minar on 2015-08-19 09:36:42 EDT ---
(In reply to Roland Huss from comment #10)
> (I reused my buildir here). So the same issue when calling it over the API
> (so URL looks fine).
Interesting. I'll investigate further but I'd like to move it to another bugzilla.
(In reply to Roland Huss from comment #12)
> Just found out also, when using a base image like 'FROM busybox' this base
> image gets pulled and stored as `docker.io/busybox:latest`
Yes, this is expected.
(In reply to Roland Huss from comment #13)
> The push only fails if there is an image with "docker.io" prefix but
> different tag.
Exactly. It's addressed here: https://github.com/rhatdan/docker/pull/111
I'll open another PRs for Fedora from 1.7 up.
So the push part should be fixed now (it will take a while before it hits
updates-testing though).
Roland, could you please open another BZ for the tagging part? Where tagging
jolokia/docker-jolokia-demo:0.13.4-SNAPSHOT as latest becomes
docker.io/jolokia/docker-jolokia-demo:latest. It's really a different issue.