RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1255051 - "docker push" selects the wrong image
Summary: "docker push" selects the wrong image
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On: 1241950
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-08-19 13:57 UTC by Daniel Walsh
Modified: 2019-03-06 01:20 UTC (History)
12 users (show)

Fixed In Version: docker-1.7.1-112.el7_1
Doc Type: Bug Fix
Doc Text:
Clone Of: 1241950
Environment:
Last Closed: 2016-01-07 21:46:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Daniel Walsh 2015-08-19 13:57:11 UTC
+++ This bug was initially created as a clone of Bug #1241950 +++

Description of problem:

I have an Image with name "fabric8/fabric8-gogs-find-projects:1.4.1" which I successfully build on the Docker host. Then I try to call the following URL for 
accessing the Remote API in order to push this image to docker-hub:

Push-URL: : http://127.0.0.1:2375/v1.15/images/fabric8%2Ffabric8-gogs-find-projects/push?tag=1.4.1&force=1

Note that I use the (undocument) query param force to allow to force to push to the default registry,

What I get is the following error:

[INFO] DOCKER> ... The push refers to a repository [docker.io/fabric8/fabric8-gogs-find-projects] (len: 1)
[ERROR] DOCKER> Error pushing to registry: Tag does not exist: 1.4.1

So it really seems to try to push docker.io/fabric8/fabric8-gogs-find-projects and not fabric8/fabric8-gogs-find-projects:1.4.1. 

When I name the image "docker.io/fabric8/fabric8-gogs-find-projects:1.4.1" it works perfectly.


Version-Release number of selected component (if applicable):

docker-selinux-1.6.2-14.el7.x86_64

How reproducible:

Try to build an image with 'docker build -t user/project:1.0 .' and then 'docker push' this to your docker hub account. It should give the same error. (I actually don't have direct access to the machine since this happens on a Jenkins slave for which I don't have karma)

--- Additional comment from Jan Kurik on 2015-07-15 09:18:48 EDT ---

This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23

--- Additional comment from Michal Minar on 2015-07-16 03:21:52 EDT ---

I couldn't reproduce the issue with mentioned docker version. Do you run it on Fedora or RHEL? The combination of Fedora:rawhide and el7 RPM is kind of strange. I did a test on RHEL7 system but I don't think it matters that much. More important is an RPM package version. You list only docker-selinux. Can you please ensure that installed docker RPM is of the same version?

Also it seems strange to me that you use 1.15 api version in you push URL. This version belongs to pre v1.4.0.

Can you please also specify command line options passed to Docker daemon?

--- Additional comment from Jimmi Dyson on 2015-07-16 03:46:51 EDT ---

The Jenkins slave Roland mentions is running Red Hat Enterprise Linux Server release 7.1 (Maipo) with docker-1.6.2-14.el7.x86_64 installed.

--- Additional comment from Jimmi Dyson on 2015-07-16 03:47:40 EDT ---

The Docker daemon running is:

/usr/bin/docker -d --selinux-enabled -H tcp://127.0.0.1:2375 -H unix://var/run/docker.sock --confirm-def-push=false --add-registry index.docker.io

--- Additional comment from Roland Huss on 2015-07-16 03:56:02 EDT ---

Yes, API 1.15 is quite old. It is used from within the docker-maven-plugin and is intentionally kept at the lowest feature set required in order to support older Docker installations, too. I don't think the problem is related to the API version, btw.

I'll try to make a better test case (which is not so easy because as I said I don't have SSH access to the Jenkins slave and can only operate over the Jenkins groovy script console).

--- Additional comment from Roland Huss on 2015-07-16 05:40:42 EDT ---

I tried to reproduced it via the command line and couldn't reproduce it, too (i.e. it works as expected). 

I will try to reproduce it via calling the remote API soon.

--- Additional comment from Roland Huss on 2015-08-18 16:25:51 EDT ---

Here are my current findings with 1.6.2.fc21 and 1.6.2.el7 :

When using these two posts

POST /v1.17/build?t=jolokia%2Fdocker-jolokia-demo%3A0.13.4-SNAPSHOT&rm=1

POST /v1.17/images/jolokia%2Fdocker-jolokia-demo%3A0.13.4-SNAPSHOT/tag?repo=jolokia%2Fdocker-jolokia-demo&force=1&tag=latest

the result is the following images:

jolokia/docker-jolokia-demo             0.13.4-SNAPSHOT     435440f7f612        28 minutes ago      3.094 MB
docker.io/jolokia/docker-jolokia-demo   latest              435440f7f612        28 minutes ago      3.094 MB 

Please note the asymmetric naming /wr to the registry.

Now, when doing a 

POST /v1.17/images/jolokia%2Fdocker-jolokia-demo/push?force=1&tag=0.13.4-SNAPSHOT

I get the following error response

The push refers to a repository [docker.io/jolokia/docker-jolokia-demo]
Error pushing to registry: Tag does not exist: 0.13.4-SNAPSHOT

Here also some strange translation between no registry and "docker.io" happens. Personally I wouldn't expect any automatic addition of any registry at all (especially the registry is part of a name which is often used as an id)

--- Additional comment from Michal Minar on 2015-08-19 04:02:36 EDT ---

Again trying with just Docker client. I've tried 1.6.2-1.el7, 1.6.2-14.el7, 1.7.1-111 and current upstream with same results:

1. docker -D build -t miminar/my-tmp-img:1-2-3-4 .
2. docker tag miminar/my-tmp-img:1-2-3-4 miminar/my-tmp-img:latest

Generating following requests:

1. POST /v1.18/build?cgroupparent=&cpusetcpus=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=miminar%2Fmy-tmp-img%3A1-2-3-4
2. POST /v1.18/images/miminar/my-tmp-img:1-2-3-4/tag?repo=miminar%2Fmy-tmp-img&tag=latest

Resulting in these images:

miminar/my-tmp-img                            1-2-3-4                      f3222ef38bd8        8 minutes ago       258.1 MB
miminar/my-tmp-img                            latest                       f3222ef38bd8        8 minutes ago       258.1 MB

The second `POST` looks like a cuplrit in your case, which is quite strange because I don't see a way, how different http headers could affect a result. Does you local image store contain any other jolokia/docker-jolokia-demo or docker.io/jolokia/docker-jolokia-demo before making those requests?

On the other hand, I'm able to reproduce your push issue. I'm working on a fix now.

--- Additional comment from Roland Huss on 2015-08-19 04:14:40 EDT ---

(In reply to Michal Minar from comment #8)

> The second `POST` looks like a cuplrit in your case, which is quite strange
> because I don't see a way, how different http headers could affect a result.
> Does you local image store contain any other jolokia/docker-jolokia-demo or
> docker.io/jolokia/docker-jolokia-demo before making those requests?

I removed all images before the test (you can infer also from the timestamp that they are results from the same experiment). Maybe the URL encoding could be an issue here ? (my request used encoded '/' in the name, you example used it literally as part of the path (but that could be also due of the way how the logging was done)

> On the other hand, I'm able to reproduce your push issue. I'm working on a
> fix now.

Thanks a lot !

--- Additional comment from Roland Huss on 2015-08-19 04:31:05 EDT ---

Indeed when I try your client commands on my environment I get the following images:

REPOSITORY                                          TAG                 IMAGE ID            CREATED              VIRTUAL SIZE
docker.io/miminar/my-tmp-img                        latest              20d5c5e57ec4        About a minute ago   3.089 MB
miminar/my-tmp-img                                  1-2-3-4             20d5c5e57ec4        About a minute ago   3.089 MB

(I reused my buildir here). So the same issue when calling it over the API (so URL looks fine).

The environment is

[root@vagrant vagrant]# uname -a
Linux vagrant.f8 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Name        : docker
Arch        : x86_64
Version     : 1.6.2
Release     : 4.gitc3ca5bb.el7
Size        : 21 M
Repo        : installed
From repo   : /docker-1.6.2-4.gitc3ca5bb.el7.x86_64
Summary     : Automates deployment of containerized applications
URL         : http://www.docker.com
License     : ASL 2.0
Description : Docker is an open-source engine that automates the deployment of any
            : application as a lightweight, portable, self-sufficient container that will
            : run virtually anywhere.
            :
            : Docker containers can encapsulate any payload, and will run consistently on
            : and between virtually any server. The same container that a developer builds
            : and tests on a laptop will run at scale, in production*, on VMs, bare-metal
            : servers, OpenStack clusters, public instances, or combinations of the above.

It is started with

OPTIONS='--selinux-enabled -H unix://var/run/docker.sock -H tcp://0.0.0.0:2375 --insecure-registry=172.0.0.0/8 --log-level=warn'
DOCKER_CERT_PATH=/etc/docker

--- Additional comment from Roland Huss on 2015-08-19 04:37:01 EDT ---

Finally here are the API calls from the Docker daemon logs

POST /v1.18/build?cpusetcpus=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=miminar%2Fmy-tmp-img%3A1-2-3-4
POST /v1.18/images/miminar/my-tmp-img:1-2-3-4/tag?repo=miminar%2Fmy-tmp-img&tag=latest

for 

docker -D build -t miminar/my-tmp-img:1-2-3-4 .
docker tag miminar/my-tmp-img:1-2-3-4 miminar/my-tmp-img:latest

resulting in (complete output):

REPOSITORY                                          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
miminar/my-tmp-img                                  1-2-3-4             e2c77d369d4d        2 minutes ago       3.089 MB
docker.io/miminar/my-tmp-img                        latest              e2c77d369d4d        2 minutes ago       3.089 MB
jolokia/docker-jolokia-demo                         0.13.4-SNAPSHOT     08dc1a7faada        About an hour ago   3.089 MB
docker.io/jolokia/docker-jolokia-demo               latest              08dc1a7faada        About an hour ago   3.089 MB
docker.io/openshift/origin-deployer                 v1.0.4              2478f82cbd24        2 weeks ago         412.2 MB
docker.io/openshift/origin-docker-registry          v1.0.4              a5c39c44a38c        2 weeks ago         279.2 MB
docker.io/openshift/origin-haproxy-router           v1.0.4              d2685ed64138        2 weeks ago         343.7 MB
docker.io/openshift/origin-pod                      v1.0.4              696923ac5cc4        2 weeks ago         1.105 MB
docker.io/fabric8/kibana-config                     2.2.16              a88d8bed5945        3 weeks ago         172.2 MB
docker.io/fabric8/elasticsearch-logstash-template   2.2.16              8dbff797b823        3 weeks ago         172.2 MB
docker.io/fabric8/fabric8-console                   v1.0.25             18eb4b7ddc30        4 weeks ago         188.4 MB
docker.io/openshift/origin-haproxy-router           v1.0.0              6bc54f04a99f        7 weeks ago         389.2 MB
docker.io/fabric8/kibana4                           4.1.0               86ae7a32a40f        7 weeks ago         210.1 MB
docker.io/fabric8/elasticsearch-k8s                 1.5.2               9f98887039cd        7 weeks ago         524.4 MB
docker.io/consol/tomcat-7.0                         latest              e88c09539e11        10 weeks ago        601.2 MB
docker.io/fabric8/fluentd-kubernetes                1.0                 191d6c962e08        10 weeks ago        203.8 MB
docker.io/busybox                                   latest              8c2e06607696        4 months ago        2.43 MB

--- Additional comment from Roland Huss on 2015-08-19 04:44:02 EDT ---

Just found out also, when using a base image like 'FROM busybox' this base image gets pulled and stored as `docker.io/busybox:latest`

--- Additional comment from Roland Huss on 2015-08-19 05:07:19 EDT ---

Another interesting observation (sorry when I'm flooding you, simply ignore this if this info is not helpful):

The push only fails if there is an image with "docker.io" prefix but different tag.

E.g. consider you have only

REPOSITORY                                          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
jolokia/docker-jolokia-demo                         0.13.4-SNAPSHOT     08dc1a7faada        About an hour ago   3.089 MB

then a 

docker push jolokia/docker-jolokia-demo:0.13.4-SNAPSHOT

works.

BUT when you have 

REPOSITORY                                          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
docker.io/jolokia/docker-jolokia-demo               latest              08dc1a7faada        About an hour ago   3.089 MB
jolokia/docker-jolokia-demo                         0.13.4-SNAPSHOT     08dc1a7faada        About an hour ago   3.089 MB

then

docker push jolokia/docker-jolokia-demo:0.13.4-SNAPSHOT 

leads to

The push refers to a repository [docker.io/jolokia/docker-jolokia-demo] (len: 1)
FATA[0000] Error pushing to registry: Tag does not exist: 0.13.4-SNAPSHOT

Seems to be some fallback in the daemon but only if there is not yet an image with the full name (so the fallback works only on image name level, not on tag level)

--- Additional comment from Michal Minar on 2015-08-19 09:36:42 EDT ---

(In reply to Roland Huss from comment #10)
> (I reused my buildir here). So the same issue when calling it over the API
> (so URL looks fine).
Interesting. I'll investigate further but I'd like to move it to another bugzilla.

(In reply to Roland Huss from comment #12)
> Just found out also, when using a base image like 'FROM busybox' this base
> image gets pulled and stored as `docker.io/busybox:latest`
Yes, this is expected.

(In reply to Roland Huss from comment #13)
> The push only fails if there is an image with "docker.io" prefix but
> different tag.
Exactly. It's addressed here: https://github.com/rhatdan/docker/pull/111
I'll open another PRs for Fedora from 1.7 up.

So the push part should be fixed now (it will take a while before it hits
updates-testing though).

Roland, could you please open another BZ for the tagging part? Where tagging
jolokia/docker-jolokia-demo:0.13.4-SNAPSHOT as latest becomes
docker.io/jolokia/docker-jolokia-demo:latest. It's really a different issue.

Comment 2 Daniel Walsh 2015-08-19 14:38:37 UTC
Fixed in git.


Note You need to log in before you can comment on or make changes to this bug.