RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1209576 - can't remove image by specifying its ID
Summary: can't remove image by specifying its ID
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Minar
QA Contact: Luwen Su
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-04-07 16:50 UTC by Tomas Tomecek
Modified: 2019-03-06 01:59 UTC (History)
6 users (show)

Fixed In Version: docker-1.6.2-8.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-23 09:29:15 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Patch for docker-1.6.2 (1.27 KB, patch)
2015-06-08 07:27 UTC, Michal Minar 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:1167 0 normal SHIPPED_LIVE docker bug fix update 2015-06-23 13:29:05 UTC

Description Tomas Tomecek 2015-04-07 16:50:16 UTC 
Fedora (docker-io-1.5.0-1.fc21.x86_64)

$ docker rmi d664714b44cbab9a9874d48d77296ca65fd188d85e828847c494baf75161865f
Untagged: my-test-image:latest
Deleted: d664714b44cbab9a9874d48d77296ca65fd188d85e828847c494baf75161865f
Deleted: e79e08ea3cdef79fa71a4c7ecfc0362c46ff28c9f0ca55c41664d2d02df6353a
Deleted: a1ca26ee4efd92901fa6d9f7c890fb4794d11c8cddceaea58ebfe8ba3c5a39d6
Deleted: 19f128206e74ed7fb1b523c7db7baa65abe3fda7c8806d44cb1ffee72ee8cd8c
Deleted: 4d3d3e07064c154e02099424834754106e0eff0fcfad506e203dfe94ff6c2883

RHEL 7 (docker-1.5.0-28.el7.x86_64)

$ docker rmi 17f055efe48dd97244767a7eaddf3b8b4206b75e9740cffbf94db9f8713a25ef
Error response from daemon: could not find image: Prefix can't be empty
FATA[0001] Error: failed to remove one or more images
$ docker rmi -f 17f055efe48dd97244767a7eaddf3b8b4206b75e9740cffbf94db9f8713a25ef
Error response from daemon: could not find image: Prefix can't be empty
FATA[0000] Error: failed to remove one or more images

My /etc/sysconfig/docker:

OPTIONS='--selinux-enabled'
DOCKER_CERT_PATH=/etc/docker
ADD_REGISTRY=''
INSECURE_REGISTRY='--insecure-registry 172.17.42.1:5000 --insecure-registry our.internal.registry'
GOTRACEBACK='crash'
Comment 2 Tomas Tomecek 2015-04-08 11:47:03 UTC 
reproducer:

dockerfile (from the other bug):

FROM rhel7
ADD tarball.tar.gz /tmp
RUN ls -lha /tmp/tarball-content

commands:

$ mkdir tarball-content
$ tar -cf tarball.tar.gz tarball-content/
$ docker build --no-cache=true --tag=test-tmp .
Sending build context to Docker daemon 13.31 kB
Sending build context to Docker daemon 
Step 0 : FROM rhel7
 ---> 10acc31def5d
Step 1 : ADD tarball.tar.gz /tmp
 ---> c219d6223e69
Removing intermediate container 3b6d063c6414
Step 2 : RUN ls -lha /tmp/tarball-content
 ---> Running in 49252fff27f6
total 8.0K
drwxr-xr-x. 2 root root 4.0K Apr  7 11:15 .
drwxrwxrwt. 8 root root 4.0K Apr  8 07:41 ..
 ---> 6691823e7113
Removing intermediate container 49252fff27f6
Successfully built 6691823e7113

$ docker inspect 6691823e7113 | grep '"Id"'
    "Id": "6691823e7113619e0ab04e93c73f1cbb8f667ebbdbbbd2d9a39c5070e7788cb3",

$ docker rmi 6691823e7113619e0ab04e93c73f1cbb8f667ebbdbbbd2d9a39c5070e7788cb3
Error response from daemon: could not find image: Prefix can't be empty
FATA[0001] Error: failed to remove one or more images   


docker-1.5.0-28.el7.x86_64
Comment 3 Michal Minar 2015-04-08 12:06:25 UTC 
Thanks Tomas for your reproducer. Unfortunately I'm still unable to reproduce it:

$ docker build --no-cache=true --tag=test-tmp .
[...]
Removing intermediate container 528c98bc307e
Successfully built af0ec874f61e
$ docker rmi af0ec874f61eca5fae1beb921bafda0c18b4c4124b18f94532ff39478c1ec101
Untagged: test-tmp:latest
Deleted: af0ec874f61eca5fae1beb921bafda0c18b4c4124b18f94532ff39478c1ec101
Deleted: c78180f66fe884cf95608d66e3a46c027e9d2fdc36d89a645b098588fef21a66

But I think I know, where the problem is ... I just need to experiment a bit.
Comment 4 Michal Minar 2015-04-09 13:46:20 UTC 
Nope, I'm still unable to reproduce it. Could you please try it again with these builds:

  1) https://brewweb.devel.redhat.com/taskinfo?taskID=8961528
  2) https://brewweb.devel.redhat.com/taskinfo?taskID=8961548

With -D flag passed to daemon and attach here corresponding logs? The first build (Docker version 1.5.0-dev, build d2b8f69/1.5.0) has registry patches and produces some additional logging messages. The second one (Docker version 1.5.0-dev, build 91abff4/1.5.0) is a clean docker master without RHEL patches.

I'm sorry to bother you this way..
Comment 5 Tomas Tomecek 2015-04-09 14:35:53 UTC 
It works fine with first build. Should I post some logs and try the other build?
Comment 6 Michal Minar 2015-04-30 06:13:19 UTC 
Tomas, do you still experience this issue? It seems to me that your image database got somehow corrupted - which is hard to reproduce. Can I close this?
Comment 7 Tomas Tomecek 2015-04-30 08:01:34 UTC 
With 1.6 I don't experience this anymore. I'll reopen once it happens again.
Comment 8 Michal Minar 2015-06-08 07:00:13 UTC 
Reopening.

Tomas provided me an access to a machine where this could be reproduced. Database there was corrupted. This is a short excerpt from Docker's init:

INFO[0000] -job init_networkdriver() = OK (0)           
INFO[0000] Loading containers: start.                   
ERRO[0000] Failed to load container 02997ccd33a7153519a57bc1ab887db3b11984ca6b054fc54675598570b9ef90: open /var/lib/docker/containers/02997ccd33a7153519a57bc1ab887db3b11984ca6b054fc54675598570b9ef90/config.json: no such file or directory 
ERRO[0000] Failed to load container 05d6b831f05a78585730621ad124d5493c0011870281842297bc4b97d1f20627: open /var/lib/docker/containers/05d6b831f05a78585730621ad124d5493c0011870281842297bc4b97d1f20627/config.json: no such file or directory 
DEBU[0000] Loaded container 07aaa7325babe2d0b49c53efb3848890ddf735702d2f39068ea2153684fc978f 
DEBU[0000] Loaded container 09e30b62285d9064a8fc32ffc6e60419bb09bde43840f914a860f90198a9b7f7 
ERRO[0000] Failed to load container 1400e4cf2833b0cf2a2bacb0e95bd0299ec00957965563f6be6ac57d89305366: open /var/lib/docker/containers/1400e4cf2833b0cf2a2bacb0e95bd0299ec00957965563f6be6ac57d89305366/config.json: no such file or directory 
DEBU[0000] Loaded container 175484be8cfb5d48edd32e7476fbb5ec7e2fedaf56eaf9940ffe7dfe01f12936 
DEBU[0000] Loaded container 2e138d33e9cae1b9e5ba070c0c18f8df7feb8c2f009155ce5d5772d4d5520994 
DEBU[0000] Loaded container 3c753b42497cd6e6150c7275e694d98843d8649451428b2cb5a601dec2f27163 
ERRO[0000] Failed to load container 45004139b1ab3498bf5e5e65c8c1924b2a79a13ef2a9bbb7d7ddd7cb52d05972: open /var/lib/docker/containers/45004139b1ab3498bf5e5e65c8c1924b2a79a13ef2a9bbb7d7ddd7cb52d05972/config.json: no such file or directory 
DEBU[0000] Loaded container 4c5a9de3d0ce7525af29f61f582dc428c7ef32f565b0109887c68307e3494c73 
ERRO[0000] Failed to load container 589f197f39f90c1b46177939b08469a4ab872a1b0a499bc14cb4e41e857bd8df: open /var/lib/docker/containers/589f197f39f90c1b46177939b08469a4ab872a1b0a499bc14cb4e41e857bd8df/config.json: no such file or directory 
ERRO[0000] Failed to load container 62ebdd22353931dc948fe35f3603b2f679117770070b03c5feb448979fb2b41b: open /var/lib/docker/containers/62ebdd22353931dc948fe35f3603b2f679117770070b03c5feb448979fb2b41b/config.json: no such file or directory 
DEBU[0000] Loaded container 6cecd3f415b422c1b19a4207ce6cb2db7e6452bd11a37c9e0672b28609c4f252 
DEBU[0000] Loaded container 7b44480222670d38105c5a693d005d464b67fad892f3baa76eb2ce346d7b066b 
DEBU[0000] Loaded container 81518de91996485de1f2abe4f4f5e3bde220174e37f3443c5c69a3815d3be2c8 
DEBU[0000] Loaded container 870e63f0888611165ae9652c30aed21e9051056197476d46e0cfac75e61849d7 
DEBU[0000] Loaded container 89ea0993a3d7d5f6e1004ad826a8ca282a975b35d0ada0a071fb849c15bd5012 
DEBU[0000] Loaded container 9ed4470c99bd42e7c13b05fdc29a689f2bfa322b2eb7b61d59a2f3c8716c299a 
DEBU[0000] Loaded container a1d08b174d48727c783a719bcdeb105600422534ce2f3a134d80de48c7a7ff80 
ERRO[0000] Failed to load container a39e83f76851bcedb63e83421805c864fc46722927ef6392e5f26e6b9f019c13: open /var/lib/docker/containers/a39e83f76851bcedb63e83421805c864fc46722927ef6392e5f26e6b9f019c13/config.json: no such file or directory 
ERRO[0000] Failed to load container abb68752e60ccbc9ab7cad326bb631107a72547f5eb9ed2bf7dbff69adc12e6a: open /var/lib/docker/containers/abb68752e60ccbc9ab7cad326bb631107a72547f5eb9ed2bf7dbff69adc12e6a/config.json: no such file or directory 
ERRO[0000] Failed to load container ad00e4c2cfe692b8a4ea4d90bf7a0bb4ab117d069e41c03cb4107717e943e847: open /var/lib/docker/containers/ad00e4c2cfe692b8a4ea4d90bf7a0bb4ab117d069e41c03cb4107717e943e847/config.json: no such file or directory 
ERRO[0000] Failed to load container b5832244af3af9392eaf2a510a4534acc9ae7c049b3e54ca0b9a7349ffa53209: open /var/lib/docker/containers/b5832244af3af9392eaf2a510a4534acc9ae7c049b3e54ca0b9a7349ffa53209/config.json: no such file or directory 
DEBU[0000] Loaded container b7010899f3234d82ff91b36051d4fc5772d50a06fd367598032e9914cb47b9f1 
DEBU[0000] Loaded container b7f7056eca6d615ea48a6b5f5393fb0b8a2b84b268d753075988af7181760c2b 
ERRO[0000] Failed to load container bf98ccfcaa150489c6f1997155bcbb0fd8ff0957200c087d5da232ec81d15a98: open /var/lib/docker/containers/bf98ccfcaa150489c6f1997155bcbb0fd8ff0957200c087d5da232ec81d15a98/config.json: no such file or directory 
DEBU[0000] Loaded container c1bc8a2376f4a57dc734e6d46564c7061347aad2b6c861725fc62365a4c98441 
ERRO[0000] Failed to load container e131c91bf23f1da7e9ee844536e27e5b1fbdfff42af54173c1ef079fc54329eb: open /var/lib/docker/containers/e131c91bf23f1da7e9ee844536e27e5b1fbdfff42af54173c1ef079fc54329eb/config.json: no such file or directory 
ERRO[0000] Failed to load container eae391b6a44046a8ac9e887fc29c3f7fcd48dc34acc07591a6f79adc7a2bdc96: open /var/lib/docker/containers/eae391b6a44046a8ac9e887fc29c3f7fcd48dc34acc07591a6f79adc7a2bdc96/config.json: no such file or directory 
DEBU[0000] Loaded container efc05ce23ae365b8c319d7bbf56e2aadb3d773b71465d786b79a927a3c18c203 
ERRO[0000] Failed to load container f8ef25089e55fa5d27181fc07ad4e300ce6c494a21972f4ba7853255bb319034: open /var/lib/docker/containers/f8ef25089e55fa5d27181fc07ad4e300ce6c494a21972f4ba7853255bb319034/config.json: no such file or directory 
DEBU[0000] Loaded container f9763cc93049b8b171a1a9594f172a942eaac9ae905bac9f6756bec3ef061718 
ERRO[0000] Failed to load container fe7a000bd1b619eaae8d23c896e8b3cb70f6d8d433e50a7eb07c38750506e9a9: open /var/lib/docker/containers/fe7a000bd1b619eaae8d23c896e8b3cb70f6d8d433e50a7eb07c38750506e9a9/config.json: no such file or directory 
DEBU[0000] Loaded container feab0478db9cd4598650aec456b21bab95405a5dd15935cc84be98ca8db43d21 
DEBU[0000] Restarting containers...                     
INFO[0000] Loading containers: done.                    
INFO[0000] docker daemon: 1.6.2 0dce5e6-dirty; execdriver: native-0.2; graphdriver: devicemapper 

The problem that caused this was a container with ImageID unset:

CONTAINER ID        IMAGE                                                                     COMMAND                CREATED             STATUS                   PORTS               NAMES
...
6cecd3f415b4        :latest                                                                   "\"/bin/rpm -qa --qf   9 days ago                                                       thirsty_darwin                                                                                                             
07aaa7325bab        811dabf4fd1547049b284a432097a0dd06a0d94c2e632c72dbe81cb9c18b12c8:latest   "/bin/sh -c 'rm -f '   9 days ago          Exited (1) 9 days ago                        pensive_bell                                                                                                               
175484be8cfb        buildroot:latest                                                          "dock --verbose insi   9 days ago          Exited (1) 9 days ago                        k8s_custom-build.86794a56_openshift-mongodb-docker-20150529-135454_default_848bf868-05f9-11e5-9eb6-fa163e1012c3_04e8af88   
...

Note the image name of 6cecd3f415b4. The question is how this could happen. Unfortunately I failed to reproduce it. It may be caused by a database corruption. But it may be also related to registry patches.
Comment 9 Michal Minar 2015-06-08 07:27:01 UTC 
Created attachment 1036154 [details]
Patch for docker-1.6.2

Don't bail out of image removing due to invalid container
    
Skip container whose ImageID is empty when checking whether image is
being used.
Comment 10 Michal Minar 2015-06-08 07:32:41 UTC 
Here's a PR for 1.6.2: https://github.com/rhatdan/docker/pull/66

Seems like upstream already has already found out: https://github.com/docker/docker/commit/71a499022904e1da074ded3e1ed874c1b75ccf23
Comment 12 Luwen Su 2015-06-15 03:14:06 UTC 
Only inspect docker-1.6.2-10.el7.x86_64's source code,the docker-b79465d2e9a11c61dcba47965fad2dad83f6c31e, and see the patch has been merged, since this issue is hard to reproduce, move to verified
Comment 13 subfuzion 2015-06-16 00:13:32 UTC 
I can confirm I'm running into this issue with Docker version 1.6.2, build 7c8fca2

Error response from daemon: could not find image: Prefix can't be empty
FATA[0000] Error: failed to remove one or more images
Comment 15 errata-xmlrpc 2015-06-23 09:29:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1167.html
Note You need to log in before you can comment on or make changes to this bug.