Bug 1209576
| Summary: | can't remove image by specifying its ID | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Tomas Tomecek <ttomecek> | ||||
| Component: | docker | Assignee: | Michal Minar <miminar> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Luwen Su <lsu> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.1 | CC: | lsm5, miabbott, miminar, sghosh, tony, ttomecek | ||||
| Target Milestone: | rc | Keywords: | Extras, Reopened | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | docker-1.6.2-8.el7 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-06-23 09:29:15 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Tomas Tomecek
2015-04-07 16:50:16 UTC
reproducer:
dockerfile (from the other bug):
FROM rhel7
ADD tarball.tar.gz /tmp
RUN ls -lha /tmp/tarball-content
commands:
$ mkdir tarball-content
$ tar -cf tarball.tar.gz tarball-content/
$ docker build --no-cache=true --tag=test-tmp .
Sending build context to Docker daemon 13.31 kB
Sending build context to Docker daemon
Step 0 : FROM rhel7
---> 10acc31def5d
Step 1 : ADD tarball.tar.gz /tmp
---> c219d6223e69
Removing intermediate container 3b6d063c6414
Step 2 : RUN ls -lha /tmp/tarball-content
---> Running in 49252fff27f6
total 8.0K
drwxr-xr-x. 2 root root 4.0K Apr 7 11:15 .
drwxrwxrwt. 8 root root 4.0K Apr 8 07:41 ..
---> 6691823e7113
Removing intermediate container 49252fff27f6
Successfully built 6691823e7113
$ docker inspect 6691823e7113 | grep '"Id"'
"Id": "6691823e7113619e0ab04e93c73f1cbb8f667ebbdbbbd2d9a39c5070e7788cb3",
$ docker rmi 6691823e7113619e0ab04e93c73f1cbb8f667ebbdbbbd2d9a39c5070e7788cb3
Error response from daemon: could not find image: Prefix can't be empty
FATA[0001] Error: failed to remove one or more images
docker-1.5.0-28.el7.x86_64
Thanks Tomas for your reproducer. Unfortunately I'm still unable to reproduce it: $ docker build --no-cache=true --tag=test-tmp . [...] Removing intermediate container 528c98bc307e Successfully built af0ec874f61e $ docker rmi af0ec874f61eca5fae1beb921bafda0c18b4c4124b18f94532ff39478c1ec101 Untagged: test-tmp:latest Deleted: af0ec874f61eca5fae1beb921bafda0c18b4c4124b18f94532ff39478c1ec101 Deleted: c78180f66fe884cf95608d66e3a46c027e9d2fdc36d89a645b098588fef21a66 But I think I know, where the problem is ... I just need to experiment a bit. Nope, I'm still unable to reproduce it. Could you please try it again with these builds: 1) https://brewweb.devel.redhat.com/taskinfo?taskID=8961528 2) https://brewweb.devel.redhat.com/taskinfo?taskID=8961548 With -D flag passed to daemon and attach here corresponding logs? The first build (Docker version 1.5.0-dev, build d2b8f69/1.5.0) has registry patches and produces some additional logging messages. The second one (Docker version 1.5.0-dev, build 91abff4/1.5.0) is a clean docker master without RHEL patches. I'm sorry to bother you this way.. It works fine with first build. Should I post some logs and try the other build? Tomas, do you still experience this issue? It seems to me that your image database got somehow corrupted - which is hard to reproduce. Can I close this? With 1.6 I don't experience this anymore. I'll reopen once it happens again. Reopening. Tomas provided me an access to a machine where this could be reproduced. Database there was corrupted. This is a short excerpt from Docker's init: INFO[0000] -job init_networkdriver() = OK (0) INFO[0000] Loading containers: start. ERRO[0000] Failed to load container 02997ccd33a7153519a57bc1ab887db3b11984ca6b054fc54675598570b9ef90: open /var/lib/docker/containers/02997ccd33a7153519a57bc1ab887db3b11984ca6b054fc54675598570b9ef90/config.json: no such file or directory ERRO[0000] Failed to load container 05d6b831f05a78585730621ad124d5493c0011870281842297bc4b97d1f20627: open /var/lib/docker/containers/05d6b831f05a78585730621ad124d5493c0011870281842297bc4b97d1f20627/config.json: no such file or directory DEBU[0000] Loaded container 07aaa7325babe2d0b49c53efb3848890ddf735702d2f39068ea2153684fc978f DEBU[0000] Loaded container 09e30b62285d9064a8fc32ffc6e60419bb09bde43840f914a860f90198a9b7f7 ERRO[0000] Failed to load container 1400e4cf2833b0cf2a2bacb0e95bd0299ec00957965563f6be6ac57d89305366: open /var/lib/docker/containers/1400e4cf2833b0cf2a2bacb0e95bd0299ec00957965563f6be6ac57d89305366/config.json: no such file or directory DEBU[0000] Loaded container 175484be8cfb5d48edd32e7476fbb5ec7e2fedaf56eaf9940ffe7dfe01f12936 DEBU[0000] Loaded container 2e138d33e9cae1b9e5ba070c0c18f8df7feb8c2f009155ce5d5772d4d5520994 DEBU[0000] Loaded container 3c753b42497cd6e6150c7275e694d98843d8649451428b2cb5a601dec2f27163 ERRO[0000] Failed to load container 45004139b1ab3498bf5e5e65c8c1924b2a79a13ef2a9bbb7d7ddd7cb52d05972: open /var/lib/docker/containers/45004139b1ab3498bf5e5e65c8c1924b2a79a13ef2a9bbb7d7ddd7cb52d05972/config.json: no such file or directory DEBU[0000] Loaded container 4c5a9de3d0ce7525af29f61f582dc428c7ef32f565b0109887c68307e3494c73 ERRO[0000] Failed to load container 589f197f39f90c1b46177939b08469a4ab872a1b0a499bc14cb4e41e857bd8df: open /var/lib/docker/containers/589f197f39f90c1b46177939b08469a4ab872a1b0a499bc14cb4e41e857bd8df/config.json: no such file or directory ERRO[0000] Failed to load container 62ebdd22353931dc948fe35f3603b2f679117770070b03c5feb448979fb2b41b: open /var/lib/docker/containers/62ebdd22353931dc948fe35f3603b2f679117770070b03c5feb448979fb2b41b/config.json: no such file or directory DEBU[0000] Loaded container 6cecd3f415b422c1b19a4207ce6cb2db7e6452bd11a37c9e0672b28609c4f252 DEBU[0000] Loaded container 7b44480222670d38105c5a693d005d464b67fad892f3baa76eb2ce346d7b066b DEBU[0000] Loaded container 81518de91996485de1f2abe4f4f5e3bde220174e37f3443c5c69a3815d3be2c8 DEBU[0000] Loaded container 870e63f0888611165ae9652c30aed21e9051056197476d46e0cfac75e61849d7 DEBU[0000] Loaded container 89ea0993a3d7d5f6e1004ad826a8ca282a975b35d0ada0a071fb849c15bd5012 DEBU[0000] Loaded container 9ed4470c99bd42e7c13b05fdc29a689f2bfa322b2eb7b61d59a2f3c8716c299a DEBU[0000] Loaded container a1d08b174d48727c783a719bcdeb105600422534ce2f3a134d80de48c7a7ff80 ERRO[0000] Failed to load container a39e83f76851bcedb63e83421805c864fc46722927ef6392e5f26e6b9f019c13: open /var/lib/docker/containers/a39e83f76851bcedb63e83421805c864fc46722927ef6392e5f26e6b9f019c13/config.json: no such file or directory ERRO[0000] Failed to load container abb68752e60ccbc9ab7cad326bb631107a72547f5eb9ed2bf7dbff69adc12e6a: open /var/lib/docker/containers/abb68752e60ccbc9ab7cad326bb631107a72547f5eb9ed2bf7dbff69adc12e6a/config.json: no such file or directory ERRO[0000] Failed to load container ad00e4c2cfe692b8a4ea4d90bf7a0bb4ab117d069e41c03cb4107717e943e847: open /var/lib/docker/containers/ad00e4c2cfe692b8a4ea4d90bf7a0bb4ab117d069e41c03cb4107717e943e847/config.json: no such file or directory ERRO[0000] Failed to load container b5832244af3af9392eaf2a510a4534acc9ae7c049b3e54ca0b9a7349ffa53209: open /var/lib/docker/containers/b5832244af3af9392eaf2a510a4534acc9ae7c049b3e54ca0b9a7349ffa53209/config.json: no such file or directory DEBU[0000] Loaded container b7010899f3234d82ff91b36051d4fc5772d50a06fd367598032e9914cb47b9f1 DEBU[0000] Loaded container b7f7056eca6d615ea48a6b5f5393fb0b8a2b84b268d753075988af7181760c2b ERRO[0000] Failed to load container bf98ccfcaa150489c6f1997155bcbb0fd8ff0957200c087d5da232ec81d15a98: open /var/lib/docker/containers/bf98ccfcaa150489c6f1997155bcbb0fd8ff0957200c087d5da232ec81d15a98/config.json: no such file or directory DEBU[0000] Loaded container c1bc8a2376f4a57dc734e6d46564c7061347aad2b6c861725fc62365a4c98441 ERRO[0000] Failed to load container e131c91bf23f1da7e9ee844536e27e5b1fbdfff42af54173c1ef079fc54329eb: open /var/lib/docker/containers/e131c91bf23f1da7e9ee844536e27e5b1fbdfff42af54173c1ef079fc54329eb/config.json: no such file or directory ERRO[0000] Failed to load container eae391b6a44046a8ac9e887fc29c3f7fcd48dc34acc07591a6f79adc7a2bdc96: open /var/lib/docker/containers/eae391b6a44046a8ac9e887fc29c3f7fcd48dc34acc07591a6f79adc7a2bdc96/config.json: no such file or directory DEBU[0000] Loaded container efc05ce23ae365b8c319d7bbf56e2aadb3d773b71465d786b79a927a3c18c203 ERRO[0000] Failed to load container f8ef25089e55fa5d27181fc07ad4e300ce6c494a21972f4ba7853255bb319034: open /var/lib/docker/containers/f8ef25089e55fa5d27181fc07ad4e300ce6c494a21972f4ba7853255bb319034/config.json: no such file or directory DEBU[0000] Loaded container f9763cc93049b8b171a1a9594f172a942eaac9ae905bac9f6756bec3ef061718 ERRO[0000] Failed to load container fe7a000bd1b619eaae8d23c896e8b3cb70f6d8d433e50a7eb07c38750506e9a9: open /var/lib/docker/containers/fe7a000bd1b619eaae8d23c896e8b3cb70f6d8d433e50a7eb07c38750506e9a9/config.json: no such file or directory DEBU[0000] Loaded container feab0478db9cd4598650aec456b21bab95405a5dd15935cc84be98ca8db43d21 DEBU[0000] Restarting containers... INFO[0000] Loading containers: done. INFO[0000] docker daemon: 1.6.2 0dce5e6-dirty; execdriver: native-0.2; graphdriver: devicemapper The problem that caused this was a container with ImageID unset: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ... 6cecd3f415b4 :latest "\"/bin/rpm -qa --qf 9 days ago thirsty_darwin 07aaa7325bab 811dabf4fd1547049b284a432097a0dd06a0d94c2e632c72dbe81cb9c18b12c8:latest "/bin/sh -c 'rm -f ' 9 days ago Exited (1) 9 days ago pensive_bell 175484be8cfb buildroot:latest "dock --verbose insi 9 days ago Exited (1) 9 days ago k8s_custom-build.86794a56_openshift-mongodb-docker-20150529-135454_default_848bf868-05f9-11e5-9eb6-fa163e1012c3_04e8af88 ... Note the image name of 6cecd3f415b4. The question is how this could happen. Unfortunately I failed to reproduce it. It may be caused by a database corruption. But it may be also related to registry patches. Created attachment 1036154 [details]
Patch for docker-1.6.2
Don't bail out of image removing due to invalid container
Skip container whose ImageID is empty when checking whether image is
being used.
Here's a PR for 1.6.2: https://github.com/rhatdan/docker/pull/66 Seems like upstream already has already found out: https://github.com/docker/docker/commit/71a499022904e1da074ded3e1ed874c1b75ccf23 Only inspect docker-1.6.2-10.el7.x86_64's source code,the docker-b79465d2e9a11c61dcba47965fad2dad83f6c31e, and see the patch has been merged, since this issue is hard to reproduce, move to verified I can confirm I'm running into this issue with Docker version 1.6.2, build 7c8fca2 Error response from daemon: could not find image: Prefix can't be empty FATA[0000] Error: failed to remove one or more images Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1167.html |