Bug 1688339

Description of problem:

The kubelet is not able to garbage collect an image referenced by multiple repositories. It throws the following error:

Type     Reason                 Age               From                                                      Message
  ----     ------                 ----              ----                                                      -------
  Warning  ImageGCFailed          1h                kubelet, ip-XXX-XXX-XXXX.compute.internal  wanted to free 9506825830 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 62a050c25c9f (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete e7e3e750297f (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 0a778da30a91 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete a7aa08dca852 (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 5f4a38b4e183 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete ddd1538e683c (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 0634b7e15ebd (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 7e87e2b3bf7a (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 6aea66716be5 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 1c57dd62d156 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete bd33314c39ad (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete fe1966f11f05 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 3cf5e5122a2e (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete e3884a17cc5c (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 325aee206677 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete e53657d9f1dc (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 26950e417ddc (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete ba228a9e6813 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 9585c45e76cd (must be forced) - image is referenced in multiple repositories]
  Warning  EvictionThresholdMet   10m (x6 over 1h)  kubelet, ip-XXX-XXX-XXXX.compute.internal  Attempting to reclaim imagefs
  Normal   NodeHasNoDiskPressure  5m (x7 over 2h)   kubelet, ip-XXX-XXX-XXXX.compute.internal  Node ip-XXX-XXX-XXXX.compute.internal status is now: NodeHasNoDiskPressure
  Normal   NodeHasDiskPressure    5m (x8 over 2h)   kubelet, ip-XXX-XXX-XXXX.compute.internal  Node ip-XXX-XXX-XXXX.compute.internal status is now: NodeHasDiskPressure
  Warning  ImageGCFailed          2m (x23 over 1h)  kubelet, iip-XXX-XXX-XXXX.compute.internal  (combined from similar events): wanted to free 13388875366 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 5f4a38b4e183 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete e3884a17cc5c (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete ddd1538e683c (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 0634b7e15ebd (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 7e87e2b3bf7a (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 6aea66716be5 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 1c57dd62d156 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 325aee206677 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete fe1966f11f05 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 3cf5e5122a2e (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete e7e3e750297f (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 62a050c25c9f (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete bd33314c39ad (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete a7aa08dca852 (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 0a778da30a91 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete e53657d9f1dc (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete ba228a9e6813 (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 26950e417ddc (must be forced) - image is referenced in multiple repositories, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 9585c45e76cd (must be forced) - image is referenced in multiple repositories]

Version-Release number of selected component (if applicable):
It was experienced with OpenShift 3.9 but I believe that it is not addressed with newer version.

How reproducible: Always

Steps to Reproduce:
1. Have a container starting on a node with an image pulled from an external registry
2. Have a container starting on the same node with the same image pulled from the internal registry 
3. # docker images | grep fe1966f11f05
docker-registry.default.svc:5000/project/test-img                      <none>                fe1966f11f05        3 weeks ago         2.42 GB
external-registry.mydomain:5000/project/test-img                                      <none>                fe1966f11f05        3 weeks ago         2.42 GB 

Actual results:

The image layers are not deleted

Expected results:

The image layers are deleted

Additional information:

It is possible to delete the image layers on the node by 
- either force removing the image: docker rmi -f test-img
- or first deleting one of the image tag.

This is a known limitation in docker:
https://github.com/moby/moby/issues/1530

This works for multiple tags from the same repo with OpenShift / Kubernetes:
https://github.com/kubernetes/kubernetes/pull/29316

But it has not been implemented for multiple repos:
https://github.com/kubernetes/kubernetes/issues/70238

This bug is to track progress on it and get it working on OpenShift.

Regards,

Frédéric