Bug 1512679
| Summary: | Failed docker builds leave temporary containers on node | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Cesar Wong <cewong> | |
| Component: | Build | Assignee: | Cesar Wong <cewong> | |
| Status: | CLOSED ERRATA | QA Contact: | Wenjing Zheng <wzheng> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 3.7.0 | CC: | aos-bugs, bparees, clichybi, dma, dyan, xtian | |
| Target Milestone: | --- | |||
| Target Release: | 3.7.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause:
The OpenShift Docker builder invokes the Docker build API without the ForceRmTemp flag
Consequence:
Containers from failed builds remain on the node where the build ran. These containers are not recognized by the kubelet for gc and are therefore accumulated until the node runs out of space.
Fix:
Modified the Docker build API call from the OpenShift Docker builder to force the removal of temporary containers.
Result:
Failed containers no longer remain on the node where a Docker build ran.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1538413 (view as bug list) | Environment: | ||
| Last Closed: | 2017-12-18 13:23:56 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1533181 | |||
PR for origin master https://github.com/openshift/origin/pull/17283 Verified # openshift version openshift v3.7.11 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 *** Bug 1515358 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3464 |
Description of problem: After running a Docker strategy build that fails on a node, a container that represents that build remains on the node. The container is not cleaned up by the Kubelet because it's not a container managed by Kubernetes. This causes the node to keep containers that will not get cleaned up, eventually causing the node to run out of space. Version-Release number of selected component (if applicable): All versions How reproducible: Always Steps to Reproduce: 1. Create a Docker build that will fail: echo "FROM openshift/origin:latest\nRUN exit 1" | oc new-build -D - --name failing-build 2. Wait for the build to finish 3. Inspect containers on the node where the build ran with 'docker ps -a' Actual results: A container that runs the last failing RUN instruction will exist ('exit 1') Expected results: No containers related to the failed build should exist on the node Additional info: