Bug 1561510
| Summary: | There is no guarantee a fluentd pod will start running on a node because Kube does not support pod preemption and priority | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Peter Portante <pportant> |
| Component: | Logging | Assignee: | ewolinet |
| Status: | CLOSED UPSTREAM | QA Contact: | Anping Li <anli> |
| Severity: | urgent | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.9.0 | CC: | aos-bugs, decarr, ewolinet, jcantril, rmeggins |
| Target Milestone: | --- | Keywords: | OpsBlocker |
| Target Release: | 3.11.0 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-05-14 18:16:52 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Peter Portante
2018-03-28 13:34:03 UTC
Here is a gist for a script that works to get fluentd running on a given node of the cluster it is labeled for currently: https://gist.github.com/portante/2b91dd7d49636c7e40fa53fb7ed1388b We should either add this script to the product, or stop using fluentd under daemonsets and use system containers deployed from ansible instead. After considering this further, pod priority and preemption won't help if there are other issues starting the fluentd pod that affect the OpenShift node or docker. We need to consider running the log collectors *out side* of OpenShift as part of the base RHEL services. We also need a work-around until then based on the gist from comment #1. This is a data-loss situation for customers. Another BZ describing how fluentd pods fail to start: https://bugzilla.redhat.com/show_bug.cgi?id=1560428 |