Bug 1616171 – logging-fluentd requires its own priority class to be scheduled on infra nodes

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1616171 - logging-fluentd requires its own priority class to be scheduled on infra nodes

Summary:	logging-fluentd requires its own priority class to be scheduled on infra nodes

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Logging (Show other bugs)
Sub Component:
Version:	3.11.0
Hardware:	Unspecified Unspecified

Priority	unspecified Severity urgent
Target Milestone:	---
Target Release:	3.11.0
Assigned To:	ewolinet
QA Contact:	Anping Li
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-08-15 03:43 EDT by Junqi Zhao
Modified:	2018-10-11 03:25 EDT (History)
CC List:	4 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: If logging was not in a namespace that began with 'openshift-' Fluentd was not able to use the "system-cluster-critical" priority class. Consequence: Fluentd would not be able to start up. Fix: We create a priority class for Cluster Logging and configure Fluentd to use that instead. Result: Fluentd is able to start up, even if not installed to an 'openshift-*' namespace.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-10-11 03:24:57 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Github	openshift/openshift-ansible/pull/9686	None	None	None	2018-08-21 14:06 EDT
Red Hat Product Errata	RHBA-2018:2652	None	None	None	2018-10-11 03:25 EDT

Groups: None (edit)

Description Junqi Zhao 2018-08-15 03:43:42 EDT

Description of problem:
free-int cluster, logging-fluentd image version is v3.10.15, it should be v3.11.0
image: registry.reg-aws.openshift.com:443/openshift3/logging-fluentd:v3.10.15

other component version is v3.11.0-0.10.0
for example:
registry.reg-aws.openshift.com:443/openshift3/ose-logging-curator5:v3.11.0-0.10.0


Version-Release number of selected component (if applicable):

OpenShift Master:v3.11.0-0.10.0 
Kubernetes Master:v1.11.0+d4cacc0 
OpenShift Web Console:v3.11.0-0.10.0

logging version: v3.11.0-0.10.0
 

How reproducible:
Always

Steps to Reproduce:
1. Check logging component images version
2.
3.

Actual results:
logging-fluentd image version is v3.10.15

Expected results:
logging-fluentd image version should be v3.11.0

Additional info:

Comment 3 Jeff Cantrill 2018-08-17 21:04:25 EDT

Do we need a special provision here or move logging to the default namespace for new deployments of 'openshift-logging' ?

Comment 4 Justin Pierce 2018-08-20 09:20:59 EDT

@liggitt seems like your name in this area of code. Can you provide some guidance?

Comment 5 Jordan Liggitt 2018-08-20 09:57:22 EDT

system-priority level pods are allowed in kube-system and openshift-... namespaces only

Comment 6 Jeff Cantrill 2018-08-21 09:17:37 EDT

Modifying this bug to reflect the need of a priority class for scheduling.  The original fact the version was wrong I'm certain was related to the deployment timing out.  Per #c5, fluent is unable to be scheduled because this is an upgrade where the logging stack still lives in a namespace that is not 'openshift-'.  At the recommendation of D.Carr, logging needs a priority class that is below system-priority level:

* reason 1 - the restriction for priority class you are hitting is specific to cluster critical
* reason 2 - fluentd should be second tier to other cluster components

We should create a priority class for cluster-logging.  We need this class regardless but is a stop gap until we can migrate deployments to 'openshift-logging' in the 4.0 uprades.

Example from avesh:
apiVersion: scheduling.k8s.io/v1beta1
kind: PriorityClass
metadata:
   name: high-priority
 value: 100000
 globalDefault: false
 description: "This priority class for high priority Applications"

ref: https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/

Comment 7 Jeff Cantrill 2018-08-21 14:06:03 EDT

https://github.com/openshift/openshift-ansible/pull/9686

Comment 9 Anping Li 2018-09-07 04:29:25 EDT

Move the verified. The logging applications in logging namespace can be upgrade to v3.11. It is using priorityClassName: cluster-logging.

$ oc get ds -o yaml |grep priorityClassName
        priorityClassName: cluster-logging

Comment 11 errata-xmlrpc 2018-10-11 03:24:57 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652

Note You need to log in before you can comment on or make changes to this bug.