Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1678640

Summary:	Running 'control-cpu-load.sh' prevents CTDB starting
Product:	[Community] GlusterFS	Reporter:	ryan
Component:	gluster-smb	Assignee:	Anoop C S <anoopcs>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.1	CC:	amukherj, anoopcs, bugs
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-11 05:23:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description ryan 2019-02-19 09:39:12 UTC

Description of problem:

After running script found here, system will not longer start CTDB:
https://github.com/gluster/glusterfs/blob/master/extras/control-cpu-load.sh


Version-Release number of selected component (if applicable):


How reproducible:
Everytime


Steps to Reproduce:
1. Run script on Gluster heal process
2. Try to start CTDB service

Actual results:
CTDB fails to start with following error:
2019/02/08 20:46:59.612215 ctdbd[2629]: Created PID file /var/run/ctdb/ctdbd.pid
2019/02/08 20:46:59.612267 ctdbd[2629]: Listening to ctdb socket /var/run/ctdb/ctdbd.socket
2019/02/08 20:46:59.612297 ctdbd[2629]: Unable to set scheduler to SCHED_FIFO (Operation not permitted)
2019/02/08 20:46:59.612304 ctdbd[2629]: CTDB daemon shutting down

Expected results:
CTDB starts


Additional info:
Other systems where script was not run, works fine.
Have set DefaultCPUAccounting=no in /etc/systemd/system.conf.

systemctl show ctdb | grep -i accounting
CPUAccounting=no
BlockIOAccounting=no
MemoryAccounting=no
TasksAccounting=no

Comment 1 Anoop C S 2019-06-07 10:55:40 UTC

(In reply to ryan from comment #0)
> Actual results:
> CTDB fails to start with following error:
> 2019/02/08 20:46:59.612215 ctdbd[2629]: Created PID file
> /var/run/ctdb/ctdbd.pid
> 2019/02/08 20:46:59.612267 ctdbd[2629]: Listening to ctdb socket
> /var/run/ctdb/ctdbd.socket
> 2019/02/08 20:46:59.612297 ctdbd[2629]: Unable to set scheduler to
> SCHED_FIFO (Operation not permitted)
> 2019/02/08 20:46:59.612304 ctdbd[2629]: CTDB daemon shutting down

Please use the following CTDB setting in /etc/sysconfig/ctdb:
CTDB_NOSETSCHED=yes

and try restarting CTDB.

Comment 2 Anoop C S 2019-06-07 11:01:49 UTC

(In reply to Anoop C S from comment #1)
> (In reply to ryan from comment #0)
> > Actual results:
> > CTDB fails to start with following error:
> > 2019/02/08 20:46:59.612215 ctdbd[2629]: Created PID file
> > /var/run/ctdb/ctdbd.pid
> > 2019/02/08 20:46:59.612267 ctdbd[2629]: Listening to ctdb socket
> > /var/run/ctdb/ctdbd.socket
> > 2019/02/08 20:46:59.612297 ctdbd[2629]: Unable to set scheduler to
> > SCHED_FIFO (Operation not permitted)
> > 2019/02/08 20:46:59.612304 ctdbd[2629]: CTDB daemon shutting down
> 
> Please use the following CTDB setting in /etc/sysconfig/ctdb:
> CTDB_NOSETSCHED=yes
> 
> and try restarting CTDB.

Copy-pasting a summary of the reason for above suggestion from a different bug:

CTDB daemon i.e, ctdbd is a service that by default requests for real-time scheduling unless it is instructed not to do so via explicit configuration parameters. By default systemd places all system services into their own control groups in the "cpu" hierarchy. But the "cpu" cgroup controller of the kernel demands absolute real-time budget to be explicitly specified. A reasonable value for required real-time cpu cycles are pre-written into corresponding configuration files. This value getting overwritten by other components in the system results in denial of real-time scheduling to services under this "cpu" hierarchy with error EPERM(Operation not permitted).

ref: https://www.freedesktop.org/wiki/Software/systemd/MyServiceCantGetRealtime/

Comment 3 ryan 2019-06-10 09:52:41 UTC

Hello,

We are currently working around this issue with the configuration option you suggested 'CTDB_NOSETSCHED=yes', and I can confirm CTDB starts successfully with this.

Best,
Ryan

Comment 4 Anoop C S 2019-06-11 05:23:17 UTC

Closing the bug report as per confirmation in comment #3.

See comment #2 for details.

Comment 5 ryan 2019-06-11 15:51:47 UTC

Is there any way to undo the changes made by the script to allow real-time scheduling?

Comment 6 Anoop C S 2019-06-12 05:32:37 UTC

(In reply to ryan from comment #5)
> Is there any way to undo the changes made by the script to allow real-time
> scheduling?

If you are concerned about performance penalty in disabling real-time scheduling for CTDB then I don't think it will be obviously visible. If not you are good to go with current setting.
On the other side you may have to ask whoever came up with this script for more details around it.