Bug 1678640 - Running 'control-cpu-load.sh' prevents CTDB starting
Summary: Running 'control-cpu-load.sh' prevents CTDB starting
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: GlusterFS
Classification: Community
Component: gluster-smb
Version: 4.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Anoop C S
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-19 09:39 UTC by ryan
Modified: 2019-06-12 05:32 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-11 05:23:17 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description ryan 2019-02-19 09:39:12 UTC
Description of problem:

After running script found here, system will not longer start CTDB:
https://github.com/gluster/glusterfs/blob/master/extras/control-cpu-load.sh


Version-Release number of selected component (if applicable):


How reproducible:
Everytime


Steps to Reproduce:
1. Run script on Gluster heal process
2. Try to start CTDB service

Actual results:
CTDB fails to start with following error:
2019/02/08 20:46:59.612215 ctdbd[2629]: Created PID file /var/run/ctdb/ctdbd.pid
2019/02/08 20:46:59.612267 ctdbd[2629]: Listening to ctdb socket /var/run/ctdb/ctdbd.socket
2019/02/08 20:46:59.612297 ctdbd[2629]: Unable to set scheduler to SCHED_FIFO (Operation not permitted)
2019/02/08 20:46:59.612304 ctdbd[2629]: CTDB daemon shutting down

Expected results:
CTDB starts


Additional info:
Other systems where script was not run, works fine.
Have set DefaultCPUAccounting=no in /etc/systemd/system.conf.

systemctl show ctdb | grep -i accounting
CPUAccounting=no
BlockIOAccounting=no
MemoryAccounting=no
TasksAccounting=no

Comment 1 Anoop C S 2019-06-07 10:55:40 UTC
(In reply to ryan from comment #0)
> Actual results:
> CTDB fails to start with following error:
> 2019/02/08 20:46:59.612215 ctdbd[2629]: Created PID file
> /var/run/ctdb/ctdbd.pid
> 2019/02/08 20:46:59.612267 ctdbd[2629]: Listening to ctdb socket
> /var/run/ctdb/ctdbd.socket
> 2019/02/08 20:46:59.612297 ctdbd[2629]: Unable to set scheduler to
> SCHED_FIFO (Operation not permitted)
> 2019/02/08 20:46:59.612304 ctdbd[2629]: CTDB daemon shutting down

Please use the following CTDB setting in /etc/sysconfig/ctdb:
CTDB_NOSETSCHED=yes

and try restarting CTDB.

Comment 2 Anoop C S 2019-06-07 11:01:49 UTC
(In reply to Anoop C S from comment #1)
> (In reply to ryan from comment #0)
> > Actual results:
> > CTDB fails to start with following error:
> > 2019/02/08 20:46:59.612215 ctdbd[2629]: Created PID file
> > /var/run/ctdb/ctdbd.pid
> > 2019/02/08 20:46:59.612267 ctdbd[2629]: Listening to ctdb socket
> > /var/run/ctdb/ctdbd.socket
> > 2019/02/08 20:46:59.612297 ctdbd[2629]: Unable to set scheduler to
> > SCHED_FIFO (Operation not permitted)
> > 2019/02/08 20:46:59.612304 ctdbd[2629]: CTDB daemon shutting down
> 
> Please use the following CTDB setting in /etc/sysconfig/ctdb:
> CTDB_NOSETSCHED=yes
> 
> and try restarting CTDB.

Copy-pasting a summary of the reason for above suggestion from a different bug:

CTDB daemon i.e, ctdbd is a service that by default requests for real-time scheduling unless it is instructed not to do so via explicit configuration parameters. By default systemd places all system services into their own control groups in the "cpu" hierarchy. But the "cpu" cgroup controller of the kernel demands absolute real-time budget to be explicitly specified. A reasonable value for required real-time cpu cycles are pre-written into corresponding configuration files. This value getting overwritten by other components in the system results in denial of real-time scheduling to services under this "cpu" hierarchy with error EPERM(Operation not permitted).

ref: https://www.freedesktop.org/wiki/Software/systemd/MyServiceCantGetRealtime/

Comment 3 ryan 2019-06-10 09:52:41 UTC
Hello,

We are currently working around this issue with the configuration option you suggested 'CTDB_NOSETSCHED=yes', and I can confirm CTDB starts successfully with this.

Best,
Ryan

Comment 4 Anoop C S 2019-06-11 05:23:17 UTC
Closing the bug report as per confirmation in comment #3.

See comment #2 for details.

Comment 5 ryan 2019-06-11 15:51:47 UTC
Is there any way to undo the changes made by the script to allow real-time scheduling?

Comment 6 Anoop C S 2019-06-12 05:32:37 UTC
(In reply to ryan from comment #5)
> Is there any way to undo the changes made by the script to allow real-time
> scheduling?

If you are concerned about performance penalty in disabling real-time scheduling for CTDB then I don't think it will be obviously visible. If not you are good to go with current setting.
On the other side you may have to ask whoever came up with this script for more details around it.


Note You need to log in before you can comment on or make changes to this bug.