Bug 1476214 - running docker containers prevents processes to use real-time scheduling when restarted
running docker containers prevents processes to use real-time scheduling when...
Status: POST
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: corosync (Show other bugs)
7.4
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Jan Friesse
cluster-qe@redhat.com
: Triaged, ZStream
Depends On: 1469170 1467919
Blocks: 1415556 1477461
  Show dependency treegraph
 
Reported: 2017-07-28 06:14 EDT by Chris Jones
Modified: 2017-08-29 05:22 EDT (History)
29 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when the corosync service was started or restarted after systemd had enabled CPU Accounting, corosync was not able to run with Real Time (RT) scheduling priority, which could reduce the stability of the High Availability (HA) cluster. This update moves corosync to the root CPU cgroup by default, and now corosync can run with Real Time priority, as expected.
Story Points: ---
Clone Of: 1467919
: 1477461 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
igkioka: needinfo-


Attachments (Terms of Use)
Proposed patch (8.20 KB, patch)
2017-07-28 11:22 EDT, Jan Friesse
no flags Details | Diff
Proposed patch v2 - upstream (8.43 KB, patch)
2017-08-01 08:44 EDT, Jan Friesse
no flags Details | Diff
Proposed patch v2 - upstream (8.43 KB, patch)
2017-08-01 08:45 EDT, Jan Friesse
no flags Details | Diff

  None (edit)
Comment 2 Jan Friesse 2017-07-28 11:20:54 EDT
@Chris
I've implemented "workaround" which just moves corosync to root cpu cgroup on start using libcgroup.

Scratch build is https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=13742381

I've tested by using CPUAccounting=True in one of unit files (not the corosync one) but it would be nice if you could test it with docker use case.
Comment 3 Jan Friesse 2017-07-28 11:22 EDT
Created attachment 1305986 [details]
Proposed patch

main: Add support for libcgroup

When corosync is started in environment where it ends in cgroup without
properly set rt_runtime_us it's impossible to get RT priority.

Already implemented workaround is to use higher non-RT priority.

This patch implements another solution. It moves corosync into root cpu
cgroup. Root cpu cgroup hopefully has enough RT budget.

Another solution was mentioned on ML
https://lists.freedesktop.org/archives/systemd-devel/2017-July/039353.html
but this means to generate some "random" values.
Comment 4 Jan Friesse 2017-07-28 11:29:47 EDT
What I've tested ("Unit test"):
- Install httpd
- copy httpd.service into /etc and add "CPUAccounting=True" line to [service] section
- systemctl daemon-reload
- service httpd restart
- service corosync restart

(Before patch):

- corosync should have standard priority (no RT)

- Install updated corosync
- service corosync restart
- corosync should have RT priority
Comment 5 Chris Jones 2017-07-28 11:54:14 EDT
Scratch build tested in two different scenarios (by poki and rasca) and worked in both :)
Comment 6 Raoul Scarazzini 2017-07-28 11:59:39 EDT
@Jan I tested the new package and it does what we need.

With the corosync shipped in osp12:

[root@overcloud-controller-0 ~]# rpm -qa corosync
corosync-2.4.0-9.el7.x86_64
[root@overcloud-controller-0 ~]# ps -eo pid,class,rtprio,command --sort=+class | grep [c]orosync
 20229 RR      99 corosync
[root@overcloud-controller-0 ~]# systemctl restart corosync
[root@overcloud-controller-0 ~]# ps -eo pid,class,rtprio,command --sort=+class | grep [c]orosync
191635 TS       - corosync

So scheduler was changed. With the new corosync package:

[root@overcloud-controller-1 ~]# rpm -Uvh /home/heat-admin/corosync*
Preparing...                          ################################# [100%]
Updating / installing...
   1:corosynclib-2.4.0-9.el7.jf1      ################################# [ 25%]
   2:corosync-2.4.0-9.el7.jf1         ################################# [ 50%]
Cleaning up / removing...
   3:corosynclib-2.4.0-9.el7          ################################# [ 75%]
   4:corosync-2.4.0-9.el7             ################################# [100%]
[root@overcloud-controller-1 ~]# ps -eo pid,class,rtprio,command --sort=+class | grep [c]orosync
 19985 RR      99 corosync
[root@overcloud-controller-1 ~]# systemctl restart corosync
[root@overcloud-controller-1 ~]# ps -eo pid,class,rtprio,command --sort=+class | grep [c]orosync
 11204 RR      99 corosync

So scheduler have been kept.
Comment 8 Jan Friesse 2017-08-01 08:44 EDT
Created attachment 1307550 [details]
Proposed patch v2 - upstream
Comment 9 Jan Friesse 2017-08-01 08:45 EDT
Created attachment 1307551 [details]
Proposed patch v2 - upstream
Comment 12 Chris Jones 2017-08-18 07:10:18 EDT
Can I suggest that the doctext be:

Previously, if corosync was started (or restarted) after systemd had enabled CPU Accounting, corosync would not be able to run with Real Time scheduling priority, which could reduce the stability of the High Availability (HA) cluster. This update moves corosync to the root CPU cgroup by default, allowing it to obtain Real Time priority.
Comment 14 Jan Friesse 2017-08-18 10:29:31 EDT
Yep, Chris description sounds much better.

Note You need to log in before you can comment on or make changes to this bug.