Description of problem: When viewing threads in system monitoring tools such as top, htop or ps, all (100's) of vdsm threads show the same name, so it is very hard to tell which are the top threads consuming most cpu time. Solution: Use pthread_set_name_np api to set thread name. This function should be available through ctypes.
See http://man7.org/linux/man-pages/man3/pthread_setname_np.3.html
patches posted and under review.
Francesco, the patch linked in the External Trackers is merged. Are we pending anything else, or could this be moved to MODIFIED?
The patch http://gerrit.ovirt.org/41052 added the infrastructure to make the names visible. We miss a sensible way to translate python names, which are arbitrarily long, in system thread names, which must be long at most 15 ASCII characters. We didn't find a good way to do so yet.
(In reply to Francesco Romani from comment #4) > The patch http://gerrit.ovirt.org/41052 added the infrastructure to make the > names visible. We miss a sensible way to translate python names, which are > arbitrarily long, in system thread names, which must be long at most 15 > ASCII characters. We didn't find a good way to do so yet. I guess it means we should move this for 4.0? As the proposed patch doesn't complete this RFE. Right?
(In reply to Oved Ourfali from comment #5) > (In reply to Francesco Romani from comment #4) > > The patch http://gerrit.ovirt.org/41052 added the infrastructure to make the > > names visible. We miss a sensible way to translate python names, which are > > arbitrarily long, in system thread names, which must be long at most 15 > > ASCII characters. We didn't find a good way to do so yet. > > I guess it means we should move this for 4.0? As the proposed patch doesn't > complete this RFE. Right? I think yes, as no simple solution emerged yet.
Francesco - would you be interested in owning that in 4.0?
(In reply to Oved Ourfali from comment #7) > Francesco - would you be interested in owning that in 4.0? I am, we are not that far from this stage after last patches landed in master (from weeks ago, not much progress lately)
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Not all patches merged yet, moving back to post.
Threads using vdsm.concurrent.thread helper are showing (truncated) python thread name now. To complete this feature: - Convert old code using threading.Thread directly to use vdsm.concurrent.thread - Modify all thread names to use short names that are not truncated, or usable when truncated. Part of the work is handled in this topic: https://gerrit.ovirt.org/#/q/topic:thread-cleanup+is:open
(In reply to Nir Soffer from comment #11) > Threads using vdsm.concurrent.thread helper are showing (truncated) python > thread > name now. > > To complete this feature: > - Convert old code using threading.Thread directly to use > vdsm.concurrent.thread > - Modify all thread names to use short names that are not truncated, or > usable > when truncated. > > Part of the work is handled in this topic: > https://gerrit.ovirt.org/#/q/topic:thread-cleanup+is:open All patches are now been merged, so I'm moving this as MODIFIED. Should we discover we forgot some threads, we'll file a new bug. Caveat: some threads may come for external packages (e.g. python-ioprocess), we should either fix those packages or skip them. To test this change, just keep "htop" running while Vdsm is working, to see the thread names. Any other tool which shows thread names besides "htop" is fine as well.
With the additional patches, all threads are named. ioprocess thread names are handled in bug 1392214. Some threads name are truncated, I suggest we open another bug for this.
The fix for this issue should be included in oVirt 4.1.0 beta 1 released on December 1st. If not included please move back to modified.
ok, vdsm-4.18.999-1173.git28e001a.el7.centos.x86_64 [root@dell-r210ii-13 ~]# systemctl status vdsmd | grep PID Main PID: 15413 (vdsm) [root@dell-r210ii-13 ~]# ps axH -q 15413 -o 'pid tid comm args' PID TID COMMAND COMMAND 15413 15413 vdsm /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15450 libvirt/events /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15451 tasks/0 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15452 tasks/1 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15453 tasks/2 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15454 tasks/3 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15455 tasks/4 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15456 tasks/5 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15457 tasks/6 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15458 tasks/7 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15459 tasks/8 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15460 tasks/9 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15464 check/loop /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15466 vdsm.Scheduler /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15474 vmchannels /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15491 JsonRpc (StompR /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15492 jsonrpc/0 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15493 jsonrpc/1 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15494 jsonrpc/2 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15495 jsonrpc/3 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15496 jsonrpc/4 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15497 jsonrpc/5 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15498 jsonrpc/6 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15499 jsonrpc/7 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15500 JsonRpcServer /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15501 BindingXMLRPC /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15502 Reactor thread /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15503 periodic/0 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15504 periodic/1 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15505 periodic/2 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15506 periodic/3 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15960 ioprocess/15959 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15966 monitor/53e447f /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15972 mailbox-hsm/0 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15973 mailbox-hsm/1 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15974 mailbox-hsm/2 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15975 mailbox-hsm/3 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15976 mailbox-hsm/4 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15983 mailbox-hsm /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15997 mailbox-spm/0 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15998 mailbox-spm/1 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 15999 mailbox-spm/2 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 16000 mailbox-spm/3 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 16001 mailbox-spm/4 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 16003 mailbox-spm /usr/bin/python2 /usr/share/vdsm/vdsm 15413 17365 ioprocess/17364 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 17436 monitor/0c78b4d /usr/bin/python2 /usr/share/vdsm/vdsm 15413 19033 ioprocess/19032 /usr/bin/python2 /usr/share/vdsm/vdsm 15413 19035 ioprocess/19034 /usr/bin/python2 /usr/share/vdsm/vdsm [root@dell-r210ii-13 ~]# rpm -q vdsm vdsm-4.18.999-1173.git28e001a.el7.centos.x86_64