Bug 463326 - [LTC 6.0 FEAT] 201722:system topology information interface
[LTC 6.0 FEAT] 201722:system topology information interface
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libtopology (Show other bugs)
6.0
ppc64 All
high Severity high
: alpha
: 6.0
Assigned To: Anton Arapov
Mike Gahagan
: FutureFeature, Reopened
Depends On:
Blocks: 391521 356741
  Show dependency treegraph
 
Reported: 2008-09-22 18:10 EDT by IBM Bug Proxy
Modified: 2014-06-18 04:02 EDT (History)
11 users (show)

See Also:
Fixed In Version: libtopology-0.3-7.el6
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-11-15 09:30:23 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description IBM Bug Proxy 2008-09-22 18:10:32 EDT
=Comment: #0=================================================
Emily J. Ratliff <emilyr@us.ibm.com> - 2008-09-16 18:29 EDT
1. Feature Overview:
Feature Id:	[201722]
a. Name of Feature:	system topology information interface
b. Feature Description
Architecture-independent interface to system topology information.  The idea is to have an
arch-independent library that presents topology information

2. Feature Details:
Sponsor:	PPC - P
Architectures:
ppc64

Arch Specificity: Both
Affects Installer: Yes
Affects Core Kernel: Yes
Delivery Mechanism: Backport
Category:	Power
Request Type:	Package - Feature from Upstream
d. Upstream Acceptance:	Not Evaluated
Sponsor Priority	1
f. Severity: High
IBM Confidential:	no
Code Contribution:	IBM code
g. Component Version Target:	this is tentatively targetted to reside on ozlabs.org.  When a library
is submitted and the page set up this area will be updated with the exact url

3. Business Case
To ease migration of apps from other architectures to Power

4. Primary contact at Red Hat: 
John Jarvis
jjarvis@redhat.com

5. Primary contacts at Partner:
Project Management Contact:
Michael Hohnbaum, hbaum@us.ibm.com, 503-578-5486

Technical contact(s):

Nathan Lynch, natlynch@us.ibm.com

IBM Manager:
Pat Gaughen, gaughen@us.ibm.com
Comment 1 Bill Nottingham 2008-10-02 11:51:18 EDT
Putting in NEEDINFO pending actual code.
Comment 2 Bill Nottingham 2008-10-02 11:52:05 EDT
Do you have a usage case for what sort of apps would use the library, and how? I'm not sure now creation of a new interface that apps need to be ported to first eases a port to another arch later.
Comment 3 IBM Bug Proxy 2008-10-03 17:15:45 EDT
> Putting in NEEDINFO pending actual code.

Development tree is available at git://ozlabs.org/srv/projects/libtopology/libtopology.git

> ------- Comment From notting@redhat.com 2008-10-02 11:52:05 EDT-------
> Do you have a usage case for what sort of apps would use the library, and how?

Sure.  Here's an extract from the README:

In HPC environments, for example, users often want to distribute jobs
within a node on separate processor cores.  On a machine with multiple
hardware threads per core, however, one must take care to bind the
tasks to the appropriate logical cpu IDs.  Consider an Intel x86
system with 4 cores, each with 2 hardware threads -- the logical CPUs
are numbered 0 through 7.  To distribute the workload equally among
cores, one would do something like this:

$ for c in 0 1 2 3 ; do taskset -c $c $my_job ; done

To perform the same task on a 4-core POWER5 system (again, two
hardware threads per core):

$ for c in 0 2 4 6 ; do taskset -c $c $my_job ; done

Why the difference in CPU numbering?  The way that the Linux kernel
assigns logical CPU numbers to hardware threads differs between
architectures (x86 and powerpc in this case).  This simple example
demonstrates that one must know rather intimately the details of a
given system in order to correctly implement a scheduling policy.  In
this case, one needs to know not only the total number of processors,
but also how CPUs are numbered on the system.

libtopology was created to address this kind of issue.  Using the
example cpumasks command included in the libtopology distribution, the same
command line works on both systems:

$ for mask in  ; do taskset $mask $my_job ; done

Additionally, this works regardless of the total number of cores.

The equivalent operation in the library's C API is provided by the
topology_for_each_core() interface (which is exactly what the cpumasks
command uses).

...

The general flow of a C program using libtopology could be something
like the following.  Assume the program wants to run on each logical
CPU in the system.  (Error handling is omitted.)

#define _GNU_SOURCE
#include <sched.h>
#include <stdlib.h>
#include <unistd.h>

#include <topology.h>

static void do_something(void) { return; }

int main(void) {
topology_thread_t thread;
topology_context_t ctx = topology_get_context();
size_t cpuset_size = topology_sizeof_cpu_set_t(ctx);
cpu_set_t *mask = malloc(cpuset_size);

topology_for_each_thread(ctx, thread) {
topology_thread_cpumask(ctx, thread, mask);
sched_setaffinity(getpid(), cpuset_size, mask);
do_something();
}

topology_release_context(ctx);

return 0;
}

> I'm not sure now creation of a new interface that apps need to be ported to
> first eases a port to another arch later.

The currently available interface (sysfs) for discovering hardware topology is not terribly easy to use, and application developers have been reluctant to use it directly because of perceived interface instability.  Often users simply "hard-code" CPU numbers (as above) or make platform- or machine-specific assumptions in their code.

The major value of the library is that it does the work of extracting the topology information from sysfs and provides an intuitive interface which allows users to formulate policy in an abstract and machine-independent way.

So, as to your concern: when porting an application from (say) x86 to POWER, it would make sense to first change the app to use libtopology and ensure it runs as expected on x86.  Then the app should run as expected on POWER with respect to topology-related behavior.  And further porting to other architectures is eased.
Comment 4 Bill Nottingham 2008-10-06 15:03:35 EDT
OK, reading the base code, it doesn't appear to be API/ABI set yet (or even set a soname)... those sorts of things obviously need done before apps can rely on it.

Also, it seems tied to the sysfs layout. Not sure that can be helped, but I can see it becoming an issue in the future.
Comment 5 IBM Bug Proxy 2008-10-06 17:15:32 EDT
(In reply to comment #8)
> ------- Comment From notting@redhat.com 2008-10-06 15:03:35 EDT-------
> OK, reading the base code, it doesn't appear to be API/ABI set yet (or even set
> a soname)... those sorts of things obviously need done before apps can rely on
> it.

Right. On the TODO list...

> Also, it seems tied to the sysfs layout. Not sure that can be helped, but I can
> see it becoming an issue in the future.

The portions of sysfs which the library uses - /sys/devices/system/{node,cpu} -  have been stable for several years (new things have been added but I'm not aware of an ABI-breaking change in the past).  There is no other interface for discovering the topology information in a platform-independent way.

If there is an incompatible sysfs change in the future, naturally the library will have to insulate users from that.  However, I view this as a low risk; sysfs developers are well aware that changing the paths in question would break existing applications and they are reluctant to change them:

http://lkml.org/lkml/2008/7/2/260
Comment 9 IBM Bug Proxy 2009-03-02 13:12:09 EST
http://libtopology.ozlabs.org/
version 0.3
Comment 10 Anton Arapov 2009-11-20 10:07:41 EST
Verdict: libtopology will be included to RHEL6 release.

Cheers! :)
Comment 12 John Jarvis 2009-12-17 15:25:00 EST
This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux major release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux major release.   
While it is a goal to include this enhancement in the next major release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next major release pending the next phase of actual 
code integration and successful Red Hat and partner testing.
Comment 13 Anton Arapov 2010-01-06 10:43:45 EST
closing this bug finally. libtopology appeared in el6 tree.
Comment 16 IBM Bug Proxy 2010-04-28 12:22:55 EDT
------- Comment From nathanl@austin.ibm.com 2010-04-28 12:18 EDT-------
Verified on JS22 POWER6 blade:

# rpm -ql libtopology
/usr/lib64/libtopology.so.0
/usr/lib64/libtopology.so.0.3
/usr/share/doc/libtopology-0.3
/usr/share/doc/libtopology-0.3/COPYING

Ran testsuite against the shipped library (100% pass); verified that sample programs in source distribution list caches and cpumasks as expected when linked against shipped library.
Comment 17 releng-rhel@redhat.com 2010-11-15 09:30:23 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.