Bug 463326

Putting in NEEDINFO pending actual code.

Do you have a usage case for what sort of apps would use the library, and how? I'm not sure now creation of a new interface that apps need to be ported to first eases a port to another arch later.

> Putting in NEEDINFO pending actual code.

Development tree is available at git://ozlabs.org/srv/projects/libtopology/libtopology.git

> ------- Comment From notting 2008-10-02 11:52:05 EDT-------
> Do you have a usage case for what sort of apps would use the library, and how?

Sure.  Here's an extract from the README:

In HPC environments, for example, users often want to distribute jobs
within a node on separate processor cores.  On a machine with multiple
hardware threads per core, however, one must take care to bind the
tasks to the appropriate logical cpu IDs.  Consider an Intel x86
system with 4 cores, each with 2 hardware threads -- the logical CPUs
are numbered 0 through 7.  To distribute the workload equally among
cores, one would do something like this:

$ for c in 0 1 2 3 ; do taskset -c $c $my_job ; done

To perform the same task on a 4-core POWER5 system (again, two
hardware threads per core):

$ for c in 0 2 4 6 ; do taskset -c $c $my_job ; done

Why the difference in CPU numbering?  The way that the Linux kernel
assigns logical CPU numbers to hardware threads differs between
architectures (x86 and powerpc in this case).  This simple example
demonstrates that one must know rather intimately the details of a
given system in order to correctly implement a scheduling policy.  In
this case, one needs to know not only the total number of processors,
but also how CPUs are numbered on the system.

libtopology was created to address this kind of issue.  Using the
example cpumasks command included in the libtopology distribution, the same
command line works on both systems:

$ for mask in  ; do taskset $mask $my_job ; done

Additionally, this works regardless of the total number of cores.

The equivalent operation in the library's C API is provided by the
topology_for_each_core() interface (which is exactly what the cpumasks
command uses).

...

The general flow of a C program using libtopology could be something
like the following.  Assume the program wants to run on each logical
CPU in the system.  (Error handling is omitted.)

#define _GNU_SOURCE
#include <sched.h>
#include <stdlib.h>
#include <unistd.h>

#include <topology.h>

static void do_something(void) { return; }

int main(void) {
topology_thread_t thread;
topology_context_t ctx = topology_get_context();
size_t cpuset_size = topology_sizeof_cpu_set_t(ctx);
cpu_set_t *mask = malloc(cpuset_size);

topology_for_each_thread(ctx, thread) {
topology_thread_cpumask(ctx, thread, mask);
sched_setaffinity(getpid(), cpuset_size, mask);
do_something();
}

topology_release_context(ctx);

return 0;
}

> I'm not sure now creation of a new interface that apps need to be ported to
> first eases a port to another arch later.

The currently available interface (sysfs) for discovering hardware topology is not terribly easy to use, and application developers have been reluctant to use it directly because of perceived interface instability.  Often users simply "hard-code" CPU numbers (as above) or make platform- or machine-specific assumptions in their code.

The major value of the library is that it does the work of extracting the topology information from sysfs and provides an intuitive interface which allows users to formulate policy in an abstract and machine-independent way.

So, as to your concern: when porting an application from (say) x86 to POWER, it would make sense to first change the app to use libtopology and ensure it runs as expected on x86.  Then the app should run as expected on POWER with respect to topology-related behavior.  And further porting to other architectures is eased.

OK, reading the base code, it doesn't appear to be API/ABI set yet (or even set a soname)... those sorts of things obviously need done before apps can rely on it.

Also, it seems tied to the sysfs layout. Not sure that can be helped, but I can see it becoming an issue in the future.

(In reply to comment #8)
> ------- Comment From notting 2008-10-06 15:03:35 EDT-------
> OK, reading the base code, it doesn't appear to be API/ABI set yet (or even set
> a soname)... those sorts of things obviously need done before apps can rely on
> it.

Right. On the TODO list...

> Also, it seems tied to the sysfs layout. Not sure that can be helped, but I can
> see it becoming an issue in the future.

The portions of sysfs which the library uses - /sys/devices/system/{node,cpu} -  have been stable for several years (new things have been added but I'm not aware of an ABI-breaking change in the past).  There is no other interface for discovering the topology information in a platform-independent way.

If there is an incompatible sysfs change in the future, naturally the library will have to insulate users from that.  However, I view this as a low risk; sysfs developers are well aware that changing the paths in question would break existing applications and they are reluctant to change them:

http://lkml.org/lkml/2008/7/2/260

http://libtopology.ozlabs.org/
version 0.3

Verdict: libtopology will be included to RHEL6 release.

Cheers! :)

This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux major release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux major release.   
While it is a goal to include this enhancement in the next major release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next major release pending the next phase of actual 
code integration and successful Red Hat and partner testing.

closing this bug finally. libtopology appeared in el6 tree.

------- Comment From nathanl.com 2010-04-28 12:18 EDT-------
Verified on JS22 POWER6 blade:

# rpm -ql libtopology
/usr/lib64/libtopology.so.0
/usr/lib64/libtopology.so.0.3
/usr/share/doc/libtopology-0.3
/usr/share/doc/libtopology-0.3/COPYING

Ran testsuite against the shipped library (100% pass); verified that sample programs in source distribution list caches and cpumasks as expected when linked against shipped library.

Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.