It looks like newer RHEL 7 kernels have included identification and pmu events for Intel KNL.  The remaining question is whether people at RH have access to KNL machine to verify functionality of the papi and libpfm support.

 papi that was built with the older libpfm-4.7.0-1 is not finding the newer knl events even when using the newer shared libraries from libpfm-4.7.0-4 that have the events:

Ah here is where things seem to go wrong in papi_avail. Using a locally built papi rpm built with "--with-debug=yes" in the configure.  Get different results for the following command. 

export PAPI_DEBUG=ALL
papi_avail >& papi.log

For the the papi built with libpfm-4.7.0-1 installed on the machine:

SUBSTRATE:papi_libpfm4_events.c:_papi_libpfm4_init:118:71093 SUBSTRATE:components/appio/appio.c:fwrite:389:71093 appio: intercepted fwrite\
(0x7fec5eeef9d4,1,18,0x7fec5ec6a1c0)
pfm_get_version()
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1374:71093 SUBSTRATE:components/appio/appio.c:fwrite:389:71093 appio:\
 intercepted fwrite(0x7fec5ef0edba,1,15,0x7fec5ec6a1c0)
Detected pmus:
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1383:71093         18 ix86arch Intel X86 architectural PMU 1
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1383:71093         51 perf perf_events generic PMU 3
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1383:71093         114 perf_raw perf_events raw PMU 3
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1406:71093 153 native events detected on 3 pmus
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1409:71093 SUBSTRATE:components/appio/appio.c:fwrite:389:71093 appio:\
 intercepted fwrite(0x7fec5ef0ee1e,1,27,0x7fec5ec6a1c0)
Could not find default PMU

For the papi built with libpfm-4.7.0-4 see:

SUBSTRATE:papi_libpfm4_events.c:_papi_libpfm4_init:118:57525 SUBSTRATE:components/appio/appio.c:fwrite:389:57525 appio: intercepted fwrite\
(0x7f686180f9d4,1,18,0x7f686158a1c0)
pfm_get_version()
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1374:57525 SUBSTRATE:components/appio/appio.c:fwrite:389:57525 appio:\
 intercepted fwrite(0x7f686182edba,1,15,0x7f686158a1c0)
Detected pmus:
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1383:57525         18 ix86arch Intel X86 architectural PMU 1
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1383:57525         51 perf perf_events generic PMU 3
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1383:57525         114 perf_raw perf_events raw PMU 3
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1383:57525         203 knl Intel Knights Landing 1
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1394:57525           knl is default
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1406:57525 184 native events detected on 4 pmus
SUBSTRATE:components/perf_event/pe_libpfm4_events.c:_pe_libpfm4_init:1423:57525 num_counters: 5
SUBSTRATE:papi_preset.c:_papi_load_preset_table:318:57525 SUBSTRATE:components/appio/appio.c:fwrite:389:57525 appio: intercepted fwrite(0x\
7f686182c22b,1,6,0x7f686158a1c0)
ENTER

Yep. And that's why it does not show any events when testing on KNL. A respin is probably unavoidable.

The problem is cause by use of the PFM_PMU_MAX enum in papi's pe_libpfm4_event.c initialization loop:

   for(i=0;i<PFM_PMU_MAX;i++) {
   ...
   }

The older libpfm-4.7.0-1 that papi was built with has a smaller value for PFM_PMU_MAX than the libpfm-4.7.0-4.  Thus the pmu entries added in the newer libpfm were not scanned by papi.

There should be a better way of doing this in libpfm.  As new pmus are added to libpfm the other code should not need to be recompiled because of a change li PFM_PMU_MAX.

Posted patch in progress for upstream at

https://groups.google.com/a/icl.utk.edu/forum/#!topic/ptools-perfapi/Dx3xjwxplWo

How is this looking so far? Has the patch a chance to get upstream? Or should we just rebuild papi once more against the proper libpfm and close this bug?

Vince Weaver expected that something like the proposed patch would be put in papi.  He did ask some questions on how to test, but I haven't heard back.  I pinged him on the mailing list.

Worst case we do a rebuild with the newest version of libpfm in rhel7 next week.  However, I would prefer to get the patch in so that we can avoid future failures because papi was built with an older version of libpfm.  That dependency on PFM_PMU_MAX really should go away and make builds less problematic because I forgot to do a buildroot override.

A patch has been included in the upstream papi to address the problem found with the Intel KNL testing where an older version of libpfm used during the build of papi prevented the discovery of the Intel KNL perf hardware.  papi would only examine up to the last pmu enum in the libpfm used during the build rather than whatever the last pmu enum is in the currently installed libpfm. There has been a patch incorporated into the upstream papi to address this problem:

https://bitbucket.org/icl/papi/commits/ba786c0c040c9caea5e9aa2aaed60cfcd1e57e3a

The patch has adapted into a backport for the papi-5.2.0 in rhel7.  However, need to have an exception flag set to allow checking in the patch and rebuilding the papi package so papi intel knl support works.

Created attachment 1288239 [details]
Backport of patch to avoid tying search to build libpfm list of pmus

This is an adaptation of the upstream patch to avoid the use of PFM_PMU_MAX from the libpfm headers used at build time.  This allows checking of any additional pmus that might not have been in the libpfm used for the build.

According to mpetlan the scratch build addressed the problem. Respun official papi-5.2.0-23.el7 build with the patch to address the problem.

Tested papi-5.2.0-23.el7 build and this one works great on Knights Landing! Thanks for fixing!

VERIFIED

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2190