Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2027428

Summary: [RFE] Ad-hoc PCP archive analysis with Grafana
Product: Red Hat Enterprise Linux 9 Reporter: Andreas Gerstmayr <agerstmayr>
Component: pcpAssignee: Andreas Gerstmayr <agerstmayr>
Status: CLOSED UPSTREAM QA Contact: Jan Kurik <jkurik>
Severity: unspecified Docs Contact: Jacob Taylor Valdez <jvaldez>
Priority: unspecified    
Version: 9.0CC: agerstmayr, chorn, cww, jkurik, nathans, peter.vreman, pportant, psatpute, surkumar
Target Milestone: rcKeywords: FutureFeature, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-01 10:54:56 UTC Type: Enhancement
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andreas Gerstmayr 2021-11-29 16:07:40 UTC
The current architecture of pmproxy discovery isn't well suited for Ad-hoc analysis of single PCP archives with Grafana.

Some issues:
- remote hosts may be air-gapped, and not part of a pmlogger farm and therefore are not discovered automatically
- after analysis, the metric values should be deleted from the Redis database again

Possible solutions:
a) a custom container with PCP, Redis, Grafana & grafana-pcp pre-installed, which loads all PCP archives from the host into the Redis database on startup (using a container bind mount) and has Grafana running on a port accessibly by the host
b) a new Grafana data source (written in Go) which uses libpcp to read the PCP archive files directly from the host

I think a) makes the most sense, as with b) we're re-inventing the wheel and having 4 datasources for PCP seems excessive (and more maintenance overhead).

Option a) could come with a custom shell script, which abstracts the container usage (it could take care of the bind mount, port forwarding and container start and stop).

Note: This is RFE based on a customer request in https://bugzilla.redhat.com/show_bug.cgi?id=2026726#c2

Comment 2 Nathan Scott 2021-11-29 22:59:18 UTC
Big fan of option a/ here - this path gives us alot of options, whereas option b/ gives us alot of development work to roughly achieve what we can already do now (from the end users perspective).

There's also been discussion in the past about us providing a (cloud based and/or internal) service where Red Hat customer support folk could be sent PCP archives (or send 'em themselves) for processing - and the output would be a running container with PCP, Redis and Grafana with all PCP metrics pre-loaded.

I like this option for several reasons:
- it removes manual effort from our support folk to get up and running with PCP, Redis and Grafana
- we can potentially deliver improvements to the analysis experience outside of the RHEL cycle (i.e. fast access to fixes for our support folk)
- in the future (with our proposed work to extend sysstat to generate PCP archives), customers running sysstat would also benefit from this service
- in the further future, we could deliver pmdiff, pcp2pdf, even AI/ML-based analysis of the uploaded archive(s) during the upload/pre-processing stage, that would further automate assisting our support folks with performance analysis of customer data

Comment 3 Andreas Gerstmayr 2021-11-30 16:06:42 UTC
(In reply to Nathan Scott from comment #2)
> Big fan of option a/ here - this path gives us alot of options, whereas
> option b/ gives us alot of development work to roughly achieve what we can
> already do now (from the end users perspective).

+1, let's go with option a)

> There's also been discussion in the past about us providing a (cloud based
> and/or internal) service where Red Hat customer support folk could be sent
> PCP archives (or send 'em themselves) for processing - and the output would
> be a running container with PCP, Redis and Grafana with all PCP metrics
> pre-loaded.

So the PCP metrics would be embedded / part of the container image?
I was thinking of a general container image which is loading the PCP archives at container startup from the host via a container bind mount.

> 
> I like this option for several reasons:
> - it removes manual effort from our support folk to get up and running with
> PCP, Redis and Grafana

+1

> - we can potentially deliver improvements to the analysis experience outside
> of the RHEL cycle (i.e. fast access to fixes for our support folk)

Yep, we could publish the container on Docker Hub or Quay.io

> - in the future (with our proposed work to extend sysstat to generate PCP
> archives), customers running sysstat would also benefit from this service

+1

> - in the further future, we could deliver pmdiff, pcp2pdf, even AI/ML-based
> analysis of the uploaded archive(s) during the upload/pre-processing stage,
> that would further automate assisting our support folks with performance
> analysis of customer data

Sounds good, that would be a custom web-based platform running in the container, with buttons to export PDFs, show diffs visually, another button to open Grafana etc.?
(fwiw, let's focus on the container with PCP+Redis+Grafana for this BZ, and add the future features (the last point above) in the upstream roadmap on GitHub?)


Afaics Mustafa from Peter's Team already worked on a container [1], we can build upon that.

[1] https://github.com/distributed-system-analysis/pbench/tree/main/agent/containers/images/visualizers (PCPGrafVisualizer)

Comment 7 Andreas Gerstmayr 2022-09-01 10:54:56 UTC
The container is now available upstream at https://quay.io/repository/performancecopilot/archive-analysis and will track the latest versions of PCP and Grafana.

Importing PCP archives created with older versions of RHEL/PCP is supported, there is no requirement of a matching version of the PCP archive and this container.
In case you encounter an error importing an archive, please open a bug report.

Comment 8 Andreas Gerstmayr 2022-09-06 10:28:54 UTC
https://access.redhat.com/solutions/6974688