RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2185803 - RFE: Allow sending events to AAP using PMIE
Summary: RFE: Allow sending events to AAP using PMIE
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: pcp
Version: 9.1
Hardware: All
OS: All
medium
low
Target Milestone: rc
: 9.3
Assignee: Nathan Scott
QA Contact: Jan Kurik
Jacob Taylor Valdez
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-11 09:26 UTC by Marko Myllynen
Modified: 2023-11-07 09:36 UTC (History)
9 users (show)

Fixed In Version: pcp-6.0.4-1.el9
Doc Type: Enhancement
Doc Text:
.PCP's `pmie` utility now supports generating webhook events The Performance Metrics Inference Engine (`pmie`) utility from Performance Co-Pilot (PCP) now supports generating webhook events. With this update, configured `pmie` rules generate events in a format consumable by Event-Driven Ansible (EDA). As a result, EDA can respond to PCP rules. To enable this feature, configure all local `pmie` rules to send to a webhook at a given endpoint (URL): ---- # pmieconf modify global webhook_endpoint https://localhost:443/<endpoint> # pmieconf modify global webhook_action yes ----
Clone Of:
Environment:
Last Closed: 2023-11-07 08:30:40 UTC
Type: Enhancement
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-154358 0 None None None 2023-04-11 09:30:20 UTC
Red Hat Product Errata RHBA-2023:6406 0 None None None 2023-11-07 08:30:52 UTC

Description Marko Myllynen 2023-04-11 09:26:13 UTC
Description of problem:
PCP provides the PMIE mechanism to e.g. run scripts on predefined conditions, for example CPU utilization over 90% for 10 minutes or more than 50% of swap space used.

Event-Driven Automation (EDA), see https://www.ansible.com/use-cases/event-driven-automation, is a new way of automation where centralized automation can dynamically react to events that happen in an environment.

It could create a very powerful combination having these two technologies integrated. For instance, if CPU usage is high for a long time an action from EDA might be to add more vCPUs to the VM or add a new VM instance to the application cluster depending on the case.

The PCP PMIE mechanism should be enhanced so that there would be an easy to way send properly formatted events from PCP to Ansible EDA with PMIE.

While this should be doable today by skilled administrators it would better that the required integration scripts would be provided and documented as part of the standard PCP installation instead of requiring each user to write them locally.

Thanks.

Comment 2 Nathan Scott 2023-04-12 01:58:25 UTC
Neat - thanks for the pointers Marko, I went through the online lab and think this is a great idea.

My limited understanding is that we want a 'source' for PCP pmie events and this would (possibly?) live here:
https://github.com/ansible/event-driven-ansible/tree/main/plugins/event_source

We'd need to define a mechanism whereby pmie could run something locally when it evaluates events and send notification through to the event source handler, over some transport/IPC mechanism, most likely written in python based on the above repo contents.  We could use the existing 'shell' action for that or we may be able to make life easier for this code by having a new pmie action that generates detailed JSON content (e.g. containing the pmieconf rule names, which are not currently made available).  That action could do a http POST perhaps .... anyway, just mulling things over here.  It'd be very good if this tied in neatly with the existing rules we have that are being consumed by Insights rather than an entire new set of rules to manage.

It'd be good to find experts from the Ansible EDA team to work with to make this happen - if you know anyone I should talk to please let me know.

Comment 3 Marko Myllynen 2023-04-12 08:54:02 UTC
Hi Nathan, I'm glad this sounds good to you.

My understanding is also far from complete but I was hoping that ideally we should be able to utilize the existing webhook event source somehow. This would mean that only PCP-side changes would be needed if that'd be possible. And as you say, reusing existing rules would make most sense at least in the beginning, no need to redo anything there, just create hopefully a small Python script to act as an alternative way to emit events. Perhaps allowing the user to define a set of variables and the target on the PMIE side which then get passed over with the event.

Perhaps this pending blog post (in raw format so a bit hard to read) and references it contains might be helpful, it describes step-by-step how the Foreman/Satellite team setup testing environment for similar kind of development work. In essence, you won't need a full-blown AAP environment to mimic the server side webhook setup and that allows concentrating on the required local changes:

https://github.com/theforeman/theforeman.org/pull/2072/files

If we could achieve this in somewhat similar fashion then the Python script should be fairly straightforward and the most important part would be to define the "interface" on the PMIE side, or what do you think?

Thanks.

Comment 7 Nathan Scott 2023-04-12 22:08:33 UTC
Hey all,

Aha - the webhook event source is news to me, thanks - yes, that should work really well and integrate nicely with existing pmie setup.  Very cool.

I think some new code in pmie to ensure cleanly formatted JSON events can be generated, some pmieconf support for that, and we'd be pretty much there.

If I work on the PCP side of things, is there anyone who can help me with integration testing?  (i.e. someone with a working Ansible EDA setup and a webhook endpoint we could point this at?)

Thanks!

Comment 10 Marko Myllynen 2023-04-17 07:02:04 UTC
The Foreman/Satellite EDA integration blog post has now been published:

https://theforeman.org/2023/04/integrating-foreman-with-event-driven-ansible.html

Comment 11 Nathan Scott 2023-04-17 07:12:13 UTC
I've opened an upstream PCP pull request (#1756) with a prototype
pmie implementation; light testing only so far but looks promising
(using the webhook mechanism described in the Foreman blog post).


commit e809fee92ba1c0913a8f9393fc161de6955fb1ac
Author: Nathan Scott <nathans>
Date:   Mon Apr 17 17:05:57 2023 +1000

    pmieconf: add support for a webhook action to pmieconf
    
    Adds support for sending JSON formatted HTTP messages to a
    webhook endpoint when a pmie predicate triggers.  This can
    be used to augment (or replace) existing pmieconf actions,
    and is based on the existing pmie_email mechanisms.
    
    Quick 2-step primer for primary pmieconf-controlled pmies:
    
    # pmieconf modify global webhook_endpoint https://localhost:443/endpoint
    # pmieconf modify global webhook_action yes
    
    Resolves Red Hat BZ #2185803

Comment 12 Marko Myllynen 2023-04-18 13:00:19 UTC
Great work! I could try to do some light testing as well in a week or so as well, I'll let you know. Thanks!

Comment 13 Nathan Scott 2023-04-18 22:28:28 UTC
Thanks Marko.  I'm planning a full recipe/writeup later this week that should assist with wider testing, hopefully I can turn this into a blog post to.

Comment 20 errata-xmlrpc 2023-11-07 08:30:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pcp bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6406


Note You need to log in before you can comment on or make changes to this bug.