Bug 2216137

Summary:	"request URL too long" when pulling data from pmproxy
Product:	Red Hat Enterprise Linux 8	Reporter:	Evgeni Golov <egolov>
Component:	pcp	Assignee:	Nathan Scott <nathans>
Status:	VERIFIED ---	QA Contact:	Jan Kurik <jkurik>
Severity:	unspecified	Docs Contact:	Jacob Taylor Valdez <jvaldez>
Priority:	unspecified
Version:	8.8	CC:	jkurik, nathans, scox
Target Milestone:	rc	Keywords:	Bugfix, Triaged
Target Release:	8.9
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	pcp-5.3.7-18.el8	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Evgeni Golov 2023-06-20 09:01:28 UTC

Description of problem:
On a busy system, loading some metrics in Grafana (like hotproc.psinfo.rss in our case) errors out with:

Jun 20 08:26:22 pcp.example.com grafana-server[2312019]: t=2023-06-20T08:26:22+0000 lvl=eror msg="Received HTTP Error Response" logger=plugins.backend pluginId=pcp-redis-datasource code=414.000 data="<html>\r\n<head><title>414 URI Too Long</title></head>\r\n<body>\r\n<h1>414 URI Too Long</h1>\r\n<p><b>unknown servlet</b>: request URL too long</p><hr>\r\n<p><small><i>pmproxy/5.3.7</i></small></p>\r\n</body>\r\n</html>\r\n" url="http://pcp.example.com:44322/series/labels?series=a3d8ad83571e11cfe488a77ab15e2262a431139c%2Cb7fd03bc96068df375e6861ee76070697b0411b0%2Ccbb004475b7b82614b2bb140a68e228200cbdc76%2C6f8fdf502cda372806448c41e40e0404c8b343fa%2C62860603d3f9c932f2703817ca21c2bd95a9cb3d%2Cecd69bdd7fa323e2ba22f675f1a50347cb557703%2C7f84e0f0a1a25eabccde2d82e315cd721ab7ae22%2C4feaff2c7ddd3486087254a92640eec2012afed2%2Cd5a51d8ced66aae90b3a65a9e80ab0aaa4d92…

Version-Release number of selected component (if applicable):
grafana-pcp-3.2.0-3.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. have many series
2. load them in grafana

Actual results:
data can't be loaded

Expected results:
data is loaded

Additional info:
This has been fixed upstream in https://github.com/performancecopilot/grafana-pcp/pull/116 which is available since grafana-pcp 4.0, however it requires a change to pcp itself (https://github.com/performancecopilot/pcp/pull/1555) which is only available in PCP 6.0+

Comment 1 Evgeni Golov 2023-06-20 09:04:14 UTC

Or should this BZ be actually against pcp itself, as grafana-pcp will get an update via BZ#2193270, but that won't help before PCP is not also updated?

Comment 2 Nathan Scott 2023-06-21 02:57:06 UTC

Thanks for the report and excellent self-diagnosis Evgeni!  You're spot on,
I think, in that this is already fixed.  The PCP issue you mentioned has been
back-ported to RHEL 8.9 - I'll assign this to Stan for double-checking though.

Comment 3 Evgeni Golov 2023-06-21 10:12:25 UTC

I didn't see the patch in pcp-5.3.7-17 (as currently in CentOS Stream 8), but I didn't look *too* closely.

Comment 4 Nathan Scott 2023-07-05 01:58:16 UTC

You're right - I thought it was in the pmproxy-rollup-fixes patch but it's not.
I'll update that one with the next rebuild, thanks.