Red Hat Bugzilla – Bug 1378184
pcp pmdas/metrics that don't support container inspection should be blocked rather than returning host results
Last modified: 2017-02-08 15:17:46 EST
Container support across the PMDAs, and even within some PMDAs, is inconsistent. When a user tries % pminfo -f -h 'pcp://....?container=HEXCODE' one gets a mishmash of host and container data, with no indication about which is which. We should audit the PCP PMDA codebase, and identify PMDAs/metrics that disregard a requested container spec. They should return "no values" or something like that instead of a bad value.
> [...] one gets a mishmash of host and container data [...] This is the nature of containers (host data is accessible from within container, esp. thru interfaces like procfs), and not something anyone is ever likely to change. If the kernel allows access to it, PCP should too. Containers setup specifically for monitoring (i.e. running monitoring tools/daemons in a container rather than on the host) will also break if the change suggested here was made.
(In reply to Nathan Scott from comment #1) > > [...] one gets a mishmash of host and container data [...] > > This is the nature of containers (host data is accessible from within > container, esp. thru interfaces like procfs) On the contrary, only limited host data is available to normal containers, by purpose. Host data that leaks through procfs is generally a bug. Host (and other containers') filesystems and processes generally do not leak through. That's the whole point. > and not something anyone is ever likely to change. If the kernel allows access to it, PCP should too. PCP presents a higher-level view of the system than the raw procfs. It can & probably should endavour to work around kernel bugs on virtualized procfs/sysfs values. We could even just blacklist procfs/sysfs metrics that we know the kernel is lying about. But that's a small point anyway. The promise of "pm$client --container=FOO" is to show all pcp statistics from within the point of view of the given container. Many PMDAs completely disregard the parameter. Instead as comment #0 says, we should audit them all and make sure PCP does not pretend. > Containers setup specifically for monitoring (i.e. running monitoring > tools/daemons in a container rather than on the host) will also break if the > change suggested here was made. I don't see how this is the case.
I don't agree, obviously. We (PCP folk) wont be working on this - unless you take this on Frank, and reassign to yourself with some realistic deliverable timeframe, this BZ will be closed shortly.
What don't you agree with? That "pm$client --container=FOO" is meant to look at the data in container FOO? That pcp does not do that currently for many metrics?
| What don't you agree with? Statements like this... (as just one example) | Host data that leaks through procfs is generally a bug. It's your opinion that the above is a (kernel) bug. IMO, this is not the case. Containers are not virtual machines. Files like /proc/stat, /proc/diskstats, etc - these often remain exposed via containers, and this is not a bug IMO. If the data is available, PCP can and should continue to export these stats. It is not our place to put restrictions above and beyond what the kernel does. I understand you have a different opinion, and that's fine. As I said before, the PCP team will not be working on this (unless someone other than me in the PCP team has a burning desire to hack on it - but it's been several months with noone else commenting) - we have so many things we need to get done that users and/or customers have requested. However, if you feel strongly about change in this area, please start providing patches upstream showing the scope and justification for change. So, if you do not wish to reassign this to yourself or someone in your team to look at, I'll be closing this one out shortly. cheers.
Your persistent skipping of other examples - like most PMDAs not participating in the container attribute logic at all - suggests that you have no intent to offer a definition of what "pm$client --container=FOO" means. The program and its user will continue to have no way of telling whether any particular metric is actually containerized or just host data.
No, the definition is the same as that of "containers" in the kernel - sometimes its container-specific (i.e. namespace/cgroup-modified) values, sometimes its not - its up to the user to know, not for us to arbitrarily decide and enforce.