Bug 1679535 - Can not collect pod's metrics with multitenant network
Summary: Can not collect pod's metrics with multitenant network
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.1.0
Assignee: Casey Callendrello
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-21 11:27 UTC by Junqi Zhao
Modified: 2019-06-04 10:44 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-04 10:44:14 UTC
Target Upstream Version:


Attachments (Terms of Use)
metrics diagram is empty for Pod (183.04 KB, image/png)
2019-02-21 11:27 UTC, Junqi Zhao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:44:20 UTC

Description Junqi Zhao 2019-02-21 11:27:37 UTC
Created attachment 1537000 [details]
metrics diagram is empty for Pod

Description of problem:
# oc get clusternetwork
NAME      CLUSTER NETWORK   SERVICE NETWORK   PLUGIN NAME
default   10.128.0.0/14     172.30.0.0/16     redhat/openshift-ovs-multitenant

# oc adm top pod --all-namespaces
nothing returned

deploy pod under project, 
# oc get pod -n test34
NAME              READY     STATUS      RESTARTS   AGE
dctest-1-deploy   0/1       Completed   0          141m
dctest-1-pjw2p    2/2       Running     0          141m

# oc -n test34 adm top pod
W0221 06:02:55.170762   15838 top_pod.go:268] Metrics not available for pod test34/dctest-1-deploy, age: 2h22m20.170747567s
error: Metrics not available for pod test34/dctest-1-deploy, age: 2h22m20.170747567s

# oc -n test34 adm top pod dctest-1-pjw2p
Error from server (NotFound): podmetrics.metrics.k8s.io "test34/dctest-1-pjw2p" not found

login web console, metrics diagram is empty for Pod, see the attached picture.
**************************************************************************
also compared with networkpolicy with the same payload, it does not have this issue, so it is related to multitenant

# oc get clusternetwork
NAME      CLUSTER NETWORK   SERVICE NETWORK   PLUGIN NAME
default   10.128.0.0/14     172.30.0.0/16     redhat/openshift-ovs-networkpolicy


Version-Release number of selected component (if applicable):
# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-20-194410   True        False         8h      Cluster version is 4.0.0-0.nightly-2019-02-20-194410


How reproducible:
Always

Steps to Reproduce:
1. Setup multitenant environment and follow steps in description part
2.
3.

Actual results:
Can not collect pod's metrics with multitenant network

Expected results:


Additional info:

Comment 1 Junqi Zhao 2019-02-22 09:44:03 UTC
Blocks collecting pod's metrics with multitenant network

Comment 3 Casey Callendrello 2019-02-22 12:53:16 UTC
Looking in to this; it *should* work, so I'm a bit surprised.

Comment 4 Casey Callendrello 2019-02-22 15:27:39 UTC
I just tested this with a Multitenant cluster, and I didn't see any issues. Can you try again? And, if there's a failure, provide "oc get netnamespaces"? Thanks!

Comment 6 Junqi Zhao 2019-02-26 04:31:36 UTC
Test again, issue is not happen
$ oc get clusternetwork
NAME      CLUSTER NETWORK   SERVICE NETWORK   PLUGIN NAME
default   10.128.0.0/14     172.30.0.0/16     redhat/openshift-ovs-multitenant

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-02-25-234632   True        False         58m     Cluster version is 4.0.0-0.nightly-2019-02-25-234632

$ oc -n test34 get pod
NAME              READY   STATUS      RESTARTS   AGE
dctest-1-deploy   0/1     Completed   0          6m27s
dctest-1-gmcbw    2/2     Running     0          6m12s


$ oc -n test34 adm top pod
NAME             CPU(cores)   MEMORY(bytes)   
dctest-1-gmcbw   0m           1Mi       

$ oc adm top pod --all-namespaces | grep -v NAME | wc -l
129

Comment 9 errata-xmlrpc 2019-06-04 10:44:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.