Bug 1646886

Summary: Empty network diagram on console UI in CRI-O env
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED ERRATA QA Contact: Sunil Choudhary <schoudha>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: aos-bugs, bjarvis, jcallen, jokerman, mpatel, nstielau, philipp.dallig, rvargasp, sjenning, sople, weinliu
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1741679 1741681 (view as bug list) Environment:
Last Closed: 2019-10-16 06:27:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1741679, 1741681    
Attachments:
Description Flags
network metrics graph could be shown - kube-system
none
There is notnetwork metrics graph - openshift-infra
none
network metrics graph could be shown after restarting atomic-openshift-node.service - openshift-infra
none
ansible logs
none
empty network diagram -- openshift-infra none

Description Junqi Zhao 2018-11-06 08:47:48 UTC
Created attachment 1502317 [details]
network metrics graph could be shown - kube-system

Description of problem:
This defect is from https://bugzilla.redhat.com/show_bug.cgi?id=1631300#c20
There are 3 namespaces(kube-system, openshift-node, openshift-sdn) could show network metrics graph, but it is empty for other namespace.
See the attached pictures.

Workaround is restart atomic-openshift-node on every nodes, then network metrics graph will be shown.

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.11.39

metrics-cassandra-v3.11.16-5
metrics-hawkular-metrics-v3.11.16-5
metrics-heapster-v3.11.16-3
metrics-schema-installer-v3.11.16-4

cri-o://1.11.8


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.11 in CRI-O env
2.
3.

Actual results:
There are 3 namespaces(kube-system, openshift-node, openshift-sdn) could show network metrics graph, but it is empty for other namespace.

Expected results:
Should show network metrics graph for all namespaces

Additional info:

Comment 1 Junqi Zhao 2018-11-06 08:48:47 UTC
Created attachment 1502318 [details]
There is notnetwork metrics graph - openshift-infra

Comment 2 Junqi Zhao 2018-11-06 08:49:08 UTC
Created attachment 1502319 [details]
network metrics graph could be shown after restarting atomic-openshift-node.service - openshift-infra

Comment 3 Junqi Zhao 2018-11-06 08:53:13 UTC
Note:
CRI-O sockert is changed to /var/run/crio/crio.sock from unix:///var/run/crio/crio.sock

we can see the warning info, it is recommend us to use format "unix:///var/run/crio/crio.sock".
# master-logs etcd etcd
W0925 07:16:36.726552   19184 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock".

Comment 11 Junqi Zhao 2019-06-11 03:18:13 UTC
Created attachment 1579226 [details]
ansible logs

Comment 15 Joseph Callen 2019-06-17 19:49:29 UTC
Fix for l_kubelet_node_name issue:
PR: https://github.com/openshift/openshift-ansible/pull/11699

Comment 17 Junqi Zhao 2019-06-19 05:58:44 UTC
Created attachment 1582082 [details]
empty network diagram -- openshift-infra

Comment 28 Seth Jennings 2019-08-08 03:11:26 UTC
Upstream fix
https://github.com/google/cadvisor/pull/2284

Comment 29 Seth Jennings 2019-08-08 03:26:58 UTC
Related origin issue
https://github.com/openshift/origin/issues/23492

Comment 30 Seth Jennings 2019-08-09 18:36:16 UTC
cherry-pick to cadvisor fork
https://github.com/openshift/google-cadvisor/pull/7

Comment 32 Seth Jennings 2019-08-12 16:16:54 UTC
origin bump
https://github.com/openshift/origin/pull/23585

Comment 34 Seth Jennings 2019-08-15 19:41:07 UTC
Brian,

Fix tracking for this issue for 3.11.z via https://bugzilla.redhat.com/show_bug.cgi?id=1741679

Comment 39 errata-xmlrpc 2019-10-16 06:27:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922