Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1646886 - Empty network diagram on console UI in CRI-O env
Summary: Empty network diagram on console UI in CRI-O env
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.2.0
Assignee: Seth Jennings
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks: 1741679 1741681
TreeView+ depends on / blocked
 
Reported: 2018-11-06 08:47 UTC by Junqi Zhao
Modified: 2019-10-16 06:27 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1741679 1741681 (view as bug list)
Environment:
Last Closed: 2019-10-16 06:27:40 UTC
Target Upstream Version:


Attachments (Terms of Use)
network metrics graph could be shown - kube-system (177.37 KB, image/png)
2018-11-06 08:47 UTC, Junqi Zhao
no flags Details
There is notnetwork metrics graph - openshift-infra (200.19 KB, image/png)
2018-11-06 08:48 UTC, Junqi Zhao
no flags Details
network metrics graph could be shown after restarting atomic-openshift-node.service - openshift-infra (195.43 KB, image/png)
2018-11-06 08:49 UTC, Junqi Zhao
no flags Details
ansible logs (1.29 MB, text/plain)
2019-06-11 03:18 UTC, Junqi Zhao
no flags Details
empty network diagram -- openshift-infra (217.56 KB, image/png)
2019-06-19 05:58 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift google-cadvisor pull 7 0 None closed UPSTREAM: google/cadvisor: 2284: container/crio: retry getting pid if 0 2020-08-25 22:15:58 UTC
Github openshift origin pull 23585 0 None closed Bug 1646886: bump cadvisor 2020-08-25 22:15:57 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:27:56 UTC

Description Junqi Zhao 2018-11-06 08:47:48 UTC
Created attachment 1502317 [details]
network metrics graph could be shown - kube-system

Description of problem:
This defect is from https://bugzilla.redhat.com/show_bug.cgi?id=1631300#c20
There are 3 namespaces(kube-system, openshift-node, openshift-sdn) could show network metrics graph, but it is empty for other namespace.
See the attached pictures.

Workaround is restart atomic-openshift-node on every nodes, then network metrics graph will be shown.

Version-Release number of selected component (if applicable):
# openshift version
openshift v3.11.39

metrics-cassandra-v3.11.16-5
metrics-hawkular-metrics-v3.11.16-5
metrics-heapster-v3.11.16-3
metrics-schema-installer-v3.11.16-4

cri-o://1.11.8


How reproducible:
Always

Steps to Reproduce:
1. Deploy metrics 3.11 in CRI-O env
2.
3.

Actual results:
There are 3 namespaces(kube-system, openshift-node, openshift-sdn) could show network metrics graph, but it is empty for other namespace.

Expected results:
Should show network metrics graph for all namespaces

Additional info:

Comment 1 Junqi Zhao 2018-11-06 08:48:47 UTC
Created attachment 1502318 [details]
There is notnetwork metrics graph - openshift-infra

Comment 2 Junqi Zhao 2018-11-06 08:49:08 UTC
Created attachment 1502319 [details]
network metrics graph could be shown after restarting atomic-openshift-node.service - openshift-infra

Comment 3 Junqi Zhao 2018-11-06 08:53:13 UTC
Note:
CRI-O sockert is changed to /var/run/crio/crio.sock from unix:///var/run/crio/crio.sock

we can see the warning info, it is recommend us to use format "unix:///var/run/crio/crio.sock".
# master-logs etcd etcd
W0925 07:16:36.726552   19184 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock".

Comment 11 Junqi Zhao 2019-06-11 03:18:13 UTC
Created attachment 1579226 [details]
ansible logs

Comment 15 Joseph Callen 2019-06-17 19:49:29 UTC
Fix for l_kubelet_node_name issue:
PR: https://github.com/openshift/openshift-ansible/pull/11699

Comment 17 Junqi Zhao 2019-06-19 05:58:44 UTC
Created attachment 1582082 [details]
empty network diagram -- openshift-infra

Comment 28 Seth Jennings 2019-08-08 03:11:26 UTC
Upstream fix
https://github.com/google/cadvisor/pull/2284

Comment 29 Seth Jennings 2019-08-08 03:26:58 UTC
Related origin issue
https://github.com/openshift/origin/issues/23492

Comment 30 Seth Jennings 2019-08-09 18:36:16 UTC
cherry-pick to cadvisor fork
https://github.com/openshift/google-cadvisor/pull/7

Comment 32 Seth Jennings 2019-08-12 16:16:54 UTC
origin bump
https://github.com/openshift/origin/pull/23585

Comment 34 Seth Jennings 2019-08-15 19:41:07 UTC
Brian,

Fix tracking for this issue for 3.11.z via https://bugzilla.redhat.com/show_bug.cgi?id=1741679

Comment 39 errata-xmlrpc 2019-10-16 06:27:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.