1607724 – [OCP on OSP via kuryr]Couldn't access 10250 in pod on OSP 14 with Kuryr

Bug 1607724 - [OCP on OSP via kuryr]Couldn't access 10250 in pod on OSP 14 with Kuryr

Summary: [OCP on OSP via kuryr]Couldn't access 10250 in pod on OSP 14 with Kuryr

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Networking
Sub Component:
Version:	3.10.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	3.10.z
Assignee:	Luis Tomas Bolivar
QA Contact:	Jon Uriarte
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-07-24 06:23 UTC by Anping Li
Modified:	2019-09-10 14:12 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-06-21 06:56:54 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Anping Li 2018-07-24 06:23:57 UTC

Description of problem:


Version-Release number of selected component (if applicable):
openstack 14
openshift v3.10.18

How reproducible:
always

Steps to Reproduce:
1. deploy metrics on the openstack 14 with kuryr
2. checking the logs for pod  heapster

#oc log heapster-m9r5n -n openshift-infra
<----snip---->
.99.5:10250: getsockopt: connection timed out
W0724 05:59:12.295130       1 manager.go:102] Failed to get kubelet_summary:192.168.99.5:10250 response in time
E0724 05:59:12.295197       1 summary.go:97] error while getting metrics summary from Kubelet infra-node-0.openshift.example.com(192.168.99.13:10250): Get https://192.168.99.13:10250/stats/summary/: dial tcp 192.168.99.13:10250: getsockopt: connection timed out
W0724 05:59:12.295213       1 manager.go:102] Failed to get kubelet_summary:192.168.99.13:10250 response in time
E0724 05:59:12.295311       1 summary.go:97] error while getting metrics summary from Kubelet master-0.openshift.example.com(192.168.99.17:10250): Get https://192.168.99.17:10250/stats/summary/: dial tcp 192.168.99.17:10250: getsockopt: connection timed out
W0724 05:59:12.295326       1 manager.go:102] Failed to get kubelet_summary:192.168.99.17:10250 response in time


3. access 10250 in pod
oc rsh $heapster-pod
$curl --insecure -H 'Authorization: Bearer $cluster-admin-token'   -k https://192.168.99.6:10250/spec/


Actual results:
The port 10250 couldn't be access in pod.

Expected results:
The port 10250 can be access in pod. 



Additional info:

Comment 1 Jon Uriarte 2018-07-25 09:19:03 UTC

Adding more info:

The heapster pod, deployed on openshift-ansible-openshift.example.com-pod-subnet (10.11.0.0/16) with IP address 10.11.0.55,
fails connecting to all the Openshift nodes for getting the metrics.

openshift-infra   heapster-m9r5n    10.11.0.55     master-2.openshift.example.com


logs from the heapster pod
--------------------------
E0725 09:01:42.310783       1 summary.go:97] error while getting metrics summary from Kubelet infra-node-1.openshift.example.com(192.168.99.4:10250): Get https://192.168.99.4:10250/stats/summary/: dial tcp 192.168.99.4:10250: getsockopt: connection timed out
E0725 09:01:42.310868       1 summary.go:97] error while getting metrics summary from Kubelet master-0.openshift.example.com(192.168.99.17:10250): Get https://192.168.99.17:10250/stats/summary/: dial tcp 192.168.99.17:10250: getsockopt: connection timed out
E0725 09:01:42.310901       1 summary.go:97] error while getting metrics summary from Kubelet master-2.openshift.example.com(192.168.99.6:10250): Get https://192.168.99.6:10250/stats/summary/: dial tcp 192.168.99.6:10250: getsockopt: connection timed out
E0725 09:01:42.310922       1 summary.go:97] error while getting metrics summary from Kubelet app-node-0.openshift.example.com(192.168.99.15:10250): Get https://192.168.99.15:10250/stats/summary/: dial tcp 192.168.99.15:10250: getsockopt: connection timed out
E0725 09:01:42.310943       1 summary.go:97] error while getting metrics summary from Kubelet master-1.openshift.example.com(192.168.99.7:10250): Get https://192.168.99.7:10250/stats/summary/: dial tcp 192.168.99.7:10250: getsockopt: connection timed out
E0725 09:01:42.310966       1 summary.go:97] error while getting metrics summary from Kubelet infra-node-0.openshift.example.com(192.168.99.13:10250): Get https://192.168.99.13:10250/stats/summary/: dial tcp 192.168.99.13:10250: getsockopt: connection timed out
E0725 09:01:42.310919       1 summary.go:97] error while getting metrics summary from Kubelet app-node-1.openshift.example.com(192.168.99.5:10250): Get https://192.168.99.5:10250/stats/summary/: dial tcp 192.168.99.5:10250: getsockopt: connection timed out

openshift nodes
---------------
192.168.99.17   master-0.openshift.example.com
192.168.99.7    master-1.openshift.example.com
192.168.99.6    master-2.openshift.example.com
192.168.99.13   infra-node-0.openshift.example.com
192.168.99.4    infra-node-1.openshift.example.com
192.168.99.15   app-node-0.openshift.example.com
192.168.99.5    app-node-1.openshift.example.com

subnets
-------
openshift-ansible-openshift.example.com-pod-subnet 10.11.0.0/16
openshift-ansible-openshift.example.com-subnet     192.168.99.0/24

Both subnets are connected to the same router (openshift-ansible-openshift.example.com-router).

connections in openshift nodes
------------------------------
master-0.openshift.example.com >>
tcp        0      0 0.0.0.0:8444            0.0.0.0:*               LISTEN      4914/openshift      
tcp        0      0 192.168.99.17:2379      0.0.0.0:*               LISTEN      5015/etcd           
tcp        0      0 192.168.99.17:2380      0.0.0.0:*               LISTEN      5015/etcd           
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      561/rpcbind         
tcp        0      0 0.0.0.0:8082            0.0.0.0:*               LISTEN      11209/python2       
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      10985/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      10804/openshift     
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      4822/openshift      
tcp        0      0 192.168.99.17:53        0.0.0.0:*               LISTEN      18119/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      18119/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1384/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1008/master         
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      4822/openshift      
tcp        0      0 192.168.99.17:53278     192.168.99.5:10250      ESTABLISHED 4822/openshift      
tcp        0      0 192.168.99.17:51726     192.168.99.15:10250     ESTABLISHED 4822/openshift      
tcp        0      0 192.168.99.17:49930     192.168.99.15:10250     ESTABLISHED 4822/openshift      
tcp6       0      0 :::10250                :::*                    LISTEN      10268/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      22405/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      561/rpcbind         
tcp6       0      0 fe80::f816:3eff:fe62:53 :::*                    LISTEN      18119/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1384/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1008/master         
tcp6       0      0 192.168.99.17:10250     192.168.99.6:37338      ESTABLISHED 10268/hyperkube     


master-1.openshift.example.com >>
tcp        0      0 0.0.0.0:8444            0.0.0.0:*               LISTEN      4980/openshift      
tcp        0      0 192.168.99.7:2379       0.0.0.0:*               LISTEN      4874/etcd           
tcp        0      0 192.168.99.7:2380       0.0.0.0:*               LISTEN      4874/etcd           
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      567/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      10884/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      10663/openshift     
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      4882/openshift      
tcp        0      0 192.168.99.7:53         0.0.0.0:*               LISTEN      18117/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      18117/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1353/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1006/master         
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      4882/openshift      
tcp        0      0 192.168.99.7:53354      192.168.99.5:10250      ESTABLISHED 4882/openshift      
tcp6       0      0 :::10250                :::*                    LISTEN      10148/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      7800/node_exporter  
tcp6       0      0 :::111                  :::*                    LISTEN      567/rpcbind         
tcp6       0      0 fe80::f816:3eff:fe81:53 :::*                    LISTEN      18117/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1353/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1006/master         


master-2.openshift.example.com >>
tcp        0      0 0.0.0.0:8444            0.0.0.0:*               LISTEN      13719/openshift     
tcp        0      0 192.168.99.6:2379       0.0.0.0:*               LISTEN      13686/etcd          
tcp        0      0 192.168.99.6:2380       0.0.0.0:*               LISTEN      13686/etcd          
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      569/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      30040/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      29494/openshift     
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      13696/openshift     
tcp        0      0 192.168.99.6:53         0.0.0.0:*               LISTEN      18500/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      18500/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1450/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1004/master         
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      13696/openshift     
tcp        0      0 192.168.99.6:37338      192.168.99.17:10250     ESTABLISHED 13696/openshift     
tcp        0      0 192.168.99.6:33362      192.168.99.5:10250      ESTABLISHED 13696/openshift     
tcp6       0      0 :::9090                 :::*                    LISTEN      1/systemd           
tcp6       0      0 :::10250                :::*                    LISTEN      26509/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      1555/node_exporter  
tcp6       0      0 :::111                  :::*                    LISTEN      569/rpcbind         
tcp6       0      0 fe80::f816:3eff:fe94:53 :::*                    LISTEN      18500/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1450/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1004/master         


infra-node-0.openshift.example.com >>
tcp        0      0 127.0.0.1:10443         0.0.0.0:*               LISTEN      29593/haproxy       
tcp        0      0 127.0.0.1:10444         0.0.0.0:*               LISTEN      29593/haproxy       
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      490/rpcbind         
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      29593/haproxy       
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      27166/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27051/openshift     
tcp        0      0 192.168.99.13:53        0.0.0.0:*               LISTEN      17504/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17504/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1195/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      963/master          
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      29593/haproxy       
tcp6       0      0 :::10250                :::*                    LISTEN      26776/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      15365/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      490/rpcbind         
tcp6       0      0 :::1936                 :::*                    LISTEN      27561/openshift-rou 
tcp6       0      0 fe80::f816:3eff:fed1:53 :::*                    LISTEN      17504/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1195/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      963/master          


infra-node-1.openshift.example.com >>
tcp        0      0 127.0.0.1:10443         0.0.0.0:*               LISTEN      8554/haproxy        
tcp        0      0 127.0.0.1:10444         0.0.0.0:*               LISTEN      8554/haproxy        
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      468/rpcbind         
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      8554/haproxy        
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      27158/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27044/openshift     
tcp        0      0 192.168.99.4:53         0.0.0.0:*               LISTEN      17499/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17499/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1193/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      961/master          
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      8554/haproxy        
tcp6       0      0 :::10250                :::*                    LISTEN      26768/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      18808/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      468/rpcbind         
tcp6       0      0 :::1936                 :::*                    LISTEN      27449/openshift-rou 
tcp6       0      0 fe80::f816:3eff:fed6:53 :::*                    LISTEN      17499/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1193/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      961/master          


app-node-0.openshift.example.com >>
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      524/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      27165/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27044/openshift     
tcp        0      0 192.168.99.15:53        0.0.0.0:*               LISTEN      17502/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17502/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1200/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      963/master          
tcp6       0      0 :::10250                :::*                    LISTEN      26796/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      29707/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      524/rpcbind         
tcp6       0      0 fe80::42:d3ff:fe14:c:53 :::*                    LISTEN      17502/dnsmasq       
tcp6       0      0 fe80::f816:3eff:feae:53 :::*                    LISTEN      17502/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1200/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      963/master          
tcp6       0      0 192.168.99.15:10250     192.168.99.17:51726     ESTABLISHED 26796/hyperkube     
tcp6       0      0 192.168.99.15:10250     192.168.99.17:49930     ESTABLISHED 26796/hyperkube     


app-node-1.openshift.example.com >>
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      522/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      15659/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27310/openshift     
tcp        0      0 192.168.99.5:53         0.0.0.0:*               LISTEN      17496/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17496/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1232/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      958/master          
tcp6       0      0 :::10250                :::*                    LISTEN      26664/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      30802/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      522/rpcbind         
tcp6       0      0 fe80::42:5aff:fea7:4:53 :::*                    LISTEN      17496/dnsmasq       
tcp6       0      0 fe80::f816:3eff:fe64:53 :::*                    LISTEN      17496/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1232/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      958/master          
tcp6       0      0 192.168.99.5:10250      192.168.99.6:33362      ESTABLISHED 26664/hyperkube     
tcp6       0      0 192.168.99.5:10250      192.168.99.17:53278     ESTABLISHED 26664/hyperkube     
tcp6       0      0 192.168.99.5:10250      192.168.99.7:53354      ESTABLISHED 26664/hyperkube

Comment 2 Luis Tomas Bolivar 2019-06-21 06:56:54 UTC

This is already working when using namespace isolation which is the default for Kuryr on OCP3.11

Note You need to log in before you can comment on or make changes to this bug.