Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1607724

Summary:	[OCP on OSP via kuryr]Couldn't access 10250 in pod on OSP 14 with Kuryr
Product:	OpenShift Container Platform	Reporter:	Anping Li <anli>
Component:	Networking	Assignee:	Luis Tomas Bolivar <ltomasbo>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Jon Uriarte <juriarte>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.10.0	CC:	aos-bugs, bschmaus, ihrachys, juriarte, lpeer, majopela, tzumainn, wsun
Target Milestone:	---
Target Release:	3.10.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-21 06:56:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Anping Li 2018-07-24 06:23:57 UTC

Description of problem:


Version-Release number of selected component (if applicable):
openstack 14
openshift v3.10.18

How reproducible:
always

Steps to Reproduce:
1. deploy metrics on the openstack 14 with kuryr
2. checking the logs for pod  heapster

#oc log heapster-m9r5n -n openshift-infra
<----snip---->
.99.5:10250: getsockopt: connection timed out
W0724 05:59:12.295130       1 manager.go:102] Failed to get kubelet_summary:192.168.99.5:10250 response in time
E0724 05:59:12.295197       1 summary.go:97] error while getting metrics summary from Kubelet infra-node-0.openshift.example.com(192.168.99.13:10250): Get https://192.168.99.13:10250/stats/summary/: dial tcp 192.168.99.13:10250: getsockopt: connection timed out
W0724 05:59:12.295213       1 manager.go:102] Failed to get kubelet_summary:192.168.99.13:10250 response in time
E0724 05:59:12.295311       1 summary.go:97] error while getting metrics summary from Kubelet master-0.openshift.example.com(192.168.99.17:10250): Get https://192.168.99.17:10250/stats/summary/: dial tcp 192.168.99.17:10250: getsockopt: connection timed out
W0724 05:59:12.295326       1 manager.go:102] Failed to get kubelet_summary:192.168.99.17:10250 response in time


3. access 10250 in pod
oc rsh $heapster-pod
$curl --insecure -H 'Authorization: Bearer $cluster-admin-token'   -k https://192.168.99.6:10250/spec/


Actual results:
The port 10250 couldn't be access in pod.

Expected results:
The port 10250 can be access in pod. 



Additional info:

Comment 1 Jon Uriarte 2018-07-25 09:19:03 UTC

Adding more info:

The heapster pod, deployed on openshift-ansible-openshift.example.com-pod-subnet (10.11.0.0/16) with IP address 10.11.0.55,
fails connecting to all the Openshift nodes for getting the metrics.

openshift-infra   heapster-m9r5n    10.11.0.55     master-2.openshift.example.com


logs from the heapster pod
--------------------------
E0725 09:01:42.310783       1 summary.go:97] error while getting metrics summary from Kubelet infra-node-1.openshift.example.com(192.168.99.4:10250): Get https://192.168.99.4:10250/stats/summary/: dial tcp 192.168.99.4:10250: getsockopt: connection timed out
E0725 09:01:42.310868       1 summary.go:97] error while getting metrics summary from Kubelet master-0.openshift.example.com(192.168.99.17:10250): Get https://192.168.99.17:10250/stats/summary/: dial tcp 192.168.99.17:10250: getsockopt: connection timed out
E0725 09:01:42.310901       1 summary.go:97] error while getting metrics summary from Kubelet master-2.openshift.example.com(192.168.99.6:10250): Get https://192.168.99.6:10250/stats/summary/: dial tcp 192.168.99.6:10250: getsockopt: connection timed out
E0725 09:01:42.310922       1 summary.go:97] error while getting metrics summary from Kubelet app-node-0.openshift.example.com(192.168.99.15:10250): Get https://192.168.99.15:10250/stats/summary/: dial tcp 192.168.99.15:10250: getsockopt: connection timed out
E0725 09:01:42.310943       1 summary.go:97] error while getting metrics summary from Kubelet master-1.openshift.example.com(192.168.99.7:10250): Get https://192.168.99.7:10250/stats/summary/: dial tcp 192.168.99.7:10250: getsockopt: connection timed out
E0725 09:01:42.310966       1 summary.go:97] error while getting metrics summary from Kubelet infra-node-0.openshift.example.com(192.168.99.13:10250): Get https://192.168.99.13:10250/stats/summary/: dial tcp 192.168.99.13:10250: getsockopt: connection timed out
E0725 09:01:42.310919       1 summary.go:97] error while getting metrics summary from Kubelet app-node-1.openshift.example.com(192.168.99.5:10250): Get https://192.168.99.5:10250/stats/summary/: dial tcp 192.168.99.5:10250: getsockopt: connection timed out

openshift nodes
---------------
192.168.99.17   master-0.openshift.example.com
192.168.99.7    master-1.openshift.example.com
192.168.99.6    master-2.openshift.example.com
192.168.99.13   infra-node-0.openshift.example.com
192.168.99.4    infra-node-1.openshift.example.com
192.168.99.15   app-node-0.openshift.example.com
192.168.99.5    app-node-1.openshift.example.com

subnets
-------
openshift-ansible-openshift.example.com-pod-subnet 10.11.0.0/16
openshift-ansible-openshift.example.com-subnet     192.168.99.0/24

Both subnets are connected to the same router (openshift-ansible-openshift.example.com-router).

connections in openshift nodes
------------------------------
master-0.openshift.example.com >>
tcp        0      0 0.0.0.0:8444            0.0.0.0:*               LISTEN      4914/openshift      
tcp        0      0 192.168.99.17:2379      0.0.0.0:*               LISTEN      5015/etcd           
tcp        0      0 192.168.99.17:2380      0.0.0.0:*               LISTEN      5015/etcd           
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      561/rpcbind         
tcp        0      0 0.0.0.0:8082            0.0.0.0:*               LISTEN      11209/python2       
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      10985/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      10804/openshift     
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      4822/openshift      
tcp        0      0 192.168.99.17:53        0.0.0.0:*               LISTEN      18119/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      18119/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1384/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1008/master         
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      4822/openshift      
tcp        0      0 192.168.99.17:53278     192.168.99.5:10250      ESTABLISHED 4822/openshift      
tcp        0      0 192.168.99.17:51726     192.168.99.15:10250     ESTABLISHED 4822/openshift      
tcp        0      0 192.168.99.17:49930     192.168.99.15:10250     ESTABLISHED 4822/openshift      
tcp6       0      0 :::10250                :::*                    LISTEN      10268/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      22405/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      561/rpcbind         
tcp6       0      0 fe80::f816:3eff:fe62:53 :::*                    LISTEN      18119/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1384/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1008/master         
tcp6       0      0 192.168.99.17:10250     192.168.99.6:37338      ESTABLISHED 10268/hyperkube     


master-1.openshift.example.com >>
tcp        0      0 0.0.0.0:8444            0.0.0.0:*               LISTEN      4980/openshift      
tcp        0      0 192.168.99.7:2379       0.0.0.0:*               LISTEN      4874/etcd           
tcp        0      0 192.168.99.7:2380       0.0.0.0:*               LISTEN      4874/etcd           
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      567/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      10884/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      10663/openshift     
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      4882/openshift      
tcp        0      0 192.168.99.7:53         0.0.0.0:*               LISTEN      18117/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      18117/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1353/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1006/master         
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      4882/openshift      
tcp        0      0 192.168.99.7:53354      192.168.99.5:10250      ESTABLISHED 4882/openshift      
tcp6       0      0 :::10250                :::*                    LISTEN      10148/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      7800/node_exporter  
tcp6       0      0 :::111                  :::*                    LISTEN      567/rpcbind         
tcp6       0      0 fe80::f816:3eff:fe81:53 :::*                    LISTEN      18117/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1353/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1006/master         


master-2.openshift.example.com >>
tcp        0      0 0.0.0.0:8444            0.0.0.0:*               LISTEN      13719/openshift     
tcp        0      0 192.168.99.6:2379       0.0.0.0:*               LISTEN      13686/etcd          
tcp        0      0 192.168.99.6:2380       0.0.0.0:*               LISTEN      13686/etcd          
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      569/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      30040/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      29494/openshift     
tcp        0      0 0.0.0.0:8053            0.0.0.0:*               LISTEN      13696/openshift     
tcp        0      0 192.168.99.6:53         0.0.0.0:*               LISTEN      18500/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      18500/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1450/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1004/master         
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      13696/openshift     
tcp        0      0 192.168.99.6:37338      192.168.99.17:10250     ESTABLISHED 13696/openshift     
tcp        0      0 192.168.99.6:33362      192.168.99.5:10250      ESTABLISHED 13696/openshift     
tcp6       0      0 :::9090                 :::*                    LISTEN      1/systemd           
tcp6       0      0 :::10250                :::*                    LISTEN      26509/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      1555/node_exporter  
tcp6       0      0 :::111                  :::*                    LISTEN      569/rpcbind         
tcp6       0      0 fe80::f816:3eff:fe94:53 :::*                    LISTEN      18500/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1450/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      1004/master         


infra-node-0.openshift.example.com >>
tcp        0      0 127.0.0.1:10443         0.0.0.0:*               LISTEN      29593/haproxy       
tcp        0      0 127.0.0.1:10444         0.0.0.0:*               LISTEN      29593/haproxy       
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      490/rpcbind         
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      29593/haproxy       
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      27166/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27051/openshift     
tcp        0      0 192.168.99.13:53        0.0.0.0:*               LISTEN      17504/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17504/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1195/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      963/master          
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      29593/haproxy       
tcp6       0      0 :::10250                :::*                    LISTEN      26776/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      15365/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      490/rpcbind         
tcp6       0      0 :::1936                 :::*                    LISTEN      27561/openshift-rou 
tcp6       0      0 fe80::f816:3eff:fed1:53 :::*                    LISTEN      17504/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1195/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      963/master          


infra-node-1.openshift.example.com >>
tcp        0      0 127.0.0.1:10443         0.0.0.0:*               LISTEN      8554/haproxy        
tcp        0      0 127.0.0.1:10444         0.0.0.0:*               LISTEN      8554/haproxy        
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      468/rpcbind         
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      8554/haproxy        
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      27158/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27044/openshift     
tcp        0      0 192.168.99.4:53         0.0.0.0:*               LISTEN      17499/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17499/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1193/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      961/master          
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      8554/haproxy        
tcp6       0      0 :::10250                :::*                    LISTEN      26768/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      18808/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      468/rpcbind         
tcp6       0      0 :::1936                 :::*                    LISTEN      27449/openshift-rou 
tcp6       0      0 fe80::f816:3eff:fed6:53 :::*                    LISTEN      17499/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1193/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      961/master          


app-node-0.openshift.example.com >>
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      524/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      27165/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27044/openshift     
tcp        0      0 192.168.99.15:53        0.0.0.0:*               LISTEN      17502/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17502/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1200/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      963/master          
tcp6       0      0 :::10250                :::*                    LISTEN      26796/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      29707/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      524/rpcbind         
tcp6       0      0 fe80::42:d3ff:fe14:c:53 :::*                    LISTEN      17502/dnsmasq       
tcp6       0      0 fe80::f816:3eff:feae:53 :::*                    LISTEN      17502/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1200/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      963/master          
tcp6       0      0 192.168.99.15:10250     192.168.99.17:51726     ESTABLISHED 26796/hyperkube     
tcp6       0      0 192.168.99.15:10250     192.168.99.17:49930     ESTABLISHED 26796/hyperkube     


app-node-1.openshift.example.com >>
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      522/rpcbind         
tcp        0      0 127.0.0.1:50036         0.0.0.0:*               LISTEN      15659/kuryr-daemon: 
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      27310/openshift     
tcp        0      0 192.168.99.5:53         0.0.0.0:*               LISTEN      17496/dnsmasq       
tcp        0      0 172.17.0.1:53           0.0.0.0:*               LISTEN      17496/dnsmasq       
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1232/sshd           
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      958/master          
tcp6       0      0 :::10250                :::*                    LISTEN      26664/hyperkube     
tcp6       0      0 :::9100                 :::*                    LISTEN      30802/node_exporter 
tcp6       0      0 :::111                  :::*                    LISTEN      522/rpcbind         
tcp6       0      0 fe80::42:5aff:fea7:4:53 :::*                    LISTEN      17496/dnsmasq       
tcp6       0      0 fe80::f816:3eff:fe64:53 :::*                    LISTEN      17496/dnsmasq       
tcp6       0      0 :::22                   :::*                    LISTEN      1232/sshd           
tcp6       0      0 ::1:25                  :::*                    LISTEN      958/master          
tcp6       0      0 192.168.99.5:10250      192.168.99.6:33362      ESTABLISHED 26664/hyperkube     
tcp6       0      0 192.168.99.5:10250      192.168.99.17:53278     ESTABLISHED 26664/hyperkube     
tcp6       0      0 192.168.99.5:10250      192.168.99.7:53354      ESTABLISHED 26664/hyperkube

Comment 2 Luis Tomas Bolivar 2019-06-21 06:56:54 UTC

This is already working when using namespace isolation which is the default for Kuryr on OCP3.11