Red Hat Bugzilla – Bug 1252386
[isolation] Cannot reach the kubernetes network from a container when using multi-tenant configuration
Last modified: 2015-09-08 16:14:02 EDT
Description of problem:
Setup origin with redhat/openshift-ovs-multitenant network plugin, create pod and try to get the kubernetes network from inside the container. Eg, master IP.
There is no route to the master/node IPs from the container.
This will cause the creation of router/registry get failed due to the deployer pod cannot talk to the master via REST API.
[root@master ~]# oc logs router-1-deploy
E0811 09:33:55.395475 1 clientcmd.go:128] Error reading BEARER_TOKEN_FILE "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
E0811 09:33:55.447713 1 clientcmd.go:146] Error reading BEARER_TOKEN_FILE "/var/run/secrets/kubernetes.io/serviceaccount/token": open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
F0811 09:33:58.455219 1 deployer.go:64] couldn't get deployment default/router-1: Get https://10.66.128.57:8443/api/v1/namespaces/default/replicationcontrollers/router-1: dial tcp 10.66.128.57:8443: no route to host
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Build the openshift-sdn with multi-tenant branch
$ git clone https://github.com/openshift/openshift-sdn -b multitenant
$ cd openshift-sdn
$ make clean build && make install
2. Build openshift binary with latest code for both master and node
3. Config the master and node to use redhat/openshift-ovs-multitenant plugin in master-config.conf and node-config.conf
4. Restart docker/openshift services
5. Create pod in the env
6. Get into the container and try to ping the master IP.
[root@rc-test-9g0r2 /]# ping 10.66.128.57
PING 10.66.128.57 (10.66.128.57) 56(84) bytes of data.
From 10.1.2.2 icmp_seq=1 Destination Host Unreachable
From 10.1.2.2 icmp_seq=2 Destination Host Unreachable
Should be able to reach the master from the container.
Instead of openshift-sdn multitenant branch, I have tried it on latest openshift/origin repo. openshift-sdn multitenant branch is obsolete now, all the changes related to multitenant are merged into origin repo.
I did *not* see any issues in my testing:
- Pods/containers that are not part of default namespace:
* Unable to talk to pods of non-default namespaces as expected.
* Able to talk to pods of default namespaces
* Able to reach master
- Pods that are part of default namespace are able to reach any pods in the cluster.
Did you switch from openshift-ovs-subnet to openshift-ovs-multitenant network plugin?
If yes, then you have to delete the lbr0 bridge otherwise sdn setup won't be performed on the node.
$ sudo systemctl stop openshift-node
$ sudo ip link set lbr0 down
$ sudo brctl delbr lbr0
Let me know, if you are still noticing this issue.
I should have missed the re-create lbr0 step in my previous try.
After I deleted the existing lbr0 and let openshift-sdn to recreate it, the issue cannot be reproduced.
Will update our test scenarios for this.