Bug 1603203
| Summary: | In DVR environment, connection over two virtual routers is not reachable. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | David Hill <dhill> |
| Component: | openstack-neutron | Assignee: | Slawek Kaplonski <skaplons> |
| Status: | CLOSED ERRATA | QA Contact: | Federico Ressi <fressi> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | amuller, bcafarel, bhaley, chrisw, dhill, hfukumot, majopela, mowens, nyechiel, pmannidi, ragiman, skaplons, srevivo |
| Target Milestone: | async | Keywords: | Triaged, ZStream |
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-neutron-9.4.1-32.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-01-16 17:11:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Here is a test scenario I've asked the customer to try to confirm this idea. If we create 2 other VMs (workaround here) of the opposite networks on the same computes, the network works as expected but as soon as we remove the 2 other VMs , network breaks because routers are no longer required. From what I can tell, in this broken case, the packets should be flowing via the centralized router if the local router has no interface in the subnet. But I did have a couple of questions: 1) Did this work in an earlier version of OSP 10 and was it broken with a recent update? 2) @dhill - do you still have this test environment setup? There was another bug filed recently that might be related, https://bugs.launchpad.net/neutron/+bug/1783654 - I am looking into whether that comes into play here as well, but it shouldn't since this customer is not using a shared network. Hideshi, thanks for the update, knowing this is also broken in OSP 13 means it's not a regression that was introduced recently. I am still looking to determine if this is a valid configuration with a bug in the code, or if this was ever intended to work like this. This means I'll also have to test the non-DVR case to see if it works with legacy routers. Hi Brian,
Thanks for your update.
> I am still looking to determine if this is a valid configuration
> with a bug in the code, or if this was ever intended to work like this.
I understand the current status.
But, my customer strongly need to know whether it's a software failure(bug)
or not ASAP.
Please let us know it when you finish the verification test.
Best Regards,
Hideshi
Short update. I was debugging this issue with Brian yesterday. We found potential solution of this problem. If we will modify logic of _get_dvr_router_ids_for_host() method: https://github.com/openstack/neutron/blob/master/neutron/db/l3_dvrscheduler_db.py#L299 that it will return as "dvr routers" routers which have got not only compute ports from same subnet but also routers which are connected to each other, then L3 agent on host should create 2 routers on each compute node and traffic between routers should go only via br-int. Then this connection should works fine. Today I tested it with some "hard coded" values on my dev environment and it worked fine. Now I will have to do it "properly" in the code and test. WIP patch pushed upstream: https://review.openstack.org/#/c/597567/ I want to wrote tempest scenario test for it also and then check if all will be fine. I continue work on fix for this bug. Today I pushed patch with scenario test which should cover this case and help me to show that fix (when will be ready) works properly. Tests patch is in https://review.openstack.org/#/c/598676/ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0080 |
Description of problem: In DVR environment, connection over two virtual routers is not reachable if VMs are on different computes but works if they are on the same compute. Here is a reproducer: $ openstack network create AP1Site $ openstack network create AP2Site $ openstack network create Wan $ openstack subnet create --gateway=10.10.210.254 --network AP1Site --subnet-range 10.10.210.0/24 AP1_sub $ openstack subnet create --gateway=10.10.220.254 --network AP2Site --subnet-range 10.10.220.0/24 AP2_sub $ openstack subnet create --gateway=10.10.200.254 --network Wan --subnet-range 10.10.200.0/24 Wan_sub $ openstack router create AP2-RT $ openstack router create AP1-RT $ openstack port create --network AP1Site --fixed-ip subnet=AP1_sub,ip-address=10.10.210.10 AP1Port-AP1Internal $ openstack port create --network AP2Site --fixed-ip subnet=AP2_sub,ip-address=10.10.220.10 AP2Port-AP2Internal $ openstack port create --network Wan --fixed-ip subnet=Wan_sub,ip-address=10.10.200.10 AP1-WANPort $ openstack port create --network Wan --fixed-ip subnet=Wan_sub,ip-address=10.10.200.20 AP2-WANPort $ nova boot --flavor <flavor> --image <image name> --nic port-id=<AP1Port-AP1Internal port id> AP1 $ nova boot --flavor <flavor> --image <image name> --nic port-id=<AP2Port-AP2Internal port id> AP2 $ openstack router add port AP2-RT AP2-WANPort $ openstack router add port AP1-RT AP1-WANPort $ openstack router add subnet AP2-RT AP2_sub $ openstack router add subnet AP1-RT AP1_sub $ openstack router set --route destination=10.10.220.0/24,gateway=10.10.200.20 AP1-RT $ openstack router set --route destination=10.10.210.0/24,gateway=10.10.200.10 AP2-RT Set default routing on AP1. $ ip route add default via 10.10.210.254 dev eth0 VMs can't ping each others and can't ping the gateways either. I thought the following BZ [1] would solve it but the customer insists on not using a shared network. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1600180 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: