Bug 972676 - [nova-network] When multi host network has created, all instances from other networks lost dhcp
[nova-network] When multi host network has created, all instances from other ...
Status: CLOSED NOTABUG
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
3.0
Unspecified Unspecified
unspecified Severity urgent
: ---
: 3.0
Assigned To: David Ripton
Ami Jeain
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-10 07:52 EDT by Rami Vaknin
Modified: 2016-04-26 11:05 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-11 11:17:23 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rami Vaknin 2013-06-10 07:52:15 EDT
Version:
Grizzly on rhel6.4 with openstack-nova-network-2013.1.1-4.el6ost.noarch (puddle 2013-06-05.2)
Two machines environment - one of them is compute node only, the second one is services + compute node.

CONFIG_NOVA_NETWORK_HOST=10.35.160.29
CONFIG_NOVA_NETWORK_PUBIF=eth3
CONFIG_NOVA_NETWORK_PRIVIF=eth3
CONFIG_NOVA_NETWORK_FIXEDRANGE=172.16.0.0/24
CONFIG_NOVA_NETWORK_FLOATRANGE=10.35.164.0/24
CONFIG_NOVA_NETWORK_DEFAULTFLOATINGPOOL=nova
CONFIG_NOVA_NETWORK_AUTOASSIGNFLOATINGIP=y

How Reproducible:
100% (2/2)

Description:
I've changed the following nova.conf configuration and restarted all running nova services:
openstack-config --set /etc/nova/nova.conf DEFAULT multi_host True  
openstack-config --set /etc/nova/nova.conf DEFAULT send_arp_for_ha True

I've created 2 instances and managed to ping to their floating ips from an outside-world machine.

I've added a multi_host network and created a new instance with nic on the multi_host network, I've noticed that all instances have not responded to ping anymore and lost their dhcp, the instance on the multi-host network also didn't have dhcp.

I've added ip and default gw to all instances manually and I've managed to ping them again from an outside-world machine.
I've tried to clear all iptables rules on both machines and request dhcp using dhclient - it didn't work, they didn't get an ip.

# nova list
+--------------------------------------+---------------------+--------+-------------------------------------+
| ID                                   | Name                | Status | Networks                            |
+--------------------------------------+---------------------+--------+-------------------------------------+
| 2676ee0a-d409-4556-8ca4-3ba36efa3c75 | instance_multi_host | ACTIVE | multi_host=172.16.1.2, 10.35.164.3  |
| 62379fb2-681a-4672-87fe-848caaf5fa7d | rhel64_1            | ACTIVE | novanetwork=172.16.0.2, 10.35.164.1 |
| 4f93082e-4f10-4bff-a729-ec0f643dd1fb | rhel64_2            | ACTIVE | novanetwork=172.16.0.3, 10.35.164.2 |
+--------------------------------------+---------------------+--------+-------------------------------------+
# nova network-list
+--------------------------------------+-------------+---------------+
| ID                                   | Label       | Cidr          |
+--------------------------------------+-------------+---------------+
| 0c27e574-d690-40a3-a132-41baaf04e13d | novanetwork | 172.16.0.0/24 |
| 490d9c8d-68e4-4d29-8a76-74e5df1d22bc | multi_host  | 172.16.1.0/24 |
+--------------------------------------+-------------+---------------+
# nova network-show 490d9c8d-68e4-4d29-8a76-74e5df1d22bc
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| bridge              | br100                                |
| vpn_public_port     | None                                 |
| dhcp_start          | 172.16.1.2                           |
| bridge_interface    | eth3                                 |
| updated_at          | None                                 |
| id                  | 490d9c8d-68e4-4d29-8a76-74e5df1d22bc |
| cidr_v6             | None                                 |
| deleted_at          | None                                 |
| gateway             | 172.16.1.1                           |
| rxtx_base           | None                                 |
| label               | multi_host                           |
| priority            | None                                 |
| project_id          | None                                 |
| vpn_private_address | None                                 |
| deleted             | 0                                    |
| vlan                | None                                 |
| broadcast           | 172.16.1.255                         |
| netmask             | 255.255.255.0                        |
| injected            | False                                |
| cidr                | 172.16.1.0/24                        |
| vpn_public_address  | None                                 |
| multi_host          | True                                 |
| dns2                | None                                 |
| created_at          | 2013-06-10T10:39:45.000000           |
| host                | None                                 |
| gateway_v6          | None                                 |
| netmask_v6          | None                                 |
| dns1                | 8.8.4.4                              |
+---------------------+--------------------------------------+
# nova network-show 0c27e574-d690-40a3-a132-41baaf04e13d
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| bridge              | br100                                |
| vpn_public_port     | None                                 |
| dhcp_start          | 172.16.0.2                           |
| bridge_interface    | eth3                                 |
| updated_at          | 2013-06-10T10:13:48.000000           |
| id                  | 0c27e574-d690-40a3-a132-41baaf04e13d |
| cidr_v6             | None                                 |
| deleted_at          | None                                 |
| gateway             | 172.16.0.1                           |
| rxtx_base           | None                                 |
| label               | novanetwork                          |
| priority            | None                                 |
| project_id          | None                                 |
| vpn_private_address | None                                 |
| deleted             | 0                                    |
| vlan                | None                                 |
| broadcast           | 172.16.0.255                         |
| netmask             | 255.255.255.0                        |
| injected            | False                                |
| cidr                | 172.16.0.0/24                        |
| vpn_public_address  | None                                 |
| multi_host          | False                                |
| dns2                | None                                 |
| created_at          | 2013-06-09T22:07:23.000000           |
| host                | puma10.scl.lab.tlv.redhat.com        |
| gateway_v6          | None                                 |
| netmask_v6          | None                                 |
| dns1                | 8.8.4.4                              |
+---------------------+--------------------------------------+
# nova boot --flavor 2 --image rhel64 --key-name oskey --nic net-id=490d9c8d-68e4-4d29-8a76-74e5df1d22bc instance_multi_host
+-------------------------------------+--------------------------------------+
| Property                            | Value                                |
+-------------------------------------+--------------------------------------+
| OS-EXT-STS:task_state               | scheduling                           |
| image                               | rhel64                               |
| OS-EXT-STS:vm_state                 | building                             |
| OS-EXT-SRV-ATTR:instance_name       | instance-00000003                    |
| flavor                              | m1.small                             |
| id                                  | 2676ee0a-d409-4556-8ca4-3ba36efa3c75 |
| security_groups                     | [{u'name': u'default'}]              |
| user_id                             | e34e389219eb4435871af5b4d8c83803     |
| OS-DCF:diskConfig                   | MANUAL                               |
| accessIPv4                          |                                      |
| accessIPv6                          |                                      |
| progress                            | 0                                    |
| OS-EXT-STS:power_state              | 0                                    |
| OS-EXT-AZ:availability_zone         | nova                                 |
| config_drive                        |                                      |
| status                              | BUILD                                |
| updated                             | 2013-06-10T10:41:18Z                 |
| hostId                              |                                      |
| OS-EXT-SRV-ATTR:host                | None                                 |
| key_name                            | oskey                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                 |
| name                                | instance_multi_host                  |
| adminPass                           | 2qSmQtTz98cz                         |
| tenant_id                           | baf098c2ce7f4d328658b9f9c0baf0a7     |
| created                             | 2013-06-10T10:41:18Z                 |
| metadata                            | {}                                   |
+-------------------------------------+--------------------------------------+
# ip a | grep -B 2 172
9: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether 00:9c:02:b0:bf:94 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.3/24 brd 172.16.1.255 scope global br100
    inet 172.16.0.1/24 brd 172.16.0.255 scope global br100


# ps -elf | grep dnsmasq
5 S nobody   14228     1  0  80   0 -  3221 poll_s 01:03 ?        00:00:01 /usr/sbin/dnsmasq --strict-order --local=// --domain-needed --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --bind-interfaces --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=253 --dhcp-no-override --dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile --addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts
5 S nobody   17199     1  0  80   0 -  3221 poll_s 13:13 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=172.16.0.1 --except-interface=lo --dhcp-range=set:novanetwork,172.16.0.2,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal
1 S root     17200 17199  0  80   0 -  3221 pipe_w 13:13 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=172.16.0.1 --except-interface=lo --dhcp-range=set:novanetwork,172.16.0.2,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal
0 S root     21782 17036  0  80   0 - 25812 pipe_w 14:38 pts/0    00:00:00 grep dnsmasq

# cat /var/lib/libvirt/dnsmasq/default.*
# cat /var/lib/nova/networks/nova-br100.conf
fa:16:3e:39:72:fd,instance-multi-host.novalocal,172.16.1.2


*** Note that the issue might be connected to another issue that I've experienced when adding a multi-host network, it's described in the following bug:
rhbz#972668 - [nova-network] multi host network has created with a default gateway different from the one appears in "nova network-show"
Comment 1 David Ripton 2013-06-11 11:17:23 EDT
Switching to multi-host on the fly is not supported.  You need to setup multi-host from the start so that all the correct settings are in the DB.

Note You need to log in before you can comment on or make changes to this bug.