Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 920282

Summary: Nova: Changes made to nova-network ip range won't take effect without manually kill the running dnsmasq process.
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: openstack-novaAssignee: Nikola Dipanov <ndipanov>
Status: CLOSED NOTABUG QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact:
Priority: medium    
Version: 2.0 (Folsom)CC: beagles, breeler, eglynn, mlopes, ndipanov, oblaut, ohochman, sgordon
Target Milestone: asyncKeywords: Triaged
Target Release: 2.1   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Release Note
Doc Text:
Changing the network address/range for a static network using FlatDHCP networking is an operation that requires Virtual Machine networking downtime. The dnsmasq process doing the DHCP is controlled by nova-networking, and it is done in such a way that it is related to the presence of a configured bridge that has been used at least once. The bridge is updated or launched when nova-network is started, or when a virtual machine instance is started. This means that Compute will not attempt to reconfigure the bridge if there is one, so changing the range would require manually killing the running dnsmasq process. Then restarting nova networking and all virtual machine instances that were previously running. Further, FlatDHCP networking will not work with multiple private networks if you are attempting to use a single bridge interface (e.g. br100). While a dnsmasq instance can support multiple simultaneous DHCP ranges, this scenario is not supported by OpenStack at this time. In order to get this to work, each distinct private network needs to have its own bridge interface. They can all connect through the same physical NIC, but they must be distinct bridge instances.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-05 17:49:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 923918    
Bug Blocks:    
Attachments:
Description Flags
nova.logs none

Description Omri Hochman 2013-03-11 17:11:23 UTC 
Nova: Changes made to nova-network ip range won't take effect without manually kill the running dnsmasq process.  


Description: 
-------------
When attempting to change network values by :
1) nova-manage network delete 192.168.32.0/24 
2) nova-manage network create --fixed_range_v4=192.168.37.0/24 --label=label
3) restart openstack-nova-network ( Or even a complete machine reboot ) 

Results: 
--------
After 'service openstack-nova-network restart' - the dnsmasq process remains running with the old "novanetwork 192.168.32.2 value". the only way to make the network changes to apply without machine reboot is to manually kill the running dnsmasq process and then restart the nova-network.  


-----------------------------------------------------------------------------

[root@puma03 ~]# ps -ef | grep dnsmasq
nobody    6889     1  0 18:13 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.32.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.32.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
root      6890  6889  0 18:13 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.32.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.32.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
root     17428 12150  0 18:47 pts/21   00:00:00 grep dnsmasq

---------------------------------------------------------------------------

root@puma03 ~]# nova-manage network list
id      IPv4                    IPv6            start address   DNS1            DNS2            VlanID          project         uuid           
1       192.168.32.0/24         None            192.168.32.2    8.8.4.4         None            None            None            e22c6739-7af3-4d97-92d8-4113d0f4f099

-----------------------------------------------------------------------------
root@puma03 ~(keystone_admin)]$ nova-manage network delete 192.168.32.0/24 e22c6739-7af3-4d97-92d8-4113d0f4f099

---------------------------------------------------------------------------[root@puma03 ~(keystone_admin)]$ nova-manage network create --fixed_range_v4=192.168.38.0/24 --label=label1

----------------------------------------------------------------------------
[root@puma03 ~(keystone_admin)]$ service openstack-nova-network restart
Stopping openstack-nova-network:                           [  OK  ]
Starting openstack-nova-network:                           [  OK  ]

---------------------------------------------------------------------------
[root@puma03 ~(keystone_admin)]$ ps -ef | grep dnsmasq
nobody    6889     1  0 18:13 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.32.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.32.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
root      6890  6889  0 18:13 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --bind-interfaces --conf-file= --domain=novalocal --pid-file=/var/lib/nova/networks/nova-br100.pid --listen-address=192.168.32.1 --except-interface=lo --dhcp-range=set:'novanetwork',192.168.32.2,static,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br100.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro
root     22098 12150  0 18:53 pts/21   00:00:00 grep dnsmasq
Comment 2 Omri Hochman 2013-03-12 08:23:50 UTC 
Created attachment 708804 [details]
nova.logs
Comment 11 Nikola Dipanov 2013-04-05 17:49:17 UTC 
I've done some investigation on this and it seems not restarting dnsmasq is by design. It will get started/restarted only once an IP adress from the network is assigned to an instance, or network service is restarted.

Please note that changing networks in the running setup *will* cause networking for running instances to fail and is something that is not really supported using FlatDHCP networking. Also multiple private networks will not work properly with FlatDHCP. At the end of the day - this is not something such a networking setup was intended to do.

I will close this as not-a-bug.

Having said that - nova should be more informative and maybe less permissive with what it allows when running FlatDHCP, so if you want to raise a bug to track these efforts - it would be a welcome thing, especially with a number of examples.
Comment 12 Brent Eagles 2013-04-05 21:36:29 UTC 
In comment #11, Nikola is correct. nova-networking controls the dnsmasq process lifetime, which can be roughly thought of as being related to the presence of a configured bridge that has been in use at least once and is updated (or launched if required) when nova-network is started or when a server (VM) instance is started. 

Also, FlatDHCP networking can work with multiple private networks, but it may not be obvious. It will not work if you are attempting to use a single bridge interface (e.g. br100). While a dnsmasq instance can support multiple simultaneous DHCP ranges, this scenario is not supported by Openstack at this time. In order to get this to work, each distinct private network needs to have its own bridge interface. They can all connect through the same physical NIC, but they do need to be distinct bridge instances. For example:

nova-manage network create --fixed_range_v4 192.168.5.0/24 --num_networks 1 --network_size 200 --bridge br101 --bridge_interface eth0 --label second

Instances that are created afterwards will be created with an additional interface and IP. Instances that have already been created will not have access to the newly configured network.

Some caveats: if you *do* create a new private network that uses the same bridge interface, it has been my experience so far that this breaks the other network that uses that bridge. Also while this seems to function "okay", there were some indications in my test environment with one NIC that this might be sensitive to race conditions, particularly with respect to DHCP. It may behave better in a multinic environment where the DHCP requests might be isolated to separate physical networks.