openstack-packstack : The second time running packstack the network creation fails over ValueError(_('cidr already in use')). Description: ------------ The first time I ran packstack the installation failed due to : iptables Command Error 'Resource temporarily unavailable'. described in Bz #965787. Then when attempted to rerun packstack with the same answer-file (attached), it seems that the installation failed when during create the network that was already exist. Failed executing the command : '/usr/bin/nova-manage network create novanetwork 192.168.35.0/22 1 1024' . When I attempted to run the command manually - I got the following trace : -------------------------------------------------------------------------- [root@puma01 yum.repos.d]# /usr/bin/nova-manage network create novanetwork 192.168.35.0/22 1 1024 2013-06-03 11:02:12.840 4031 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net' Command failed, please check log for more info 2013-06-03 11:02:13.248 CRITICAL nova [req-580e8cb9-1ea8-4b07-ab98-aa3cd4c9fce7 None None] cidr already in use 2013-06-03 11:02:13.248 4031 TRACE nova Traceback (most recent call last): 2013-06-03 11:02:13.248 4031 TRACE nova File "/usr/bin/nova-manage", line 1263, in <module> 2013-06-03 11:02:13.248 4031 TRACE nova main() 2013-06-03 11:02:13.248 4031 TRACE nova File "/usr/bin/nova-manage", line 1255, in main 2013-06-03 11:02:13.248 4031 TRACE nova fn(*fn_args, **fn_kwargs) 2013-06-03 11:02:13.248 4031 TRACE nova File "/usr/bin/nova-manage", line 494, in create 2013-06-03 11:02:13.248 4031 TRACE nova net_manager.create_networks(context.get_admin_context(), **kwargs) 2013-06-03 11:02:13.248 4031 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1056, in create_networks 2013-06-03 11:02:13.248 4031 TRACE nova return self._do_create_networks(context, **kwargs) 2013-06-03 11:02:13.248 4031 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1116, in _do_create_networks 2013-06-03 11:02:13.248 4031 TRACE nova raise ValueError(_('cidr already in use')) 2013-06-03 11:02:13.248 4031 TRACE nova ValueError: cidr already in use 2013-06-03 11:02:13.248 4031 TRACE nova packstack-setup.log : ----------------------- 2013-06-03 10:49:49::INFO::run_setup::359::root:: Comparing pre- conditions, value: 'yes', and match: 'yes' 2013-06-03 10:49:49::INFO::run_setup::359::root:: Comparing pre- conditions, value: 'yes', and match: 'yes' 2013-06-03 10:51:00::ERROR::ospluginutils::143::root:: Error during remote puppet apply of /var/tmp/packstack/20130603-104949-TaUccG/manifests/10.35.160.11_nova.pp 2013-06-03 10:51:00::ERROR::ospluginutils::144::root:: ESC[0;36mnotice: /Service[messagebus]/ensure: ensure changed 'stopped' to 'running'ESC[0m ESC[0;36mnotice: /Stage[main]//Exec[load_kvm]/returns: executed successfullyESC[0m ESC[1;35merr: /Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure: change from absent to present failed: Execution of '/usr/bin/nova-manage network create novanetwork 192.168.35.0/22 1 1024' returned 1: 2013-06-03 10:50:52.962 3068 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net' 2013-06-03 10:50:53.369 CRITICAL nova [req-337bc22d-2472-49f2-b6be-4a4d1a8f0f8d None None] cidr already in use 2013-06-03 10:50:53.369 3068 TRACE nova Traceback (most recent call last): 2013-06-03 10:50:53.369 3068 TRACE nova File "/usr/bin/nova-manage", line 1263, in <module> 2013-06-03 10:50:53.369 3068 TRACE nova main() 2013-06-03 10:50:53.369 3068 TRACE nova File "/usr/bin/nova-manage", line 1255, in main 2013-06-03 10:50:53.369 3068 TRACE nova fn(*fn_args, **fn_kwargs) 2013-06-03 10:50:53.369 3068 TRACE nova File "/usr/bin/nova-manage", line 494, in create 2013-06-03 10:50:53.369 3068 TRACE nova net_manager.create_networks(context.get_admin_context(), **kwargs) 2013-06-03 10:50:53.369 3068 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1056, in create_networks 2013-06-03 10:50:53.369 3068 TRACE nova return self._do_create_networks(context, **kwargs) 2013-06-03 10:50:53.369 3068 TRACE nova File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1116, in _do_create_networks 2013-06-03 10:50:53.369 3068 TRACE nova raise ValueError(_('cidr already in use')) 2013-06-03 10:50:53.369 3068 TRACE nova ValueError: cidr already in use 2013-06-03 10:50:53.369 3068 TRACE nova Command failed, please check log for more info ESC[0m ESC[0;36mnotice: Finished catalog run in 5.22 secondsESC[0m 2013-06-03 10:51:00::ERROR::run_setup::889::root:: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 884, in main _main(confFile) File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 577, in _main runSequences() File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 554, in runSequences controller.runAllSequences() File "/usr/lib/python2.6/site-packages/packstack/installer/setup_controller.py", line 90, in runAllSequences sequence.run(self.CONF) File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 105, in run step.run(config=config) File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 52, in run raise SequenceError(str(ex)) SequenceError: Error during puppet run : err: /Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure: change from absent to present failed: Execution of '/usr/bin/nova-manage network create novanetwork 192.168.35.0/22 1 1024' returned 1: 2013-06-03 10:50:52.962 3068 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net' 2013-06-03 10:51:00::INFO::run_setup::598::root:: Removing /var/tmp/packstack/e415e53c62e1441cb98f3cca07fc20c4 on 10.35.160.11 (if it is a remote host) 2013-06-03 10:51:00::INFO::run_setup::598::root:: Removing /var/tmp/packstack/326eb350f71448c1bbef6564e0929c4e on 10.35.160.13 (if it is a remote host) 2013-06-03 10:51:00::INFO::run_setup::532::root:: * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. 2013-06-03 10:51:00::INFO::run_setup::532::root:: * Did not create a cinder volume group, one already existed 2013-06-03 10:51:00::INFO::run_setup::532::root:: * To use the command line tools you need to source the file /root/keystonerc_admin created on 10.35.160.11 2013-06-03 10:51:00::INFO::run_setup::532::root:: * To use the console, browse to http://10.35.160.11/dashboard
Created attachment 756186 [details] setup-logs
Created attachment 756187 [details] openstack-setup.log
first attachment is the answerfile.
As I wrote in comment #5 for some reason nova-manage network create fails when A.B.C.D/22 network is used. This wasn't happening in the past, so adding needifo for somebody from nova team. [para@virtual-rhel packstack]$ packstack --answer-file=ans.txt Welcome to Installer setup utility Installing: Clean Up... [ DONE ] Adding pre install manifest entries... [ DONE ] Setting up ssh keys...root.122.110's password: root.122.253's password: root.122.39's password: [ DONE ] Adding MySQL manifest entries... [ DONE ] Adding QPID manifest entries... [ DONE ] Adding Keystone manifest entries... [ DONE ] Adding Glance Keystone manifest entries... [ DONE ] Adding Glance manifest entries... [ DONE ] Adding Cinder Keystone manifest entries... [ DONE ] Installing dependencies for Cinder... [ DONE ] Checking if the Cinder server has a cinder-volumes vg...[ DONE ] Adding Cinder manifest entries... [ DONE ] Adding Nova API manifest entries... [ DONE ] Adding Nova Keystone manifest entries... [ DONE ] Adding Nova Cert manifest entries... [ DONE ] Adding Nova Conductor manifest entries... [ DONE ] Adding Nova Compute manifest entries... [ DONE ] Adding Nova Scheduler manifest entries... [ DONE ] Adding Nova VNC Proxy manifest entries... [ DONE ] Adding Nova Common manifest entries... [ DONE ] Adding Nova Network manifest entries... [ DONE ] Adding OpenStack Client manifest entries... [ DONE ] Adding Horizon manifest entries... [ DONE ] Preparing servers... [ DONE ] Adding post install manifest entries... [ DONE ] Installing Dependencies... [ DONE ] Copying Puppet modules and manifests... [ DONE ] Applying Puppet manifests... Applying 192.168.122.110_prescript.pp Applying 192.168.122.253_prescript.pp Applying 192.168.122.39_prescript.pp 192.168.122.253_prescript.pp : [ DONE ] 192.168.122.110_prescript.pp : [ DONE ] 192.168.122.39_prescript.pp : [ DONE ] Applying 192.168.122.110_mysql.pp Applying 192.168.122.110_qpid.pp 192.168.122.110_mysql.pp : [ DONE ] 192.168.122.110_qpid.pp : [ DONE ] Applying 192.168.122.110_keystone.pp Applying 192.168.122.110_glance.pp Applying 192.168.122.110_cinder.pp 192.168.122.110_keystone.pp : [ DONE ] 192.168.122.110_glance.pp : [ DONE ] 192.168.122.110_cinder.pp : [ DONE ] Applying 192.168.122.110_api_nova.pp 192.168.122.110_api_nova.pp : [ DONE ] Applying 192.168.122.110_nova.pp Applying 192.168.122.253_nova.pp Applying 192.168.122.39_nova.pp Applying 192.168.122.110_osclient.pp Applying 192.168.122.110_horizon.pp 192.168.122.253_nova.pp : [ DONE ] 192.168.122.39_nova.pp : [ DONE ] 192.168.122.110_nova.pp : [ DONE ] 192.168.122.110_osclient.pp : [ DONE ] 192.168.122.110_horizon.pp : [ DONE ] Applying 192.168.122.110_postscript.pp Applying 192.168.122.39_postscript.pp Applying 192.168.122.253_postscript.pp 192.168.122.39_postscript.pp : [ DONE ] 192.168.122.110_postscript.pp : [ DONE ] 192.168.122.253_postscript.pp : [ DONE ] [ DONE ] **** Installation completed successfully ****** Additional information: * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * To use the command line tools you need to source the file /root/keystonerc_admin created on 192.168.122.110 * To use the console, browse to http://192.168.122.110/dashboard * The installation log file is available at: /var/tmp/packstack/20130605-143405-ttngRq/openstack-setup.log [para@virtual-rhel packstack]$ packstack --answer-file=ans.txt Welcome to Installer setup utility Installing: Clean Up... [ DONE ] Adding pre install manifest entries... [ DONE ] Setting up ssh keys... [ DONE ] Adding MySQL manifest entries... [ DONE ] Adding QPID manifest entries... [ DONE ] Adding Keystone manifest entries... [ DONE ] Adding Glance Keystone manifest entries... [ DONE ] Adding Glance manifest entries... [ DONE ] Adding Cinder Keystone manifest entries... [ DONE ] Installing dependencies for Cinder... [ DONE ] Checking if the Cinder server has a cinder-volumes vg...[ DONE ] Adding Cinder manifest entries... [ DONE ] Adding Nova API manifest entries... [ DONE ] Adding Nova Keystone manifest entries... [ DONE ] Adding Nova Cert manifest entries... [ DONE ] Adding Nova Conductor manifest entries... [ DONE ] Adding Nova Compute manifest entries... [ DONE ] Adding Nova Scheduler manifest entries... [ DONE ] Adding Nova VNC Proxy manifest entries... [ DONE ] Adding Nova Common manifest entries... [ DONE ] Adding Nova Network manifest entries... [ DONE ] Adding OpenStack Client manifest entries... [ DONE ] Adding Horizon manifest entries... [ DONE ] Preparing servers... [ DONE ] Adding post install manifest entries... [ DONE ] Installing Dependencies... [ DONE ] Copying Puppet modules and manifests... [ DONE ] Applying Puppet manifests... Applying 192.168.122.110_prescript.pp Applying 192.168.122.253_prescript.pp Applying 192.168.122.39_prescript.pp 192.168.122.253_prescript.pp : [ DONE ] 192.168.122.110_prescript.pp : [ DONE ] 192.168.122.39_prescript.pp : [ DONE ] Applying 192.168.122.110_mysql.pp Applying 192.168.122.110_qpid.pp 192.168.122.110_mysql.pp : [ DONE ] 192.168.122.110_qpid.pp : [ DONE ] Applying 192.168.122.110_keystone.pp Applying 192.168.122.110_glance.pp Applying 192.168.122.110_cinder.pp 192.168.122.110_keystone.pp : [ DONE ] 192.168.122.110_cinder.pp : [ DONE ] 192.168.122.110_glance.pp : [ DONE ] Applying 192.168.122.110_api_nova.pp 192.168.122.110_api_nova.pp : [ DONE ] Applying 192.168.122.110_nova.pp Applying 192.168.122.253_nova.pp Applying 192.168.122.39_nova.pp Applying 192.168.122.110_osclient.pp Applying 192.168.122.110_horizon.pp 192.168.122.253_nova.pp : [ DONE ] [ ERROR ] ERROR : Error during puppet run : err: /Stage[main]/Nova::Network/Nova::Manage::Network[nova-vm-net]/Nova_network[nova-vm-net]/ensure: change from absent to present failed: Execution of '/usr/bin/nova-manage network create novanetwork 192.168.35.0/22 1 1024' returned 1: 2013-06-05 14:52:34.968 14355 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net' Please check log file /var/tmp/packstack/20130605-145135-EomONw/openstack-setup.log for more information Additional information: * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * Did not create a cinder volume group, one already existed * To use the command line tools you need to source the file /root/keystonerc_admin created on 192.168.122.110 * To use the console, browse to http://192.168.122.110/dashboard [para@virtual-rhel packstack]$ [para@virtual-rhel packstack]$ vi ans.txt >>> Changed values to: # IP Range for Flat DHCP CONFIG_NOVA_NETWORK_FIXEDRANGE=192.168.10.0/24 # IP Range for Floating IP's CONFIG_NOVA_NETWORK_FLOATRANGE=10.3.10.0/24 >>> [para@virtual-rhel packstack]$ packstack --answer-file=ans.txt Welcome to Installer setup utility Installing: Clean Up... [ DONE ] Adding pre install manifest entries... [ DONE ] Setting up ssh keys... [ DONE ] Adding MySQL manifest entries... [ DONE ] Adding QPID manifest entries... [ DONE ] Adding Keystone manifest entries... [ DONE ] Adding Glance Keystone manifest entries... [ DONE ] Adding Glance manifest entries... [ DONE ] Adding Cinder Keystone manifest entries... [ DONE ] Installing dependencies for Cinder... [ DONE ] Checking if the Cinder server has a cinder-volumes vg...[ DONE ] Adding Cinder manifest entries... [ DONE ] Adding Nova API manifest entries... [ DONE ] Adding Nova Keystone manifest entries... [ DONE ] Adding Nova Cert manifest entries... [ DONE ] Adding Nova Conductor manifest entries... [ DONE ] Adding Nova Compute manifest entries... [ DONE ] Adding Nova Scheduler manifest entries... [ DONE ] Adding Nova VNC Proxy manifest entries... [ DONE ] Adding Nova Common manifest entries... [ DONE ] Adding Nova Network manifest entries... [ DONE ] Adding OpenStack Client manifest entries... [ DONE ] Adding Horizon manifest entries... [ DONE ] Preparing servers... [ DONE ] Adding post install manifest entries... [ DONE ] Installing Dependencies... [ DONE ] Copying Puppet modules and manifests... [ DONE ] Applying Puppet manifests... Applying 192.168.122.110_prescript.pp Applying 192.168.122.253_prescript.pp Applying 192.168.122.39_prescript.pp 192.168.122.253_prescript.pp : [ DONE ] 192.168.122.110_prescript.pp : [ DONE ] 192.168.122.39_prescript.pp : [ DONE ] Applying 192.168.122.110_mysql.pp Applying 192.168.122.110_qpid.pp 192.168.122.110_mysql.pp : [ DONE ] 192.168.122.110_qpid.pp : [ DONE ] Applying 192.168.122.110_keystone.pp Applying 192.168.122.110_glance.pp Applying 192.168.122.110_cinder.pp 192.168.122.110_glance.pp : [ DONE ] 192.168.122.110_keystone.pp : [ DONE ] 192.168.122.110_cinder.pp : [ DONE ] Applying 192.168.122.110_api_nova.pp 192.168.122.110_api_nova.pp : [ DONE ] Applying 192.168.122.110_nova.pp Applying 192.168.122.253_nova.pp Applying 192.168.122.39_nova.pp Applying 192.168.122.110_osclient.pp Applying 192.168.122.110_horizon.pp 192.168.122.253_nova.pp : [ DONE ] 192.168.122.39_nova.pp : [ DONE ] 192.168.122.110_nova.pp : [ DONE ] 192.168.122.110_horizon.pp : [ DONE ] 192.168.122.110_osclient.pp : [ DONE ] Applying 192.168.122.110_postscript.pp Applying 192.168.122.39_postscript.pp Applying 192.168.122.253_postscript.pp 192.168.122.39_postscript.pp : [ DONE ] 192.168.122.110_postscript.pp : [ DONE ] 192.168.122.253_postscript.pp : [ DONE ] [ DONE ] **** Installation completed successfully ****** Additional information: * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * Did not create a cinder volume group, one already existed * To use the command line tools you need to source the file /root/keystonerc_admin created on 192.168.122.110 * To use the console, browse to http://192.168.122.110/dashboard * The installation log file is available at: /var/tmp/packstack/20130605-150601-nmMTtC/openstack-setup.log [para@virtual-rhel packstack]$ packstack --answer-file=ans.txt Welcome to Installer setup utility Installing: Clean Up... [ DONE ] Adding pre install manifest entries... [ DONE ] Setting up ssh keys... [ DONE ] Adding MySQL manifest entries... [ DONE ] Adding QPID manifest entries... [ DONE ] Adding Keystone manifest entries... [ DONE ] Adding Glance Keystone manifest entries... [ DONE ] Adding Glance manifest entries... [ DONE ] Adding Cinder Keystone manifest entries... [ DONE ] Installing dependencies for Cinder... [ DONE ] Checking if the Cinder server has a cinder-volumes vg...[ DONE ] Adding Cinder manifest entries... [ DONE ] Adding Nova API manifest entries... [ DONE ] Adding Nova Keystone manifest entries... [ DONE ] Adding Nova Cert manifest entries... [ DONE ] Adding Nova Conductor manifest entries... [ DONE ] Adding Nova Compute manifest entries... [ DONE ] Adding Nova Scheduler manifest entries... [ DONE ] Adding Nova VNC Proxy manifest entries... [ DONE ] Adding Nova Common manifest entries... [ DONE ] Adding Nova Network manifest entries... [ DONE ] Adding OpenStack Client manifest entries... [ DONE ] Adding Horizon manifest entries... [ DONE ] Preparing servers... [ DONE ] Adding post install manifest entries... [ DONE ] Installing Dependencies... [ DONE ] Copying Puppet modules and manifests... [ DONE ] Applying Puppet manifests... Applying 192.168.122.110_prescript.pp Applying 192.168.122.253_prescript.pp Applying 192.168.122.39_prescript.pp 192.168.122.253_prescript.pp : [ DONE ] 192.168.122.110_prescript.pp : [ DONE ] 192.168.122.39_prescript.pp : [ DONE ] Applying 192.168.122.110_mysql.pp Applying 192.168.122.110_qpid.pp 192.168.122.110_mysql.pp : [ DONE ] 192.168.122.110_qpid.pp : [ DONE ] Applying 192.168.122.110_keystone.pp Applying 192.168.122.110_glance.pp Applying 192.168.122.110_cinder.pp 192.168.122.110_keystone.pp : [ DONE ] 192.168.122.110_cinder.pp : [ DONE ] 192.168.122.110_glance.pp : [ DONE ] Applying 192.168.122.110_api_nova.pp 192.168.122.110_api_nova.pp : [ DONE ] Applying 192.168.122.110_nova.pp Applying 192.168.122.253_nova.pp Applying 192.168.122.39_nova.pp Applying 192.168.122.110_osclient.pp Applying 192.168.122.110_horizon.pp 192.168.122.253_nova.pp : [ DONE ] 192.168.122.110_nova.pp : [ DONE ] 192.168.122.110_osclient.pp : [ DONE ] 192.168.122.39_nova.pp : [ DONE ] 192.168.122.110_horizon.pp : [ DONE ] Applying 192.168.122.110_postscript.pp Applying 192.168.122.39_postscript.pp Applying 192.168.122.253_postscript.pp 192.168.122.39_postscript.pp : [ DONE ] 192.168.122.110_postscript.pp : [ DONE ] 192.168.122.253_postscript.pp : [ DONE ] [ DONE ] **** Installation completed successfully ****** Additional information: * Time synchronization installation was skipped. Please note that unsynchronized time on server instances might be problem for some OpenStack components. * Did not create a cinder volume group, one already existed * To use the command line tools you need to source the file /root/keystonerc_admin created on 192.168.122.110 * To use the console, browse to http://192.168.122.110/dashboard * The installation log file is available at: /var/tmp/packstack/20130605-150805-is3oQP/openstack-setup.log [para@virtual-rhel packstack]$
So ... after thinking about this issue a little bit with using parts of my brain long time not used (facts what they were teaching us on high school about subnets) I must confess that I forgot even basics of that and how lame I become :). First of all, this is configuration problem, not Packstack's or Nova's. I will show it on example with failed command from description "/usr/bin/nova-manage network create novanetwork 192.168.35.0/22 1 1024": 1. CIDR 192.168.35.0/22 is invalid for network here, because you are using 22 bits of IP address for network address and the rest 10 bits for host address. 2. The point above means that correct CIDR should be 192.168.32.0/22 (all bits of host address are set to 0). 3. nova-manage is doing such correction when creating new network first time, so that's the reason why with second installation run puppet doesn't see network "192.168.35.0/22" created ... it's because network "192.168.32.0/22" was created first time. So IMHO the first aid fix would be to fix Packstack's defaults (for RHOS-3.0) and QEs should start using correct CIDRs. For RHOS-4.0 could be implemented CIDR correction on Packstack/Puppet side also.
Looking at Packstack's source, CONFIG_NOVA_NETWORK_FIXEDRANGE defaults to "192.168.32.0/22", so nothing need's to be done for RHOS-3.0
verified with:openstack-packstack-2013.2.1-0.10.dev763.el6.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html