Bug 1066519 - Packstack failed to configure ML2 to work with OVS and VXLAN
Summary: Packstack failed to configure ML2 to work with OVS and VXLAN
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-packstack
Version: 4.0
Hardware: Unspecified
OS: Linux
unspecified
urgent
Target Milestone: z3
: 4.0
Assignee: Martin Magr
QA Contact: Nir Magnezi
URL:
Whiteboard:
Depends On:
Blocks: 1017144 1056612
TreeView+ depends on / blocked
 
Reported: 2014-02-18 14:56 UTC by Nir Magnezi
Modified: 2014-03-25 19:24 UTC (History)
8 users (show)

Fixed In Version: openstack-packstack-2013.2.1-0.27.dev989.el6ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-03-25 19:24:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 74753 0 None MERGED Ask for CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT with ML2 2021-02-05 09:28:05 UTC
OpenStack gerrit 76168 0 None MERGED Ask for CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT with ML2 2021-02-05 09:28:05 UTC
OpenStack gerrit 77634 0 None MERGED Fixed Neutron ML2 issues 2021-02-05 09:28:04 UTC
Red Hat Product Errata RHBA-2014:0334 0 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform 4 Bug Fix and Enhancement Advisory 2014-03-25 23:22:45 UTC

Description Nir Magnezi 2014-02-18 14:56:13 UTC
Description of problem:
=======================
Packstack failed to configure ML2 to work with OVS and VXLAN.
I did not change the default value of CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT, yet I get:

Adding Neutron L2 Agent manifest entries...         [ ERROR ]

ERROR : 'CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT'

The answer file neutron related configuration I used:

CONFIG_NEUTRON_INSTALL=y
CONFIG_NEUTRON_SERVER_HOST=192.168.160.21
CONFIG_NEUTRON_KS_PW=123456
CONFIG_NEUTRON_DB_PW=123456
CONFIG_NEUTRON_L3_HOSTS=192.168.160.29
CONFIG_NEUTRON_L3_EXT_BRIDGE=br-ex
CONFIG_NEUTRON_DHCP_HOSTS=192.168.160.29
CONFIG_NEUTRON_LBAAS_HOSTS=192.168.160.21
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_METADATA_HOSTS=192.168.160.29
CONFIG_NEUTRON_METADATA_PW=123456
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vxlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_FLAT_NETWORKS=*
CONFIG_NEUTRON_ML2_VLAN_RANGES=
CONFIG_NEUTRON_ML2_TUNNEL_ID_RANGES=208:210
CONFIG_NEUTRON_ML2_VXLAN_GROUP=185,208,209,210
CONFIG_NEUTRON_ML2_VNI_RANGES=185:185,208:210
CONFIG_NEUTRON_L2_AGENT=openvswitch
CONFIG_NEUTRON_LB_TENANT_NETWORK_TYPE=local
CONFIG_NEUTRON_LB_VLAN_RANGES=
CONFIG_NEUTRON_LB_INTERFACE_MAPPINGS=
CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vxlan
CONFIG_NEUTRON_OVS_VLAN_RANGES=
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=
CONFIG_NEUTRON_OVS_BRIDGE_IFACES=
CONFIG_NEUTRON_OVS_TUNNEL_RANGES=208:210
CONFIG_NEUTRON_OVS_TUNNEL_IF=eth3
CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT=4789

Version-Release number of selected component (if applicable):
=============================================================
openstack-packstack-2013.2.1-0.24.dev982.el6ost.noarch

How reproducible:
=================
Always

Steps to Reproduce:
===================
Use packstack with the Neutron entries I mentioned above.

Actual results:
===============
Snipped from the log:
2014-02-18 16:40:42::DEBUG::sequences::48::root:: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 46, in run
    self.function(config)
  File "/usr/lib/python2.6/site-packages/packstack/plugins/neutron_350.py", line 769, in create_l2_agent_manifests
    manifestdata = getManifestTemplate(template_name)
  File "/usr/lib/python2.6/site-packages/packstack/modules/ospluginutils.py", line 76, in getManifestTemplate
    return fp.read() % controller.CONF
KeyError: 'CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT'

2014-02-18 16:40:42::ERROR::run_setup::912::root:: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 907, in main
    _main(confFile)
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 573, in _main
    runSequences()
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 552, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.6/site-packages/packstack/installer/setup_controller.py", line 84, in runAllSequences
    sequence.run(self.CONF)
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 105, in run
    step.run(config=config)
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 52, in run
    raise SequenceError(str(ex))
SequenceError: 'CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT'

Expected results:
=================
Packstack should install OpenStack with Neutron ML2 plugin with OVS as L2 agent and VXLAN as type driver

Additional Info:
================
https://github.com/stackforge/packstack/blob/master/packstack/plugins/neutron_350.py#L274-L277

Comment 5 Nir Magnezi 2014-03-02 10:56:26 UTC
Reopening.
Tested NVR: openstack-packstack-2013.2.1-0.25.dev987.el6ost.noarch

Tested packstack with ML2 VXLAN with the following neutron configs:

CONFIG_NEUTRON_INSTALL=y
CONFIG_NEUTRON_SERVER_HOST=192.168.160.29
CONFIG_NEUTRON_KS_PW=123456
CONFIG_NEUTRON_DB_PW=123456
CONFIG_NEUTRON_L3_HOSTS=192.168.160.27
CONFIG_NEUTRON_L3_EXT_BRIDGE=br-ex
CONFIG_NEUTRON_DHCP_HOSTS=192.168.160.27
CONFIG_NEUTRON_LBAAS_HOSTS=192.168.160.29
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_METADATA_HOSTS=192.168.160.27
CONFIG_NEUTRON_METADATA_PW=123456
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vxlan,vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_FLAT_NETWORKS=*
CONFIG_NEUTRON_ML2_VLAN_RANGES=ext_net:185:185
CONFIG_NEUTRON_ML2_TUNNEL_ID_RANGES=1:1000
CONFIG_NEUTRON_ML2_VXLAN_GROUP=208,209,210
CONFIG_NEUTRON_ML2_VNI_RANGES=3000:3100,2000:2100
CONFIG_NEUTRON_L2_AGENT=openvswitch
CONFIG_NEUTRON_LB_TENANT_NETWORK_TYPE=local
CONFIG_NEUTRON_LB_VLAN_RANGES=
CONFIG_NEUTRON_LB_INTERFACE_MAPPINGS=
CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vxlan
CONFIG_NEUTRON_OVS_VLAN_RANGES=ext_net:185:185
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=ext_net:br-ex
CONFIG_NEUTRON_OVS_BRIDGE_IFACES=br-ex:eth3.185
CONFIG_NEUTRON_OVS_TUNNEL_RANGES=1:1000
CONFIG_NEUTRON_OVS_TUNNEL_IF=eth3
CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT=4789


This resulted the following error (snipped from packstack log):

2014-03-02 12:10:18::DEBUG::sequences::48::root:: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 46, in run
    self.function(config)
  File "/usr/lib/python2.6/site-packages/packstack/plugins/puppet_950.py", line 187, in applyPuppetManifest
    waitforpuppet(currently_running)
  File "/usr/lib/python2.6/site-packages/packstack/plugins/puppet_950.py", line 167, in waitforpuppet
    validate_logfile(log)
  File "/usr/lib/python2.6/site-packages/packstack/modules/puppet.py", line 90, in validate_logfile
    raise PuppetError(message)
PuppetError: Error appeared during Puppet run: 192.168.160.21_neutron.pp
Error: Could not parse for environment production: Syntax error at ','; expected '}' at /var/tmp/packstack/06111c690eb84f16aa1dc086cbca6555/manifests/192.168.160.21_neutron.pp:50 on node puma06.scl.lab.tlv.redhat.com^[[0m
You will find full trace in log /var/tmp/packstack/20140302-115102-8QDjBq/manifests/192.168.160.21_neutron.pp.log

2014-03-02 12:10:18::ERROR::run_setup::912::root:: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 907, in main
    _main(confFile)
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 573, in _main
    runSequences()
  File "/usr/lib/python2.6/site-packages/packstack/installer/run_setup.py", line 552, in runSequences
    controller.runAllSequences()
  File "/usr/lib/python2.6/site-packages/packstack/installer/setup_controller.py", line 84, in runAllSequences
    sequence.run(self.CONF)
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 105, in run
    step.run(config=config)
  File "/usr/lib/python2.6/site-packages/packstack/installer/core/sequences.py", line 52, in run
    raise SequenceError(str(ex))
SequenceError: Error appeared during Puppet run: 192.168.160.21_neutron.pp
Error: Could not parse for environment production: Syntax error at ','; expected '}' at /var/tmp/packstack/06111c690eb84f16aa1dc086cbca6555/manifests/192.168.160.21_neutron.pp:50 on node puma06.scl.lab.tlv.redhat.com^[[0m
You will find full trace in log /var/tmp/packstack/20140302-115102-8QDjBq/manifests/192.168.160.21_neutron.pp.log

2014-03-02 12:10:18::DEBUG::run_setup::594::root:: Removing /var/tmp/packstack/e984500258e84f2685e04b980e78a022 on 192.168.160.29 (if it is a remote host)
2014-03-02 12:10:18::INFO::shell::78::root:: [192.168.160.29] Executing script:
rm -rf /var/tmp/packstack/e984500258e84f2685e04b980e78a022
2014-03-02 12:10:20::DEBUG::shell::96::root::

Comment 7 Martin Magr 2014-03-03 13:08:16 UTC
Nir this is also configuration problem. I admit that we should have implemented validation on that parameter, but you should try correct values. Following is what answer file says:

# Multicast group for VXLAN. If unset, disables VXLAN enable sending
# allocate broadcast traffic to this multicast group. When left
# unconfigured, will disable multicast VXLAN mode. Should be an
# Multicast IP (v4 or v6) address.
CONFIG_NEUTRON_ML2_VXLAN_GROUP=208,209,210


Nevertheless there are missing quotes in resulting neutron.pp:50 manifest:

43: class { 'neutron::plugins::ml2':
44:    type_drivers         => ['vxlan'],
45:    tenant_network_types => ['vxlan', 'vlan'],
46:    mechanism_drivers    => ['openvswitch'],
47:    flat_networks        => ['*'],
48:    network_vlan_ranges  => ['ext_net:100:200'],
49:    tunnel_id_ranges     => ['1:1000'],
50:    vxlan_group          => 208,209,210,
51:    vni_ranges           => ['3000:3100', '2000:2100'],
52: }

So we will have to fix that. Thanks for discovering this bug.

Comment 11 Nir Magnezi 2014-03-12 20:46:14 UTC
Verified NVR: openstack-packstack-2013.2.1-0.28.dev989.el6ost

Verification Steps:
===================
1. enabled tunnel interface on at least 2 compute nodes and the network node (l3,dhcp.neutron metadata).
2. enabled a sub-interface for external network provider
3. used the following answer file with packstack:

CONFIG_NEUTRON_INSTALL=y
CONFIG_NEUTRON_SERVER_HOST=192.168.160.29
CONFIG_NEUTRON_KS_PW=123456
CONFIG_NEUTRON_DB_PW=123456
CONFIG_NEUTRON_L3_HOSTS=192.168.160.27
CONFIG_NEUTRON_L3_EXT_BRIDGE=br-ex
CONFIG_NEUTRON_DHCP_HOSTS=192.168.160.27
CONFIG_NEUTRON_LBAAS_HOSTS=
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_METADATA_HOSTS=192.168.160.27
CONFIG_NEUTRON_METADATA_PW=123456
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan,vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vxlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_FLAT_NETWORKS=*
CONFIG_NEUTRON_ML2_VLAN_RANGES=ext_net:185:185
CONFIG_NEUTRON_ML2_TUNNEL_ID_RANGES=1:1000
CONFIG_NEUTRON_ML2_VXLAN_GROUP=224.0.0.1
CONFIG_NEUTRON_ML2_VNI_RANGES=3000:3100,2000:2100
CONFIG_NEUTRON_L2_AGENT=openvswitch
CONFIG_NEUTRON_LB_TENANT_NETWORK_TYPE=local
CONFIG_NEUTRON_LB_VLAN_RANGES=
CONFIG_NEUTRON_LB_INTERFACE_MAPPINGS=
CONFIG_NEUTRON_OVS_TENANT_NETWORK_TYPE=vxlan
CONFIG_NEUTRON_OVS_VLAN_RANGES=ext_net:185:185
CONFIG_NEUTRON_OVS_BRIDGE_MAPPINGS=ext_net:br-ex
CONFIG_NEUTRON_OVS_BRIDGE_IFACES=br-ex:eth3.185
CONFIG_NEUTRON_OVS_TUNNEL_RANGES=1:1000
CONFIG_NEUTRON_OVS_TUNNEL_IF=eth3
CONFIG_NEUTRON_OVS_VXLAN_UDP_PORT=4789

4. on the network node, added a link between the br-ex bridge to the eth3.185 sub-inteface (for external network). this is done manually due to Bug #1072268
   # ovs-vsctl add-port br-ex eth3.185

5. Created an external network:
   # neutron net-create ext_net --provider:network_type vlan --provider:physical_network ext_net --provider:segmentation_id 185 --router:external=True

6. Added a routeable subnet to that network:
   # neutron subnet-create ext_net --gateway 11.11.170.254 11.11.170.0/24 -- --enable_dhcp=False --name subnet_ext_net

7. Created a neutron router:
   # neutron router-create router1

8. Created a VXLAN tenant network:
   # neutron net-create int_net --provider:network_type vxlan --provider:segmentation_id

9. Added a subnet to that network
   # neutron subnet-create int_net 44.44.44.0/24 -- --enable_dhcp=True --name subnet_int_net

10. Configuted router's gw:
    # neutron router-gateway-set router1 ext_net

11. Added a router interface for the internal VXLAN network:
    # neutron router-interface-add router1 subnet_int_net

12.Uploaded an image to glance.

13. Booted 4 instances.

14. Verified network connectivity between instances both on the same and on different compute nodes.

15. Verified that the instances can reach the internet (pinged 8.8.8.8)

16. Assiged floating IP addresses and checked that they are pingable from outside of the OpenStack deployment.

Comment 12 Nir Magnezi 2014-03-12 20:49:48 UTC
(In reply to Martin Magr from comment #7)
> Nir this is also configuration problem. I admit that we should have
> implemented validation on that parameter, but you should try correct values.
> Following is what answer file says:
> 
> # Multicast group for VXLAN. If unset, disables VXLAN enable sending
> # allocate broadcast traffic to this multicast group. When left
> # unconfigured, will disable multicast VXLAN mode. Should be an
> # Multicast IP (v4 or v6) address.
> CONFIG_NEUTRON_ML2_VXLAN_GROUP=208,209,210
> 
> 
> Nevertheless there are missing quotes in resulting neutron.pp:50 manifest:
> 
> 43: class { 'neutron::plugins::ml2':
> 44:    type_drivers         => ['vxlan'],
> 45:    tenant_network_types => ['vxlan', 'vlan'],
> 46:    mechanism_drivers    => ['openvswitch'],
> 47:    flat_networks        => ['*'],
> 48:    network_vlan_ranges  => ['ext_net:100:200'],
> 49:    tunnel_id_ranges     => ['1:1000'],
> 50:    vxlan_group          => 208,209,210,
> 51:    vni_ranges           => ['3000:3100', '2000:2100'],
> 52: }
> 
> So we will have to fix that. Thanks for discovering this bug.

I confirm that packstack validates the correctness of vxlan_group:
                                                                                           [ ERROR ]

ERROR : Error appeared during Puppet run: 192.168.160.21_neutron.pp
Error: 208,209,210 is not valid for vxlan_group. at /var/tmp/packstack/463e04f9f17541acbbb4826980b716fe/modules/neutron/manifests/plugins/ml2/driver.pp:72 on node <FQDN>

Comment 14 errata-xmlrpc 2014-03-25 19:24:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0334.html


Note You need to log in before you can comment on or make changes to this bug.