RHEL6.5 Controller (lyn05) Neutron Networker (lynx06) 2 Compute nodes Description of problem: During installation, puppet neutron destroys "br-ex" bridge (external bridge for public network) Even though br-ex is connected to physical NIC, it is immediately destroyed and recreated without that connection As a result, external network and floating IPs don't work [root@lynx06 ~]# grep "br-ex\|br-int\|br-tun" /var/log/messages | head -n200 Aug 24 19:01:14 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-br br-int Aug 24 19:01:14 lynx06 kernel: device br-int entered promiscuous mode Aug 24 19:01:14 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-br br-ex Aug 24 19:01:14 lynx06 kernel: device br-ex entered promiscuous mode Aug 24 19:01:14 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port br-ex eth3.341 Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex eth3.341 Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex Aug 24 19:01:15 lynx06 kernel: device br-ex left promiscuous mode Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex Aug 24 19:01:15 lynx06 kernel: device br-ex entered promiscuous mode Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex Aug 24 19:01:15 lynx06 kernel: device br-ex left promiscuous mode Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-ex eth3.341 Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00002|vsctl|ERR|no bridge named br-ex Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex Aug 24 19:01:15 lynx06 kernel: device br-ex entered promiscuous mode Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex Aug 24 19:01:15 lynx06 kernel: device br-ex left promiscuous mode Aug 24 19:01:15 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex Aug 24 19:01:16 lynx06 kernel: device br-ex entered promiscuous mode Aug 24 19:01:16 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex Aug 24 19:01:16 lynx06 kernel: device br-ex left promiscuous mode Aug 24 19:01:16 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex Aug 24 19:01:16 lynx06 kernel: device br-ex entered promiscuous mode [root@lynx05 manifests]# grep "br-ex\|br-in\|br-tun" 10.35.117.12_neutron.pp.log Debug: Adding relationship from Service[openvswitch] to Vs_bridge[br-int] with 'before' Debug: Adding relationship from Service[openvswitch] to Vs_bridge[br-ex] with 'before' Debug: Adding relationship from Service[openvswitch] to Vs_bridge[br-tun] with 'before' Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_bridge[br-int] Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_bridge[br-ex] Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_bridge[br-tun] Debug: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]/before: requires Service[neutron-ovs-agent-service] Debug: /Stage[main]/Neutron::Agents::Ml2::Ovs/Vs_bridge[br-tun]/before: requires Service[neutron-ovs-agent-service] Debug: /Stage[main]/Neutron::Agents::Ml2::Ovs/Vs_bridge[br-int]/before: requires Service[neutron-ovs-agent-service] Debug: /Stage[main]/Main/Vs_port[eth3.341]: Autorequiring Vs_bridge[br-ex] Debug: Executing '/usr/bin/ovs-vsctl br-exists br-int' Debug: Executing '/usr/bin/ovs-vsctl add-br br-int' Debug: Executing '/sbin/ip link set br-int up' Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Vs_bridge[br-int]/ensure: created Debug: /Stage[main]/Neutron::Agents::Ml2::Ovs/Vs_bridge[br-int]: The container Class[Neutron::Agents::Ml2::Ovs] will propagate my refresh event Debug: Executing '/usr/bin/ovs-vsctl br-exists br-ex' Debug: Executing '/usr/bin/ovs-vsctl add-br br-ex' Debug: Executing '/sbin/ip link set br-ex up' Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]/Vs_bridge[br-ex]/ensure: created Debug: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]/Vs_bridge[br-ex]: The container Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex] will propagate my refresh event Debug: Executing '/usr/bin/ovs-vsctl list-ports br-ex' Debug: Executing '/usr/bin/ovs-vsctl list-ports br-ex' Debug: Executing '/usr/bin/ovs-vsctl add-port br-ex eth3.341' Debug: Executing '/sbin/ifdown br-ex' Debug: Executing '/sbin/ifup br-ex' Debug: Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]: The container Class[Neutron::Agents::Ml2::Ovs] will propagate my refresh event Debug: Executing '/usr/bin/ovs-vsctl br-exists br-tun' Debug: Executing '/usr/bin/ovs-vsctl add-br br-tun' Debug: Executing '/sbin/ip link set br-tun up' Notice: /Stage[main]/Neutron::Agents::Ml2::Ovs/Vs_bridge[br-tun]/ensure: created Debug: /Stage[main]/Neutron::Agents::Ml2::Ovs/Vs_bridge[br-tun]: The container Class[Neutron::Agents::Ml2::Ovs] will p /var/log/neutron/l3-agent.log: 2014-08-24 19:24:39.679 10347 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:48 Command: ['ip', '-o', 'link', 'show', 'br-ex'] Stderr: 'Device "br-ex" does not exist.\n' execute /usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py:74 2014-08-24 19:24:39.686 10347 ERROR neutron.agent.l3_agent [-] The external network bridge 'br-ex' does not exist Version-Release number of selected component (if applicable): [root@lynx05 manifests]# rpm -qa | grep "neutron\|packstack" openstack-packstack-puppet-2014.1.1-0.40.dev1239.el6ost.noarch python-neutronclient-2.3.4-2.el6ost.noarch python-neutron-2014.1.2-2.el6ost.noarch openstack-neutron-ml2-2014.1.2-2.el6ost.noarch openstack-packstack-2014.1.1-0.40.dev1239.el6ost.noarch openstack-neutron-2014.1.2-2.el6ost.noarch [root@lynx06 ~]# rpm -qa | grep "neutron\|packstack" python-neutron-2014.1.2-2.el6ost.noarch python-neutronclient-2.3.4-2.el6ost.noarch openstack-neutron-2014.1.2-2.el6ost.noarch openstack-neutron-openvswitch-2014.1.2-2.el6ost.noarch Steps to Reproduce: 1. install openstack+neutron with packstack 2. create floating IPs and try to connect to them Additional info: Doesn't seem to happen in RHEL7
Created attachment 930349 [details] answer file
Created attachment 930351 [details] packstack log
More details: NetworkManager is off puppet creates file "ifcfg-br-ex" which I believe shouldn't exist [root@lynx06 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex TYPE=OVSBridge DEVICE=br-ex OVSBOOTPROTO= DEVICETYPE=ovs
Created attachment 930360 [details] Networker puppet log
Created attachment 930361 [details] neutron networker log files (l3-agent, messages, ovs)
This is not a regression. Module puppet-vswitch is creating ifcfg-br-ex config also on RHEL-7. Also, created bridge can be connected to physical interface after Packstack run, so IMHO this is also not blocking issue. Assigned to Gilles, who is working on puppet-vswitch refactor.
this is a regression because the same workflow (on automation) worked before, and still works on el7 I don't know why it works on el7 and not on el6 but that doesn't mean ifcfg-br-ex file should exist. We have said before that it's problematic, even though it currently works on el7. According to neutron devs (amuller) it should not be created unless you assign IP to br-ex. So if you aren't - please don't create it. From the logs it seems like br-ex is being constantly recreated. Don't know why, but if that is the case, there is no use for a workaround to reattach NIC, because it will still be deleted later. So still AutomationBlocker
(In reply to yfried from comment #9) > According to neutron devs (amuller) > it should not be created unless you assign IP to br-ex. So if you aren't - > please don't create it. > The vswitch module creates the br-ex ifcfg file using the assigned physical interface configuration as a template. Therefore if an IP exist on the physical interface, it's used for the bridge configuration. > From the logs it seems like br-ex is being constantly recreated. Don't know > why, but if that is the case, there is no use for a workaround to reattach > NIC, because it will still be deleted later. So still AutomationBlocker I can't see that from the attached log.
Identified the reason it doesn't behave the same way on RHEL6 vs RHEL7: Respective Ruby versions are 1.8.7 and 2.0. and Regexp match method behaves differently after Ruby 1.8.7. This is breaking puppet-vswitch implementation when assigning a network interface to the bridge. The existing network interface configuration is not used to populate the bridge configuration and the generated content is incomplete. Upstream puppet-vswitch patch 18 now supports older Ruby version as well.
looks a bit similar to staypuft bug that we had before: https://bugzilla.redhat.com/show_bug.cgi?id=1122142 Tested with staypuft using: ruby193-rubygem-staypuft-0.2.5-1.el6ost.noarch (from puddle 2014-08-19.1), the br-ex bridge exist on the neutron-networker, As well assigning floating IPs to instances works well. > > assigned a public address 10.8.30.103 to one instance (checked it was > > available/unreachable prior to assignment). > > > > ping 10.8.30.103 > > PING 10.8.30.103 (10.8.30.103) 56(84) bytes of data. > > 64 bytes from 10.8.30.103: icmp_seq=1 ttl=59 time=77.7 ms > > 64 bytes from 10.8.30.103: icmp_seq=2 ttl=59 time=73.3 ms > > 64 bytes from 10.8.30.103: icmp_seq=3 ttl=59 time=68.5 ms > > 64 bytes from 10.8.30.103: icmp_seq=4 ttl=59 time=64.4 ms > > 64 bytes from 10.8.30.103: icmp_seq=5 ttl=59 time=63.9 ms > > > > > > Pinging outside also works:
BTW Patch 19 changed the ifdown/ifup order so br-ex (default) comes up on rhel6
[root@lynx05 ~]# rpm -qa | grep "packstack\|neutron" python-neutronclient-2.3.4-2.el6ost.noarch openstack-packstack-2014.1.1-0.40.dev1239.el6ost.noarch openstack-neutron-2014.1.2-2.el6ost.noarch openstack-packstack-puppet-2014.1.1-0.40.dev1239.el6ost.noarch python-neutron-2014.1.2-2.el6ost.noarch openstack-neutron-openvswitch-2014.1.2-2.el6ost.noarch [root@lynx05 ~]# rpm -qa | grep openstack-puppet openstack-puppet-modules-2014.1-21.4.el6ost.noarch [root@lynx06 ~]# grep -n "br-ex\|eth3" /var/log/messages | head -n100 | grep ovs 1827:Aug 28 10:08:01 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-br br-ex 1829:Aug 28 10:08:02 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port br-ex eth3 1833:Aug 28 10:08:02 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-port br-ex eth3 1835:Aug 28 10:08:02 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1837:Aug 28 10:08:02 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1841:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1842:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-port br-ex eth3 1844:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00002|vsctl|ERR|no bridge named br-ex 1846:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1848:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1850:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1852:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1854:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1858:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1860:Aug 28 10:08:03 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1862:Aug 28 10:08:04 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1864:Aug 28 10:08:04 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1866:Aug 28 10:08:04 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1868:Aug 28 10:08:04 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1871:Aug 28 10:08:04 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1873:Aug 28 10:08:04 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1877:Aug 28 10:08:05 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex 1879:Aug 28 10:08:05 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --may-exist add-br br-ex 1881:Aug 28 10:08:05 lynx06 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl -t 10 -- --if-exists del-br br-ex [root@lynx06 ~]# ovs-vsctl show 66db9bc2-d54b-4df0-b3bb-423280672621 Bridge br-ex Port br-ex Interface br-ex type: internal Bridge br-int fail_mode: secure Port "qg-2f524977-b5" Interface "qg-2f524977-b5" type: internal [root@lynx05 ~]# grep -n "br-ex\|eth3" /var/tmp/packstack/20140828-094304-3E5SRv/manifests/10.35.117.12_neutron.pp.log --color 453:Debug: Adding relationship from Service[openvswitch] to Vs_port[eth3] with 'before' 454:Debug: Adding relationship from Service[openvswitch] to Vs_bridge[br-ex] with 'before' 504:Debug: Adding relationship from Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex] to Service[neutron-plugin-ovs-service] with 'before' 1005:Debug: /Stage[main]/Neutron::Agents::Ovs/Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]/before: requires Service[neutron-plugin-ovs-service] 1037:Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_port[eth3] 1038:Debug: /Stage[main]/Vswitch::Ovs/Service[openvswitch]/before: requires Vs_bridge[br-ex] 1275:Debug: /Stage[main]/Main/Vs_port[eth3]: Autorequiring Vs_bridge[br-ex] 1633:Debug: Executing '/usr/bin/ovs-vsctl br-exists br-int' 1674:Debug: Executing '/usr/bin/ovs-vsctl br-exists br-ex' 1675:Debug: Executing '/usr/bin/ovs-vsctl add-br br-ex' 1676:Debug: Executing '/sbin/ip link set br-ex up' 1677:Notice: /Stage[main]/Neutron::Agents::Ovs/Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]/Vs_bridge[br-ex]/ensure: created 1678:Debug: /Stage[main]/Neutron::Agents::Ovs/Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]/Vs_bridge[br-ex]: The container Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex] will propagate my refresh event 1679:Debug: Neutron::Plugins::Ovs::Bridge[ext_vlan:br-ex]: The container Class[Neutron::Agents::Ovs] will propagate my refresh event 1680:Debug: Executing '/usr/bin/ovs-vsctl list-ports br-ex' 1681:Debug: Executing '/usr/bin/ovs-vsctl list-ports br-ex' 1682:Debug: Executing '/usr/bin/ovs-vsctl add-port br-ex eth3' 1683:Debug: Executing '/sbin/ifdown eth3' 1684:Debug: Executing '/sbin/ifdown br-ex' 1685:Debug: Executing '/sbin/ifup eth3' 1686:Debug: Executing '/sbin/ifup br-ex' 1687:Notice: /Stage[main]/Main/Vs_port[eth3]/ensure: created 1688:Debug: /Stage[main]/Main/Vs_port[eth3]: The container Class[Main] will propagate my refresh event 1778:Debug: Executing '/usr/bin/ovs-vsctl br-exists br-tun'
Created attachment 931766 [details] qa failure logs Adding logs from failedQA failure. I really think this is problems with ifcfg-br-ex file. I believe this file shouldn't be created (the same way we don't have files for br-int and br-tun) [root@lynx06 ~]# ll /etc/sysconfig/network-scripts/ifcfg-br-* -rw-r--r--. 1 root root 115 Aug 28 10:08 /etc/sysconfig/network-scripts/ifcfg-br-ex [root@lynx06 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex PEERDNS=no ONBOOT=yes NOZEROCONF=yes DEVICETYPE=ovs NM_CONTROLLED=no OVSBOOTPROTO=dhcp TYPE=OVSBridge DEVICE=br-ex
Patch 19 is not in openstack-puppet-modules-2014.1-21.4.el6ost. The reasons: Tests results on RHEL65 show the bridge interface must be put down first prior to physical interface, more likely because the configuration file exists when the command ifdown command is executed. Patch 18 and 19 have been tested on rhel6 (and rhel7 for non regression although out of scope here).
(In reply to yfried from comment #20) > I believe this file shouldn't be created (the same way we > don't have files for br-int and br-tun) The only way, as far as I know to have a resilient physical interface attached to an OVS bridge is to have both ifcfg files defined. If you know another way, please share the details. > [root@lynx06 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br-ex > PEERDNS=no > ONBOOT=yes > NOZEROCONF=yes > DEVICETYPE=ovs > NM_CONTROLLED=no > OVSBOOTPROTO=dhcp > TYPE=OVSBridge > DEVICE=br-ex Also keep in mind that when a physical interface is attached to an OVS bridge, any existing configuration of such physical interface is going to be used into the bridge definition (ifcfg). So the necessary steps must be done prior to the installation in the case where no IP address is needed.
I've rebuilt to include patch #19
Verified by automation
ip command (iproute2) doesn't behave as expected in regards of dynamic interfaces, created BZ#1135877 accordingly. So when an Openvswitch bridge is configured using a physical interface initially configured with DHCP, the bridge won't start on RHEL6 because the interface is not detected as dynamic and therefore OVSDHCPINTERFACES option is missing. In the meantime, BZ#1135877 gets eventually addressed, upstream puppet-vswitch patch 20 provides a workaround for the DHCP detection issue through a rhel6 vs_port provider. Just tested patch 20 on rhel6 (and rhel7 for non regression) for following cases: - Dynamic (DHCP) - Static IP - No IP
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1124.html
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days