Bug 1482604 - [osp12] After successful deployment cannot ping external network ovs-system state DOWN on the overcloud nodes.
Summary: [osp12] After successful deployment cannot ping external network ovs-system s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 12.0 (Pike)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: beta
: 12.0 (Pike)
Assignee: Daniel Alvarez Sanchez
QA Contact: Toni Freger
URL:
Whiteboard:
: 1484853 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-17 17:20 UTC by Omri Hochman
Modified: 2018-02-05 19:12 UTC (History)
10 users (show)

Fixed In Version: openstack-neutron-11.0.0-0.20170821141644.3441b3f.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 21:53:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 494046 0 None MERGED Add a target to Sinkhole so it's compatible 2020-04-09 12:07:34 UTC
Red Hat Product Errata RHEA-2017:3462 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC

Description Omri Hochman 2017-08-17 17:20:02 UTC
[osp12] After successful deployment cannot ping external network ovs-system state DOWN on the overcloud nodes.



Environment: 
-------------
container images with TAG 2017-08-16.1
openstack-tripleo-heat-templates-7.0.0-0.20170805163048.el7ost.noarch
openvswitch-ovn-common-2.7.2-1.git20170719.el7fdp.x86_64
openvswitch-2.7.2-1.git20170719.el7fdp.x86_64
openvswitch-ovn-host-2.7.2-1.git20170719.el7fdp.x86_64
openstack-neutron-openvswitch-11.0.0-0.20170807223712.el7ost.noarch
python-openvswitch-2.7.2-1.git20170719.el7fdp.noarch
openvswitch-ovn-central-2.7.2-1.git20170719.el7fdp.x86_64
puppet-neutron-11.3.0-0.20170805104936.743dde6.el7ost.noarch
openstack-neutron-lbaas-11.0.0-0.20170807144457.c9adfd4.el7ost.noarch
openstack-neutron-linuxbridge-11.0.0-0.20170807223712.el7ost.noarch
python-neutronclient-6.5.0-0.20170807200849.355983d.el7ost.noarch
python-neutron-11.0.0-0.20170807223712.el7ost.noarch
python-neutron-lbaas-11.0.0-0.20170807144457.c9adfd4.el7ost.noarch
openstack-neutron-sriov-nic-agent-11.0.0-0.20170807223712.el7ost.noarch
openstack-neutron-ml2-11.0.0-0.20170807223712.el7ost.noarch
openstack-neutron-11.0.0-0.20170807223712.el7ost.noarch
openstack-neutron-openvswitch-11.0.0-0.20170807223712.el7ost.noarch
openstack-neutron-metering-agent-11.0.0-0.20170807223712.el7ost.noarch
python-neutron-lib-1.9.1-0.20170731102145.0ef54c3.el7ost.noarch
openstack-neutron-common-11.0.0-0.20170807223712.el7ost.noarch


Description:
-------------
after successful deployment on Bare-metal :  3 controller 1 compute 3 ceph 
ovs-system state DOWN on the overcloud nodes and it's impossible to ping any external server. 


| 358440d3-ed9b-4ff9-ad6f-6835954ecce0 | overcloud-cephstorage-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.20 |
| 118c3073-3e3a-4c91-9ab8-3bfe681e9ee1 | overcloud-cephstorage-1 | ACTIVE | -          | Running     | ctlplane=192.168.24.6  |
| 7bb1b4de-d848-4980-8311-e9d7550f7810 | overcloud-cephstorage-2 | ACTIVE | -          | Running     | ctlplane=192.168.24.18 |
| 8ef8ba97-9db5-4fa0-b957-192ab308ac5b | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=192.168.24.8  |
| 6dcd4597-a4dd-4e48-8f3e-41845065c4cb | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.168.24.12 |
| 70c2b99f-159c-4873-928e-5db68003eca3 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.168.24.7  |
| 62057c81-63d4-4965-b5a5-eee84d78e3be | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.168.24.10 |


undercloud) [stack@undercloud ~]$ ssh heat-admin.24.8
[heat-admin@overcloud-compute-0 ~]$ ping 8.8.8.8
ping: socket: Operation not permitted
[heat-admin@overcloud-compute-0 ~]$ sudo ping 8.8.8.8
connect: Network is unreachable
[heat-admin@overcloud-compute-0 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether c8:1f:66:c7:eb:60 brd ff:ff:ff:ff:ff:ff
    inet6 2620:52:0:13b8:ca1f:66ff:fec7:eb60/64 scope global mngtmpaddr dynamic
       valid_lft 2591809sec preferred_lft 604609sec
    inet6 fe80::ca1f:66ff:fec7:eb60/64 scope link
       valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 1000
    link/ether c8:1f:66:c7:eb:61 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ca1f:66ff:fec7:eb61/64 scope link
       valid_lft forever preferred_lft forever
4: p2p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:0a:f7:79:93:1a brd ff:ff:ff:ff:ff:ff
    inet 192.168.24.8/24 brd 192.168.24.255 scope global p2p1
       valid_lft forever preferred_lft forever
    inet6 fe80::20a:f7ff:fe79:931a/64 scope link
       valid_lft forever preferred_lft forever
5: p2p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP qlen 1000
    link/ether 00:0a:f7:79:93:1b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20a:f7ff:fe79:931b/64 scope link
       valid_lft forever preferred_lft forever
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 36:42:c2:e9:56:31 brd ff:ff:ff:ff:ff:ff
7: br-nic4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 00:0a:f7:79:93:1b brd ff:ff:ff:ff:ff:ff
    inet 192.168.150.15/24 brd 192.168.150.255 scope global br-nic4
       valid_lft forever preferred_lft forever
    inet6 fe80::20a:f7ff:fe79:931b/64 scope link
       valid_lft forever preferred_lft forever
8: br-nic2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether c8:1f:66:c7:eb:61 brd ff:ff:ff:ff:ff:ff
    inet 10.19.94.11/24 brd 10.19.94.255 scope global br-nic2
       valid_lft forever preferred_lft forever
    inet6 2620:52:0:135e:ca1f:66ff:fec7:eb61/64 scope global mngtmpaddr dynamic
       valid_lft 2591879sec preferred_lft 604679sec
    inet6 fe80::ca1f:66ff:fec7:eb61/64 scope link
       valid_lft forever preferred_lft forever
9: vlan183: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 16:9b:8b:c0:00:20 brd ff:ff:ff:ff:ff:ff
    inet 10.19.95.14/24 brd 10.19.95.255 scope global vlan183
       valid_lft forever preferred_lft forever
    inet6 2620:52:0:135f:149b:8bff:fec0:20/64 scope global mngtmpaddr dynamic
       valid_lft 2591879sec preferred_lft 604679sec
    inet6 fe80::149b:8bff:fec0:20/64 scope link
       valid_lft forever preferred_lft forever
10: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:a8:c1:7f:39 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:a8ff:fec1:7f39/64 scope link
       valid_lft forever preferred_lft forever
13: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether da:58:7f:06:e2:49 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d858:7fff:fe06:e249/64 scope link
       valid_lft forever preferred_lft forever
14: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether ae:74:3f:94:bc:44 brd ff:ff:ff:ff:ff:ff
15: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether b6:f7:09:6d:3f:43 brd ff:ff:ff:ff:ff:ff
16: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master ovs-system state UNKNOWN qlen 1000
    link/ether e6:64:5d:15:18:c0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e464:5dff:fe15:18c0/64 scope link
       valid_lft forever preferred_lft forever
[heat-admin@overcloud-compute-0 ~]$ ip a |grep down
[heat-admin@overcloud-compute-0 ~]$ ip a |grep DOWN
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
10: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
14: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
15: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
[heat-admin@overcloud-compute-0 ~]$ sudo ping ya.ru
ping: ya.ru: Name or service not known
[heat-admin@overcloud-compute-0 ~]$ sudo ping 8.8.8.8
connect: Network is unreachable
[heat-admin@overcloud-compute-0 ~]$ sudo ping 10.0.0.1
connect: Network is unreachable
[heat-admin@overcloud-compute-0 ~]$ sudo ping undercloud-0
ping: undercloud-0: Name or service not known
[heat-admin@overcloud-compute-0 ~]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# HEAT_HOSTS_START - Do not edit manually within this section!
10.19.184.187  overcloud.localdomain
192.168.24.15  overcloud.ctlplane.localdomain
10.19.94.15  overcloud.internalapi.localdomain
10.19.95.13  overcloud.storage.localdomain
192.168.200.13  overcloud.storagemgmt.localdomain
10.19.94.10 overcloud-controller-0.localdomain overcloud-controller-0
10.19.184.189 overcloud-controller-0.external.localdomain overcloud-controller-0.external
10.19.94.10 overcloud-controller-0.internalapi.localdomain overcloud-controller-0.internalapi
10.19.95.24 overcloud-controller-0.storage.localdomain overcloud-controller-0.storage
192.168.200.15 overcloud-controller-0.storagemgmt.localdomain overcloud-controller-0.storagemgmt
192.168.150.13 overcloud-controller-0.tenant.localdomain overcloud-controller-0.tenant
192.168.24.12 overcloud-controller-0.management.localdomain overcloud-controller-0.management
192.168.24.12 overcloud-controller-0.ctlplane.localdomain overcloud-controller-0.ctlplane
10.19.94.13 overcloud-controller-1.localdomain overcloud-controller-1
10.19.184.192 overcloud-controller-1.external.localdomain overcloud-controller-1.external
10.19.94.13 overcloud-controller-1.internalapi.localdomain overcloud-controller-1.internalapi
10.19.95.16 overcloud-controller-1.storage.localdomain overcloud-controller-1.storage
192.168.200.14 overcloud-controller-1.storagemgmt.localdomain overcloud-controller-1.storagemgmt
192.168.150.10 overcloud-controller-1.tenant.localdomain overcloud-controller-1.tenant
192.168.24.7 overcloud-controller-1.management.localdomain overcloud-controller-1.management
192.168.24.7 overcloud-controller-1.ctlplane.localdomain overcloud-controller-1.ctlplane
10.19.94.23 overcloud-controller-2.localdomain overcloud-controller-2
10.19.184.191 overcloud-controller-2.external.localdomain overcloud-controller-2.external
10.19.94.23 overcloud-controller-2.internalapi.localdomain overcloud-controller-2.internalapi
10.19.95.11 overcloud-controller-2.storage.localdomain overcloud-controller-2.storage
192.168.200.12 overcloud-controller-2.storagemgmt.localdomain overcloud-controller-2.storagemgmt
192.168.150.11 overcloud-controller-2.tenant.localdomain overcloud-controller-2.tenant
192.168.24.10 overcloud-controller-2.management.localdomain overcloud-controller-2.management
192.168.24.10 overcloud-controller-2.ctlplane.localdomain overcloud-controller-2.ctlplane

10.19.94.11 overcloud-compute-0.localdomain overcloud-compute-0
192.168.24.8 overcloud-compute-0.external.localdomain overcloud-compute-0.external
10.19.94.11 overcloud-compute-0.internalapi.localdomain overcloud-compute-0.internalapi
10.19.95.14 overcloud-compute-0.storage.localdomain overcloud-compute-0.storage
192.168.24.8 overcloud-compute-0.storagemgmt.localdomain overcloud-compute-0.storagemgmt
192.168.150.15 overcloud-compute-0.tenant.localdomain overcloud-compute-0.tenant
192.168.24.8 overcloud-compute-0.management.localdomain overcloud-compute-0.management
192.168.24.8 overcloud-compute-0.ctlplane.localdomain overcloud-compute-0.ctlplane



10.19.95.19 overcloud-cephstorage-0.localdomain overcloud-cephstorage-0
192.168.24.20 overcloud-cephstorage-0.external.localdomain overcloud-cephstorage-0.external
192.168.24.20 overcloud-cephstorage-0.internalapi.localdomain overcloud-cephstorage-0.internalapi
10.19.95.19 overcloud-cephstorage-0.storage.localdomain overcloud-cephstorage-0.storage
192.168.200.10 overcloud-cephstorage-0.storagemgmt.localdomain overcloud-cephstorage-0.storagemgmt
192.168.24.20 overcloud-cephstorage-0.tenant.localdomain overcloud-cephstorage-0.tenant
192.168.24.20 overcloud-cephstorage-0.management.localdomain overcloud-cephstorage-0.management
192.168.24.20 overcloud-cephstorage-0.ctlplane.localdomain overcloud-cephstorage-0.ctlplane
10.19.95.18 overcloud-cephstorage-1.localdomain overcloud-cephstorage-1
192.168.24.6 overcloud-cephstorage-1.external.localdomain overcloud-cephstorage-1.external
192.168.24.6 overcloud-cephstorage-1.internalapi.localdomain overcloud-cephstorage-1.internalapi
10.19.95.18 overcloud-cephstorage-1.storage.localdomain overcloud-cephstorage-1.storage
192.168.200.19 overcloud-cephstorage-1.storagemgmt.localdomain overcloud-cephstorage-1.storagemgmt
192.168.24.6 overcloud-cephstorage-1.tenant.localdomain overcloud-cephstorage-1.tenant
192.168.24.6 overcloud-cephstorage-1.management.localdomain overcloud-cephstorage-1.management
192.168.24.6 overcloud-cephstorage-1.ctlplane.localdomain overcloud-cephstorage-1.ctlplane
10.19.95.15 overcloud-cephstorage-2.localdomain overcloud-cephstorage-2
192.168.24.18 overcloud-cephstorage-2.external.localdomain overcloud-cephstorage-2.external
192.168.24.18 overcloud-cephstorage-2.internalapi.localdomain overcloud-cephstorage-2.internalapi
10.19.95.15 overcloud-cephstorage-2.storage.localdomain overcloud-cephstorage-2.storage
192.168.200.17 overcloud-cephstorage-2.storagemgmt.localdomain overcloud-cephstorage-2.storagemgmt
192.168.24.18 overcloud-cephstorage-2.tenant.localdomain overcloud-cephstorage-2.tenant
192.168.24.18 overcloud-cephstorage-2.management.localdomain overcloud-cephstorage-2.management
192.168.24.18 overcloud-cephstorage-2.ctlplane.localdomain overcloud-cephstorage-2.ctlplane

# HEAT_HOSTS_END

[heat-admin@overcloud-compute-0 ~]$ cat /etc/hosts |grep undercloud
[heat-admin@overcloud-compute-0 ~]$ ping compute-1
ping: socket: Operation not permitted
[heat-admin@overcloud-compute-0 ~]$ sudo ping compute-1
ping: compute-1: Name or service not known
[heat-admin@overcloud-compute-0 ~]$ hostname -f
overcloud-compute-0.localdomain
[heat-admin@overcloud-compute-0 ~]$ ping overcloud-compute-1.localdomain
ping: socket: Operation not permitted
[heat-admin@overcloud-compute-0 ~]$ sudo ping overcloud-compute-1.localdomain
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.056 ms
^C
--- localhost ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.056/0.056/0.056/0.000 ms
[heat-admin@overcloud-compute-0 ~]$ sudo ping overcloud-cephstorage-2.localdomain
PING overcloud-cephstorage-2.localdomain (10.19.95.15) 56(84) bytes of data.
64 bytes from overcloud-cephstorage-2.localdomain (10.19.95.15): icmp_seq=1 ttl=64 time=1.82 ms
64 bytes from overcloud-cephstorage-2.localdomain (10.19.95.15): icmp_seq=2 ttl=64 time=0.217 ms
^C
--- overcloud-cephstorage-2.localdomain ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.217/1.020/1.824/0.804 ms
[heat-admin@overcloud-compute-0 ~]$ exit
logout
Connection to 192.168.24.8 closed.
(undercloud) [stack@undercloud ~]$ sudo iptables -nL -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
REDIRECT   tcp  --  0.0.0.0/0            169.254.169.254      tcp dpt:80 redir ports 8775

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
BOOTSTACK_MASQ  all  --  0.0.0.0/0            0.0.0.0/0

Chain BOOTSTACK_MASQ (1 references)
target     prot opt source               destination
MASQUERADE  all  --  192.168.24.0/24     !192.168.24.0/24
(undercloud) [stack@undercloud ~]$ ssh heat-admin.24.8
Last login: Thu Aug 17 16:20:41 2017 from 192.168.24.1
[heat-admin@overcloud-compute-0 ~]$ systemctl status ovs*
● ovs-vswitchd.service - Open vSwitch Forwarding Unit
   Loaded: loaded (/usr/lib/systemd/system/ovs-vswitchd.service; static; vendor preset: disabled)
   Active: active (running) since Thu 2017-08-17 14:27:07 UTC; 1h 57min ago
 Main PID: 1009 (ovs-vswitchd)
   Memory: 96.5M
   CGroup: /system.slice/ovs-vswitchd.service
           └─1009 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.lo...

Aug 17 14:27:06 localhost.localdomain systemd[1]: Starting Open vSwitch Forwarding Unit...
Aug 17 14:27:06 localhost.localdomain ovs-ctl[967]: Inserting openvswitch module [  OK  ]
Aug 17 14:27:07 localhost.localdomain ovs-ctl[967]: Starting ovs-vswitchd [  OK  ]
Aug 17 14:27:07 localhost.localdomain ovs-ctl[967]: Enabling remote OVSDB managers [  OK  ]
Aug 17 14:27:07 localhost.localdomain systemd[1]: Started Open vSwitch Forwarding Unit.

● ovsdb-server.service - Open vSwitch Database Unit
   Loaded: loaded (/usr/lib/systemd/system/ovsdb-server.service; static; vendor preset: disabled)
   Active: active (running) since Thu 2017-08-17 14:27:06 UTC; 1h 57min ago
 Main PID: 945 (ovsdb-server)
   Memory: 72.0K
   CGroup: /system.slice/ovsdb-server.service
           └─945 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,p...

Aug 17 14:27:05 localhost.localdomain systemd[1]: Starting Open vSwitch Database Unit...
Aug 17 14:27:05 localhost.localdomain ovs-ctl[884]: /etc/openvswitch/conf.db does not exist ... (warning).
Aug 17 14:27:05 localhost.localdomain ovs-ctl[884]: Creating empty database /etc/openvswitch/conf.db [  OK  ]
Aug 17 14:27:06 localhost.localdomain ovs-ctl[884]: Starting ovsdb-server [  OK  ]
Aug 17 14:27:06 localhost.localdomain ovs-vsctl[946]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait -- init -- set Open_vSwitch . db-version=7.15.0
Aug 17 14:27:06 localhost.localdomain ovs-vsctl[957]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --no-wait set Open_vSwitch . ovs-version=2.7.2 "external-ids:system-i...n=\"7.4\""
Aug 17 14:27:06 localhost.localdomain ovs-ctl[884]: Configuring Open vSwitch system IDs [  OK  ]
Aug 17 14:27:06 localhost.localdomain systemd[1]: Started Open vSwitch Database Unit.
Hint: Some lines were ellipsized, use -l to show in full.
[heat-admin@overcloud-compute-0 ~]$ systemctl |grep ovs
  sys-devices-virtual-net-ovs\x2dsystem.device                                             loaded active plugged   /sys/devices/virtual/net/ovs-system
  sys-subsystem-net-devices-ovs\x2dsystem.device                                           loaded active plugged   /sys/subsystem/net/devices/ovs-system
● dhcp-interface                                                        loaded failed failed    DHCP interface ovs-system
  ovs-vswitchd.service                                                                     loaded active running   Open vSwitch Forwarding Unit
  ovsdb-server.service                                                                     loaded active running   Open vSwitch Database Unit
[heat-admin@overcloud-compute-0 ~]$ ip a |grep DOWN
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
10: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
14: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
15: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
[heat-admin@overcloud-compute-0 ~]$ ping example.com
ping: socket: Operation not permitted
[heat-admin@overcloud-compute-0 ~]$ sudo ping example.com
ping: example.com: Name or service not known
[heat-admin@overcloud-compute-0 ~]$ exit
logout

Comment 1 Omri Hochman 2017-08-17 17:24:41 UTC
from /var/log/neutron/openvswitch-agent.log:

2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server [-] Exception during message handling: AttributeError: 'function' object has no attribute 'accepted_namespaces'
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 207, in dispatch
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server     if not (self._is_namespace(target, namespace) and
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 170, in _is_namespace
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server     return namespace in target.accepted_namespaces
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server AttributeError: 'function' object has no attribute 'accepted_namespaces'
2017-06-14 03:49:33.512 24833 ERROR oslo_messaging.rpc.server

Comment 2 Omri Hochman 2017-08-17 19:23:46 UTC
Adding sosreport from all nodes : 

The reports should be available here: 
http://rhos-release.virt.bos.redhat.com/log/bz148260

Your sosreport has been generated and saved in:
  /var/tmp/sosreport-undercloud.localdomain-20170614083245.tar.xz

Comment 3 Assaf Muller 2017-08-24 17:49:53 UTC
*** Bug 1484853 has been marked as a duplicate of this bug. ***

Comment 6 Omri Hochman 2017-10-26 19:31:53 UTC
unable to reproduce with : 
OpenStack-12.0-RHEL-7 Puddle: 2017-10-24.3

Comment 9 errata-xmlrpc 2017-12-13 21:53:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.