Bug 1671347

ovs-vswitchd logs during the crash (find attached complete ovs-vswitchd.log file):

2019-01-31T11:27:40.741Z|00001|ofproto_dpif_xlate(handler34)|WARN|Invalid Geneve tunnel metadata while processing udp,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:60:c5:f0,dl_dst=fa:16:3e:41:cd:ed,nw_src=192.168.99.4,nw_dst=10.46.22.50,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=42305,tp_dst=53 on bridge br-int
2019-01-31T11:27:41.348Z|00001|ofproto_dpif_xlate(handler40)|WARN|Invalid Geneve tunnel metadata while processing tcp,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:be:76:44,dl_dst=52:54:00:08:3e:d7,nw_src=172.16.40.10,nw_dst=10.46.22.24,nw_tos=0,nw_ecn=0,nw_ttl=63,tp_src=57364,tp_dst=5000,tcp_flags=ack on bridge br-int
2019-01-31T11:27:42.826Z|00001|ofproto_dpif_xlate(handler38)|WARN|Invalid Geneve tunnel metadata while processing tcp,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:18:ea:9c,dl_dst=fa:16:3e:f0:7d:3f,nw_src=172.24.0.14,nw_dst=172.24.0.5,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=9443,tp_dst=37734,tcp_flags=psh|ack on bridge br-int
2019-01-31T11:27:42.826Z|00001|ofproto_dpif_xlate(handler41)|WARN|Invalid Geneve tunnel metadata while processing tcp,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:18:ea:9c,dl_dst=fa:16:3e:f0:7d:3f,nw_src=172.24.0.14,nw_dst=172.24.0.5,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=9443,tp_dst=37734,tcp_flags=psh|ack on bridge br-int

Created attachment 1525373 [details]
ovs-vswitchd logs

It seems it was fixed [1] in ovs master and 2.10 branches, but not in 2.9


[1] https://patchwork.ozlabs.org/patch/979576/

Same behaviour in OSP 14, almost the same core dump (find it attached):

#0  tun_metadata_to_geneve__ (flow=flow@entry=0x7ffc8183c5d8, b=b@entry=0x7ffc81820288, crit_opt=crit_opt@entry=0x7ffc8181ff67) at ../lib/tun-metadata.c:676
#1  0x00005624864a8f6b in tun_metadata_to_geneve_nlattr_flow (b=0x7ffc81820288, flow=0x7ffc8183c590) at ../lib/tun-metadata.c:706
#2  tun_metadata_to_geneve_nlattr (tun=tun@entry=0x7ffc8183c590, flow=flow@entry=0x7ffc8183c590, key=key@entry=0x0, b=b@entry=0x7ffc81820288) at ../lib/tun-metadata.c:810
#3  0x000056248642a4cc in tun_key_to_attr (a=a@entry=0x7ffc81820288, tun_key=tun_key@entry=0x7ffc8183c590, tun_flow_key=tun_flow_key@entry=0x7ffc8183c590, key_buf=key_buf@entry=0x0, tnl_type=<optimized out>, 
    tnl_type@entry=0x0) at ../lib/odp-util.c:2885
#4  0x0000562486435ce2 in odp_key_from_dp_packet (buf=buf@entry=0x7ffc81820288, packet=0x7ffc8183c480) at ../lib/odp-util.c:5908
#5  0x00005624864b8df0 in dpif_netlink_encode_execute (buf=0x7ffc81820288, d_exec=0x7ffc8183c0e8, dp_ifindex=<optimized out>) at ../lib/dpif-netlink.c:1742
#6  dpif_netlink_operate__ (dpif=dpif@entry=0x56248833ca50, ops=ops@entry=0x7ffc8183c0d8, n_ops=n_ops@entry=1) at ../lib/dpif-netlink.c:1828
#7  0x00005624864b95ce in dpif_netlink_operate_chunks (n_ops=1, ops=0x7ffc8183c0d8, dpif=<optimized out>) at ../lib/dpif-netlink.c:2120
#8  dpif_netlink_operate (dpif_=0x56248833ca50, ops=0x7ffc8183c0d8, n_ops=<optimized out>) at ../lib/dpif-netlink.c:2156
#9  0x00005624863fac53 in dpif_operate (dpif=0x56248833ca50, ops=ops@entry=0x7ffc8183c0d8, n_ops=n_ops@entry=1) at ../lib/dpif.c:1349
#10 0x00005624863fb438 in dpif_execute (dpif=<optimized out>, execute=execute@entry=0x7ffc8183c170) at ../lib/dpif.c:1314
#11 0x00005624863a8951 in nxt_resume (ofproto_=0x56248835fea0, pin=0x7ffc8183cbe0) at ../ofproto/ofproto-dpif.c:5085
#12 0x00005624863946f6 in handle_nxt_resume (ofconn=ofconn@entry=0x5624882fe690, oh=oh@entry=0x562488326ce0) at ../ofproto/ofproto.c:3619
#13 0x00005624863a08cb in handle_openflow__ (msg=0x5624883262c0, ofconn=0x5624882fe690) at ../ofproto/ofproto.c:8167
#14 handle_openflow (ofconn=0x5624882fe690, ofp_msg=0x5624883262c0) at ../ofproto/ofproto.c:8288
#15 0x00005624863d2583 in ofconn_run (handle_openflow=0x5624863a0600 <handle_openflow>, ofconn=0x5624882fe690) at ../ofproto/connmgr.c:1446
#16 connmgr_run (mgr=0x562488329bd0, handle_openflow=handle_openflow@entry=0x5624863a0600 <handle_openflow>) at ../ofproto/connmgr.c:365
#17 0x000056248639a4ce in ofproto_run (p=0x56248835fea0) at ../ofproto/ofproto.c:1825
#18 0x0000562486387bac in bridge_run__ () at ../vswitchd/bridge.c:2944
#19 0x000056248638da08 in bridge_run () at ../vswitchd/bridge.c:3002
#20 0x00005624861cce15 in main (argc=12, argv=0x7ffc8183e088) at ../vswitchd/ovs-vswitchd.c:125

Version-Release number of selected component (if applicable):
OSP 14 2019-01-17.2 puddle

Controller (RHEL 7.6):
[root@controller-0 tmp]# rpm -qa | grep openvswitch
openvswitch2.10-2.10.0-28.el7fdp.1.x86_64
rhosp-openvswitch-2.10-0.1.el7ost.noarch
openvswitch2.10-debuginfo-2.10.0-28.el7fdp.1.x86_64
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch

[heat-admin@controller-0 ~]$ sudo docker images | grep ovn
192.168.24.1:8787/rhosp14/openstack-ovn-northd                2019-01-16.1        03bdcefcab6c        8 days ago          752 MB
192.168.24.1:8787/rhosp14/openstack-ovn-northd                pcmklatest          03bdcefcab6c        8 days ago          752 MB
192.168.24.1:8787/rhosp14/openstack-neutron-server-ovn        2019-01-16.1        29207e903690        8 days ago          920 MB
192.168.24.1:8787/rhosp14/openstack-nova-novncproxy           2019-01-16.1        9803d4995c22        2 weeks ago         880 MB
192.168.24.1:8787/rhosp14/openstack-ovn-controller            2019-01-16.1        7cde11f5e83e        2 weeks ago         626 MB


neutron_api container:
python-neutron-fwaas-13.0.1-0.20180910012802.3c6cfbc.el7ost.noarch
openstack-neutron-13.0.2-0.20180929022427.c7f970c.el7ost.noarch
openstack-neutron-common-13.0.2-0.20180929022427.c7f970c.el7ost.noarch
openstack-neutron-ml2-13.0.2-0.20180929022427.c7f970c.el7ost.noarch
puppet-neutron-13.3.1-0.20181013115834.el7ost.noarch
python-neutron-13.0.2-0.20180929022427.c7f970c.el7ost.noarch
openstack-neutron-lbaas-13.0.1-0.20180831185310.e0cca6e.el7ost.noarch
python2-neutronclient-6.9.1-0.20180925041810.7eba94e.el7ost.noarch
python-neutron-lbaas-13.0.1-0.20180831185310.e0cca6e.el7ost.noarch
openstack-neutron-fwaas-13.0.1-0.20180910012802.3c6cfbc.el7ost.noarch
python2-neutron-lib-1.18.0-0.20180816094046.67865c7.el7ost.noarch


ovn_controller container:
puppet-ovn-13.3.1-0.20181013120724.38e2e33.el7ost.noarch
openvswitch2.10-ovn-common-2.10.0-28.el7fdp.1.x86_64
rhosp-openvswitch-ovn-host-2.10-0.1.el7ost.noarch
rhosp-openvswitch-ovn-common-2.10-0.1.el7ost.noarch
openvswitch2.10-ovn-host-2.10.0-28.el7fdp.1.x86_64

Created attachment 1525778 [details]
ovs-vswitchd core dump

Looking into the backtrace, the patch [1] should fix the issue

[1] - https://github.com/openvswitch/ovs/commit/bed941ba0f14854683c241fa9bff3d49dd2efeee

(In reply to Numan Siddique from comment #6)
> Looking into the backtrace, the patch [1] should fix the issue
> 
> [1] -
> https://github.com/openvswitch/ovs/commit/
> bed941ba0f14854683c241fa9bff3d49dd2efeee

Comment #3 already says that :)

@juriarte from shiftstack team confirmed that the crash doesn't happen with this version. Still working on validation
Please Jon, update this after your findings.

Waiting to [1] to be fixed, as it blocks Openshift installation.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1684077

Verified in OSP13 2019-02-25.2 puddle.

[stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 
13  -p 2019-02-25.2

[heat-admin@controller-0 ~]$ rpm -qa | grep openvswitch
openvswitch-2.9.0-95.el7fdp.x86_64
openvswitch-ovn-host-2.9.0-95.el7fdp.x86_64
openvswitch-ovn-common-2.9.0-95.el7fdp.x86_64
openstack-neutron-openvswitch-12.0.5-4.el7ost.noarch
python-openvswitch-2.9.0-95.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
openvswitch-ovn-central-2.9.0-95.el7fdp.x86_64

[heat-admin@controller-0 ~]$ sudo grep -r service_plugins /var/lib/config-data/puppet-generated/neutron/etc/                                                                                                       
/var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf:#service_plugins =
/var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf:service_plugins=qos,ovn-router,trunk

[heat-admin@controller-0 ~]$ sudo docker images | grep ovn
192.168.24.1:8787/rhosp13/openstack-neutron-server-ovn        2019-02-24.1        9626dfbcacff        4 days ago          856 MB
192.168.24.1:8787/rhosp13/openstack-nova-novncproxy           2019-02-24.1        8e3ba946872f        4 days ago          808 MB
192.168.24.1:8787/rhosp13/openstack-ovn-controller            2019-02-24.1        5ff907fbf00f        4 days ago          590 MB
192.168.24.1:8787/rhosp13/openstack-ovn-northd                2019-02-24.1        2efa8840e5bf        4 days ago          716 MB
192.168.24.1:8787/rhosp13/openstack-ovn-northd                pcmklatest          2efa8840e5bf        4 days ago          716 MB

- neutron_api container:
[heat-admin@controller-0 ~]$ sudo docker exec -it neutron_api rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
openvswitch-2.9.0-95.el7fdp.x86_64
python-openvswitch-2.9.0-95.el7fdp.x86_64

- ovn_controller container:
[heat-admin@controller-0 ~]$ sudo docker exec -it ovn_controller rpm -qa | grep openvswitch
openvswitch-selinux-extra-policy-1.0-9.el7fdp.noarch
openvswitch-2.9.0-95.el7fdp.x86_64
python-openvswitch-2.9.0-95.el7fdp.x86_64
openvswitch-ovn-common-2.9.0-95.el7fdp.x86_64
openvswitch-ovn-host-2.9.0-95.el7fdp.x86_64


Verification steps:

1. Deploy OSP 13 with octavia and ovn backend in a hybrid environment
  · Set in Jenkins job:
    ir_tripleo_overcloud_templates: 'ovn-extras,octavia'
    ir_tripleo_overcloud_network_backend:
        - 'geneve'
    ir_tripleo_overcloud_deploy_override_options: |-
        --config-heat ControllerExtraConfig.neutron::agents::l3::extensions="fip_qos" \
        --network-ovn yes \
        --extra-deploy-params='-e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml'

   OSP installation must set - "service_plugins=qos,ovn-router,trunk" in neutron.conf

2. Deploy OCP 3.11 with Kuryr:
Note: A warkaround for [1] has been applied - "yum install openshift-ansible --enablerepo=rhelosp-rhel-7.6-server-opt"

   ansible-playbook --user openshift -i "/usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py" -i inventory "/usr/share/ansible/openshift-ansible/playbooks/openstack/openshift-cluster/provision.yml"

3. Check the provision playbook finishes successfully and the stack openshift.example.com is created in the overcloud.

(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack stack list
+--------------------------------------+-----------------------+-----------------+----------------------+--------------+
| ID                                   | Stack Name            | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+-----------------------+-----------------+----------------------+--------------+
| 2fa65297-882c-4a2f-8b5f-dfe8aeca1711 | openshift.example.com | CREATE_COMPLETE | 2019-03-01T13:39:49Z | None         |
+--------------------------------------+-----------------------+-----------------+----------------------+--------------+

(shiftstack) [cloud-user@ansible-host-0 ~]$ openstack server list
+--------------------------------------+------------------------------------+--------+------------------------------------------------------------------------+---------------------------------------+-----------+
| ID                                   | Name                               | Status | Networks                                                               | Image                                 | Flavor    |
+--------------------------------------+------------------------------------+--------+------------------------------------------------------------------------+---------------------------------------+-----------+
| 080e3c8e-b644-489e-bc0c-97d1fc13b8b9 | infra-node-0.openshift.example.com | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.9, 10.46.22.113 | rhel-guest-image-7.6-210.x86_64.qcow2 | m1.node   |
| eb91ed29-a48b-49f8-8ce6-b32f4f8e5531 | master-0.openshift.example.com     | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.8, 10.46.22.103 | rhel-guest-image-7.6-210.x86_64.qcow2 | m1.master |
| 8092cf88-3623-47bc-98e5-2b01df2c3e8b | app-node-0.openshift.example.com   | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.23, 10.46.22.96 | rhel-guest-image-7.6-210.x86_64.qcow2 | m1.node   |
| 96db78db-1c70-4d47-886b-d0f3461a5f22 | app-node-1.openshift.example.com   | ACTIVE | openshift-ansible-openshift.example.com-net=192.168.99.4, 10.46.22.91  | rhel-guest-image-7.6-210.x86_64.qcow2 | m1.node   |
| c89b10ce-148d-47ba-889e-5ba3c6228737 | ansible_host-0                     | ACTIVE | private_openshift=172.16.40.18, 10.46.22.105                           | rhel-guest-image-7.6-210.x86_64.qcow2 | m1.small  |
| c5b6b356-42c0-456f-8be1-28e1090aedd2 | openshift_dns-0                    | ACTIVE | openshift_dns=192.168.23.13, 10.46.22.95                               | rhel-guest-image-7.6-210.x86_64.qcow2 | m1.small  |
+--------------------------------------+------------------------------------+--------+------------------------------------------------------------------------+---------------------------------------+-----------+


4. Check ovs-vswitchd service has not been restarted in the controller:

[heat-admin@controller-0 ~]$ sudo grep vswitchd /var/log/messages 
Mar  1 06:27:18 controller-0 ovs-ctl: Starting ovs-vswitchd [  OK  ]

[heat-admin@controller-0 ~]$ sudo grep -r "ovs-ctl: " /var/log/messages 
Mar  1 06:27:18 controller-0 ovs-ctl: /etc/openvswitch/conf.db does not exist ... (warning).
Mar  1 06:27:18 controller-0 ovs-ctl: Creating empty database /etc/openvswitch/conf.db [  OK  ]
Mar  1 06:27:18 controller-0 ovs-ctl: Starting ovsdb-server [  OK  ]
Mar  1 06:27:18 controller-0 ovs-ctl: Configuring Open vSwitch system IDs [  OK  ]
Mar  1 06:27:18 controller-0 ovs-ctl: Inserting openvswitch module [  OK  ]
Mar  1 06:27:18 controller-0 ovs-ctl: Starting ovs-vswitchd [  OK  ]
Mar  1 06:27:18 controller-0 ovs-ctl: Enabling remote OVSDB managers [  OK  ]

[heat-admin@controller-0 ~]$ sudo systemctl status ovs-vswitchd
● ovs-vswitchd.service - Open vSwitch Forwarding Unit
   Loaded: loaded (/usr/lib/systemd/system/ovs-vswitchd.service; static; vendor preset: disabled)
   Active: active (running) since vie 2019-03-01 11:27:18 UTC; 3h 50min ago
 Main PID: 4672 (ovs-vswitchd)
    Tasks: 18
   Memory: 147.8M
   CGroup: /system.slice/ovs-vswitchd.service
           └─4672 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --user openvswitch:hugetlbfs --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --p...

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1684077

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0552