Bug 1533653 - VXLAN port type cast error - impossible to change VXLAN port from defaults in neutron-openvswitch-agent
Summary: VXLAN port type cast error - impossible to change VXLAN port from defaults in...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z7
: 10.0 (Newton)
Assignee: Jakub Libosvar
QA Contact: Roee Agiman
URL:
Whiteboard:
Depends On:
Blocks: 1533883 1533884 1533885 1533886 1533887
TreeView+ depends on / blocked
 
Reported: 2018-01-11 21:57 UTC by Andreas Karis
Modified: 2018-02-27 16:42 UTC (History)
5 users (show)

Fixed In Version: openstack-neutron-9.4.1-11.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1533883 1533884 1533885 1533886 1533887 (view as bug list)
Environment:
Last Closed: 2018-02-27 16:42:42 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Launchpad 1742931 None None None 2018-01-12 13:02:50 UTC
OpenStack gerrit 533176 'None' 'MERGED' 'ovs-lib: Pass string as udp port to ovsdb' 2019-12-03 23:01:58 UTC
Red Hat Product Errata RHBA-2018:0357 normal SHIPPED_LIVE openstack-neutron bug fix advisory 2018-02-27 21:38:41 UTC

Description Andreas Karis 2018-01-11 21:57:55 UTC
Description of problem:
Hi,

I don't know when and how this bug was introduced nor how nobody caught it so far (is everybody using VXLANs on the default port?). But we miss an important cast of the VXLAN port to type String when we pass the data into OVS. I tested this on OSP 10 and OSP 11, both of which use OVS 2.6.1. The bug perhaps doesn't exist in 7/8/9, I'm currently testing. I'm setting this as high priority because I have a customer with a custom VXLAN port who upgraded from OSP 8 to 10 and they are upgrading multiple environments with the same setting. It seems that they are not running into any issues right now, but I suppose that this is because the VXLAN tunnels were already configured prior. If they deleted the ports and restarted neutron-openvswitch-agent, I'm sure they'd run into the same issue.

Version-Release number of selected component (if applicable):
OSP 10 and 11 latest RPMs
openstack-neutron-openvswitch-9.4.1-5.el7ost.noarch
openstack-neutron-openvswitch-10.0.3-1.el7ost.noarch

Additional info:

Reproducer:

1) Failure:
~~~
ovs-vsctl del-port vxlan-ac100007
ovs-vsctl del-port vxlan-ac100006
crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 50000
 strace -f -tt -s1000 -e trace=network /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log 2>&1 | tee /tmp/strace1.txt
~~~

2) Good:
~~~
ovs-vsctl del-port vxlan-ac100007
ovs-vsctl del-port vxlan-ac100006
crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789
 strace -f -tt -s1000 -e trace=network /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log 2>&1 | tee /tmp/strace2.txt
~~~

3) Failure:
~~~
ovs-vsctl del-port vxlan-ac100007
ovs-vsctl del-port vxlan-ac100006
crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 9999
 strace -f -tt -s1000 -e trace=network /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log 2>&1 | tee /tmp/strace3.txt
~~~

~~~
 strace -f -tt -s1000 -e trace=network /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log 2>&1 | tee /tmp/strace2.txt
~~~

The strace shows that neutron is not sending the needed data down to OVS:
1)
~~~
[pid 10232] 18:21:55.283512 sendto(5, "{\"id\":8,\"method\":\"transact\",\"params\":[\"Open_vSwitch\",{\"op\":\"insert\",\"row\":{\"name\":\"vxlan-ac100006\",\"type\":\"vxlan\"},\"table\":\"Interface\",\"uuid-name\":\"rowc635862c_106c_446c_baf2_c3960c6124d6\"},{\"mutations\":[[\"ports\",\"insert\",[\"set\",[[\"named-uuid\",\"row5fa82848_a6d4_4329_91a2_a5dfb9e52786\"]]]]],\"op\":\"mutate\",\"table\":\"Bridge\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"54841c9d-09e5-4422-90f9-f86ec50ce39b\"]]]},{\"op\":\"insert\",\"row\":{\"interfaces\":[\"named-uuid\",\"rowc635862c_106c_446c_baf2_c3960c6124d6\"],\"name\":\"vxlan-ac100006\"},\"table\":\"Port\",\"uuid-name\":\"row5fa82848_a6d4_4329_91a2_a5dfb9e52786\"},{\"mutations\":[[\"next_cfg\",\"+=\",1]],\"op\":\"mutate\",\"table\":\"Open_vSwitch\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"b1edb573-bdfa-493d-8960-b68bef1c88d2\"]]]},{\"columns\":[\"next_cfg\"],\"op\":\"select\",\"table\":\"Open_vSwitch\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"b1edb573-bdfa-493d-8960-b68bef1c88d2\"]]]}]}", 865, 0, NULL, 0) = 865
[pid 10232] 18:21:55.283685 recvfrom(5, 0x5cca754, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10232] 18:21:55.284935 recvfrom(17, 0x5dce834, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10269] 18:21:55.285918 recvfrom(3, "{\"id\":null,\"method\":\"update\",\"params\":[null,{\"Interface\":{\"79c3a374-7027-488c-b7b4-6aa40a3cd9ea\":{\"new\":{\"name\":\"vxlan-ac100006\",\"ofport\":[\"set\",[]],\"external_ids\":[\"map\",[]]}}}}]}", 324, 0, NULL, NULL) = 180
[pid 10269] 18:21:55.286037 recvfrom(3, 0x557aa69cdf40, 144, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10232] 18:21:55.286338 recvfrom(5, "{\"id\":null,\"method\":\"update2\",\"params\":[\"4a744222-f6fc-11e7-8ea4-5254006c0999\",{\"Interface\":{\"79c3a374-7027-488c-b7b4-6aa40a3cd9ea\":{\"insert\":{\"name\":\"vxlan-ac100006\",\"type\":\"vxlan\"}}},\"Port\":{\"413602c2-1fdc-4b30-b552-df359e202035\":{\"insert\":{\"name\":\"vxlan-ac100006\",\"interfaces\":[\"uuid\",\"79c3a374-7027-488c-b7b4-6aa40a3cd9ea\"]}}},\"Bridge\":{\"54841c9d-09e5-4422-90f9-f86ec50ce39b\":{\"modify\":{\"ports\":[\"uuid\",\"413602c2-1fdc-4b30-b552-df359e202035\"]}}},\"Open_vSwitch\":{\"b1edb573-bdfa-493d-8960-b68bef1c88d2\":{\"modify\":{\"next_cfg\":227}}}}]}{\"id\":8,\"result\":[{\"uuid\":[\"uuid\",\"79c3a374-7027-488c-b7b4-6aa40a3cd9ea\"]},{\"count\":1},{\"uuid\":[\"uuid\",\"413602c2-1fdc-4b30-b552-df359e202035\"]},{\"count\":1},{\"rows\":[{\"next_cfg\":227}]}],\"error\":null}", 4096, 0, NULL, NULL) = 734
[pid 10269] 18:21:55.287328 recvfrom(3, "{\"id\":null,\"method\":\"update\",\"params\":[null,{\"Interface\":{\"79c3a374-7027-488c-b7b4-6aa40a3cd9ea\":{\"new\":{\"name\":\"vxlan-ac100006\",\"ofport\":-1,\"ex", 144, 0, NULL, NULL) = 144
[pid 10269] 18:21:55.287392 recvfrom(3, "ternal_ids\":[\"map\",[]]},\"old\":{\"ofport\":[\"set\",[]]}}}}]}", 512, 0, NULL, NULL) = 56
[pid 10269] 18:21:55.287505 recvfrom(3, 0x557aa69cde08, 456, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10232] 18:21:55.289877 recvfrom(5, "{\"id\":null,\"method\":\"update2\",\"params\":[\"4a744222-f6fc-11e7-8ea4-5254006c0999\",{\"Interface\":{\"79c3a374-7027-488c-b7b4-6aa40a3cd9ea\":{\"modify\":{\"ofport\":-1,\"error\":\"vxlan-ac100006: could not set configuration (Invalid argument)\"}}},\"Open_vSwitch\":{\"b1edb573-bdfa-493d-8960-b68bef1c88d2\":{\"modify\":{\"cur_cfg\":227}}}}]}", 4096, 0, NULL, NULL) = 316
~~~

vs.
2)
~~~
[pid 10394] 18:24:36.665762 sendto(5, "{\"id\":8,\"method\":\"transact\",\"params\":[\"Open_vSwitch\",{\"op\":\"insert\",\"row\":{\"name\":\"vxlan-ac100006\",\"options\":[\"map\",[[\"df_default\",\"true\"],[\"in_key\",\"flow\"],[\"local_ip\",\"172.16.0.5\"],[\"out_key\",\"flow\"],[\"remote_ip\",\"172.16.0.6\"]]],\"type\":\"vxlan\"},\"table\":\"Interface\",\"uuid-name\":\"rowe4a99e68_affc_4870_aa92_e77073088d29\"},{\"mutations\":[[\"ports\",\"insert\",[\"set\",[[\"named-uuid\",\"rowe179f202_f36b_44a5_9221_1132ff81d261\"]]]]],\"op\":\"mutate\",\"table\":\"Bridge\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"54841c9d-09e5-4422-90f9-f86ec50ce39b\"]]]},{\"op\":\"insert\",\"row\":{\"interfaces\":[\"named-uuid\",\"rowe4a99e68_affc_4870_aa92_e77073088d29\"],\"name\":\"vxlan-ac100006\"},\"table\":\"Port\",\"uuid-name\":\"rowe179f202_f36b_44a5_9221_1132ff81d261\"},{\"mutations\":[[\"next_cfg\",\"+=\",1]],\"op\":\"mutate\",\"table\":\"Open_vSwitch\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"b1edb573-bdfa-493d-8960-b68bef1c88d2\"]]]},{\"columns\":[\"next_cfg\"],\"op\":\"select\",\"table\":\"Open_vSwitch\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"b1edb573-bdfa-493d-8960-b68bef1c88d2\"]]]}]}", 997, 0, NULL, 0) = 997
[pid 10394] 18:24:36.666003 recvfrom(5, 0x5c293c4, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.667035 recvfrom(17, 0x5e3cbf4, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10444] 18:24:36.667212 recvfrom(3, "{\"id\":null,\"method\":\"update\",\"params\":[null,{\"Interface\":{\"9af5627b-17ef-42d3-966c-4fe15b4c81e5\":{\"new\":{\"name\":\"vxlan-ac100006\",\"ofport\":[\"set\",[]],\"external_ids\":[\"map\",[]]}}}}]}", 324, 0, NULL, NULL) = 180
[pid 10444] 18:24:36.667357 recvfrom(3, 0x55a877ed1f40, 144, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.667748 recvfrom(5, "{\"id\":null,\"method\":\"update2\",\"params\":[\"aa1bc5c4-f6fc-11e7-b6ca-5254006c0999\",{\"Interface\":{\"9af5627b-17ef-42d3-966c-4fe15b4c81e5\":{\"insert\":{\"name\":\"vxlan-ac100006\",\"options\":[\"map\",[[\"df_default\",\"true\"],[\"in_key\",\"flow\"],[\"local_ip\",\"172.16.0.5\"],[\"out_key\",\"flow\"],[\"remote_ip\",\"172.16.0.6\"]]],\"type\":\"vxlan\"}}},\"Port\":{\"574e729f-be5c-486b-ab30-fb2a610f9a95\":{\"insert\":{\"name\":\"vxlan-ac100006\",\"interfaces\":[\"uuid\",\"9af5627b-17ef-42d3-966c-4fe15b4c81e5\"]}}},\"Bridge\":{\"54841c9d-09e5-4422-90f9-f86ec50ce39b\":{\"modify\":{\"ports\":[\"uuid\",\"574e729f-be5c-486b-ab30-fb2a610f9a95\"]}}},\"Open_vSwitch\":{\"b1edb573-bdfa-493d-8960-b68bef1c88d2\":{\"modify\":{\"next_cfg\":237}}}}]}{\"id\":8,\"result\":[{\"uuid\":[\"uuid\",\"9af5627b-17ef-42d3-966c-4fe15b4c81e5\"]},{\"count\":1},{\"uuid\":[\"uuid\",\"574e729f-be5c-486b-ab30-fb2a610f9a95\"]},{\"count\":1},{\"rows\":[{\"next_cfg\":237}]}],\"error\":null}", 4096, 0, NULL, NULL) = 866
[pid 10444] 18:24:36.671816 recvfrom(3, "{\"id\":null,\"method\":\"update\",\"params\":[null,{\"Interface\":{\"9af5627b-17ef-42d3-966c-4fe15b4c81e5\":{\"new\":{\"name\":\"vxlan-ac100006\",\"ofport\":4,\"ext", 144, 0, NULL, NULL) = 144
[pid 10444] 18:24:36.671872 recvfrom(3, "ernal_ids\":[\"map\",[]]},\"old\":{\"ofport\":[\"set\",[]]}}}}]}", 512, 0, NULL, NULL) = 55
[pid 10444] 18:24:36.671962 recvfrom(3, 0x55a877ed1e07, 457, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.672171 recvfrom(5, "{\"id\":null,\"method\":\"update2\",\"params\":[\"aa1bc5c4-f6fc-11e7-b6ca-5254006c0999\",{\"Interface\":{\"9af5627b-17ef-42d3-966c-4fe15b4c81e5\":{\"modify\":{\"statistics\":[\"map\",[[\"rx_bytes\",0],[\"rx_packets\",0],[\"tx_bytes\",0],[\"tx_packets\",0]]],\"mac_in_use\":\"b2:60:8a:db:9f:64\",\"link_resets\":0,\"status\":[\"map\",[[\"tunnel_egress_iface\",\"vlan902\"],[\"tunnel_egress_iface_carrier\",\"up\"]]],\"ifindex\":0,\"link_state\":\"up\",\"ofport\":4,\"admin_state\":\"up\"}}},\"Open_vSwitch\":{\"b1edb573-bdfa-493d-8960-b68bef1c88d2\":{\"modify\":{\"cur_cfg\":237}}}}]}", 4096, 0, NULL, NULL) = 517
[pid 10394] 18:24:36.675117 recvfrom(5, 0x657cdb4, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.684522 recvfrom(8, "\4\f\0P\0\0\0\0", 8, 0, NULL, NULL) = 8
[pid 10394] 18:24:36.684685 recvfrom(8, "\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\0\262`\212\333\237d\0\0vxlan-ac100006\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 72, 0, NULL, NULL) = 72
[pid 10394] 18:24:36.685360 recvfrom(8, 0x60cdc84, 8, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.775627 recvfrom(5, 0x606c6d4, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.776795 recvfrom(5, 0x606c6d4, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.778846 sendto(8, "\4\16\0H]S\275E\275\215j\37_B\273\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\377\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\f\200\0\0\4\0\0\0\4\0\0\0\0\0\1\0\10\4\0\0\0", 72, 0, NULL, 0) = 72
[pid 10394] 18:24:36.778979 sendto(8, "\4\24\0\10]S\275F", 8, 0, NULL, 0) = 8
[pid 10394] 18:24:36.779359 recvfrom(8, "\4\25\0\10]S\275F", 8, 0, NULL, NULL) = 8
[pid 10394] 18:24:36.779526 recvfrom(8, 0x5e3cd44, 8, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.780332 recvfrom(5, 0x606c6d4, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.782569 sendto(5, "{\"id\":9,\"method\":\"transact\",\"params\":[\"Open_vSwitch\",{\"op\":\"insert\",\"row\":{\"name\":\"vxlan-ac100007\",\"options\":[\"map\",[[\"df_default\",\"true\"],[\"in_key\",\"flow\"],[\"local_ip\",\"172.16.0.5\"],[\"out_key\",\"flow\"],[\"remote_ip\",\"172.16.0.7\"]]],\"type\":\"vxlan\"},\"table\":\"Interface\",\"uuid-name\":\"row7273ebb8_fb5b_4dec_831c_4892044e349f\"},{\"op\":\"insert\",\"row\":{\"interfaces\":[\"named-uuid\",\"row7273ebb8_fb5b_4dec_831c_4892044e349f\"],\"name\":\"vxlan-ac100007\"},\"table\":\"Port\",\"uuid-name\":\"rowe714b933_f844_494e_acc9_ce24e12704fe\"},{\"mutations\":[[\"ports\",\"insert\",[\"set\",[[\"named-uuid\",\"rowe714b933_f844_494e_acc9_ce24e12704fe\"]]]]],\"op\":\"mutate\",\"table\":\"Bridge\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"54841c9d-09e5-4422-90f9-f86ec50ce39b\"]]]},{\"mutations\":[[\"next_cfg\",\"+=\",1]],\"op\":\"mutate\",\"table\":\"Open_vSwitch\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"b1edb573-bdfa-493d-8960-b68bef1c88d2\"]]]},{\"columns\":[\"next_cfg\"],\"op\":\"select\",\"table\":\"Open_vSwitch\",\"where\":[[\"_uuid\",\"==\",[\"uuid\",\"b1edb573-bdfa-493d-8960-b68bef1c88d2\"]]]}]}", 997, 0, NULL, 0) = 997
[pid 10394] 18:24:36.782757 recvfrom(5, 0x6570064, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 10394] 18:24:36.783279 recvfrom(5, "{\"id\":null,\"method\":\"update2\",\"params\":[\"aa1bc5c4-f6fc-11e7-b6ca-5254006c0999\",{\"Interface\":{\"eabb20d5-cc05-486b-ab67-a465b86dc68f\":{\"insert\":{\"name\":\"vxlan-ac100007\",\"options\":[\"map\",[[\"df_default\",\"true\"],[\"in_key\",\"flow\"],[\"local_ip\",\"172.16.0.5\"],[\"out_key\",\"flow\"],[\"remote_ip\",\"172.16.0.7\"]]],\"type\":\"vxlan\"}}},\"Port\":{\"397a8e35-9b8c-4ed2-a61d-ce8eccd688c7\":{\"insert\":{\"name\":\"vxlan-ac100007\",\"interfaces\":[\"uuid\",\"eabb20d5-cc05-486b-ab67-a465b86dc68f\"]}}},\"Bridge\":{\"54841c9d-09e5-4422-90f9-f86ec50ce39b\":{\"modify\":{\"ports\":[\"uuid\",\"397a8e35-9b8c-4ed2-a61d-ce8eccd688c7\"]}}},\"Open_vSwitch\":{\"b1edb573-bdfa-493d-8960-b68bef1c88d2\":{\"modify\":{\"next_cfg\":238}}}}]}{\"id\":9,\"result\":[{\"uuid\":[\"uuid\",\"eabb20d5-cc05-486b-ab67-a465b86dc68f\"]},{\"uuid\":[\"uuid\",\"397a8e35-9b8c-4ed2-a61d-ce8eccd688c7\"]},{\"count\":1},{\"count\":1},{\"rows\":[{\"next_cfg\":238}]}],\"error\":null}", 4096, 0, NULL, NULL) = 866
~~~

Comment 1 Andreas Karis 2018-01-11 21:58:48 UTC
Error in neutron-openvswitch-agent logs:

/var/log/neutron/openvswitch-agent.log
~~~
[root@overcloud-compute-0 site-packages]# grep 'tunnel port to' /var/log/neutron/openvswitch-agent.log
2018-01-11 18:21:11.684 10070 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8eea42df-f5b3-4011-90e4-2a3bd31f3caa - - - - -] Failed to set-up vxlan tunnel port to 172.16.0.6
2018-01-11 18:21:11.795 10070 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-8eea42df-f5b3-4011-90e4-2a3bd31f3caa - - - - -] Failed to set-up vxlan tunnel port to 172.16.0.7
2018-01-11 18:21:55.393 10232 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-480ebd2c-7b63-44d4-b271-8d980aa19755 - - - - -] Failed to set-up vxlan tunnel port to 172.16.0.6
2018-01-11 18:21:55.509 10232 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-480ebd2c-7b63-44d4-b271-8d980aa19755 - - - - -] Failed to set-up vxlan tunnel port to 172.16.0.7
2018-01-11 18:34:28.145 10782 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-101d6067-a9e3-4803-b24b-1b1b2dae6292 - - - - -] Failed to set-up vxlan tunnel port to 172.16.0.6
2018-01-11 18:34:28.255 10782 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-101d6067-a9e3-4803-b24b-1b1b2dae6292 - - - - -] Failed to set-up vxlan tunnel port to 172.16.0.7
~~~

Code:

/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
~~~
   1444             LOG.error(_LE("Invalid local or remote IP, cannot create tunnel: "
   1445                           "local_ip=%(lip)s remote_ip=%(rip)s"),
   1446                       {'lip': self.local_ip, 'rip': remote_ip})
   1447             return 0
   1448         ofport = br.add_tunnel_port(port_name,
   1449                                     remote_ip,
   1450                                     self.local_ip,
   1451                                     tunnel_type,
   1452                                     self.vxlan_udp_port,
   1453                                     self.dont_fragment,
   1454                                     self.tunnel_csum)
   1455         if ofport == ovs_lib.INVALID_OFPORT:
   1456             LOG.error(_LE("Failed to set-up %(type)s tunnel port to %(ip)s"),
   1457                       {'type': tunnel_type, 'ip': remote_ip})
   1458             return 0
~~~

~~~
[root@overcloud-compute-0 neutron]# grep add_tunnel_port -RI | grep def
tests/functional/agent/test_ovs_lib.py:    def _test_add_tunnel_port(self, attrs):
tests/functional/agent/test_ovs_lib.py:    def test_add_tunnel_port_ipv4(self):
tests/functional/agent/test_ovs_lib.py:    def test_add_tunnel_port_ipv6(self):
tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/test_br_tun.py:    def _mock_add_tunnel_port(self, deferred_br=False):
tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/test_br_tun.py:                ofport = deferred_br.add_tunnel_port(port_name, remote_ip,
tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/test_br_tun.py:    def test_add_tunnel_port(self):
tests/unit/plugins/ml2/drivers/openvswitch/agent/openflow/ovs_ofctl/test_br_tun.py:    def test_deferred_br_add_tunnel_port(self):
tests/unit/agent/common/test_ovs_lib.py:    def test_add_tunnel_port(self):
agent/common/ovs_lib.py:    def add_tunnel_port(self, port_name, remote_ip, local_ip,
~~~

Made the following change:

/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py
~~~
    351     def add_tunnel_port(self, port_name, remote_ip, local_ip,
    352                         tunnel_type=p_const.TYPE_GRE,
    353                         vxlan_udp_port=p_const.VXLAN_UDP_PORT,
    354                         dont_fragment=True,
    355                         tunnel_csum=False):
    356         attrs = [('type', tunnel_type)]
    357         # TODO(twilson) This is an OrderedDict solely to make a test happy
    358         options = collections.OrderedDict()
    359         vxlan_uses_custom_udp_port = (
    360             tunnel_type == p_const.TYPE_VXLAN and
    361             vxlan_udp_port != p_const.VXLAN_UDP_PORT
    362         )
    363         if vxlan_uses_custom_udp_port:
    364             options['dst_port'] = vxlan_udp_port
    365         options['df_default'] = str(dont_fragment).lower()
    366         options['remote_ip'] = remote_ip
    367         options['local_ip'] = local_ip
    368         options['in_key'] = 'flow'
    369         options['out_key'] = 'flow'
    370         if tunnel_csum:
    371             options['csum'] = str(tunnel_csum).lower()
    372         attrs.append(('options', options))
++    373         print(repr(attrs))
    374         return self.add_port(port_name, *attrs)
~~~

~~~
ovs-vsctl del-port vxlan-ac100007
ovs-vsctl del-port vxlan-ac100006
crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 9999
/usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log
~~~

Output:
~~~
Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
[('type', 'vxlan'), ('options', OrderedDict([('dst_port', 9999), ('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')]))]
[('type', 'vxlan'), ('options', OrderedDict([('dst_port', 9999), ('df_default', 'true'), ('remote_ip', u'172.16.0.7'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')]))]
~~~

~~~
ovs-vsctl del-port vxlan-ac100007
ovs-vsctl del-port vxlan-ac100006
crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789
/usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log
~~~

Output: 
~~~
[root@overcloud-compute-0 neutron]# ovs-vsctl del-port vxlan-ac100007
ovs-vsctl: no port named vxlan-ac100007
[root@overcloud-compute-0 neutron]# ovs-vsctl del-port vxlan-ac100006
ovs-vsctl: no port named vxlan-ac100006
[root@overcloud-compute-0 neutron]# crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789
[root@overcloud-compute-0 neutron]# /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log
Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
[('type', 'vxlan'), ('options', OrderedDict([('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')]))]
[('type', 'vxlan'), ('options', OrderedDict([('df_default', 'true'), ('remote_ip', u'172.16.0.7'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')]))]
^C[root@overcloud-compute-0 neutron]# 
~~~

Comment 2 Andreas Karis 2018-01-11 22:00:09 UTC
The problem is that 9999 is an integer and not a string:
agent/common/ovs_lib.py
~~~
    240     def add_port(self, port_name, *interface_attr_tuples):
    241         with self.ovsdb.transaction() as txn:
    242             txn.add(self.ovsdb.add_port(self.br_name, port_name))
    243             if interface_attr_tuples:
    244                 if interface_attr_tuples[0][1] == 'vxlan':
    245                   import pdb; pdb.set_trace()
    246                 txn.add(self.ovsdb.db_set('Interface', port_name,
    247                                           *interface_attr_tuples))
    248         return self.get_port_ofport(port_name)
~~~

~~~
[root@overcloud-compute-0 neutron]# /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log
Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
> /usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py(246)add_port()
-> txn.add(self.ovsdb.db_set('Interface', port_name,
(Pdb) ?

Documented commands (type help <topic>):
========================================
EOF    bt         cont      enable  jump  pp       run      unt   
a      c          continue  exit    l     q        s        until 
alias  cl         d         h       list  quit     step     up    
args   clear      debug     help    n     r        tbreak   w     
b      commands   disable   ignore  next  restart  u        whatis
break  condition  down      j       p     return   unalias  where 

Miscellaneous help topics:
==========================
exec  pdb

Undocumented commands:
======================
retval  rv

(Pdb) s
> /usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py(247)add_port()
-> *interface_attr_tuples))
(Pdb) s
--Call--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(252)db_set()
-> def db_set(self, table, record, *col_values):
(Pdb) a
self = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
table = Interface
record = vxlan-ac100006
col_values = (('type', 'vxlan'), ('options', OrderedDict([('dst_port', 9999), ('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')])))
(Pdb) s
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(253)db_set()
-> return cmd.DbSetCommand(self, table, record, *col_values)
(Pdb) a
self = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
table = Interface
record = vxlan-ac100006
col_values = (('type', 'vxlan'), ('options', OrderedDict([('dst_port', 9999), ('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')])))
(Pdb) s
--Call--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(238)__init__()
-> def __init__(self, api, table, record, *col_values):
(Pdb) a
self = DbSetCommand()
api = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
table = Interface
record = vxlan-ac100006
col_values = (('type', 'vxlan'), ('options', OrderedDict([('dst_port', 9999), ('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')])))
(Pdb) s
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(239)__init__()
-> super(DbSetCommand, self).__init__(api)
(Pdb) a
self = DbSetCommand()
api = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
table = Interface
record = vxlan-ac100006
col_values = (('type', 'vxlan'), ('options', OrderedDict([('dst_port', 9999), ('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')])))
(Pdb) s
--Call--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(28)__init__()
-> def __init__(self, api):
(Pdb) s
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(29)__init__()
-> self.api = api
(Pdb) a
self = DbSetCommand()
api = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
(Pdb) s
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(30)__init__()
-> self.result = None
(Pdb) a
self = DbSetCommand()
api = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
(Pdb) s
--Return--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(30)__init__()->None
-> self.result = None
(Pdb) a
self = DbSetCommand()
api = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
(Pdb) s
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(240)__init__()
-> self.table = table
(Pdb) s
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(241)__init__()
-> self.record = record
(Pdb) n
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(242)__init__()
-> self.col_values = col_values
(Pdb) n
--Return--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/native/commands.py(242)__init__()->None
-> self.col_values = col_values
(Pdb) n
--Return--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(253)db_set()->DbSetCom...ac100006)
-> return cmd.DbSetCommand(self, table, record, *col_values)
(Pdb) a
self = <neutron.agent.ovsdb.impl_idl.OvsdbIdl object at 0x5a2e450>
table = Interface
record = vxlan-ac100006
col_values = (('type', 'vxlan'), ('options', OrderedDict([('dst_port', 9999), ('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')])))
(Pdb) p
*** SyntaxError: SyntaxError('unexpected EOF while parsing', ('<string>', 0, 0, ''))
(Pdb) ?

Documented commands (type help <topic>):
========================================
EOF    bt         cont      enable  jump  pp       run      unt   
a      c          continue  exit    l     q        s        until 
alias  cl         d         h       list  quit     step     up    
args   clear      debug     help    n     r        tbreak   w     
b      commands   disable   ignore  next  restart  u        whatis
break  condition  down      j       p     return   unalias  where 

Miscellaneous help topics:
==========================
exec  pdb

Undocumented commands:
======================
retval  rv

(Pdb) w
  /usr/lib/python2.7/site-packages/eventlet/greenthread.py(214)main()
-> result = function(*args, **kwargs)
  /usr/lib/python2.7/site-packages/ryu/lib/hub.py(54)_launch()
-> return func(*args, **kwargs)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py(43)agent_main_wrapper()
-> ovs_agent.main(bridge_classes)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(2185)main()
-> agent.daemon_loop()
  /usr/lib/python2.7/site-packages/osprofiler/profiler.py(154)wrapper()
-> return f(*args, **kwargs)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(2103)daemon_loop()
-> self.rpc_loop(polling_manager=pm)
  /usr/lib/python2.7/site-packages/osprofiler/profiler.py(154)wrapper()
-> return f(*args, **kwargs)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(2007)rpc_loop()
-> tunnel_sync = self.tunnel_sync()
  /usr/lib/python2.7/site-packages/osprofiler/profiler.py(154)wrapper()
-> return f(*args, **kwargs)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(1734)tunnel_sync()
-> tunnel_type)
  /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(1454)_setup_tunnel_port()
-> self.tunnel_csum)
  /usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py(375)add_tunnel_port()
-> return self.add_port(port_name, *attrs)
  /usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py(247)add_port()
-> *interface_attr_tuples))
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(253)db_set()->DbSetCom...ac100006)
-> return cmd.DbSetCommand(self, table, record, *col_values)
(Pdb) n
--Call--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(55)add()
-> def add(self, command):
(Pdb) n
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(61)add()
-> self.commands.append(command)
(Pdb) n
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(62)add()
-> return command
(Pdb) n
--Return--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py(62)add()->DbSetCom...ac100006)
-> return command
(Pdb) 
--Call--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py(74)__exit__()
-> def __exit__(self, exc_type, exc_val, tb):
(Pdb) 
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py(75)__exit__()
-> if exc_type is None:
(Pdb) 
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py(76)__exit__()
-> self.result = self.commit()
(Pdb) 




--Return--
> /usr/lib/python2.7/site-packages/neutron/agent/ovsdb/api.py(76)__exit__()->None
-> self.result = self.commit()
(Pdb) 
> /usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py(248)add_port()
-> return self.get_port_ofport(port_name)
(Pdb) 
--Return--
> /usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py(248)add_port()->-1
-> return self.get_port_ofport(port_name)
(Pdb) 
--Return--
> /usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py(375)add_tunnel_port()->-1
-> return self.add_port(port_name, *attrs)
(Pdb) 
> /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(1455)_setup_tunnel_port()
-> if ofport == ovs_lib.INVALID_OFPORT:
(Pdb) 
> /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(1456)_setup_tunnel_port()
-> LOG.error(_LE("Failed to set-up %(type)s tunnel port to %(ip)s"),
(Pdb) 
> /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py(1457)_setup_tunnel_port()
-> {'type': tunnel_type, 'ip': remote_ip})
(Pdb) 
~~~

Comment 3 Andreas Karis 2018-01-11 22:01:57 UTC
I changed the code to convert the input into a string, and it works now:

plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
~~~
    197         self.tunnel_count = 0
    198         # akaris
    199         self.vxlan_udp_port = str(agent_conf.vxlan_udp_port)
    200         self.dont_fragment = agent_conf.dont_fragment
    201         self.tunnel_csum = agent_conf.tunnel_csum
    202         self.tun_br = None
    203         self.patch_int_ofport = constants.OFPORT_INVALID
~~~

~~~
[root@overcloud-compute-0 neutron]# /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/openvswitch_agent.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-openvswitch-agent --log-file /var/log/neutron/openvswitch-agent.log
Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
(('type', 'patch'), ('options', {'peer': 'patch-int'}))
(('type', 'patch'), ('options', {'peer': 'patch-tun'}))
(('type', 'vxlan'), ('options', OrderedDict([('dst_port', '9999'), ('df_default', 'true'), ('remote_ip', u'172.16.0.6'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')])))
(('type', 'vxlan'), ('options', OrderedDict([('dst_port', '9999'), ('df_default', 'true'), ('remote_ip', u'172.16.0.7'), ('local_ip', '172.16.0.5'), ('in_key', 'flow'), ('out_key', 'flow')])))
^C[root@overcloud-compute-0 neutron]# 
~~~

~~~
[root@overcloud-compute-0 site-packages]# ovs-vsctl show | grep vxlan
        Port "vxlan-ac100007"
            Interface "vxlan-ac100007"
                type: vxlan
        Port "vxlan-ac100006"
            Interface "vxlan-ac100006"
                type: vxlan
[root@overcloud-compute-0 site-packages]# 
~~~

Comment 4 Andreas Karis 2018-01-11 22:02:14 UTC
I confirmed the behavior on a second OSP 10 test cluster:
~~~
[root@overcloud-compute-0 ~]# crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 9999
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac100006
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac10000e
[root@overcloud-compute-0 ~]# systemctl restart neutron-openvswitch-agent
[root@overcloud-compute-0 ~]# ovs-vsctl show | grep vxlan
        Port "vxlan-ac100006"
            Interface "vxlan-ac100006"
                type: vxlan
                error: "vxlan-ac100006: could not set configuration (Invalid argument)"
        Port "vxlan-ac10000e"
            Interface "vxlan-ac10000e"
                type: vxlan
                error: "vxlan-ac10000e: could not set configuration (Invalid argument)"
[root@overcloud-compute-0 ~]# 
~~~

~~~
[root@overcloud-compute-0 ~]# crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 4789
[root@overcloud-compute-0 ~]# systemctl restart neutron-openvswitch-agent
[root@overcloud-compute-0 ~]# ovs-vsctl show | grep vxlan
        Port "vxlan-ac100006"
            Interface "vxlan-ac100006"
                type: vxlan
        Port "vxlan-ac10000e"
            Interface "vxlan-ac10000e"
                type: vxlan
~~~

RPM version: openstack-neutron-openvswitch-9.4.1-5.el7ost.noarch

Comment 5 Andreas Karis 2018-01-11 22:02:32 UTC
It should be an integer, without quotes, according to:
https://docs.openstack.org/mitaka/config-reference/networking/networking_options_reference.html
https://docs.openstack.org/newton/config-reference/networking/samples/openvswitch_agent.ini.html

It's the same in OSP 11:
~~~
[root@overcloud-compute-0 ~]# ovs-vsctl show | grep vxlan
        Port "vxlan-ac100009"
            Interface "vxlan-ac100009"
                type: vxlan
        Port "vxlan-ac10000e"
            Interface "vxlan-ac10000e"
                type: vxlan
        Port "vxlan-ac10000a"
            Interface "vxlan-ac10000a"
                type: vxlan
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac100009
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac10000e
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac10000a
[root@overcloud-compute-0 ~]# crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port 9999
[root@overcloud-compute-0 ~]# systemctl restart neutron-openvswitch-agent
[root@overcloud-compute-0 ~]# sleep 15 ; ovs-vsctl show | grep vxlan

        Port "vxlan-ac100009"
            Interface "vxlan-ac100009"
                type: vxlan
                error: "vxlan-ac100009: could not set configuration (Invalid argument)"
        Port "vxlan-ac10000e"
            Interface "vxlan-ac10000e"
                type: vxlan
                error: "vxlan-ac10000e: could not set configuration (Invalid argument)"
        Port "vxlan-ac10000a"
            Interface "vxlan-ac10000a"
                type: vxlan
                error: "vxlan-ac10000a: could not set configuration (Invalid argument)"
[root@overcloud-compute-0 ~]# 
[root@overcloud-compute-0 ~]# ovs-vsctl list interface vxlan-ac100009 | grep options
options             : {}
~~~

~~~
[root@overcloud-compute-0 ~]# rpm -qa | grep neutron-open
openstack-neutron-openvswitch-10.0.3-1.el7ost.noarch
~~~

Indeed, passing anything other than an integer causes a type verification error - we expect an integer.
~~~
[root@overcloud-compute-0 ~]# crudini --set /etc/neutron/plugins/ml2/openvswitch_agent.ini agent vxlan_udp_port "'\"9999\"'"
(reverse-i-search)`rest': systemctl ^Cstart neutron-openvswitch-agent
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac100009
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac10000e
[root@overcloud-compute-0 ~]# ovs-vsctl del-port vxlan-ac10000a
(reverse-i-search)`re': ovs-vsctl show | g^Cp vxlan
[root@overcloud-compute-0 ~]# ^C
[root@overcloud-compute-0 ~]# systemctl restart neutron-openvswitch-agent
~~~

~~~
2018-01-11 21:42:29.259 132901 ERROR neutron 
2018-01-11 21:42:30.801 132923 ERROR neutron Traceback (most recent call last):
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2018-01-11 21:42:30.801 132923 ERROR neutron     sys.exit(main())
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
2018-01-11 21:42:30.801 132923 ERROR neutron     agent_main.main()
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 49, in main
2018-01-11 21:42:30.801 132923 ERROR neutron     n_utils.log_opt_values(LOG)
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 207, in log_opt_values
2018-01-11 21:42:30.801 132923 ERROR neutron     cfg.CONF.log_opt_values(log, logging.DEBUG)
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2748, in log_opt_values
2018-01-11 21:42:30.801 132923 ERROR neutron     _sanitize(opt, getattr(group_attr, opt_name)))
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 3198, in __getattr__
2018-01-11 21:42:30.801 132923 ERROR neutron     return self._conf._get(name, self._group)
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2790, in _get
2018-01-11 21:42:30.801 132923 ERROR neutron     value = self._do_get(name, group, namespace)
2018-01-11 21:42:30.801 132923 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_config/cfg.py", line 2833, in _do_get
2018-01-11 21:42:30.801 132923 ERROR neutron     % (opt.name, str(ve)))
2018-01-11 21:42:30.801 132923 ERROR neutron ConfigFileValueError: Value for option vxlan_udp_port is not valid: invalid literal for int() with base 10: '"9999"'
2018-01-11 21:42:30.801 132923 ERROR neutron 
~~~

~~~
[root@overcloud-compute-0 ~]# grep vxlan_udp_port /usr/lib/python2.7/site-packages/neutron/ -R
/usr/lib/python2.7/site-packages/neutron/conf/plugins/ml2/drivers/ovs_conf.py:    cfg.PortOpt('vxlan_udp_port', default=p_const.VXLAN_UDP_PORT,
(...)
~~~

~~~
   1358 class PortOpt(Opt):
   1359 
   1360     """Option for a TCP/IP port number.  Ports can range from 0 to 65535.
   1361 
   1362     Option with ``type`` :class:`oslo_config.types.Integer`
   1363 
   1364     :param name: the option's name
   1365     :param choices: Optional sequence of valid values.
   1366     :param \*\*kwargs: arbitrary keyword arguments passed to :class:`Opt`
   1367     :param min: minimum value the port can take
   1368     :param max: maximum value the port can take
   1369 
   1370     .. versionadded:: 2.6
   1371     .. versionchanged:: 3.2
   1372        Added *choices* parameter.
   1373     .. versionchanged:: 3.4
   1374        Allow port number with 0.
   1375     .. versionchanged:: 3.16
   1376        Added *min* and *max* parameters.
   1377     """
   1378 
   1379     def __init__(self, name, min=None, max=None, choices=None, **kwargs):
   1380         type = types.Port(min=min, max=max, choices=choices,
   1381                           type_name='port value')
   1382         super(PortOpt, self).__init__(name, type=type, **kwargs)
~~~

~~~
class Port(Integer):

    """Port type

    Represents a L4 Port.

    :param type_name: Type name to be used in the sample config file.
    :param choices: Optional sequence of valid values.
    :param min: Optional check that value is greater than or equal to min.
    :param max: Optional check that value is less than or equal to max.

    .. versionadded:: 3.16
    """

    PORT_MIN = 0
    PORT_MAX = 65535

    def __init__(self, min=None, max=None, type_name='port', choices=None):
        min = self.PORT_MIN if min is None else min
        max = self.PORT_MAX if max is None else max
        if min < self.PORT_MIN:
            raise ValueError('Min value cannot be less than %(min)d',
                             {'min': self.PORT_MIN})
        if max > self.PORT_MAX:
            raise ValueError('Max value cannot be more than %(max)d',
                             {'max': self.PORT_MAX})

        super(Port, self).__init__(min=min, max=max, type_name=type_name,
                                   choices=choices)
~~~

The problem is that we have to send a sting to the OVS DB, and we're not doing that.

Comment 7 Andreas Karis 2018-01-11 22:29:43 UTC
By the way, if we didn't catch this, then it means that our unit tests are flawed, and we should fix them as well.

Comment 18 errata-xmlrpc 2018-02-27 16:42:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0357


Note You need to log in before you can comment on or make changes to this bug.