Bug 1786135 - Port creation fails sometimes after OVN minor update
Summary: Port creation fails sometimes after OVN minor update
Keywords:
Status: CLOSED DUPLICATE of bug 1766575
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-networking-ovn
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Assaf Muller
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-23 14:49 UTC by Eduardo Olivares
Modified: 2020-01-08 09:10 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-08 09:10:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Eduardo Olivares 2019-12-23 14:49:37 UTC
Description of problem:
After minor update (executed by an OVN Jenkins job) many tempest tests failed when requesting the creating of a network port with server response 500.
Scenario was reproduced and showed that, after OVN minor update was completed, creating ports failed sometimes (and a few times it was successful). According to the logs, when executing command "openstack port create --network xxx testportyyy", it succeeded only when the request was managed by the controller who was acting as cluster's DC (according to "pcs status" command).
In other cases, when any other of the two controllers received the request, it failed showing the following error logs:
2019-12-23 14:33:21.442 29 DEBUG neutron.api.v2.base [req-9ff749fa-150f-4a2d-b131-a9ab732fab13 a932cd6699cf485d826c708328cda5db 35cb54f52bf54dd9a3b136dc9bd895b6 - default default] Request body: {u'port': {u'network_id': u'de7722f1-be91-4
28f-ae47-c4727cbb7081', u'name': u'testport02', u'admin_state_up': True}} prepare_request_body /usr/lib/python2.7/site-packages/neutron/api/v2/base.py:690
...
2019-12-23 14:33:22.971 29 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=0): AddLSwitchPortCommand(lswitch=neutron-de7722f1-be91-428f-ae47-c4727cbb7081, may_exist=True, lport=617ca044-9a90-456b-9f98-ea7911a00789, columns={'parent_name': [], 'addresses': [], 'enabled': True, 'tag': [], 'port_security': [u'fa:16:3e:6a:96:27 192.168.0.11'], 'dhcpv6_options': [], 'external_ids': {'neutron:cidrs': '192.168.0.11/24', 'neutron:device_id': '', 'neutron:revision_number': '6', 'neutron:port_name': u'testport02', 'neutron:network_name': u'neutron-de7722f1-be91-428f-ae47-c4727cbb7081', 'neutron:project_id': u'35cb54f52bf54dd9a3b136dc9bd895b6', 'neutron:device_owner': '', 'neutron:security_group_ids': u'5421ef09-35fd-43c7-946c-d675880218b6'}, 'type': '', 'options': {'requested-chassis': ''}, 'dhcpv4_options': [UUID('c571d486-f726-47fa-ae15-c47960aa4ba2')]}) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-12-23 14:33:22.973 29 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=1): PgAddPortCommand(if_exists=False, lsp=[<networking_ovn.ovsdb.commands.AddLSwitchPortCommand object at 0x7fb97de5d650>], port_group=neutron_pg_drop) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-12-23 14:33:22.974 29 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn command(idx=2): PgAddPortCommand(if_exists=False, lsp=[<networking_ovn.ovsdb.commands.AddLSwitchPortCommand object at 0x7fb97de5d650>], port_group=pg_5421ef09_35fd_43c7_946c_d675880218b6) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-12-23 14:33:22.976 29 ERROR ovsdbapp.backend.ovs_idl.transaction [req-9ff749fa-150f-4a2d-b131-a9ab732fab13 a932cd6699cf485d826c708328cda5db 35cb54f52bf54dd9a3b136dc9bd895b6 - default default] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python2.7/site-packages/networking_ovn/ovsdb/commands.py", line 1242, in run_idl
    raise RuntimeError(msg)
RuntimeError: Port group pg_5421ef09_35fd_43c7_946c_d675880218b6 does not exist


Job (artifacts include logs): https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/network/view/networking-ovn/job/DFG-network-networking-ovn-update-13_director-rhel-virthost-3cont_2comp_2net-ipv4-geneve-composable/73/



Version-Release number of selected component (if applicable):
Red Hat OpenStack Platform release 13.0.10 (Queens)
core_puddle=2019-11-04.1 - before update
core_puddle=2019-12-13.1 - after update


How reproducible:
100% (as long as CI job reaches 2nd tempest phase, right after update)

Steps to Reproduce:
1. Run CI job
2.
3.

Actual results:
162 tempest tests failed (maybe not all of them due to this)

Expected results:


Additional info:

Comment 1 Eduardo Olivares 2019-12-23 15:21:56 UTC
Workaround:
after restarting neutron_api container from controller-2 (cluster's DC and the only controller that was managing successfully port creation requests), all controllers are processing correctly port creation requests.

Might this BZ be a duplicated of https://bugzilla.redhat.com/show_bug.cgi?id=1766575?
If so, a fix for that bug (https://code.engineering.redhat.com/gerrit/#/c/184199/) was tested here and is not working.

Comment 2 Eduardo Olivares 2020-01-08 09:10:02 UTC
As mentioned in previous comment, this bug is a duplicate of 1766575.
Something wrong from previous comment: The puddle I was testing did not include the mentioned fix.

*** This bug has been marked as a duplicate of bug 1766575 ***


Note You need to log in before you can comment on or make changes to this bug.