Bug 1468048 - openvswitch constantly querying DHCP ports [NEEDINFO]
openvswitch constantly querying DHCP ports
Status: NEW
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron (Show other bugs)
10.0 (Newton)
All Unspecified
medium Severity medium
: ---
: ---
Assigned To: Jakub Libosvar
Toni Freger
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-05 18:25 EDT by bigswitch
Modified: 2017-08-17 11:20 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
majopela: needinfo? (rhosp-bugs-internal)


Attachments (Terms of Use)

  None (edit)
Description bigswitch 2017-07-05 18:25:45 EDT
Description of problem:
We see it in scale setup , where Openvswitch is constantly quering DHCP port information for every segment regularly. This triggers constant port_update function to Bigswitch plugin 

We do not see any error in the ports or update , in the stable stage , OVS is constantly quering the DB , this creates more activity. We would like to know is there any tuning paramters to make it less frequency or query or check the status of the ports.

Version-Release number of selected component (if applicable):
RHOSP 10

How reproducible:

RHOSP 10 with scale tenants , networks , DHCP Ports.
Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

2017-07-05 15:11:18.167 295659 DEBUG neutron.plugins.ml2.rpc [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] Device 7b237e0b-3f1f-4027-a1ae-80882933b1c2 details requested by agent ovs-agent-overcloud-controller-0.localdomain with host overcloud-controller-0.localdomain get_device_details /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:71
2017-07-05 15:11:18.241 295659 DEBUG neutron.plugins.ml2.db [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] For port 7b237e0b-3f1f-4027-a1ae-80882933b1c2, host overcloud-controller-0.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 6852190] {port_id=u'7b237e0b-3f1f-4027-a1ae-80882933b1c2', host=u'overcloud-controller-0.localdomain', level=0, driver=u'openvswitch', segment_id=u'60873bfb-6ba0-4df9-8241-6c97aecc2b55'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123
2017-07-05 15:11:18.275 295659 DEBUG neutron.plugins.ml2.db [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] For port 7b237e0b-3f1f-4027-a1ae-80882933b1c2, host overcloud-controller-0.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 6852190] {port_id=u'7b237e0b-3f1f-4027-a1ae-80882933b1c2', host=u'overcloud-controller-0.localdomain', level=0, driver=u'openvswitch', segment_id=u'60873bfb-6ba0-4df9-8241-6c97aecc2b55'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123
2017-07-05 15:11:18.283 295659 DEBUG networking_bigswitch.plugins.bigswitch.plugin [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] Function: update_port_postcommit called wrapper /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/plugin.py:858
Add Comment Collapse

All the ports are DHCP ports.


[stack@undercloud ~]$ neutron port-show 7b237e0b-3f1f-4027-a1ae-80882933b1c2
^[[O+-----------------------+--------------------------------------------------------------------------------+
| Field                 | Value                                                                          |
+-----------------------+--------------------------------------------------------------------------------+
| admin_state_up        | True                                                                           |
| allowed_address_pairs |                                                                                |
| binding:host_id       | overcloud-controller-0.localdomain                                             |
| binding:profile       | {}                                                                             |
| binding:vif_details   | {"port_filter": true, "ovs_hybrid_plug": true}                                 |
| binding:vif_type      | ovs                                                                            |
| binding:vnic_type     | normal                                                                         |
| created_at            | 2017-07-01T12:43:55Z                                                           |
| description           |                                                                                |
| device_id             | dhcp827da361-9c56-50f7-913f-5a01f7bfed2c-025bc979-650c-4921-b212-371a9e5b1a15  |
| device_owner          | network:dhcp                                                                   |
| extra_dhcp_opts       |                                                                                |
| fixed_ips             | {"subnet_id": "5e55bd27-dfb7-4519-a308-4b0170b648f3", "ip_address": "1.1.1.2"} |
| id                    | 7b237e0b-3f1f-4027-a1ae-80882933b1c2                                           |
| mac_address           | fa:16:3e:79:9f:bf                                                              |
| name                  |                                                                                |
| network_id            | 025bc979-650c-4921-b212-371a9e5b1a15                                           |
| port_security_enabled | False                                                                          |
| project_id            | 3f171277202745208c4b90d8bdb5126c                                               |
| revision_number       | 171                                                                            |
| security_groups       |                                                                                |
| status                | ACTIVE                                                                         |
| tenant_id             | 3f171277202745208c4b90d8bdb5126c                                               |
| updated_at            | 2017-07-05T22:11:19Z                                                           |
Comment 1 bigswitch 2017-07-06 12:21:51 EDT
The below port request by OVS keep happening for all the ports in scale setup , we would like to reduce this churn.

2017-07-06 09:15:10.791 119678 DEBUG neutron.plugins.ml2.rpc [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] Device 5b7abded-715c-4dee-9e87-b74170889195 details requested by agent ovs-agent-overcloud-controller-1.localdomain with host overcloud-controller-1.localdomain get_device_details /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:71
2017-07-06 09:15:10.870 119678 DEBUG neutron.plugins.ml2.db [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] For port 5b7abded-715c-4dee-9e87-b74170889195, host overcloud-controller-1.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 73e73d0] {port_id=u'5b7abded-715c-4dee-9e87-b74170889195', host=u'overcloud-controller-1.localdomain', level=0, driver=u'openvswitch', segment_id=u'308a29f8-faf7-4c84-8ee8-11c11bc544d8'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123
2017-07-06 09:15:10.906 119678 DEBUG neutron.plugins.ml2.db [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] For port 5b7abded-715c-4dee-9e87-b74170889195, host overcloud-controller-1.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 73e73d0] {port_id=u'5b7abded-715c-4dee-9e87-b74170889195', host=u'overcloud-controller-1.localdomain', level=0, driver=u'openvswitch', segment_id=u'308a29f8-faf7-4c84-8ee8-11c11bc544d8'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123
2017-07-06 09:15:10.914 119678 DEBUG networking_bigswitch.plugins.bigswitch.plugin [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] Function: update_port_postcommit called wrapper /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/plugin.py:858
Comment 2 Assaf Muller 2017-07-09 10:00:15 EDT
Can we get an sosreport from a node with the DHCP, OVS agents as well as neutron-server? I'm assuming you're using a combined controller/network node, if not please upload sosreports from both the controller and network node. If you can please mark the timestamps where the system is idle but you're seeing these queries I'd appreciate it.
Comment 3 bigswitch 2017-07-10 12:20:48 EDT
I could not attach sos report as file is too large , I have copied google drive 

https://drive.google.com/drive/u/0/folders/0B07f4p28b_XUTlBpbEdlcEVSQkE
Comment 4 bigswitch 2017-07-10 13:10:51 EDT
I also updated OVS to 2.6.1 , Still i see constant update of DHCP port once controller node rebooted
Comment 5 bigswitch 2017-07-10 13:18:39 EDT
I also updated new sos report 

Name: sosreport-overcloud-controller-0.localdomain-20170710094011.tar.xz
Comment 7 Mike Burns 2017-07-28 14:31:56 EDT
Assaf, could you (or someone) take a look at the sosreport
Comment 9 Assaf Muller 2017-08-07 09:22:47 EDT
Assigned to Miguel for triage.
Comment 10 Miguel Angel Ajo 2017-08-09 06:14:11 EDT
Hey, The sosreport on google drive is not accessible by default, I asked for permission from majopela@redhat.com , can you please provide me with access?

I'm looking at the source code trying to understand why getting the binding levels, from a get_device_details rpc triggers the update_port_postcommit on the bigswitch mech driver.
Comment 11 Miguel Angel Ajo 2017-08-09 07:52:28 EDT
Ok, the neutron-openvswitch-agent calls neutron-server via rpc on the 
get_devices_details_list_and_failed_devices rpc method [1] [2]which in turn calls get_device_details, which under some condition [5] updates the port status, triggering the calls to the plugin (port_update).

I don't understand the logic of that part [5], I'm leaving that analysis to my colleagues on the neutron team.

neutron-openvswitch-agent:

[1] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1506
[2] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1560


neutron-server:
[3] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/rpc.py#L146

[4] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/rpc.py#L61

see:
[5] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/rpc.py#L103

Note You need to log in before you can comment on or make changes to this bug.