Description of problem: We see it in scale setup , where Openvswitch is constantly quering DHCP port information for every segment regularly. This triggers constant port_update function to Bigswitch plugin We do not see any error in the ports or update , in the stable stage , OVS is constantly quering the DB , this creates more activity. We would like to know is there any tuning paramters to make it less frequency or query or check the status of the ports. Version-Release number of selected component (if applicable): RHOSP 10 How reproducible: RHOSP 10 with scale tenants , networks , DHCP Ports. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: 2017-07-05 15:11:18.167 295659 DEBUG neutron.plugins.ml2.rpc [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] Device 7b237e0b-3f1f-4027-a1ae-80882933b1c2 details requested by agent ovs-agent-overcloud-controller-0.localdomain with host overcloud-controller-0.localdomain get_device_details /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:71 2017-07-05 15:11:18.241 295659 DEBUG neutron.plugins.ml2.db [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] For port 7b237e0b-3f1f-4027-a1ae-80882933b1c2, host overcloud-controller-0.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 6852190] {port_id=u'7b237e0b-3f1f-4027-a1ae-80882933b1c2', host=u'overcloud-controller-0.localdomain', level=0, driver=u'openvswitch', segment_id=u'60873bfb-6ba0-4df9-8241-6c97aecc2b55'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123 2017-07-05 15:11:18.275 295659 DEBUG neutron.plugins.ml2.db [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] For port 7b237e0b-3f1f-4027-a1ae-80882933b1c2, host overcloud-controller-0.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 6852190] {port_id=u'7b237e0b-3f1f-4027-a1ae-80882933b1c2', host=u'overcloud-controller-0.localdomain', level=0, driver=u'openvswitch', segment_id=u'60873bfb-6ba0-4df9-8241-6c97aecc2b55'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123 2017-07-05 15:11:18.283 295659 DEBUG networking_bigswitch.plugins.bigswitch.plugin [req-e0e2b467-0e14-4764-89b6-24bf551eaf28 - - - - -] Function: update_port_postcommit called wrapper /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/plugin.py:858 Add Comment Collapse All the ports are DHCP ports. [stack@undercloud ~]$ neutron port-show 7b237e0b-3f1f-4027-a1ae-80882933b1c2 ^[[O+-----------------------+--------------------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | overcloud-controller-0.localdomain | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2017-07-01T12:43:55Z | | description | | | device_id | dhcp827da361-9c56-50f7-913f-5a01f7bfed2c-025bc979-650c-4921-b212-371a9e5b1a15 | | device_owner | network:dhcp | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "5e55bd27-dfb7-4519-a308-4b0170b648f3", "ip_address": "1.1.1.2"} | | id | 7b237e0b-3f1f-4027-a1ae-80882933b1c2 | | mac_address | fa:16:3e:79:9f:bf | | name | | | network_id | 025bc979-650c-4921-b212-371a9e5b1a15 | | port_security_enabled | False | | project_id | 3f171277202745208c4b90d8bdb5126c | | revision_number | 171 | | security_groups | | | status | ACTIVE | | tenant_id | 3f171277202745208c4b90d8bdb5126c | | updated_at | 2017-07-05T22:11:19Z |
The below port request by OVS keep happening for all the ports in scale setup , we would like to reduce this churn. 2017-07-06 09:15:10.791 119678 DEBUG neutron.plugins.ml2.rpc [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] Device 5b7abded-715c-4dee-9e87-b74170889195 details requested by agent ovs-agent-overcloud-controller-1.localdomain with host overcloud-controller-1.localdomain get_device_details /usr/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:71 2017-07-06 09:15:10.870 119678 DEBUG neutron.plugins.ml2.db [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] For port 5b7abded-715c-4dee-9e87-b74170889195, host overcloud-controller-1.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 73e73d0] {port_id=u'5b7abded-715c-4dee-9e87-b74170889195', host=u'overcloud-controller-1.localdomain', level=0, driver=u'openvswitch', segment_id=u'308a29f8-faf7-4c84-8ee8-11c11bc544d8'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123 2017-07-06 09:15:10.906 119678 DEBUG neutron.plugins.ml2.db [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] For port 5b7abded-715c-4dee-9e87-b74170889195, host overcloud-controller-1.localdomain, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 73e73d0] {port_id=u'5b7abded-715c-4dee-9e87-b74170889195', host=u'overcloud-controller-1.localdomain', level=0, driver=u'openvswitch', segment_id=u'308a29f8-faf7-4c84-8ee8-11c11bc544d8'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123 2017-07-06 09:15:10.914 119678 DEBUG networking_bigswitch.plugins.bigswitch.plugin [req-31829b15-4f12-4882-a938-13a0a1cfc3b1 - - - - -] Function: update_port_postcommit called wrapper /usr/lib/python2.7/site-packages/networking_bigswitch/plugins/bigswitch/plugin.py:858
Can we get an sosreport from a node with the DHCP, OVS agents as well as neutron-server? I'm assuming you're using a combined controller/network node, if not please upload sosreports from both the controller and network node. If you can please mark the timestamps where the system is idle but you're seeing these queries I'd appreciate it.
I could not attach sos report as file is too large , I have copied google drive https://drive.google.com/drive/u/0/folders/0B07f4p28b_XUTlBpbEdlcEVSQkE
I also updated OVS to 2.6.1 , Still i see constant update of DHCP port once controller node rebooted
I also updated new sos report Name: sosreport-overcloud-controller-0.localdomain-20170710094011.tar.xz
https://drive.google.com/drive/u/0/folders/0B07f4p28b_XUTlBpbEdlcEVSQkE
Assaf, could you (or someone) take a look at the sosreport
Assigned to Miguel for triage.
Hey, The sosreport on google drive is not accessible by default, I asked for permission from majopela , can you please provide me with access? I'm looking at the source code trying to understand why getting the binding levels, from a get_device_details rpc triggers the update_port_postcommit on the bigswitch mech driver.
Ok, the neutron-openvswitch-agent calls neutron-server via rpc on the get_devices_details_list_and_failed_devices rpc method [1] [2]which in turn calls get_device_details, which under some condition [5] updates the port status, triggering the calls to the plugin (port_update). I don't understand the logic of that part [5], I'm leaving that analysis to my colleagues on the neutron team. neutron-openvswitch-agent: [1] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1506 [2] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1560 neutron-server: [3] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/rpc.py#L146 [4] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/rpc.py#L61 see: [5] https://github.com/openstack/neutron/blob/stable/newton/neutron/plugins/ml2/rpc.py#L103
(In reply to bigswitch from comment #5) > I also updated new sos report > > Name: sosreport-overcloud-controller-0.localdomain-20170710094011.tar.xz Can you please set debug = True in /etc/neutron/dhcp_agent.ini and /etc/neutron/plugins/ml2/openvswitch_agent.ini and then restart the dhcp and ovs agents? It would be also very helpful to get an ID of problematic port that DHCP agent keeps asking for details for, I didn't find the one from description in the logs. One thing I noticed was that at sosreport-overcloud-controller-0.localdomain-20170710094011/var/log/neutron/openvswitch-agent.log , agent attempts to sync but can't talk to server over RPC.
I asked for additional information about a month ago and they were not provided. I'm closing this bug for now. If you hit the issue again, feel free to re-open.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days