Bug 1653057
Summary: | Container is unhealthy, error is "DPID for bridge br-int is NULL" | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Noam Manos <nmanos> |
Component: | opendaylight | Assignee: | lpeer <lpeer> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | Noam Manos <nmanos> |
Severity: | high | Docs Contact: | |
Priority: | unspecified | ||
Version: | 14.0 (Rocky) | CC: | mkolesni, nmanos, nyechiel |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-12-03 16:36:28 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Noam Manos
2018-11-25 10:00:11 UTC
Karaf log from controller-0: http://file.tlv.redhat.com/~nmanos/ctl-0_logs.tgz After another restart to Controller-0 VM, the container is healthy. However, I noticed this error in neutron/server.log: Sending METHOD (get) URL (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7-4672-9d35-408781631727) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 ERROR networking_odl.journal.periodic_task Failed during periodic task operation journal_recovery.: InvalidRequestError: Object '<OpenDaylightJournal>' is already attached to session '3786' (this is '3795') ###### controller-0: ssh heat-admin.24.14 ###### *** /var/log/containers/neutron/server.log *** 2018-11-25 16:29:18.740 36 INFO networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Finished full_sync phase of maintenance task. 2018-11-25 16:29:18.740 36 INFO networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Starting journal_recovery phase of periodic task maintenance. 2018-11-25 16:29:18.759 36 DEBUG networking_odl.journal.recovery [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Attempting recovery of journal entry <networking_odl.db.models.OpenDaylightJournal[object at 7fc6b1ccebd0] {seqnum=232, object_type=u'network', object_uuid=u'd7b0355e-59c7-4672-9d35-408781631727', operation=u'update', data={'provider:physical_network': None, 'ipv6_address_scope': None, 'revision_number': 4, 'port_security_enabled': True, 'mtu': 1450, 'id': u'd7b0355e-59c7-4672-9d35-408781631727', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': [u'nova'], 'ipv4_address_scope': None, 'shared': False, 'project_id': u'64e4ba4aea9641e99fcd8915b8e45e65', 'status': u'ACTIVE', 'subnets': [], 'description': u'', 'tags': [], 'updated_at': '2018-11-22T11:32:16Z', 'provider:segmentation_id': None, 'name': u'tempest-TestMinimumBasicScenario-257073058-network', 'admin_state_up': True, 'tenant_id': u'64e4ba4aea9641e99fcd8915b8e45e65', 'created_at': '2018-11-22T11:27:10Z', 'provider:network_type': None, 'vlan_transparent': None}, state='failed', retry_count=5, last_retried=datetime.datetime(2018, 11, 22, 11, 32, 17), version_id=13}>. journal_recovery /usr/lib/python2.7/site-packages/networking_odl/journal/recovery.py:39 2018-11-25 16:29:18.760 36 DEBUG networking_odl.common.client [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Sending METHOD (get) URL (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7-4672-9d35-408781631727) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Failed during periodic task operation journal_recovery.: InvalidRequestError: Object '<OpenDaylightJournal at 0x7fc6b1ccebd0>' is already attached to session '3786' (this is '3795') 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task Traceback (most recent call last): 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/networking_odl/journal/periodic_task.py", line 65, in _execute_op 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task operation(context) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 233, in wrapped 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task return method(*args, **kwargs) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task setattr(e, '_RETRY_EXCEEDED', True) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task self.force_reraise() 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task six.reraise(self.type_, self.value, self.tb) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task return f(*args, **kwargs) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task ectxt.value = e.inner_exc 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task self.force_reraise() 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task six.reraise(self.type_, self.value, self.tb) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task return f(*args, **kwargs) It seems that the "DPID for bridge br-int is NULL" error happened at approximately 11:06 on the 22nd, before the host was restarted at approximately 20:00 on the 22nd So probably the problem was there before and the restart didn't help to fix it. It's not really clear what cause it though, please see if you can reproduce this error again so we can better understand the root cause. (In reply to Noam Manos from comment #2) > After another restart to Controller-0 VM, the container is healthy. > > However, I noticed this error in neutron/server.log: > > Sending METHOD (get) URL > (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7- > 4672-9d35-408781631727) JSON (None) request > /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 > > ERROR networking_odl.journal.periodic_task Failed during periodic task > operation journal_recovery.: InvalidRequestError: Object > '<OpenDaylightJournal>' is already attached to session '3786' (this is > '3795') > > For the record, this is a different issue however I don't believe it manifests in any noticeable difference in the system's overall behavior (i.e. even if it happens everything would still work). It's not related to the original error you've encountered. On OSP14, puddle 2018-11-29.3, the issue was not reproduced. ODL container was healthy on all controller after deployment, and also after host reboot. VMs were up and running also: openstack server list --all +--------------------------------------+-----------------+--------+-------------- | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------+--------+-------------- | 62b409c1-295c-4385-b5ad-a5743a5e25e0 | rhel75_vm1_net2 | ACTIVE | net_ipv64_2=10.0.2.8, 2002::f816:3eff:fef2:6f01, 10.0.0.228 | rhel75 | rhel_flavor | | db1408c6-9051-4ff5-999a-e841aa653513 | rhel75_vm1_net1 | ACTIVE | net_ipv64_1=10.0.1.5, 2001::f816:3eff:fe9d:825, 10.0.0.234 | rhel75 | rhel_flavor | +--------------------------------------+-----------------+--------+-------------- |