1653057 – Container is unhealthy, error is "DPID for bridge br-int is NULL"

Bug 1653057 - Container is unhealthy, error is "DPID for bridge br-int is NULL"

Summary: Container is unhealthy, error is "DPID for bridge br-int is NULL"

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	opendaylight
Sub Component:
Version:	14.0 (Rocky)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	lpeer
QA Contact:	Noam Manos
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-11-25 10:00 UTC by Noam Manos
Modified:	2018-12-03 16:36 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-12-03 16:36:28 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Description Noam Manos 2018-11-25 10:00:11 UTC

Description of problem:
Restarting Controllers in ODL Deployment with 3 controllers, shows that opendaylight_api container on controller-0 is up but unhealthy.

Karaf.log shows errors:

org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | Failed to register the entity arpmonitoring

org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.10 cannot be added. DPID for bridge br-int is NULL.

org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 189671357144136:br-ex-patch:trunk not ready, holding on 

org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | Local DPNID is empty for rd 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab, vpnId 103003, vrfEntry VrfEntry{getDestPrefix=10.100.0.3/32, getEncapType=Mplsgre, getGatewayMacAddress=fa:16:3e:1f:2b:fc, getOrigin=l, getRoutePaths=[RoutePaths{getLabel=103005, getNexthopAddress=172.17.2.18, augmentations={}}], augmentations={}}


Version-Release number of selected component (if applicable):
OSP14   2018-11-21.2

How reproducible:
Sometimes.

Steps to Reproduce:
1. Install OSP14 + ODL
2. Restart Controllers by rebooting host.
3. Check on each controller: sudo docker ps -a | grep -E '(Down|unhealthy)'

Actual results:

###### controller-0: ssh heat-admin@192.168.24.14 ######

192.168.24.1:8787/rhosp14/openstack-opendaylight:2018-11-20.1  "kolla_start"  2 days ago  Up 2 days (unhealthy)  opendaylight_api

[heat-admin@controller-0 ~]$ date
Sun Nov 25 09:26:13 GMT 2018

[heat-admin@controller-0 ~]$ uptime
 09:26:20 up 2 days, 13:21,  1 user,  load average: 4.60, 4.13, 4.25

Karaf Errors:


2018-11-22T10:50:29,948 | WARN  | Aliveness Callback Handler-1 | AlivenessMonitor                 | 247 - org.opendaylight.genius.alivenessmonitor-impl - 0.4.4.redhat-7 | Monitor profile MonitorProfileCreateInput [_profile=Profile [_failureThreshold=2, _monitorInterval=120000, _monitorWindow=4, _protocolType=Arp, augmentation=[]], augmentation=[]] already present for the given input
2018-11-22T10:50:29,951 | DEBUG | Aliveness Callback Handler-2 | AlivenessMonitor                 | 247 - org.opendaylight.genius.alivenessmonitor-impl - 0.4.4.redhat-7 | Successfully created monitor Profile MonitorProfileCreateInput [_profile=Profile [_failureThreshold=2, _monitorInterval=120000, _monitorWindow=4, _protocolType=Arp, augmentation=[]], augmentation=[]] 
2018-11-22T10:50:29,953 | INFO  | Blueprint Extender: 1 | VpnClusterShardBasedOwnershipDriver | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | VpnClusterShardBasedOwnershipDriver start
2018-11-22T10:50:29,956 | INFO  | Blueprint Extender: 1 | AsyncClusteredDataTreeChangeListenerBase | 261 - org.opendaylight.genius.mdsalutil-api - 0.4.4.redhat-7 | ThreadFactory created: org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.l3vpn.rev130911.learnt.vpn.vip.to.port.event.data.LearntVpnVipToPortEvent_AsyncClusteredDataTreeChangeListenerBase-DataTreeChangeHandler
2018-11-22T10:50:29,965 | ERROR | Blueprint Extender: 1 | LearntVpnVipToPortEventProcessor | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | Failed to register the entity arpmonitoring
2018-11-22T10:50:29,968 | INFO  | Blueprint Extender: 1 | AsyncDataTreeChangeListenerBase  | 261 - org.opendaylight.genius.mdsalutil-api - 0.4.4.redhat-7 | ThreadFactory created: org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.elan.rev150602.elan.interfaces.ElanInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler
...
2018-11-22T11:05:57,841 | DEBUG | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener                | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | OtherConfigs list does not exist in the OVSDB Node Augmentation.
2018-11-22T11:05:57,844 | ERROR | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener                | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.10 cannot be added. DPID for bridge br-int is NULL.
2018-11-22T11:05:57,847 | INFO  | org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | SouthboundUtils                  | 400 - org.opendaylight.ovsdb.utils.southbound-utils - 1.6.4.redhat-7 | addBridge: result: true
...
2018-11-22T11:06:01,087 | DEBUG | ForkJoinPool-2-worker-5 | IdUtils                          | 252 - org.opendaylight.genius.idmanager-impl - 0.4.4.redhat-7 | Unlocked elan.ids.poolarp.responder.group.id
2018-11-22T11:06:01,103 | INFO  | ForkJoinPool-2-worker-0 | AclNodeListener                  | 334 - org.opendaylight.netvirt.aclservice-impl - 0.6.4.redhat-7 | Adding default ACL flows for dpId=71081822344240
2018-11-22T11:06:01,106 | INFO  | org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | SouthboundUtils                  | 400 - org.opendaylight.ovsdb.utils.southbound-utils - 1.6.4.redhat-7 | addBridge: result: true
2018-11-22T11:06:01,107 | DEBUG | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener                | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | OtherConfigs list does not exist in the OVSDB Node Augmentation.
...
2018-11-22T11:06:01,108 | ERROR | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener                | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.25 cannot be added. DPID for bridge br-int is NULL.
...
2018-11-22T11:06:01,301 | DEBUG | opendaylight-cluster-data-notification-dispatcher-78 | OvsdbNodeListener                | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | OtherConfigs list does not exist in the OVSDB Node Augmentation.
2018-11-22T11:06:01,302 | ERROR | opendaylight-cluster-data-notification-dispatcher-78 | OvsdbNodeListener                | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.16 cannot be added. DPID for bridge br-int is NULL.
2018-11-22T11:06:01,303 | DEBUG | org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | ElanOvsdbNodeListener   
...
2018-11-22T11:16:30,325 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 189671357144136:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746]
2018-11-22T11:16:30,327 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 189671357144136:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746
2018-11-22T11:16:30,327 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 189671357144136:br-ex-patch:trunk not ready, holding on 
2018-11-22T11:16:30,327 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 71081822344240:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746]
2018-11-22T11:16:30,327 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 71081822344240:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746
2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 71081822344240:br-ex-patch:trunk not ready, holding on 
2018-11-22T11:16:30,328 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 237564874001843:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746]
2018-11-22T11:16:30,328 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 237564874001843:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746
2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 237564874001843:br-ex-patch:trunk not ready, holding on 
2018-11-22T11:16:30,328 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 181347047273174:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746]
2018-11-22T11:16:30,328 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 181347047273174:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746
2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 181347047273174:br-ex-patch:trunk not ready, holding on 
2018-11-22T11:16:30,328 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 75656706099449:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746]
2018-11-22T11:16:30,328 | INFO  | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 75656706099449:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746
2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 75656706099449:br-ex-patch:trunk not ready, holding on 
2018-11-22T11:16:30,359 | INFO  | ForkJoinPool-2-worker-5 | VpnInstanceListener              | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | VPN-ADD: addVpnInstance: VpnInstanceOpData populated successfully for vpn 49b1c270-a2c1-4caf-b328-20e1951ed746 rd 49b1c270-a2c1-4caf-b328-20e1951ed746
...
2018-11-22T11:27:20,623 | DEBUG | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.fibmanager.rev150330.vrfentries.VrfEntry_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VrfEntryListener                 | 346 - org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | Creating Local fib entry with vpnName 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab usedRds [] localNextHopIP 10.100.0.3/32 vpnExtraRoutes []
2018-11-22T11:27:20,623 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.fibmanager.rev150330.vrfentries.VrfEntry_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VrfEntryListener                 | 346 - org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | Local DPNID is empty for rd 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab, vpnId 103003, vrfEntry VrfEntry{getDestPrefix=10.100.0.3/32, getEncapType=Mplsgre, getGatewayMacAddress=fa:16:3e:1f:2b:fc, getOrigin=l, getRoutePaths=[RoutePaths{getLabel=103005, getNexthopAddress=172.17.2.18, augmentations={}}], augmentations={}}
2018-11-22T11:27:20,623 | INFO  | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.fibmanager.rev150330.vrfentries.VrfEntry_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VrfEntryListener                 | 346 - org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | ADD: Added Fib Entry rd 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab prefix 10.100.0.3/32 route-paths [RoutePaths{getLabel=103005, getNexthopAddress=172.17.2.18, augmentations={}}]




Expected results:
After restart, opendaylight_api on all controllers should be up and healthy.

Additional info:
Adding full Karaf.log

Comment 1 Noam Manos 2018-11-25 10:37:47 UTC

Karaf log from controller-0:
http://file.tlv.redhat.com/~nmanos/ctl-0_logs.tgz

Comment 2 Noam Manos 2018-11-25 16:41:01 UTC

After another restart to Controller-0 VM, the container is healthy.

However, I noticed this error in neutron/server.log:

Sending METHOD (get) URL (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7-4672-9d35-408781631727) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89

ERROR networking_odl.journal.periodic_task Failed during periodic task operation journal_recovery.: InvalidRequestError: Object '<OpenDaylightJournal>' is already attached to session '3786' (this is '3795')



###### controller-0: ssh heat-admin@192.168.24.14 ######


*** /var/log/containers/neutron/server.log ***

2018-11-25 16:29:18.740 36 INFO networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Finished full_sync phase of maintenance task.
2018-11-25 16:29:18.740 36 INFO networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Starting journal_recovery phase of periodic task maintenance.
2018-11-25 16:29:18.759 36 DEBUG networking_odl.journal.recovery [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Attempting recovery of journal entry <networking_odl.db.models.OpenDaylightJournal[object at 7fc6b1ccebd0] {seqnum=232, object_type=u'network', object_uuid=u'd7b0355e-59c7-4672-9d35-408781631727', operation=u'update', data={'provider:physical_network': None, 'ipv6_address_scope': None, 'revision_number': 4, 'port_security_enabled': True, 'mtu': 1450, 'id': u'd7b0355e-59c7-4672-9d35-408781631727', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': [u'nova'], 'ipv4_address_scope': None, 'shared': False, 'project_id': u'64e4ba4aea9641e99fcd8915b8e45e65', 'status': u'ACTIVE', 'subnets': [], 'description': u'', 'tags': [], 'updated_at': '2018-11-22T11:32:16Z', 'provider:segmentation_id': None, 'name': u'tempest-TestMinimumBasicScenario-257073058-network', 'admin_state_up': True, 'tenant_id': u'64e4ba4aea9641e99fcd8915b8e45e65', 'created_at': '2018-11-22T11:27:10Z', 'provider:network_type': None, 'vlan_transparent': None}, state='failed', retry_count=5, last_retried=datetime.datetime(2018, 11, 22, 11, 32, 17), version_id=13}>. journal_recovery /usr/lib/python2.7/site-packages/networking_odl/journal/recovery.py:39
2018-11-25 16:29:18.760 36 DEBUG networking_odl.common.client [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Sending METHOD (get) URL (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7-4672-9d35-408781631727) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Failed during periodic task operation journal_recovery.: InvalidRequestError: Object '<OpenDaylightJournal at 0x7fc6b1ccebd0>' is already attached to session '3786' (this is '3795')
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task Traceback (most recent call last):
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/networking_odl/journal/periodic_task.py", line 65, in _execute_op
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     operation(context)
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     return method(*args, **kwargs)
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     setattr(e, '_RETRY_EXCEEDED', True)
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     self.force_reraise()
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     six.reraise(self.type_, self.value, self.tb)
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     return f(*args, **kwargs)
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     ectxt.value = e.inner_exc
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     self.force_reraise()
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     six.reraise(self.type_, self.value, self.tb)
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper
2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task     return f(*args, **kwargs)

Comment 3 Mike Kolesnik 2018-11-26 12:08:22 UTC

It seems that the "DPID for bridge br-int is NULL" error happened at approximately 11:06 on the 22nd, before the host was restarted at approximately 20:00 on the 22nd

So probably the problem was there before and the restart didn't help to fix it.

It's not really clear what cause it though, please see if you can reproduce this error again so we can better understand the root cause.

Comment 4 Mike Kolesnik 2018-11-26 12:10:34 UTC

(In reply to Noam Manos from comment #2)
> After another restart to Controller-0 VM, the container is healthy.
> 
> However, I noticed this error in neutron/server.log:
> 
> Sending METHOD (get) URL
> (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7-
> 4672-9d35-408781631727) JSON (None) request
> /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89
> 
> ERROR networking_odl.journal.periodic_task Failed during periodic task
> operation journal_recovery.: InvalidRequestError: Object
> '<OpenDaylightJournal>' is already attached to session '3786' (this is
> '3795')
> 
> 

For the record, this is a different issue however I don't believe it manifests in any noticeable difference in the system's overall behavior (i.e. even if it happens everything would still work).
It's not related to the original error you've encountered.

Comment 5 Noam Manos 2018-12-03 16:36:28 UTC

On OSP14, puddle 2018-11-29.3, the issue was not reproduced.

ODL container was healthy on all controller after deployment, and also after host reboot. VMs were up and running also:

openstack server list --all
+--------------------------------------+-----------------+--------+--------------
| ID                                   | Name            | Status | Networks                                                    | Image  | Flavor      |
+--------------------------------------+-----------------+--------+--------------
| 62b409c1-295c-4385-b5ad-a5743a5e25e0 | rhel75_vm1_net2 | ACTIVE | net_ipv64_2=10.0.2.8, 2002::f816:3eff:fef2:6f01, 10.0.0.228 | rhel75 | rhel_flavor |
| db1408c6-9051-4ff5-999a-e841aa653513 | rhel75_vm1_net1 | ACTIVE | net_ipv64_1=10.0.1.5, 2001::f816:3eff:fe9d:825, 10.0.0.234  | rhel75 | rhel_flavor |
+--------------------------------------+-----------------+--------+--------------

Note You need to log in before you can comment on or make changes to this bug.