Hide Forgot
Description of problem: Restarting Controllers in ODL Deployment with 3 controllers, shows that opendaylight_api container on controller-0 is up but unhealthy. Karaf.log shows errors: org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | Failed to register the entity arpmonitoring org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.10 cannot be added. DPID for bridge br-int is NULL. org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 189671357144136:br-ex-patch:trunk not ready, holding on org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | Local DPNID is empty for rd 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab, vpnId 103003, vrfEntry VrfEntry{getDestPrefix=10.100.0.3/32, getEncapType=Mplsgre, getGatewayMacAddress=fa:16:3e:1f:2b:fc, getOrigin=l, getRoutePaths=[RoutePaths{getLabel=103005, getNexthopAddress=172.17.2.18, augmentations={}}], augmentations={}} Version-Release number of selected component (if applicable): OSP14 2018-11-21.2 How reproducible: Sometimes. Steps to Reproduce: 1. Install OSP14 + ODL 2. Restart Controllers by rebooting host. 3. Check on each controller: sudo docker ps -a | grep -E '(Down|unhealthy)' Actual results: ###### controller-0: ssh heat-admin@192.168.24.14 ###### 192.168.24.1:8787/rhosp14/openstack-opendaylight:2018-11-20.1 "kolla_start" 2 days ago Up 2 days (unhealthy) opendaylight_api [heat-admin@controller-0 ~]$ date Sun Nov 25 09:26:13 GMT 2018 [heat-admin@controller-0 ~]$ uptime 09:26:20 up 2 days, 13:21, 1 user, load average: 4.60, 4.13, 4.25 Karaf Errors: 2018-11-22T10:50:29,948 | WARN | Aliveness Callback Handler-1 | AlivenessMonitor | 247 - org.opendaylight.genius.alivenessmonitor-impl - 0.4.4.redhat-7 | Monitor profile MonitorProfileCreateInput [_profile=Profile [_failureThreshold=2, _monitorInterval=120000, _monitorWindow=4, _protocolType=Arp, augmentation=[]], augmentation=[]] already present for the given input 2018-11-22T10:50:29,951 | DEBUG | Aliveness Callback Handler-2 | AlivenessMonitor | 247 - org.opendaylight.genius.alivenessmonitor-impl - 0.4.4.redhat-7 | Successfully created monitor Profile MonitorProfileCreateInput [_profile=Profile [_failureThreshold=2, _monitorInterval=120000, _monitorWindow=4, _protocolType=Arp, augmentation=[]], augmentation=[]] 2018-11-22T10:50:29,953 | INFO | Blueprint Extender: 1 | VpnClusterShardBasedOwnershipDriver | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | VpnClusterShardBasedOwnershipDriver start 2018-11-22T10:50:29,956 | INFO | Blueprint Extender: 1 | AsyncClusteredDataTreeChangeListenerBase | 261 - org.opendaylight.genius.mdsalutil-api - 0.4.4.redhat-7 | ThreadFactory created: org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.l3vpn.rev130911.learnt.vpn.vip.to.port.event.data.LearntVpnVipToPortEvent_AsyncClusteredDataTreeChangeListenerBase-DataTreeChangeHandler 2018-11-22T10:50:29,965 | ERROR | Blueprint Extender: 1 | LearntVpnVipToPortEventProcessor | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | Failed to register the entity arpmonitoring 2018-11-22T10:50:29,968 | INFO | Blueprint Extender: 1 | AsyncDataTreeChangeListenerBase | 261 - org.opendaylight.genius.mdsalutil-api - 0.4.4.redhat-7 | ThreadFactory created: org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.elan.rev150602.elan.interfaces.ElanInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler ... 2018-11-22T11:05:57,841 | DEBUG | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | OtherConfigs list does not exist in the OVSDB Node Augmentation. 2018-11-22T11:05:57,844 | ERROR | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.10 cannot be added. DPID for bridge br-int is NULL. 2018-11-22T11:05:57,847 | INFO | org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | SouthboundUtils | 400 - org.opendaylight.ovsdb.utils.southbound-utils - 1.6.4.redhat-7 | addBridge: result: true ... 2018-11-22T11:06:01,087 | DEBUG | ForkJoinPool-2-worker-5 | IdUtils | 252 - org.opendaylight.genius.idmanager-impl - 0.4.4.redhat-7 | Unlocked elan.ids.poolarp.responder.group.id 2018-11-22T11:06:01,103 | INFO | ForkJoinPool-2-worker-0 | AclNodeListener | 334 - org.opendaylight.netvirt.aclservice-impl - 0.6.4.redhat-7 | Adding default ACL flows for dpId=71081822344240 2018-11-22T11:06:01,106 | INFO | org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | SouthboundUtils | 400 - org.opendaylight.ovsdb.utils.southbound-utils - 1.6.4.redhat-7 | addBridge: result: true 2018-11-22T11:06:01,107 | DEBUG | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | OtherConfigs list does not exist in the OVSDB Node Augmentation. ... 2018-11-22T11:06:01,108 | ERROR | opendaylight-cluster-data-notification-dispatcher-80 | OvsdbNodeListener | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.25 cannot be added. DPID for bridge br-int is NULL. ... 2018-11-22T11:06:01,301 | DEBUG | opendaylight-cluster-data-notification-dispatcher-78 | OvsdbNodeListener | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | OtherConfigs list does not exist in the OVSDB Node Augmentation. 2018-11-22T11:06:01,302 | ERROR | opendaylight-cluster-data-notification-dispatcher-78 | OvsdbNodeListener | 258 - org.opendaylight.genius.itm-impl - 0.4.4.redhat-7 | TEP 172.17.2.16 cannot be added. DPID for bridge br-int is NULL. 2018-11-22T11:06:01,303 | DEBUG | org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | ElanOvsdbNodeListener ... 2018-11-22T11:16:30,325 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 189671357144136:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746] 2018-11-22T11:16:30,327 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 189671357144136:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746 2018-11-22T11:16:30,327 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 189671357144136:br-ex-patch:trunk not ready, holding on 2018-11-22T11:16:30,327 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 71081822344240:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746] 2018-11-22T11:16:30,327 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 71081822344240:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746 2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 71081822344240:br-ex-patch:trunk not ready, holding on 2018-11-22T11:16:30,328 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 237564874001843:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746] 2018-11-22T11:16:30,328 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 237564874001843:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746 2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 237564874001843:br-ex-patch:trunk not ready, holding on 2018-11-22T11:16:30,328 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 181347047273174:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746] 2018-11-22T11:16:30,328 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 181347047273174:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746 2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 181347047273174:br-ex-patch:trunk not ready, holding on 2018-11-22T11:16:30,328 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: intfName 75656706099449:br-ex-patch:trunk onto vpnName [49b1c270-a2c1-4caf-b328-20e1951ed746] 2018-11-22T11:16:30,328 | INFO | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | addToUnprocessedVpnInterfaces: Saved unhandled vpn interface 75656706099449:br-ex-patch:trunk in vpn instance 49b1c270-a2c1-4caf-b328-20e1951ed746 2018-11-22T11:16:30,328 | ERROR | org.opendaylight.yang.gen.v1.urn.huawei.params.xml.ns.yang.l3vpn.rev140815.vpn.interfaces.VpnInterface_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceManager | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | add: VpnInstance 49b1c270-a2c1-4caf-b328-20e1951ed746 for vpnInterface 75656706099449:br-ex-patch:trunk not ready, holding on 2018-11-22T11:16:30,359 | INFO | ForkJoinPool-2-worker-5 | VpnInstanceListener | 361 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.4.redhat-7 | VPN-ADD: addVpnInstance: VpnInstanceOpData populated successfully for vpn 49b1c270-a2c1-4caf-b328-20e1951ed746 rd 49b1c270-a2c1-4caf-b328-20e1951ed746 ... 2018-11-22T11:27:20,623 | DEBUG | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.fibmanager.rev150330.vrfentries.VrfEntry_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VrfEntryListener | 346 - org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | Creating Local fib entry with vpnName 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab usedRds [] localNextHopIP 10.100.0.3/32 vpnExtraRoutes [] 2018-11-22T11:27:20,623 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.fibmanager.rev150330.vrfentries.VrfEntry_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VrfEntryListener | 346 - org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | Local DPNID is empty for rd 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab, vpnId 103003, vrfEntry VrfEntry{getDestPrefix=10.100.0.3/32, getEncapType=Mplsgre, getGatewayMacAddress=fa:16:3e:1f:2b:fc, getOrigin=l, getRoutePaths=[RoutePaths{getLabel=103005, getNexthopAddress=172.17.2.18, augmentations={}}], augmentations={}} 2018-11-22T11:27:20,623 | INFO | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.fibmanager.rev150330.vrfentries.VrfEntry_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VrfEntryListener | 346 - org.opendaylight.netvirt.fibmanager-impl - 0.6.4.redhat-7 | ADD: Added Fib Entry rd 3513e8a7-f6e7-4759-99c6-3a3fa8c39eab prefix 10.100.0.3/32 route-paths [RoutePaths{getLabel=103005, getNexthopAddress=172.17.2.18, augmentations={}}] Expected results: After restart, opendaylight_api on all controllers should be up and healthy. Additional info: Adding full Karaf.log
Karaf log from controller-0: http://file.tlv.redhat.com/~nmanos/ctl-0_logs.tgz
After another restart to Controller-0 VM, the container is healthy. However, I noticed this error in neutron/server.log: Sending METHOD (get) URL (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7-4672-9d35-408781631727) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 ERROR networking_odl.journal.periodic_task Failed during periodic task operation journal_recovery.: InvalidRequestError: Object '<OpenDaylightJournal>' is already attached to session '3786' (this is '3795') ###### controller-0: ssh heat-admin@192.168.24.14 ###### *** /var/log/containers/neutron/server.log *** 2018-11-25 16:29:18.740 36 INFO networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Finished full_sync phase of maintenance task. 2018-11-25 16:29:18.740 36 INFO networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Starting journal_recovery phase of periodic task maintenance. 2018-11-25 16:29:18.759 36 DEBUG networking_odl.journal.recovery [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Attempting recovery of journal entry <networking_odl.db.models.OpenDaylightJournal[object at 7fc6b1ccebd0] {seqnum=232, object_type=u'network', object_uuid=u'd7b0355e-59c7-4672-9d35-408781631727', operation=u'update', data={'provider:physical_network': None, 'ipv6_address_scope': None, 'revision_number': 4, 'port_security_enabled': True, 'mtu': 1450, 'id': u'd7b0355e-59c7-4672-9d35-408781631727', 'router:external': False, 'availability_zone_hints': [], 'availability_zones': [u'nova'], 'ipv4_address_scope': None, 'shared': False, 'project_id': u'64e4ba4aea9641e99fcd8915b8e45e65', 'status': u'ACTIVE', 'subnets': [], 'description': u'', 'tags': [], 'updated_at': '2018-11-22T11:32:16Z', 'provider:segmentation_id': None, 'name': u'tempest-TestMinimumBasicScenario-257073058-network', 'admin_state_up': True, 'tenant_id': u'64e4ba4aea9641e99fcd8915b8e45e65', 'created_at': '2018-11-22T11:27:10Z', 'provider:network_type': None, 'vlan_transparent': None}, state='failed', retry_count=5, last_retried=datetime.datetime(2018, 11, 22, 11, 32, 17), version_id=13}>. journal_recovery /usr/lib/python2.7/site-packages/networking_odl/journal/recovery.py:39 2018-11-25 16:29:18.760 36 DEBUG networking_odl.common.client [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Sending METHOD (get) URL (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7-4672-9d35-408781631727) JSON (None) request /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task [req-b494ed11-27ca-4905-a827-a0f9dfc2b3d7 - - - - -] Failed during periodic task operation journal_recovery.: InvalidRequestError: Object '<OpenDaylightJournal at 0x7fc6b1ccebd0>' is already attached to session '3786' (this is '3795') 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task Traceback (most recent call last): 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/networking_odl/journal/periodic_task.py", line 65, in _execute_op 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task operation(context) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 233, in wrapped 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task return method(*args, **kwargs) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 140, in wrapped 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task setattr(e, '_RETRY_EXCEEDED', True) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task self.force_reraise() 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task six.reraise(self.type_, self.value, self.tb) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/neutron_lib/db/api.py", line 136, in wrapped 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task return f(*args, **kwargs) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 154, in wrapper 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task ectxt.value = e.inner_exc 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task self.force_reraise() 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task six.reraise(self.type_, self.value, self.tb) 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 142, in wrapper 2018-11-25 16:29:18.785 36 ERROR networking_odl.journal.periodic_task return f(*args, **kwargs)
It seems that the "DPID for bridge br-int is NULL" error happened at approximately 11:06 on the 22nd, before the host was restarted at approximately 20:00 on the 22nd So probably the problem was there before and the restart didn't help to fix it. It's not really clear what cause it though, please see if you can reproduce this error again so we can better understand the root cause.
(In reply to Noam Manos from comment #2) > After another restart to Controller-0 VM, the container is healthy. > > However, I noticed this error in neutron/server.log: > > Sending METHOD (get) URL > (http://172.17.1.11:8081/controller/nb/v2/neutron/networks/d7b0355e-59c7- > 4672-9d35-408781631727) JSON (None) request > /usr/lib/python2.7/site-packages/networking_odl/common/client.py:89 > > ERROR networking_odl.journal.periodic_task Failed during periodic task > operation journal_recovery.: InvalidRequestError: Object > '<OpenDaylightJournal>' is already attached to session '3786' (this is > '3795') > > For the record, this is a different issue however I don't believe it manifests in any noticeable difference in the system's overall behavior (i.e. even if it happens everything would still work). It's not related to the original error you've encountered.
On OSP14, puddle 2018-11-29.3, the issue was not reproduced. ODL container was healthy on all controller after deployment, and also after host reboot. VMs were up and running also: openstack server list --all +--------------------------------------+-----------------+--------+-------------- | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------+--------+-------------- | 62b409c1-295c-4385-b5ad-a5743a5e25e0 | rhel75_vm1_net2 | ACTIVE | net_ipv64_2=10.0.2.8, 2002::f816:3eff:fef2:6f01, 10.0.0.228 | rhel75 | rhel_flavor | | db1408c6-9051-4ff5-999a-e841aa653513 | rhel75_vm1_net1 | ACTIVE | net_ipv64_1=10.0.1.5, 2001::f816:3eff:fe9d:825, 10.0.0.234 | rhel75 | rhel_flavor | +--------------------------------------+-----------------+--------+--------------