Bug 1653773

Summary: Thread terminated due to uncaught null pointer exception
Product: Red Hat OpenStack Reporter: Sai Sindhur Malleni <smalleni>
Component: opendaylightAssignee: lpeer <lpeer>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Noam Manos <nmanos>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 13.0 (Queens)CC: jhershbe, mkolesni, nyechiel, smalleni
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-17 10:16:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sai Sindhur Malleni 2018-11-27 15:45:35 UTC
Description of problem:
In a clustered ODL+OSP environment, we are seeing the following in logs
2018-11-27T15:38:46,408 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.OvsdbBridgeAugmentation_AsyncClusteredDataTreeChangeListenerBase-DataTreeChangeHandler-325593 | AsyncClusteredDataTreeChangeListenerBase | 261 - org.opendaylight.genius.mdsalutil-api - 0.4.4.redhat-4 | Thread terminated due to uncaught exception: org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.OvsdbBridgeAugmentation_AsyncClusteredDataTreeChangeListenerBase-DataTreeChangeHandler-325593
java.lang.NullPointerException: null
        at org.opendaylight.genius.interfacemanager.InterfacemgrProvider.addBridgeForNodeIid(InterfacemgrProvider.java:670) [255:org.opendaylight.genius.interfacemanager-impl:0.4.4.redhat-4]
        at org.opendaylight.genius.interfacemanager.listeners.InterfaceTopologyStateListener.update(InterfaceTopologyStateListener.java:135) [255:org.opendaylight.genius.interfacemanager-impl:0.4.4.redhat-4]
        at org.opendaylight.genius.interfacemanager.listeners.InterfaceTopologyStateListener.update(InterfaceTopologyStateListener.java:47) [255:org.opendaylight.genius.interfacemanager-impl:0.4.4.redhat-4]
        at org.opendaylight.genius.datastoreutils.AsyncClusteredDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncClusteredDataTreeChangeListenerBase.java:154) [261:org.opendaylight.genius.mdsalutil-api:0.4.4.redhat-4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
        at java.lang.Thread.run(Thread.java:748) [?:?]


Version-Release number of selected component (if applicable):
OSP13

How reproducible:


Steps to Reproduce:
1. Deploy OSP+ODL clustered setup
2. Run some load tests to stress the system
3. Look at logs

Actual results:
NullPointer exceptions

Expected results:
No exceptions

Additional info:

Comment 1 Josh Hershberg 2018-11-29 08:44:56 UTC
This exception happens here:
    public void addBridgeForNodeIid(InstanceIdentifier<Node> nodeIid, OvsdbBridgeAugmentation bridge) {
        if (nodeIid != null && bridge != null) {
            nodeIidToBridgeMap.put(nodeIid, bridge);
        }
    }

It would seem the only way this can happen is if nodeIidToBridgeMap is null. nodeIidToEtc is initialized here:

    public void start() {
        try {
            createIdPool();
            configEntityCandidate = entityOwnershipService.registerCandidate(
                    new Entity(IfmConstants.INTERFACE_CONFIG_ENTITY, IfmConstants.INTERFACE_CONFIG_ENTITY));
            bindingEntityCandidate = entityOwnershipService.registerCandidate(
                    new Entity(IfmConstants.INTERFACE_SERVICE_BINDING_ENTITY,
                            IfmConstants.INTERFACE_SERVICE_BINDING_ENTITY));
            this.ifaceToTpMap = new ConcurrentHashMap<>();
            this.ifaceToNodeIidMap = new ConcurrentHashMap<>();
            this.nodeIidToBridgeMap = new ConcurrentHashMap<>();
            ifmStatusProvider.reportStatus(ServiceState.OPERATIONAL);
            LOG.info("InterfacemgrProvider Started");
        } catch (CandidateAlreadyRegisteredException e) {
            LOG.error("Failed to register entity {} with EntityOwnershipService", e.getEntity());
            ifmStatusProvider.reportStatus(e);
        } catch (InterruptedException | ExecutionException e) {
            LOG.error("Failed to create idPool for InterfaceMgr", e);
        }
    }

If an exception is thrown in the first three lines of the try block the hash maps will never be initialized. Can you please check if any of the LOG.error messages are present higher up in the log? Also, please check for "InterfacemgrProvider Started"?

NB: In general there is no reason for the hash map initializations to be in the try block. A patch should be raised to move them.

Comment 2 Mike Kolesnik 2018-12-03 11:46:59 UTC
Sai, can you please attach the relevant logs from when this happened?

Comment 3 Mike Kolesnik 2018-12-17 10:16:55 UTC
Closing since there are no logs attached.

Please attach the relevant logs and reopen the bug should you encounter it again.