Created attachment 1274607 [details]
Hung dockerd core file generated by gcore

Description of problem:

On a Docker environment with Cluster Store, Cluster Advertise, and some Overlay networks, if an EventMemberJoin event is received while processing a Node found in KV store, dockerd process hangs indefinitely.


Version-Release number of selected component (if applicable):

Tested with docker-1.12.6-11.el7.x86_64


How reproducible:

Sometimes, depends on Nodes coordination.


Steps to Reproduce:
1. Prepare a 3 node Docker environment with Cluster Store and Cluster Advertise.
2. Create some Overlay Networks.
3. Create a few containers with an automatic restart policy.
4. Restart dockerd a few times.


Actual results:

dockerd becomes unresponsive (service socket is created, but doesn't answer to connection requests).


Expected results:

dockerd shouldn't hang.


Additional info:

The root cause is that peerDbNetworkWalk iterates through the map calling the passed function while holding pMap.Lock. One of the functions passed to peerDbNetworkWalk is pushLocalEndpointEvent, which sends an event to the serf channel.

Meanwhile, if an EventMemberJoin event has been received by ov_serf event channel, it will eventually call peerAdd, which tries to hold the same pMap.Lock.

So we have startSerfLoop busy waiting for pMap.Lock to be released, and peerDbNetworkWalk holding the lock while waiting in pushLocalEndpointEvent, causing the deadlock.

This has been addressed upstream by this commit: https://github.com/docker/libnetwork/commit/760abd041b3100007987423140db1f169c84dab9