It turns out that this issue was caused by too many entries in the event_streams table. This was causing some queries to take ~12 hours to run. The backend connection to the database would remain open throughout the runtime of the query even though the worker which issued the query was killed.

When the workers issuing these queries were killed they would be restarted and then attempt to run the same query again. These connections associated with long running queries were accumulating until we hit the limit of 1000.

To solve this issue we truncated the event_streams table which allowed the workers to run and the number of connections stabilized.

As a fix, we should allow blacklisting events for openshift (which accounted for the majority of the 28 million rows in the table). But I'm not sure if we should attach that fix to this bug as the description here is not directly related to the fix.

*** This bug has been marked as a duplicate of bug 1653871 ***