I think this is either related to, or a dup of, https://bugzilla.redhat.com/show_bug.cgi?id=1399388

Rich,

Is this resolved with our 3.6.1 changes to address some perf issues and dropped messages possibly?

(In reply to Jeff Cantrill from comment #2)
> Rich,
> 
> Is this resolved with our 3.6.1 changes to address some perf issues and
> dropped messages possibly?

I don't know because it is incredibly difficult to reproduce this problem.

*** This bug has been marked as a duplicate of bug 1489533 ***

(In reply to Jeff Cantrill from comment #4)
> 
> *** This bug has been marked as a duplicate of bug 1489533 ***

Earlier in comment 1 it was asserted this is might be a duplicate of, or at least related to, a different bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=1399388

    Failed to ship logs by "Cannot get new connection from pool." to
    AWS Elasticsearch after start logging-fluentd pod for a while

  Resolution: change fluentd config to use:

    reload_connections false
    reload_on_failure false

This bug had been closed as a duplicate of:

  https://bugzilla.redhat.com/show_bug.cgi?id=1489533

  logging-fluentd needs to periodically reconnect to logging-mux
  or elasticsearch to help balance sessions

But it does not appear to be related to either bug described above.  Instead, this bug is likely related to the proper use of the Ruby API stack to talk to Elasticsearch 2.x.  We might be able to close this bug as resolved by that work to correct the use of the proper ruby gems.

I think if we fix https://bugzilla.redhat.com/show_bug.cgi?id=1489533 by making the reload behavior work for our case, then this bug goes away.

Then, there may still be some underlying bug in elasticsearch-ruby, or in fluent-plugin-elasticsearch, related to connection reload, but we won't hit it because we won't be using that mechanism.

Otherwise, what is our resolution to this bug?  Are we going to submit a PR to fix https://github.com/elastic/elasticsearch-ruby/issues/428?

(In reply to Rich Megginson from comment #6)
> I think if we fix https://bugzilla.redhat.com/show_bug.cgi?id=1489533 by
> making the reload behavior work for our case, then this bug goes away.

Not sure I follow.  This BZ references an issue upstream which references to the use of gems:

  fluentd-0.12.29
  fluent-plugin-elasticsearch-1.9.1
  elasticsearch-api-1.0.18
  elasticsearch-transport-1.0.18

I believe this stack works for our 3.2 and 3.3 product versions, since they are based on 1.5.x of Elasticsearch, but won't work properly for 3.4 and later.

Since we have already corrected this stack to use the proper versions intended for Elasticsearch 2.x with releases 3.4.z and later, I am guessing this bug can be closed as a duplicate whatever BZ drove those original changes.

> Then, there may still be some underlying bug in elasticsearch-ruby, or in
> fluent-plugin-elasticsearch, related to connection reload, but we won't hit
> it because we won't be using that mechanism.

FWIW, the code path where we hit this bug does not appear to be on a reload case.  This looks like a simple bug in the error handling logic, where it expects a "response" object when a "ServerError" is raised, but no object is present.