Description of problem:

LogForwarding configured with forward protocol tries to send a heartbeat to the configured output and will fail if the output does not respond:


Version-Release number of selected component (if applicable):

This causes the pods to error and go into a crash-loop

How reproducible:

Always

Steps to Reproduce:
1. Deploy ClusterLogForwarding with a fluent forward output that does not exist
2.
3.

Actual results:

2020-11-23 20:26:38 +0000 [error]: unexpected error error_class=SocketError error="getaddrinfo: Name or service not known"
  2020-11-23 20:26:38 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:680:in `getaddrinfo'
  2020-11-23 20:26:38 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:680:in `resolve_dns!'
  2020-11-23 20:26:38 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:666:in `resolved_host'
  2020-11-23 20:26:38 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:516:in `validate_host_resolution!'
  2020-11-23 20:26:38 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:237:in `block in configure'
  2020-11-23 20:26:38 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:227:in `each'
  2020-11-23 20:26:38 +0000 [error]: /usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/out_forward.rb:227:in `configure'

Expected results:

The pod should start and continue to process log to other endpoints