Thanks, I'll look into reproducing the symptoms. Meanwhile, you can prevent the clients from being unresponsive with a bit of a configuration. In plain English: configure each of the forwarding actions to have an in-memory queue that spools messages to disk if it gets full. This way you won't loose messages (as long as there is free disk space), the queue won't get full and the logger won't jam as you've experienced and rsyslog won't eat all your ram.

Here's a link to some upstream docs and a config snippet:
http://www.rsyslog.com/doc/rsyslog_conf_global.html
http://www.rsyslog.com/doc/queues.html

--- 8< ---
$ActionQueueType LinkedList
$ActionQueueFileName fwd-spool-file
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
$ActionQueueMaxDiskSpace 15g
*.* @@server.com:1234 # some fwd action
--- 8< ---

I haven't managed to reproduce the symptoms yet. I'm using openssl's s_client as the sender and a variation of this script:

for j in {1..99}; do for i in {1..999}; do echo "<13>clihost clitag: asdf_${j}_${i}"; done | openssl s_client -connect localhost:1234 & done

Could you please provide more details on the configuration of the server and the steps you're using to reproduce the fault?

Updating ticket to remove needinfo? flag, I have provided him with access to a server that I ws able to reproduce this on and he was able to get what he needed from there.

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Tomas - do you know if this is still a problem with rsyslog v7 or rsyslog v8?

Two recent fixes for gnutls code in rsyslog to be aware of:

1. https://github.com/rsyslog/rsyslog/pull/801
   Fix for: "Errors with TLS connections in 8.15" #732
2. https://github.com/rsyslog/rsyslog/pull/649
   gnutls: fixed memory leak when gtlsRecordRecv returned a failure.

(In reply to Rich Megginson from comment #6)
> Tomas - do you know if this is still a problem with rsyslog v7 or rsyslog v8?

Can't really say. AFAIR, we never really found the actual cause and there is a giant diff between v5 and v8. We'd have to run the test again but I've never been able to reproduce it in my environment.

Have you run into any TLS related issues? There are still some outstanding bugs.

(In reply to Tomas Heinrich from comment #8)
> (In reply to Rich Megginson from comment #6)
> > Tomas - do you know if this is still a problem with rsyslog v7 or rsyslog v8?
> 
> Can't really say. AFAIR, we never really found the actual cause and there is
> a giant diff between v5 and v8. We'd have to run the test again but I've
> never been able to reproduce it in my environment.
> 
> Have you run into any TLS related issues? There are still some outstanding
> bugs.

I have not done any testing with secure connections or HTTPS.  I need to do that.