Bug 2166108

Hello,

We introduced openssl support for relp input and output modules in 8.2102.0-2. So the feature should be available in the version of rsyslog you are using.
What is the exact output when it fails? You can validate the config via rsyslogd -N3 -f /path/to/rsyslog.conf
Can you provide us the minimal configuration that is needed to reproduce the issue?

We are using that version on the EL8 hosts, but we still have a significant number of RHEL7 clients which only have:
rsyslog-relp-8.24.0-57.el7_9.3.x86_64 
rsyslog-8.24.0-57.el7_9.3.x86_64 

That version does not seem to have openssl support
I was hoping that a version with openssl support would be backported for RHEL7/ EL7 releases. 

Notice it RHEL 7 version has no support for tls.tlslib
 
RHEL 7:
strings /usr/lib64/rsyslog/omrelp.so  | grep -i tls
relpCltEnableTLS
relpCltEnableTLSZip
relpCltSetGnuTLSPriString
tls.compression
tls.prioritystring
tls.cacert
tls.mycert
tls.myprivkey
tls.authmode
tls.permittedpeer
omrelp: Could not connect, librelp does NOT does not support TLS (most probably GnuTLS lib is too old)!

RHEL8:
strings /usr/lib64/rsyslog/omrelp.so   |grep -i tls
relpEngineSetTLSLibByName
relpCltEnableTLS
relpCltSetGnuTLSPriString
relpCltSetTlsConfigCmd
relpCltEnableTLSZip
tls.tlslib
tls.compression
tls.prioritystring
tls.cacert
tls.mycert
tls.myprivkey
tls.tlscfgcmd
tls.authmode
tls.permittedpeer
omrelp: Could not connect, librelp does NOT support TLS (most probably GnuTLS lib is too old)!
omrelp: could not activate relp TLS with authentication, librelp does not support it (most probably GnuTLS lib is too old)! Note: anonymous TLS is probably supported.
omrelp: tlslib '%s' not accepted as valid by librelp - using default

(In reply to Gregg Leventhal from comment #4)
> We are using that version on the EL8 hosts, but we still have a significant
> number of RHEL7 clients which only have:
> rsyslog-relp-8.24.0-57.el7_9.3.x86_64 
> rsyslog-8.24.0-57.el7_9.3.x86_64 
> 
> That version does not seem to have openssl support
> I was hoping that a version with openssl support would be backported for
> RHEL7/ EL7 releases. 
> 
> Notice it RHEL 7 version has no support for tls.tlslib
>  
> RHEL 7:
> strings /usr/lib64/rsyslog/omrelp.so  | grep -i tls
> relpCltEnableTLS
> relpCltEnableTLSZip
> relpCltSetGnuTLSPriString
> tls.compression
> tls.prioritystring
> tls.cacert
> tls.mycert
> tls.myprivkey
> tls.authmode
> tls.permittedpeer
> omrelp: Could not connect, librelp does NOT does not support TLS (most
> probably GnuTLS lib is too old)!
> 
> RHEL8:
> strings /usr/lib64/rsyslog/omrelp.so   |grep -i tls
> relpEngineSetTLSLibByName
> relpCltEnableTLS
> relpCltSetGnuTLSPriString
> relpCltSetTlsConfigCmd
> relpCltEnableTLSZip
> tls.tlslib
> tls.compression
> tls.prioritystring
> tls.cacert
> tls.mycert
> tls.myprivkey
> tls.tlscfgcmd
> tls.authmode
> tls.permittedpeer
> omrelp: Could not connect, librelp does NOT support TLS (most probably
> GnuTLS lib is too old)!
> omrelp: could not activate relp TLS with authentication, librelp does not
> support it (most probably GnuTLS lib is too old)! Note: anonymous TLS is
> probably supported.
> omrelp: tlslib '%s' not accepted as valid by librelp - using default

The strings just print the sequence of printable characters and we are checking at startup if tls version is usable or not. This does not mean that it's not usable. Please provide a minimal configuration for rsyslog (/etc/rsyslog.conf + rsyslog.d) that reproduces the bug and the error messages that rsyslog produces.

Sure, let me know if this suffices please:

EL7 build:
```
# rsyslogd -N1
rsyslogd: version 8.24.0-57.el7_9.3, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 13: parameter 'tls.tlslib' not known -- typo in config file? [v8.24.0-57.el7_9.3 try http://www.rsyslog.com/e/2207 ]
igm-qws-u12504e:1# rsyslogd -v
rsyslogd 8.24.0-57.el7_9.3, compiled with:
	PLATFORM:				x86_64-redhat-linux-gnu
	PLATFORM (lsb_release -d):		Description:	CentOS Linux release 7.9.2009 (Core)
	FEATURE_REGEXP:				Yes
	GSSAPI Kerberos 5 support:		Yes
	FEATURE_DEBUG (debug build, slow code):	No
	32bit Atomic operations supported:	Yes
	64bit Atomic operations supported:	Yes
	memory allocator:			system default
	Runtime Instrumentation (slow code):	No
	uuid support:				Yes
	Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.
```

EL8 build:
```
0# rsyslogd -N1
rsyslogd: version 8.2102.0-7.el8_6.1, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
ves-oib-syslog99:0# rsyslogd -v
rsyslogd  8.2102.0-7.el8_6.1 (aka 2021.02) compiled with:
	PLATFORM:				x86_64-redhat-linux-gnu
	PLATFORM (lsb_release -d):		Description:	Rocky Linux release 8.6 (Green Obsidian)
	FEATURE_REGEXP:				Yes
	GSSAPI Kerberos 5 support:		Yes
	FEATURE_DEBUG (debug build, slow code):	No
	32bit Atomic operations supported:	Yes
	64bit Atomic operations supported:	Yes
	memory allocator:			system default
	Runtime Instrumentation (slow code):	No
	uuid support:				Yes
	systemd support:			Yes
	Config file:				/etc/rsyslog.conf
	PID file:				/var/run/rsyslogd.pid
	Number of Bits in RainerScript integers: 64

See https://www.rsyslog.com for more information.

```

Both have this in their respective configs but it only causes an issue on the EL7 (8.24.0-57.el7_9.3) build:
module( load="omrelp" tls.tlslib="openssl" )

I need to edit my previous comment, how do I do that please?

I wanted to follow up and see if I've provided enough information.  Essentially tls.tlslib is not understood in the EL7 release, and I would like to be able to use openssl.

You can not edit a comment.

On RHEL 7 you can not specify the TLS library because only gnutls is available there and we can not introduce new features for RHEL 7. RHEL 7 is currently in Maintenance Support 2 Phase, please check out comment 1 what that actually means. What you can do is remove the tls.tlslib option from your configuration file and use the default gnutls.

Hello,

Is there any update on this? Have you tried my suggestion?
Thanks.

Using the default gnutls is the problem, I believe.

To summarize the issue:

* Running RHEL8 with 8.2102.0-7.el8_6.1 and GNUTLS relp causes the server to crash repeatedly and corrupt log files.
* The RHEL8 8.2102.0-7.el8_6.1 version supports openssl TLS for relp but The RHEL7 clients which run 8.24.0-57.el7_9.3 do not
* From what I can see, you cannot run openssl on the server and gnutls on the clients, it doesnt seem to work, and I tried a few configs to make it work, and no luck.
* This is a blocker from running RHEL8 on the syslog server
* It will be a while before all of the clients are running RHEL8 and can use openssl relp TLS
* This is why I was hoping to get help for porting the openssl TLS relp support to the Rhel7 rsyslog version (8.24.0-57.el7_9.3)

What solution is it that you are suggesting that would help me run RHEL8 on the server without the problems I am seeing?  
Do you have a solution for making RHEL7 rsyslog (8.24.0-57.el7_9.3) run gnutls relp successfully with a RHEL 8 (rsyslog version 8.2102.0-7.el8_6.1) using openssl ?

Thanks for the summarization. Unfortunately, we can not introduce support for openssl in rhel-7.9 due to the Maintenance 2 phase.

What we can do now is to identify why you are experiencing crashes with gnutls and fix it. If it's indeed reproducible then we might get an exception for this bz. Please provide us the minimal rsyslog configuration file(/etc/rsyslog.conf and /etc/rsyslog.d/*.conf) that can be used to reproduce the problem for both client and server side.

It would also be great if you could give us a core dump file, which catches the rsyslog crash [1].

[1] https://access.redhat.com/solutions/56021

There is no core dump because it's not exiting with a fatal signal, something catches the error and exits status 1
Here is the log pattern when a crash/failure happens.  On busier servers, the crashes happen so frequently that the log files that the syslog server is writing to get corrupted, and lose the data afterwards because it doesnt flush the gzip/deflate data, so the compression is info corrupted (so the log continues to grow but isnt readable)

Mar 04 14:15:08 syslog1 rsyslogd[2384698]: rsyslogd: imrelp[20514]: error 'TLS record reception failed [gnutls error -54: Error in the pull function.]', object  'lstn 20514: conn to clt <ip-address>/<hostname.domain>' - input may not work as intended [v8.2102.0-7.el8_6.1 try https://www.rsyslog.com/e/2353 ]
Mar 04 14:15:08 syslog1 rsyslogd[2384698]: imrelp[20514]: error 'TLS record reception failed [gnutls error -54: Error in the pull function.]', object  'lstn 20514: conn to clt <ip-address>/<hostname.domain>' - input may not work as intended [v8.2102.0-7.el8_6.1 try https://www.rsyslog.com/e/2353 ]
Mar 04 14:15:08 syslog1 systemd[1]: rsyslog_server.service: Main process exited, code=exited, status=1/FAILURE
Mar 04 14:15:08 syslog1 systemd[1]: rsyslog_server.service: Failed with result 'exit-code'.

I almost forgot, this log message is often seen preceding the issue:
Mar 04 15:56:39 syslog1 rsyslogd[233070]: main Q:Reg: high activity - starting 1 additional worker thread(s), currently 1 active worker threads. [v8.2102.0-7.el8_6.1 try https://www.rsyslog.com/e/2439 ]