Bug 2140043

Summary: [jitter-9.2] rngd fails during start with *** stack smashing detected ***
Product: Red Hat Enterprise Linux 9 Reporter: Filip Dvorak <fdvorak>
Component: jitterentropyAssignee: Vladis Dronov <vdronov>
Status: CLOSED ERRATA QA Contact: Vilém Maršík <vmarsik>
Severity: high Docs Contact:
Priority: high    
Version: 9.2CC: chwen, core-kernel-mgr, dzheng, hkario, omoris, smitterl, vmarsik
Target Milestone: rcKeywords: Regression, TestBlocker, Triaged
Target Release: 9.2Flags: pm-rhel: mirror+
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: jitterentropy-3.4.1-2.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2141379 (view as bug list) Environment:
Last Closed: 2023-05-09 08:15:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2141379, 2144442    

Description Filip Dvorak 2022-11-04 09:40:33 UTC
Description of problem:
Rngd fails to start with an error "*** stack smashing detected ***: terminated" on s390x machine.

Version-Release number of selected component (if applicable):
RHEL-9.2.0-20221103.0
rng-tools-6.15-2.el9.s390x

How reproducible:
always

Steps to Reproduce:
1. dnf install rng-tools
2. systemctl start rngd

Actual results:
# grep rngd /var/log/messages
Nov  4 05:27:54  rngd[48761]: Disabling 7: PKCS11 Entropy generator (pkcs11)
Nov  4 05:27:54  rngd[48761]: Disabling 5: NIST Network Entropy Beacon (nist)
Nov  4 05:27:54  rngd[48761]: Disabling 9: Qrypt quantum entropy beacon (qrypt)
Nov  4 05:27:54  rngd[48761]: Initializing available sources
Nov  4 05:27:54  rngd[48761]: [hwrng ]: Initialized
Nov  4 05:27:54  rngd[48761]: [jitter]: JITTER timeout set to 5 sec
Nov  4 05:27:54  rngd[48761]: *** stack smashing detected ***: terminated
Nov  4 05:27:55  systemd-coredump[48763]: Resource limits disable core dumping for process 48761 (rngd).
Nov  4 05:27:55  systemd-coredump[48763]: Process 48761 (rngd) of user 0 dumped core.
Nov  4 05:27:55  systemd[1]: rngd.service: Main process exited, code=dumped, status=6/ABRT
Nov  4 05:27:55  systemd[1]: rngd.service: Failed with result 'core-dump'.


Expected results:
rngd should start without any problems


Additional info:
- the issue is related only s390x arch
- coredump file was attached

Comment 3 smitterl 2022-11-07 16:12:10 UTC
Our test cases run into this. Given that I hit this every time and it renders rngd unusable on s390x without a known workaround, I'm setting severity urgent, please shout if anybody disagrees. Thanks.

The info from coredump that I see is

Reading symbols from /usr/sbin/rngd...
Reading symbols from /usr/lib/debug/usr/sbin/rngd-6.15-2.el9.s390x.debug...
[New LWP 279710]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/rngd -f'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=<optimized out>, no_tid=no_tid@entry=0)
    at pthread_kill.c:44
44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;

Comment 4 smitterl 2022-11-09 16:31:04 UTC
reproduces with rng-tools-6.15-2.el8.s390x  creating clone

Comment 5 smitterl 2022-11-09 16:46:11 UTC
This doesn't reproduce with rng-tools-6.15-1.el9.s390x - hence the Regression keyword.

Comment 8 Vladis Dronov 2022-11-29 12:46:33 UTC
(In reply to smitterl from comment #3)
> Our test cases run into this. Given that I hit this every time and it
> renders rngd unusable on s390x without a known workaround, I'm setting
> severity urgent, please shout if anybody disagrees. Thanks.

Hi,
1) the workaround should be disabling jitter entropy source by adding "-x jitter" to rngd options.

2) if you still have a coredump file, could you please provide full output of "bt" command in gdb?
unfotunately, "#0 __pthread_kill_implementation" entry does not give us anything.

3)  if you still have a coredump file, can you please provide it and a package "rngd" binary is from?
i mean, not just a package name, but where to get it.

Comment 9 Ondrej Moriš 2022-12-08 18:42:35 UTC
Hi Vladis,

> 2) if you still have a coredump file, could you please provide full output
> of "bt" command in gdb?
> unfotunately, "#0 __pthread_kill_implementation" entry does not give us
> anything.

# ulimit -c unlimited

# /usr/sbin/rngd

# coredumpctl dump -o core

# gdb /usr/sbin/rngd core 
...
Core was generated by `/usr/sbin/rngd'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=<optimized out>, no_tid=no_tid@entry=0) at pthread_kill.c:44
44	      return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=<optimized out>, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x000003ff93da0196 in __pthread_kill_internal (signo=<optimized out>, threadid=<optimized out>) at pthread_kill.c:78
#2  0x000003ff93d508f0 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x000003ff93d2a340 in __GI_abort () at abort.c:79
#4  0x000003ff93d92e60 in __libc_message (fmt=fmt@entry=0x3ff93e8e088 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:150
#5  0x000003ff93e256fc in __GI___fortify_fail (msg=msg@entry=0x3ff93e8e070 "stack smashing detected") at fortify_fail.c:24
#6  0x000003ff93e256d8 in __stack_chk_fail () at stack_chk_fail.c:24
#7  0x000003ff94686d00 in jent_get_nstime (out=0x3ffe17f8b28) at ./jitterentropy-base-user.h:139
#8  0x000003ff94687200 in jent_get_nstime_internal (ec=0x2aa1a3cf1c0, out=0x3ffe17f8b28) at src/jitterentropy-timer.c:181
#9  0x000003ff946846b4 in jent_memaccess (ec=0x2aa1a3cf1c0, loop_cnt=0) at src/jitterentropy-noise.c:233
#10 0x000003ff94684804 in jent_measure_jitter (ec=0x2aa1a3cf1c0, loop_cnt=0, ret_current_delta=0x0) at src/jitterentropy-noise.c:353
#11 0x000003ff94682b34 in jent_time_entropy_init (osr=0, flags=48) at src/jitterentropy-base.c:578
#12 0x000003ff94682e5e in jent_entropy_init () at src/jitterentropy-base.c:712
#13 0x000002aa19a0e024 in init_jitter_entropy_source (ent_src=0x2aa19a172b0 <entropy_sources+576>) at rngd_jitter.c:432
#14 init_jitter_entropy_source (ent_src=0x2aa19a172b0 <entropy_sources+576>) at rngd_jitter.c:399
#15 0x000002aa19a049f8 in main (argc=<optimized out>, argv=<optimized out>) at rngd.c:1102

Comment 11 Ondrej Moriš 2022-12-08 18:44:46 UTC
(In reply to Vladis Dronov from comment #8)
 
> 3)  if you still have a coredump file, can you please provide it and a
> package "rngd" binary is from?
> i mean, not just a package name, but where to get it.

Attached new one with actual command that coredumped (see c#0).

Comment 12 Ondrej Moriš 2022-12-08 18:45:19 UTC
(In reply to Ondrej Moriš from comment #11)
> (In reply to Vladis Dronov from comment #8)
>  
> > 3)  if you still have a coredump file, can you please provide it and a
> > package "rngd" binary is from?
> > i mean, not just a package name, but where to get it.
> 
> Attached new one with actual command that coredumped (see c#0).

Typo - c#10. Sorry.

Comment 13 Vladis Dronov 2022-12-27 13:56:59 UTC
(In reply to Ondrej Moriš from comment #9)
> # gdb /usr/sbin/rngd core 
Filip, thanks for reporting this issue, Ondrej thanks for reproducing and providing full stack stace, this was a great help.

Research details see: https://github.com/smuellerDD/jitterentropy-library/pull/95

Comment 27 Vilém Maršík 2023-01-25 23:52:50 UTC
Reproduced with jitterentropy-3.4.1-1.el9.s390x and rng-tools-6.15-2.el9.s390x, verified with rng-tools-6.15-3.el9.s390x.rpm and jitterentropy-3.4.1-2.el9.s390x.rpm :

# uname -r
5.14.0-242.el9.s390x
# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.2 Beta (Plow)
# dnf install rng-tools
(...)
Installed:
  jitterentropy-3.4.1-1.el9.s390x
  rng-tools-6.15-2.el9.s390x
Complete!
# systemctl start rngd
# grep rngd /var/log/messages
Jan 25 12:23:23 ibm-z-536 dracut[1286]: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
Jan 25 12:23:24 ibm-z-536 dracut[1286]: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
Jan 25 18:42:41 ibm-z-536 rngd[79567]: Disabling 7: PKCS11 Entropy generator (pkcs11)
Jan 25 18:42:41 ibm-z-536 rngd[79567]: Disabling 5: NIST Network Entropy Beacon (nist)
Jan 25 18:42:41 ibm-z-536 rngd[79567]: Disabling 9: Qrypt quantum entropy beacon (qrypt)
Jan 25 18:42:41 ibm-z-536 rngd[79567]: Initializing available sources
Jan 25 18:42:41 ibm-z-536 rngd[79567]: [hwrng ]: Initialized
Jan 25 18:42:41 ibm-z-536 rngd[79567]: [jitter]: JITTER timeout set to 5 sec
Jan 25 18:42:41 ibm-z-536 rngd[79567]: *** stack smashing detected ***: terminated
Jan 25 18:42:41 ibm-z-536 systemd-coredump[79569]: Resource limits disable core dumping for process 79567 (rngd).
Jan 25 18:42:41 ibm-z-536 systemd-coredump[79569]: Process 79567 (rngd) of user 0 dumped core.
Jan 25 18:42:41 ibm-z-536 systemd[1]: rngd.service: Main process exited, code=dumped, status=6/ABRT
Jan 25 18:42:41 ibm-z-536 systemd[1]: rngd.service: Failed with result 'core-dump'.

# rpm -U rng-tools-6.15-3.el9.s390x.rpm jitterentropy-3.4.1-2.el9.s390x.rpm
# systemctl stop rngd
# systemctl start rngd
# systemctl status rngd
● rngd.service - Hardware RNG Entropy Gatherer Daemon
     Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-01-25 18:46:39 EST; 5s ago
   Main PID: 79888 (rngd)
      Tasks: 3 (limit: 36540)
     Memory: 1.5M
        CPU: 10.715s
     CGroup: /system.slice/rngd.service
             └─79888 /usr/sbin/rngd -f --fill-watermark=0 -x pkcs11 -x nist -x qrypt -D daemon:daemon

Jan 25 18:46:39 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: Disabling 7: PKCS11 Entropy generator (pkcs11)
Jan 25 18:46:39 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: Disabling 5: NIST Network Entropy Beacon (nist)
Jan 25 18:46:39 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: Disabling 9: Qrypt quantum entropy beacon (qrypt)
Jan 25 18:46:39 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: Initializing available sources
Jan 25 18:46:39 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: [hwrng ]: Initialized
Jan 25 18:46:39 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: [jitter]: JITTER timeout set to 5 sec
Jan 25 18:46:39 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: [jitter]: Initializing AES buffer
Jan 25 18:46:42 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: [jitter]: Enabling JITTER rng support
Jan 25 18:46:42 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: [jitter]: Initialized
Jan 25 18:46:42 ibm-z-536.lab.eng.rdu2.redhat.com rngd[79888]: Process privileges have been dropped to 2:2

Comment 30 errata-xmlrpc 2023-05-09 08:15:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (jitterentropy bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2472