Bug 1793577

Summary: glibc: Parsing of /etc/gshadow can return incorrect pointers causing application segfaults
Product: Red Hat Enterprise Linux 8 Reporter: David Galloway <dgallowa>
Component: glibcAssignee: glibc team <glibc-bugzilla>
Status: CLOSED DUPLICATE QA Contact: qe-baseos-tools-bugs
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: ashankar, bstinson, carl, codonell, dj, dtardon, fweimer, jwboyer, mnewsome, pfrankli, sipoyare, systemd-maint-list
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-08-25 13:33:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Galloway 2020-01-21 15:31:31 UTC
Description of problem:
`yum -y install dnsmasq` throws a few segfaults and dnsmasq is non-operational after installation.


Version-Release number of selected component (if applicable):
2.79-6.el8

How reproducible:
100%

Steps to Reproduce:
1. Install CentOS 8.0
2. `yum -y install dnsmasq`

Actual results:
[dgalloway@smithi074 ~]$ sudo yum -y install dnsmasq
CentOS-8 - AppStream                                                                                                                                                                                                                             17 kB/s | 4.3 kB     00:00    
CentOS-8 - AppStream                                                                                                                                                                                                                            6.6 MB/s | 5.9 MB     00:00    
CentOS-8 - Base                                                                                                                                                                                                                                  34 kB/s | 3.8 kB     00:00    
CentOS-8 - Base                                                                                                                                                                                                                                  15 MB/s | 4.0 MB     00:00    
CentOS-8 - Extras                                                                                                                                                                                                                               7.8 kB/s | 1.5 kB     00:00    
CentOS-8 - Extras                                                                                                                                                                                                                               7.9 kB/s | 2.1 kB     00:00    
Extra Packages for Enterprise Linux 8 - x86_64                                                                                                                                                                                                  195 kB/s |  19 kB     00:00    
Extra Packages for Enterprise Linux 8 - x86_64                                                                                                                                                                                                  7.2 MB/s | 5.2 MB     00:00    
Dependencies resolved.
================================================================================================================================================================================================================================================================================
 Package                                                          Arch                                                            Version                                                              Repository                                                          Size
================================================================================================================================================================================================================================================================================
Installing:
 dnsmasq                                                          x86_64                                                          2.79-6.el8                                                           AppStream                                                          317 k

Transaction Summary
================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 317 k
Installed size: 736 k
Downloading Packages:
dnsmasq-2.79-6.el8.x86_64.rpm                                                                                                                                                                                                                   2.9 MB/s | 317 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                           1.7 MB/s | 317 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                        1/1 
  Running scriptlet: dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 
/var/tmp/rpm-tmp.jB0zF0: line 5:  7161 Segmentation fault      (core dumped) systemd-sysusers -  &> /dev/null <<SYSTEMD_INLINE_EOF
u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq 
SYSTEMD_INLINE_EOF


  Installing       : dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 
  Running scriptlet: dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 
warning: group dnsmasq does not exist - using root
warning: group dnsmasq does not exist - using root
warning: group dnsmasq does not exist - using root
/var/tmp/rpm-tmp.TYAf4s: line 3:  7170 Segmentation fault      (core dumped) systemd-sysusers &> /dev/null

/var/tmp/rpm-tmp.aWTxqn: line 6:  7208 Segmentation fault      (core dumped) /usr/bin/systemd-sysusers
warning: %triggerin(systemd-239-13.el8_0.5.x86_64) scriptlet failed, exit status 139

Error in <unknown> scriptlet in rpm package dnsmasq
  Verifying        : dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 

Installed:
  dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                                                     

Complete!


Additional info:
https://tracker.ceph.com/issues/43744

Comment 1 Tomáš Hozza 2020-02-05 15:38:16 UTC
From the log, the issue happened in systemd-sysusers

Comment 2 David Tardon 2020-05-18 08:00:30 UTC
I'm not getting any crash. If you can reproduce this reliably, please run:

# yum install valgrind
# yum debuginfo-install systemd
# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'

and paste the output here.

Comment 3 David Galloway 2020-05-18 13:18:29 UTC
Sure, I'm not sure if the output is valuable before or after the segfault (I would guess after but it appears to be the same at a cursory glance.  Although I have no idea what I'm looking at.

[root@smithi017 ~]# yum install valgrind
CentOS-8 - AppStream                                                                                                                                                                                                                             23 MB/s | 7.0 MB     00:00    
CentOS-8 - Base                                                                                                                                                                                                                                  25 MB/s | 2.2 MB     00:00    
CentOS-8 - Extras                                                                                                                                                                                                                                86 kB/s | 5.5 kB     00:00    
Copr repo for ceph-el8 owned by ktdreyer                                                                                                                                                                                                        457 kB/s |  47 kB     00:00    
Extra Packages for Enterprise Linux                                                                                                                                                                                                              19 MB/s | 6.6 MB     00:00    
lab-extras                                                                                                                                                                                                                                      3.9 MB/s |  20 kB     00:00    
Package valgrind-1:3.15.0-9.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!



[root@smithi017 ~]# yum debuginfo-install systemd
Last metadata expiration check: 0:00:08 ago on Mon 18 May 2020 01:15:02 PM UTC.
Could not find debuginfo for package: systemd-239-18.el8_1.5.x86_64
Could not find debuginfo for package: systemd-239-18.el8_1.5.x86_64
Could not find debuginfo for package: systemd-239-18.el8_1.5.i686
No debuginfo packages available to install
Dependencies resolved.
Nothing to do.
Complete!



[root@smithi017 ~]# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==7433== Memcheck, a memory error detector
==7433== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==7433== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==7433== Command: systemd-sysusers -
==7433== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==7433== Invalid read of size 1
==7433==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==7433==    by 0x5602EAB: __nss_valid_list_field (in /usr/lib64/libc-2.28.so)
==7433==    by 0x55E4A7E: putsgent (in /usr/lib64/libc-2.28.so)
==7433==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==7433==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==7433==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==7433==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==7433==    by 0x5505872: (below main) (in /usr/lib64/libc-2.28.so)
==7433==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==7433== 
==7433== 
==7433== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==7433==  General Protection Fault
==7433==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==7433==    by 0x5602EAB: __nss_valid_list_field (in /usr/lib64/libc-2.28.so)
==7433==    by 0x55E4A7E: putsgent (in /usr/lib64/libc-2.28.so)
==7433==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==7433==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==7433==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==7433==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==7433==    by 0x5505872: (below main) (in /usr/lib64/libc-2.28.so)
==7433== 
==7433== HEAP SUMMARY:
==7433==     in use at exit: 1,322,581 bytes in 9,278 blocks
==7433==   total heap usage: 10,378 allocs, 1,100 frees, 3,446,165 bytes allocated
==7433== 
==7433== LEAK SUMMARY:
==7433==    definitely lost: 0 bytes in 0 blocks
==7433==    indirectly lost: 0 bytes in 0 blocks
==7433==      possibly lost: 0 bytes in 0 blocks
==7433==    still reachable: 1,322,581 bytes in 9,278 blocks
==7433==         suppressed: 0 bytes in 0 blocks
==7433== Rerun with --leak-check=full to see details of leaked memory
==7433== 
==7433== For lists of detected and suppressed errors, rerun with: -s
==7433== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)




[root@smithi017 ~]# yum -y install dnsmasq
Last metadata expiration check: 0:00:44 ago on Mon 18 May 2020 01:15:02 PM UTC.
Dependencies resolved.
================================================================================================================================================================================================================================================================================
 Package                                                        Architecture                                                  Version                                                             Repository                                                               Size
================================================================================================================================================================================================================================================================================
Installing:
 dnsmasq                                                        x86_64                                                        2.79-6.el8                                                          CentOS-AppStream                                                        317 k

Transaction Summary
================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 317 k
Installed size: 736 k
Downloading Packages:
dnsmasq-2.79-6.el8.x86_64.rpm                                                                                                                                                                                                                   4.0 MB/s | 317 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                           3.9 MB/s | 317 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                        1/1 
  Running scriptlet: dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 
/var/tmp/rpm-tmp.fmneLs: line 5:  7453 Segmentation fault      (core dumped) systemd-sysusers -  &> /dev/null <<SYSTEMD_INLINE_EOF
u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq 
SYSTEMD_INLINE_EOF


  Installing       : dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 
warning: group dnsmasq does not exist - using root
warning: group dnsmasq does not exist - using root
warning: group dnsmasq does not exist - using root

  Running scriptlet: dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 
/var/tmp/rpm-tmp.JD5Cir: line 3:  7462 Segmentation fault      (core dumped) systemd-sysusers &> /dev/null

/var/tmp/rpm-tmp.gjF8ys: line 6:  7501 Segmentation fault      (core dumped) /usr/bin/systemd-sysusers
warning: %triggerin(systemd-239-18.el8_1.5.x86_64) scriptlet failed, exit status 139

Error in <unknown> scriptlet in rpm package dnsmasq
  Verifying        : dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                              1/1 

Installed:
  dnsmasq-2.79-6.el8.x86_64                                                                                                                                                                                                                                                     

Complete!



[root@smithi017 ~]# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==8412== Memcheck, a memory error detector
==8412== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==8412== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==8412== Command: systemd-sysusers -
==8412== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==8412== Invalid read of size 1
==8412==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==8412==    by 0x5602EAB: __nss_valid_list_field (in /usr/lib64/libc-2.28.so)
==8412==    by 0x55E4A7E: putsgent (in /usr/lib64/libc-2.28.so)
==8412==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==8412==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==8412==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==8412==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==8412==    by 0x5505872: (below main) (in /usr/lib64/libc-2.28.so)
==8412==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==8412== 
==8412== 
==8412== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==8412==  General Protection Fault
==8412==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==8412==    by 0x5602EAB: __nss_valid_list_field (in /usr/lib64/libc-2.28.so)
==8412==    by 0x55E4A7E: putsgent (in /usr/lib64/libc-2.28.so)
==8412==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==8412==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==8412==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==8412==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==8412==    by 0x5505872: (below main) (in /usr/lib64/libc-2.28.so)
==8412== 
==8412== HEAP SUMMARY:
==8412==     in use at exit: 1,322,581 bytes in 9,278 blocks
==8412==   total heap usage: 10,378 allocs, 1,100 frees, 3,446,165 bytes allocated
==8412== 
==8412== LEAK SUMMARY:
==8412==    definitely lost: 0 bytes in 0 blocks
==8412==    indirectly lost: 0 bytes in 0 blocks
==8412==      possibly lost: 0 bytes in 0 blocks
==8412==    still reachable: 1,322,581 bytes in 9,278 blocks
==8412==         suppressed: 0 bytes in 0 blocks
==8412== Rerun with --leak-check=full to see details of leaked memory
==8412== 
==8412== For lists of detected and suppressed errors, rerun with: -s
==8412== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

Comment 4 David Tardon 2020-05-19 11:14:08 UTC
(In reply to David Galloway from comment #3)
> Sure, I'm not sure if the output is valuable before or after the segfault (I
> would guess after but it appears to be the same at a cursory glance. 

Eh, sorry, I thought it was obvious (it is to me :-) It's the command that segfaults during installation, so it's not necessary to run yum at all. Just the command(s) I posted.

> [root@smithi017 ~]# yum debuginfo-install systemd
> Last metadata expiration check: 0:00:08 ago on Mon 18 May 2020 01:15:02 PM
> UTC.
> Could not find debuginfo for package: systemd-239-18.el8_1.5.x86_64
> Could not find debuginfo for package: systemd-239-18.el8_1.5.x86_64
> Could not find debuginfo for package: systemd-239-18.el8_1.5.i686
> No debuginfo packages available to install

Blah. I foolishly expected that the debuginfo-install command enables debuginfo repositories automatically, like it does on Fedora... This makes the backtrace almost useless. Could you try once more, please?

# sed -i -e /enabled=/s/0/1/ /etc/yum.repos.d/CentOS-Debuginfo.repo
# yum debuginfo-install systemd glibc
# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'

Comment 5 David Galloway 2020-05-19 23:33:33 UTC
# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==12115== Memcheck, a memory error detector
==12115== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12115== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12115== Command: systemd-sysusers -
==12115== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==12115== Invalid read of size 1
==12115==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12115==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12115==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12115==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12115==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x5505872: (below main) (libc-start.c:308)
==12115==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==12115== 
==12115== 
==12115== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==12115==  General Protection Fault
==12115==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12115==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12115==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12115==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12115==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x5505872: (below main) (libc-start.c:308)
==12115== 
==12115== HEAP SUMMARY:
==12115==     in use at exit: 1,322,581 bytes in 9,278 blocks
==12115==   total heap usage: 10,378 allocs, 1,100 frees, 3,446,165 bytes allocated
==12115== 
==12115== LEAK SUMMARY:
==12115==    definitely lost: 0 bytes in 0 blocks
==12115==    indirectly lost: 0 bytes in 0 blocks
==12115==      possibly lost: 0 bytes in 0 blocks
==12115==    still reachable: 1,322,581 bytes in 9,278 blocks
==12115==         suppressed: 0 bytes in 0 blocks
==12115== Rerun with --leak-check=full to see details of leaked memory
==12115== 
==12115== For lists of detected and suppressed errors, rerun with: -s
==12115== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Comment 6 David Galloway 2020-05-19 23:36:32 UTC
It looks the same.  Did I do something wrong?

[root@smithi017 ~]# sed -i -e /enabled=/s/0/1/ /etc/yum.repos.d/CentOS-Debuginfo.repo

[root@smithi017 ~]# yum debuginfo-install systemd glibc
CentOS-8 - AppStream                                                                                                                                                                                                                             37 MB/s | 7.0 MB     00:00    
CentOS-8 - Base                                                                                                                                                                                                                                  25 MB/s | 2.2 MB     00:00    
CentOS-8 - Debuginfo                                                                                                                                                                                                                            1.6 MB/s |  10 MB     00:06    
CentOS-8 - Extras                                                                                                                                                                                                                                73 kB/s | 5.5 kB     00:00    
Copr repo for ceph-el8 owned by ktdreyer                                                                                                                                                                                                        471 kB/s |  47 kB     00:00    
Extra Packages for Enterprise Linux                                                                                                                                                                                                              19 MB/s | 6.6 MB     00:00    
lab-extras                                                                                                                                                                                                                                      4.1 MB/s |  20 kB     00:00    
Dependencies resolved.
================================================================================================================================================================================================================================================================================
 Package                                                                   Architecture                                              Version                                                            Repository                                                         Size
================================================================================================================================================================================================================================================================================
Installing:
 glibc-debuginfo                                                           x86_64                                                    2.28-72.el8_1.1                                                    base-debuginfo                                                     13 M
 systemd-debuginfo                                                         x86_64                                                    239-18.el8_1.4                                                     base-debuginfo                                                    7.2 M
Installing dependencies:
 glibc-debuginfo-common                                                    x86_64                                                    2.28-72.el8_1.1                                                    base-debuginfo                                                     11 M
Installing weak dependencies:
 systemd-debugsource                                                       x86_64                                                    239-18.el8_1.4                                                     base-debuginfo                                                    2.3 M

Transaction Summary
================================================================================================================================================================================================================================================================================
Install  4 Packages

Total download size: 34 M
Installed size: 237 M
Is this ok [y/N]: y
Downloading Packages:
(1/4): systemd-debuginfo-239-18.el8_1.4.x86_64.rpm                                                                                                                                                                                              1.5 MB/s | 7.2 MB     00:04    
(2/4): systemd-debugsource-239-18.el8_1.4.x86_64.rpm                                                                                                                                                                                            1.6 MB/s | 2.3 MB     00:01    
(3/4): glibc-debuginfo-common-2.28-72.el8_1.1.x86_64.rpm                                                                                                                                                                                        1.5 MB/s |  11 MB     00:07    
(4/4): glibc-debuginfo-2.28-72.el8_1.1.x86_64.rpm                                                                                                                                                                                               1.5 MB/s |  13 MB     00:08    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                           3.8 MB/s |  34 MB     00:08     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                        1/1 
  Installing       : systemd-debugsource-239-18.el8_1.4.x86_64                                                                                                                                                                                                              1/4 
  Installing       : glibc-debuginfo-common-2.28-72.el8_1.1.x86_64                                                                                                                                                                                                          2/4 
  Installing       : glibc-debuginfo-2.28-72.el8_1.1.x86_64                                                                                                                                                                                                                 3/4 
  Installing       : systemd-debuginfo-239-18.el8_1.4.x86_64                                                                                                                                                                                                                4/4 
  Running scriptlet: systemd-debuginfo-239-18.el8_1.4.x86_64                                                                                                                                                                                                                4/4 
  Verifying        : glibc-debuginfo-2.28-72.el8_1.1.x86_64                                                                                                                                                                                                                 1/4 
  Verifying        : glibc-debuginfo-common-2.28-72.el8_1.1.x86_64                                                                                                                                                                                                          2/4 
  Verifying        : systemd-debuginfo-239-18.el8_1.4.x86_64                                                                                                                                                                                                                3/4 
  Verifying        : systemd-debugsource-239-18.el8_1.4.x86_64                                                                                                                                                                                                              4/4 

Installed:
  glibc-debuginfo-2.28-72.el8_1.1.x86_64                           systemd-debuginfo-239-18.el8_1.4.x86_64                           systemd-debugsource-239-18.el8_1.4.x86_64                           glibc-debuginfo-common-2.28-72.el8_1.1.x86_64                          

Complete!

[root@smithi017 ~]# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==12096== Memcheck, a memory error detector
==12096== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12096== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12096== Command: systemd-sysusers -
==12096== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==12096== Invalid read of size 1
==12096==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12096==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12096==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12096==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12096==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12096==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12096==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12096==    by 0x5505872: (below main) (libc-start.c:308)
==12096==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==12096== 
==12096== 
==12096== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==12096==  General Protection Fault
==12096==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12096==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12096==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12096==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12096==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12096==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12096==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12096==    by 0x5505872: (below main) (libc-start.c:308)
==12096== 
==12096== HEAP SUMMARY:
==12096==     in use at exit: 1,322,581 bytes in 9,278 blocks
==12096==   total heap usage: 10,378 allocs, 1,100 frees, 3,446,165 bytes allocated
==12096== 
==12096== LEAK SUMMARY:
==12096==    definitely lost: 0 bytes in 0 blocks
==12096==    indirectly lost: 0 bytes in 0 blocks
==12096==      possibly lost: 0 bytes in 0 blocks
==12096==    still reachable: 1,322,581 bytes in 9,278 blocks
==12096==         suppressed: 0 bytes in 0 blocks
==12096== Rerun with --leak-check=full to see details of leaked memory
==12096== 
==12096== For lists of detected and suppressed errors, rerun with: -s
==12096== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)



[root@smithi017 ~]# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==12115== Memcheck, a memory error detector
==12115== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12115== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12115== Command: systemd-sysusers -
==12115== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==12115== Invalid read of size 1
==12115==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12115==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12115==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12115==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12115==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x5505872: (below main) (libc-start.c:308)
==12115==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==12115== 
==12115== 
==12115== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==12115==  General Protection Fault
==12115==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12115==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12115==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12115==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12115==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12115==    by 0x5505872: (below main) (libc-start.c:308)
==12115== 
==12115== HEAP SUMMARY:
==12115==     in use at exit: 1,322,581 bytes in 9,278 blocks
==12115==   total heap usage: 10,378 allocs, 1,100 frees, 3,446,165 bytes allocated
==12115== 
==12115== LEAK SUMMARY:
==12115==    definitely lost: 0 bytes in 0 blocks
==12115==    indirectly lost: 0 bytes in 0 blocks
==12115==      possibly lost: 0 bytes in 0 blocks
==12115==    still reachable: 1,322,581 bytes in 9,278 blocks
==12115==         suppressed: 0 bytes in 0 blocks
==12115== Rerun with --leak-check=full to see details of leaked memory
==12115== 
==12115== For lists of detected and suppressed errors, rerun with: -s
==12115== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)



[root@smithi017 ~]# valgrind --leak-check=full systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==12132== Memcheck, a memory error detector
==12132== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12132== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==12132== Command: systemd-sysusers -
==12132== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==12132== Invalid read of size 1
==12132==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12132==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12132==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12132==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12132==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12132==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12132==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12132==    by 0x5505872: (below main) (libc-start.c:308)
==12132==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==12132== 
==12132== 
==12132== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==12132==  General Protection Fault
==12132==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==12132==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==12132==    by 0x55E4A7E: putsgent (putsgent.c:37)
==12132==    by 0x4F3C500: putsgent_sane (in /usr/lib/systemd/libsystemd-shared-239.so)
==12132==    by 0x10D360: ??? (in /usr/bin/systemd-sysusers)
==12132==    by 0x10FFF8: ??? (in /usr/bin/systemd-sysusers)
==12132==    by 0x10C2F1: ??? (in /usr/bin/systemd-sysusers)
==12132==    by 0x5505872: (below main) (libc-start.c:308)
==12132== 
==12132== HEAP SUMMARY:
==12132==     in use at exit: 1,322,581 bytes in 9,278 blocks
==12132==   total heap usage: 10,378 allocs, 1,100 frees, 3,446,165 bytes allocated
==12132== 
==12132== LEAK SUMMARY:
==12132==    definitely lost: 0 bytes in 0 blocks
==12132==    indirectly lost: 0 bytes in 0 blocks
==12132==      possibly lost: 0 bytes in 0 blocks
==12132==    still reachable: 1,322,581 bytes in 9,278 blocks
==12132==         suppressed: 0 bytes in 0 blocks
==12132== Reachable blocks (those to which a pointer was found) are not shown.
==12132== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==12132== 
==12132== For lists of detected and suppressed errors, rerun with: -s
==12132== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

Comment 7 David Tardon 2020-05-21 13:44:49 UTC
(In reply to David Galloway from comment #6)
> It looks the same.  Did I do something wrong?
> ...
> [root@smithi017 ~]# yum debuginfo-install systemd glibc
> Installing:
>  systemd-debuginfo                                                        
> x86_64                                                    239-18.el8_1.4    

No, you didn't do anything wrong. Just the systemd-debuginfo version doesn't match the installed systemd version, shown in comment 3:

> # yum debuginfo-install systemd
> Last metadata expiration check: 0:00:08 ago on Mon 18 May 2020 01:15:02 PM UTC.
> Could not find debuginfo for package: systemd-239-18.el8_1.5.x86_64

It appears that the debuginfo repo is lagging :-( You can use the following command to downgrade systemd to the 219-18.el_8_1.4 version:

# yum downgrade http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/systemd-{,{libs,pam,udev}-}239-18.el8_1.4.x86_64.rpm

Comment 8 David Tardon 2020-06-18 14:01:28 UTC
It seems systemd-debuginfo-239-18.el8_1.5 is available now. Could you try again?

Comment 9 David Galloway 2020-06-29 18:30:15 UTC
[root@smithi153 ~]# rpm -q systemd-debuginfo
systemd-debuginfo-239-18.el8_1.5.x86_64

[root@smithi153 ~]# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==10699== Memcheck, a memory error detector
==10699== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==10699== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==10699== Command: systemd-sysusers -
==10699== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==10699== Invalid read of size 1
==10699==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==10699==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==10699==    by 0x55E4A7E: putsgent (putsgent.c:37)
==10699==    by 0x4F3C500: putsgent_sane (user-util.c:758)
==10699==    by 0x10D360: putsgent_with_members (sysusers.c:321)
==10699==    by 0x10FFF8: write_temporary_gshadow (sysusers.c:679)
==10699==    by 0x10FFF8: write_files (sysusers.c:737)
==10699==    by 0x10C2F1: main (sysusers.c:1996)
==10699==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==10699== 
==10699== 
==10699== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==10699==  General Protection Fault
==10699==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==10699==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==10699==    by 0x55E4A7E: putsgent (putsgent.c:37)
==10699==    by 0x4F3C500: putsgent_sane (user-util.c:758)
==10699==    by 0x10D360: putsgent_with_members (sysusers.c:321)
==10699==    by 0x10FFF8: write_temporary_gshadow (sysusers.c:679)
==10699==    by 0x10FFF8: write_files (sysusers.c:737)
==10699==    by 0x10C2F1: main (sysusers.c:1996)
==10699== 
==10699== HEAP SUMMARY:
==10699==     in use at exit: 1,322,581 bytes in 9,278 blocks
==10699==   total heap usage: 10,378 allocs, 1,100 frees, 3,446,171 bytes allocated
==10699== 
==10699== LEAK SUMMARY:
==10699==    definitely lost: 0 bytes in 0 blocks
==10699==    indirectly lost: 0 bytes in 0 blocks
==10699==      possibly lost: 0 bytes in 0 blocks
==10699==    still reachable: 1,322,581 bytes in 9,278 blocks
==10699==         suppressed: 0 bytes in 0 blocks
==10699== Rerun with --leak-check=full to see details of leaked memory
==10699== 
==10699== For lists of detected and suppressed errors, rerun with: -s
==10699== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)

Comment 10 David Tardon 2020-07-01 14:43:27 UTC
Thanks! I don't see any problem on systemd's side. For this simple case write_temporary_gshadow() esentially just runs putsgent(getsgent(stream1), stream2) in a loop, so if there's any garbage in one of the returned struct sgrp objects, it must be glibc's fault...

Comment 11 Florian Weimer 2020-07-01 14:58:15 UTC
Do you have a user called "zack" on the system? What is the contents of /etc/nsswitch.conf? Thanks.

Comment 12 David Galloway 2020-07-09 14:28:06 UTC
Yes, there is a zack user.

[root@smithi153 ~]# cat /etc/nsswitch.conf
# Generated by authselect on Sun Apr 26 15:38:07 2020
# Do not modify this file manually.

# If you want to make changes to nsswitch.conf please modify
# /etc/authselect/user-nsswitch.conf and run 'authselect apply-changes'.
#
# Note that your changes may not be applied as they may be
# overwritten by selected profile. Maps set in the authselect
# profile takes always precedence and overwrites the same maps
# set in the user file. Only maps that are not set by the profile
# are applied from the user file.
#
# For example, if the profile sets:
#     passwd: sss files
# and /etc/authselect/user-nsswitch.conf contains:
#     passwd: files
#     hosts: files dns
# the resulting generated nsswitch.conf will be:
#     passwd: sss files # from profile
#     hosts: files dns  # from user file

passwd:     sss files systemd
group:      sss files systemd
netgroup:   sss files
automount:  sss files
services:   sss files

# Included from /etc/authselect/user-nsswitch.conf

#
# /etc/nsswitch.conf
#
# An example Name Service Switch config file. This file should be
# sorted with the most-used services at the beginning.
#
# The entry '[NOTFOUND=return]' means that the search for an
# entry should stop if the search in the previous entry turned
# up nothing. Note that if the search failed due to some other reason
# (like no NIS server responding) then the search continues with the
# next entry.
#
# Valid entries include:
#
#	nisplus			Use NIS+ (NIS version 3)
#	nis			Use NIS (NIS version 2), also called YP
#	dns			Use DNS (Domain Name Service)
#	files			Use the local files in /etc
#	db			Use the pre-processed /var/db files
#	compat			Use /etc files plus *_compat pseudo-databases
#	hesiod			Use Hesiod (DNS) for user lookups
#	sss			Use sssd (System Security Services Daemon)
#	[NOTFOUND=return]	Stop searching if not found so far
#
# 'sssd' performs its own 'files'-based caching, so it should
# generally come before 'files'.

# To use 'db', install the nss_db package, and put the 'db' in front
# of 'files' for entries you want to be looked up first in the
# databases, like this:
#
# passwd:    db files
# shadow:    db files
# group:     db files

shadow:     files sss

hosts:      files dns myhostname

bootparams: files

ethers:     files
netmasks:   files
networks:   files
protocols:  files
rpc:        files


publickey:  files

aliases:    files

Comment 13 Florian Weimer 2020-07-09 14:37:05 UTC
Thanks. I assume that sssd is running. Can you reproduce this without sss?

Comment 14 David Galloway 2020-07-09 14:42:33 UTC
Looks like yes.

[root@smithi153 ~]# valgrind systemd-sysusers - <<<'u dnsmasq - "Dnsmasq DHCP and DNS server" /var/lib/dnsmasq'
==6147== Memcheck, a memory error detector
==6147== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6147== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==6147== Command: systemd-sysusers -
==6147== 
Creating group dnsmasq with gid 983.
Creating user dnsmasq (Dnsmasq DHCP and DNS server) with uid 983 and gid 983.
==6147== Invalid read of size 1
==6147==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==6147==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==6147==    by 0x55E4A7E: putsgent (putsgent.c:37)
==6147==    by 0x4F3C500: putsgent_sane (user-util.c:758)
==6147==    by 0x10D360: putsgent_with_members (sysusers.c:321)
==6147==    by 0x10FFF8: write_temporary_gshadow (sysusers.c:679)
==6147==    by 0x10FFF8: write_files (sysusers.c:737)
==6147==    by 0x10C2F1: main (sysusers.c:1996)
==6147==  Address 0x72646b006b63617a is not stack'd, malloc'd or (recently) free'd
==6147== 
==6147== 
==6147== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==6147==  General Protection Fault
==6147==    at 0x4C393A2: strpbrk (vg_replace_strmem.c:1691)
==6147==    by 0x5602EAB: __nss_valid_list_field (valid_list_field.c:32)
==6147==    by 0x55E4A7E: putsgent (putsgent.c:37)
==6147==    by 0x4F3C500: putsgent_sane (user-util.c:758)
==6147==    by 0x10D360: putsgent_with_members (sysusers.c:321)
==6147==    by 0x10FFF8: write_temporary_gshadow (sysusers.c:679)
==6147==    by 0x10FFF8: write_files (sysusers.c:737)
==6147==    by 0x10C2F1: main (sysusers.c:1996)
==6147== 
==6147== HEAP SUMMARY:
==6147==     in use at exit: 1,322,581 bytes in 9,278 blocks
==6147==   total heap usage: 10,369 allocs, 1,091 frees, 3,446,103 bytes allocated
==6147== 
==6147== LEAK SUMMARY:
==6147==    definitely lost: 0 bytes in 0 blocks
==6147==    indirectly lost: 0 bytes in 0 blocks
==6147==      possibly lost: 0 bytes in 0 blocks
==6147==    still reachable: 1,322,581 bytes in 9,278 blocks
==6147==         suppressed: 0 bytes in 0 blocks
==6147== Rerun with --leak-check=full to see details of leaked memory
==6147== 
==6147== For lists of detected and suppressed errors, rerun with: -s
==6147== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Segmentation fault (core dumped)



[root@smithi153 ~]# systemctl status sssd
● sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2020-07-09 14:37:42 UTC; 1min 9s ago
  Process: 2393 ExecStart=/usr/sbin/sssd -i ${DEBUG_LOGGER} (code=exited, status=0/SUCCESS)
 Main PID: 2393 (code=exited, status=0/SUCCESS)

Jun 26 16:07:36 smithi179 systemd[1]: Starting System Security Services Daemon...
Jun 26 16:07:38 smithi179 sssd[2393]: Starting up
Jun 26 16:07:38 smithi179 sssd[be[implicit_files]][2521]: Starting up
Jun 26 16:07:38 smithi179 sssd[nss][2528]: Starting up
Jun 26 16:07:39 smithi179 systemd[1]: Started System Security Services Daemon.
Jul 09 14:37:42 smithi153 systemd[1]: Stopping System Security Services Daemon...
Jul 09 14:37:42 smithi153 sssd[nss][2528]: Shutting down
Jul 09 14:37:42 smithi153 sssd[be[implicit_files]][2521]: Shutting down
Jul 09 14:37:42 smithi153 systemd[1]: Stopped System Security Services Daemon.


[root@smithi153 ~]# grep 983 /etc/passwd
[root@smithi153 ~]# grep 983 /etc/group
[root@smithi153 ~]# grep 983 /etc/shadow


A colleague said he was able to get around this by manually creating the user and group.  He also linked me this: https://github.com/systemd/systemd/issues/6512

Comment 15 Carlos O'Donell 2020-07-09 15:22:37 UTC
I've reviewed the systemd bug, thanks for that reference, and there are claims that the posted fix for upstream bug 20338 fixes the issue.

We're going to need to review this more thoroughly and see if they are related and if it does indeed fix the issue.

I've linked the other trackers to this issue and posted upstream on the systemd issue.

Comment 20 Florian Weimer 2020-07-17 08:34:01 UTC
Upstream patches posted for review: https://sourceware.org/pipermail/libc-alpha/2020-July/116430.html

Comment 22 Carlos O'Donell 2020-08-25 13:33:04 UTC
We are going to track this in bug 1871397.

*** This bug has been marked as a duplicate of bug 1871397 ***