Bug 2120222

Summary: systemd 250-9 crashes when dnf update cloud-init is called
Product: Red Hat Enterprise Linux 9 Reporter: Florence Blanc-Renaud <frenaud>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Frantisek Sumsal <fsumsal>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.1CC: jamacku, pholica, systemd-maint-list
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-250-10.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-15 11:18:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Florence Blanc-Renaud 2022-08-22 08:55:06 UTC
Description of problem:
systemd coredumps when cloud-init is updated to 22.1-5.

Install RHEL 9.1 with systemd 250-9, then update cloud-init to 22.1-5.el9
The dnf update operation causes systemd to crash


Version-Release number of selected component (if applicable):
systemd-250-9.el9.x86_64
cloud-init-22.1-5.el9.noarch

How reproducible:
Always

Steps to Reproduce:
1. update to systemd 250-9
2. run "dnf update cloud-init" to update to 22.1-5

Actual results:
systemd coredumps


Additional info:
The issue does not happen with systemd-250-7.

Output of dnf update:
[root@master ~]# dnf update cloud-init
Last metadata expiration check: 0:01:23 ago on Mon Aug 22 04:42:08 2022.
Dependencies resolved.
==========================================================================================================================================================================================================================================================================
 Package                                                         Architecture                                                Version                                                         Repository                                                              Size
==========================================================================================================================================================================================================================================================================
Upgrading:
 cloud-init                                                      noarch                                                      22.1-5.el9                                                      rhelidm-AppStream                                                      1.1 M

Transaction Summary
==========================================================================================================================================================================================================================================================================
Upgrade  1 Package

Total download size: 1.1 M
Is this ok [y/N]: y
Downloading Packages:
cloud-init-22.1-5.el9.noarch.rpm                                                                                                                                                                                                           40 MB/s | 1.1 MB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                      36 MB/s | 1.1 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                                  1/1 
  Upgrading        : cloud-init-22.1-5.el9.noarch                                                                                                                                                                                                                     1/2 
  Running scriptlet: cloud-init-22.1-5.el9.noarch                                                                                                                                                                                                                     1/2 

Broadcast message from systemd-journald.test (Mon 2022-08-22 04:43:36 EDT):

systemd[1]: Caught <ABRT>, dumped core as pid 25505.


Broadcast message from systemd-journald.test (Mon 2022-08-22 04:43:36 EDT):

systemd[1]: Freezing execution.


Message from syslogd@localhost at Aug 22 04:43:36 ...
 systemd[1]:Caught <ABRT>, dumped core as pid 25505.

Message from syslogd@localhost at Aug 22 04:43:36 ...
 systemd[1]:Freezing execution.
  Running scriptlet: cloud-init-22.1-4.el9.noarch                                                                                                                                                                                                                     2/2 
  Cleanup          : cloud-init-22.1-4.el9.noarch                                                                                                                                                                                                                     2/2 
  Running scriptlet: cloud-init-22.1-4.el9.noarch                                                                                                                                                                                                                     2/2 
Failed to reload daemon: Connection timed out
Failed to start jobs: Connection timed out

Failed to reload daemon: Connection timed out

Failed to start jobs: Connection timed out

  Verifying        : cloud-init-22.1-5.el9.noarch                                                                                                                                                                                                                     1/2 
  Verifying        : cloud-init-22.1-4.el9.noarch                                                                                                                                                                                                                     2/2 

Upgraded:
  cloud-init-22.1-5.el9.noarch                                                                                                                                                                                                                                            

Complete!
[root@master ~]# coredumpctl info
Failed to check if any systemd-coredump@.service units are running: Connection timed out
           PID: 25505 (systemd)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Mon 2022-08-22 04:43:36 EDT (9min ago)
  Command Line: /usr/lib/systemd/systemd --system --deserialize 37
    Executable: /usr/lib/systemd/systemd
 Control Group: /init.scope
          Unit: init.scope
         Slice: -.slice
       Boot ID: 14c14a33ea5b4fcbb55e8f3ecba879f3
    Machine ID: b64b347e0a1f422eada91fc20f22a0a3
      Hostname: master.testrealm.test
       Storage: /var/lib/systemd/coredump/core.systemd.0.14c14a33ea5b4fcbb55e8f3ecba879f3.25505.1661157816000000.zst (present)
     Disk Size: 668.0K
       Message: Process 25505 (systemd) of user 0 dumped core.
                
                Module linux-vdso.so.1 with build-id 5e66e8e0b643ad635d35b7568b73d9e88f8c9222
                Module libffi.so.8 with build-id 48e3675db4765a2e42729140922e11a10016f7ab
                Module libgpg-error.so.0 with build-id 9d27198f0ca61c66cd921675219dffc0bad16a1a
                Module libattr.so.1 with build-id e67f1fc89e8ac6a35f6fda914bcf6144b9ccb99c
                Module libz.so.1 with build-id 25486226566596e403da5485fb0ec85deed6b9fa
                Module libcap-ng.so.0 with build-id fdca0a301667e15db99d726152b57feeb35e4dbe
                Module libeconf.so.0 with build-id cbc97ad9626c64a49d712bdd3460812cc2d57027
                Module libpcre2-8.so.0 with build-id dac773591ff85ee4d18b00795d8bca123f3d5d66
                Module ld-linux-x86-64.so.2 with build-id df9c6b298bf5e3c1d0eb6a0911f3f561908a704d
                Module libm.so.6 with build-id c0eb573a2171d96b1aa970edb07f3368573bf845
                Module liblzma.so.5 with build-id 330eb2fe0769e5466e2e0ac1b158e1e8452738c9
                Module libzstd.so.1 with build-id f0c68ad1b3f8941857af47c6887736d835317ccc
                Module libp11-kit.so.0 with build-id 5e20d86b92c9f913571338c18cb70f74da7d3c0e
                Module libcrypto.so.3 with build-id cf4309c0f1575da5c049b4952352744019992bf7
                Module liblz4.so.1 with build-id 4d32cb5fa39c86b05cc10cc380f3a8a0d6d9d648
                Module libgcrypt.so.20 with build-id 739fe88859e158a6f549030c694c2e9c8070d75f
                Module libcrypt.so.2 with build-id 6ce4e5eb200e61d07398af52f8bcb316cf8466e0
                Module libcap.so.2 with build-id c7625c8524a3d7756043555a1e7b1c3cb56fabbe
                Module libblkid.so.1 with build-id d2ca5e74365b5a58e5dca6b672513f4a576048e2
                Module libacl.so.1 with build-id afd5611fa888ba30a62a2109d13bd3903d84c61c
                Module libc.so.6 with build-id b439a356c78dfa4bd24c75a16f564540db2a30ad
                Module libgcc_s.so.1 with build-id 9526c65fed0e95fbb6b988476cc811ca19d5c9c9
                Module libkmod.so.2 with build-id 722d3027106c561018756766be8bed5215ed0f31
                Module libaudit.so.1 with build-id 199420e5d951d173a9975ba0b2b7c5d3d8e5c68d
                Module libpam.so.0 with build-id 763a5df38e5fc1e505f194974f3728919f30185b
                Module libmount.so.1 with build-id 2cb3370d622a16c653f486ac9d15a63902d0f73e
                Module libselinux.so.1 with build-id 321a1f9b5537883ee8ec04c65a9edbaefcc7b5aa
                Module libseccomp.so.2 with build-id 5a5b4ee3bb99ee0378bd937fdef7a7b7255d0428
                Module libsystemd-shared-250.so with build-id 44b7a28b1071d92710d10f3c715a9a134bc275b2
                Module systemd with build-id f55670c75144a84195097f24f88768d5707454f3
                Stack trace of thread 25505:
                #0  0x00007f3cd855276b kill (libc.so.6 + 0x2976b)
                #1  0x00005632941db178 crash (systemd + 0x49178)
                #2  0x00007f3cd857dd90 __restore_rt (libc.so.6 + 0x54d90)
                #3  0x00007f3cd85ca54c __pthread_kill_implementation (libc.so.6 + 0xa154c)
                #4  0x00007f3cd857dce6 raise (libc.so.6 + 0x54ce6)
                #5  0x00007f3cd85517f3 abort (libc.so.6 + 0x287f3)
                #6  0x00007f3cd88c34dc log_assert_failed.cold (libsystemd-shared-250.so + 0x834dc)
                #7  0x00007f3cd8a004ff startswith (libsystemd-shared-250.so + 0x1c04ff)
                #8  0x00007f3cd893ebd4 unit_file_reenable (libsystemd-shared-250.so + 0xfebd4)
                #9  0x00005632941eb75c method_enable_unit_files_generic (systemd + 0x5975c)
                #10 0x00007f3cd8a209b8 object_find_and_run (libsystemd-shared-250.so + 0x1e09b8)
                #11 0x00007f3cd8a21e4d bus_process_object (libsystemd-shared-250.so + 0x1e1e4d)
                #12 0x00007f3cd8a3524e bus_process_internal (libsystemd-shared-250.so + 0x1f524e)
                #13 0x00007f3cd8a35ca1 io_callback (libsystemd-shared-250.so + 0x1f5ca1)
                #14 0x00007f3cd8a8a345 source_dispatch (libsystemd-shared-250.so + 0x24a345)
                #15 0x00007f3cd8a8a51d sd_event_dispatch (libsystemd-shared-250.so + 0x24a51d)
                #16 0x00007f3cd8a8bbe8 sd_event_run (libsystemd-shared-250.so + 0x24bbe8)
                #17 0x000056329422dcd9 manager_loop (systemd + 0x9bcd9)
                #18 0x00005632941d976d main (systemd + 0x4776d)
                #19 0x00007f3cd8568eb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                #20 0x00007f3cd8568f60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                #21 0x00005632941dad25 _start (systemd + 0x48d25)
                ELF object binary architecture: AMD x86-64
[root@master ~]#

Comment 1 Jan Macku 2022-08-22 10:35:48 UTC
Output of gdb bt:

#0  0x00007fc1e260776b in kill () at ../sysdeps/unix/syscall-template.S:120
#1  0x000055b9f410a178 in crash (sig=6) at ../src/core/main.c:267
#2  <signal handler called>
#3  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#4  0x00007fc1e267f5b3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#5  0x00007fc1e2632ce6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#6  0x00007fc1e26067f3 in __GI_abort () at abort.c:79
#7  0x00007fc1e29784dc in log_assert_failed (text=<optimized out>, file=<optimized out>, line=<optimized out>, func=<optimized out>) at ../src/basic/log.c:866
#8  0x00007fc1e2ab54ff in startswith (s=<optimized out>, prefix=0x0) at ../src/fundamental/string-util-fundamental.c:14
#9  0x00007fc1e29f3bd4 in normalize_linked_files (ret_files=0x7ffef29059f0, ret_names=0x7ffef29059f8, names_or_paths=<optimized out>, lp=0x7ffef2905a40, scope=LOOKUP_SCOPE_SYSTEM) at ../src/shared/install.c:2839
#10 unit_file_reenable (scope=LOOKUP_SCOPE_SYSTEM, flags=0, root_dir=<optimized out>, names_or_paths=<optimized out>, changes=0x7ffef2905b10, n_changes=0x7ffef2905b08) at ../src/shared/install.c:2874
#11 0x000055b9f411a75c in method_enable_unit_files_generic (message=0x55b9f55b0040, m=0x55b9f5418670, call=0x7fc1e29f3990 <unit_file_reenable>, carries_install_info=<optimized out>, error=0x7ffef2905bf0) at ../src/core/dbus-manager.c:2259
#12 0x00007fc1e2ad59b8 in method_callbacks_run (found_object=0x7ffef2905cb7, require_fallback=false, c=0x55b9f552d030, m=0x55b9f55b0040, bus=0x55b9f54cfd90) at ../src/libsystemd/sd-bus/bus-objects.c:413
#13 object_find_and_run (bus=0x55b9f54cfd90, m=0x55b9f55b0040, p=0x55b9f555d378 "/org/freedesktop/systemd1", require_fallback=false, found_object=0x7ffef2905cb7) at ../src/libsystemd/sd-bus/bus-objects.c:1322
#14 0x00007fc1e2ad6e4d in bus_process_object (bus=0x55b9f54cfd90, m=0x55b9f55b0040) at ../src/libsystemd/sd-bus/bus-objects.c:1442
#15 0x00007fc1e2aea24e in process_message (m=0x55b9f55b0040, bus=0x55b9f54cfd90) at ../src/libsystemd/sd-bus/sd-bus.c:2965
#16 process_running (ret=0x0, bus=<optimized out>) at ../src/libsystemd/sd-bus/sd-bus.c:3007
#17 bus_process_internal (bus=<optimized out>, ret=0x0) at ../src/libsystemd/sd-bus/sd-bus.c:3227
#18 0x00007fc1e2aeaca1 in io_callback (s=<optimized out>, fd=<optimized out>, revents=<optimized out>, userdata=0x55b9f54cfd90) at ../src/libsystemd/sd-bus/sd-bus.c:3605
#19 0x00007fc1e2b3f345 in source_dispatch (s=0x55b9f5540ae0) at ../src/libsystemd/sd-event/sd-event.c:3600
#20 0x00007fc1e2b3f51d in sd_event_dispatch (e=0x55b9f5417bf0) at ../src/libsystemd/sd-event/sd-event.c:4185
#21 0x00007fc1e2b40be8 in sd_event_run (e=0x55b9f5417bf0, timeout=18446744073709551615) at ../src/libsystemd/sd-event/sd-event.c:4246
#22 0x000055b9f415ccd9 in manager_loop (m=0x55b9f5418670) at ../src/core/manager.c:3049
#23 0x000055b9f410876d in invoke_main_loop (ret_error_message=0x7ffef2906090, ret_switch_root_init=<synthetic pointer>, ret_switch_root_dir=<synthetic pointer>, ret_fds=0x7ffef2906080, ret_shutdown_verb=<synthetic pointer>, ret_retval=<synthetic pointer>, ret_reexecute=<synthetic pointer>, 
    saved_rlimit_memlock=0x7ffef29060a0, saved_rlimit_nofile=0x7ffef29060b0, m=0x55b9f5418670) at ../src/core/main.c:1976
#24 main (argc=5, argv=0x7ffef2906378) at ../src/core/main.c:3022

Comment 2 Jan Macku 2022-08-22 11:17:46 UTC
*** Bug 2120206 has been marked as a duplicate of this bug. ***

Comment 3 Plumber Bot 2022-08-22 11:21:33 UTC
fix merged to github main branch -> https://github.com/redhat-plumbers/systemd-rhel9/pull/101

Comment 8 errata-xmlrpc 2022-11-15 11:18:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (systemd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8327