Bug 2034715

Hmm, so the code it's ostensibly failing on just does:

    from ctypes.util import find_library
    find_library("udev")

So it'd be interesting to know if just doing that on a Rawhide armv7 box fails. The fact that it's apparently happening during some sort of teardown, though, does mean it could be something ickier and...Python-ier. Maybe we should pull in Miro for that angle, CCing.

Marking as an accepted blocker as this hits the automatic blocker policy:

"Bugs which entirely prevent the composition of one or more of the release-blocking images required to be built for a currently-pending (pre-)release"

https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process#Automatic_blockers

So this broke between 20211217.n.1 and 20211219.n.0 it seems (17.n.1 completed, 19.n.0 failed, there doesn't appear to have been an 18.n.0 compose at all). libudev is in systemd-libs, and there was a new systemd build on 2021-12-18, though at first blush the changes don't seem very relevant...

https://koji.fedoraproject.org/koji/buildinfo?buildID=1868311

Yeah, they really don't. ;( I suppose it could be a compiler or tool chain issue? 
I could try untagging all the systemd's since the one in 20211217.n.1 and re-run?
It would be nice to know if that fixed it first tho.

Adding zbyszek

I'm not aware of any changes that could be relevant. Upstream we did some tweaks to the
build system, but nothing that would touch libudev directly. Downstream also nothing relevant.
And abidiff says that there have been no ABI changes in systemd-250.
In mock on armv7-test01.fp.o the results are the same, and commands from comment #1 above work just fine.
'from blivet import udev' also works.

Hmm, how would one go about doing a "scratch image build" ?

huh, so I tried another scratch image build with: 

koji image-build Fedora-Minimal-Rawhide 20211225.n.0.kevin f36 "https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20211225.n.0/compose/Everything/armhfp/os" --disk-size 4 --distro Fedora-30 --format raw-xz --ksurl "git+https://pagure.io/fedora-kickstarts.git?#1bfe603576516ee1396d89e3e0f4f9113fa5e20a" --kickstart fedora-disk-minimal.ks --release 20211225.n.0.kevin --scratch armhfp

But I think we restricted all image builds to admins. ;( also, in order to get the vm console you have to go to the builder and virsh console, which needs root on builders. ;( 

Anyhow, now the image fails sooner. It can't start dbus and errors out. 

[Wed Dec 22 23:59:59 2021] systemd[1]: Detected virtualization kvm.                                 
[Wed Dec 22 23:59:59 2021] systemd[1]: Detected architecture arm.                                    
[Wed Dec 22 23:59:59 2021] systemd[1]: Running in initial RAM disk.                                 
[Wed Dec 22 23:59:59 2021] systemd[1]: No hostname configured, using default hostname.               
[Wed Dec 22 23:59:59 2021] systemd[1]: Hostname set to <fedora>.                                     
[Wed Dec 22 23:59:59 2021] systemd[1]: Initializing machine ID from VM UUID.                         
[Wed Dec 22 23:59:59 2021] hrtimer: interrupt took 10982525 ns                                       
[Wed Dec 22 23:59:59 2021] systemd-gpt-auto-generator[213]: EFI loader partition unknown, exiting.   
[Wed Dec 22 23:59:59 2021] systemd-gpt-auto-generator[213]: (The boot loader did not set EFI variable
 LoaderDevicePartUUID.)                                                                             
[Thu Dec 23 00:00:00 2021] systemd[1]: Queued start job for default target initrd.target.            
[Thu Dec 23 00:00:00 2021] systemd[1]: Reached target initrd-root-device.target - Initrd Root Device.
[Thu Dec 23 00:00:00 2021] systemd[1]: Reached target initrd-usr-fs.target - Initrd /usr File System.
[Thu Dec 23 00:00:00 2021] systemd[1]: Reached target slices.target - Slice Units.
[Thu Dec 23 00:00:00 2021] systemd[1]: Reached target swap.target - Swaps.
[Thu Dec 23 00:00:00 2021] systemd[1]: Reached target timers.target - Timer Units.
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on dbus.socket - D-Bus System Message Bus Socket.
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on iscsid.socket - Open-iSCSI iscsid Socket.
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on iscsiuio.socket - Open-iSCSI iscsiuio Socket.
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-journald-audit.socket - Journal Audit Soc
ket.
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-journald-dev-log.socket - Journal Socket
(/dev/log).
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-journald.socket - Journal Socket.
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socke
t.
[Thu Dec 23 00:00:00 2021] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket.
[Thu Dec 23 00:00:00 2021] systemd[1]: Reached target sockets.target - Socket Units.
[Thu Dec 23 00:00:00 2021] systemd[1]: Starting kmod-static-nodes.service - Create List of Static Dev
ice Nodes...
[Thu Dec 23 00:00:00 2021] systemd[1]: Started rngd.service - Hardware RNG Entropy Gatherer Daemon.
[Thu Dec 23 00:00:00 2021] systemd[1]: Starting systemd-journald.service - Journal Service...
[Thu Dec 23 00:00:00 2021] systemd[1]: systemd-modules-load.service - Load Kernel Modules was skipped
 because all trigger condition checks failed.
[Thu Dec 23 00:00:00 2021] systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[Thu Dec 23 00:00:00 2021] systemd[1]: systemd-sysusers.service - Create System Users was skipped bec
ause of a failed condition check (ConditionNeedsUpdate=/etc).
[Thu Dec 23 00:00:00 2021] systemd[1]: Starting systemd-vconsole-setup.service - Setup Virtual Consol
e...
[Thu Dec 23 00:00:00 2021] systemd[1]: Finished kmod-static-nodes.service - Create List of Static Dev
ice Nodes.
[Thu Dec 23 00:00:00 2021] systemd[1]: systemd-journald.service: Main process exited, code=exited, st
atus=127/n/a
[Thu Dec 23 00:00:00 2021] systemd[1]: systemd-journald.service: Failed with result 'exit-code'.
[Thu Dec 23 00:00:00 2021] systemd[1]: Failed to start systemd-journald.service - Journal Service.
...
[Thu Dec 23 00:17:28 2021] systemd[1]: Listening on dbus.socket - D-Bus System Message Bus Socket.  
[Thu Dec 23 00:17:28 2021] systemd[1]: Starting dbus-broker.service - D-Bus System Message Bus...   
[Thu Dec 23 00:17:28 2021] systemd[1]: dbus-broker.service: Main process exited, code=exited, status=1/FAILURE
[Thu Dec 23 00:17:28 2021] systemd[1]: dbus-broker.service: Failed with result 'exit-code'.         
[Thu Dec 23 00:17:28 2021] systemd[1]: Failed to start dbus-broker.service - D-Bus System Message Bus.


Could this be https://github.com/systemd/systemd/issues/21665 ? perhaps this landed after the initial anaconda/udev thing, but we likely need to fix it before we can see the initial problem again. And I don't understand why it only happens on armv7... ;( 

Happy to try and gather more debug info...

In the snippet, systemd-journald.service fails first. It runs as root and I don't think it has any
dependence on non-root users. So I think this must be something else. dbus-broker also runs
as root, and shouldn't really care about sysusers. So I don't think 21665 is related.
(Also, dracut didn't install sysusers binary afaik, so for our initrd's those changes are not relevant.)

status=127 is strange.

I'd say that failure to start systemd-journald and dbus-broker looks like it could be selinux.

(N.B.: isn't the new output format nicer to read?)

Exception ignored in: <function Context.__del__ at 0xb29a67c0>
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pyudev/core.py", line 65, in __del__
    self._libudev.udev_unref(self)
AttributeError: 'Context' object has no attribute '_libudev'

That part is https://github.com/pyudev/pyudev/issues/421



The main issue seems like https://github.com/pyudev/pyudev/issues/83 to me.

https://docs.python.org/3/library/ctypes.html#finding-shared-libraries ays that:

> On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, objdump and ld) to find the library file.

Is it possible that started to happen when one or more tools that find_library() uses under the hood stopped being available and hence it falbacks to another one (or the other way around) and hence hitting the ARM issue as described in https://bugs.python.org/issue13508 ?

So, I think we are dealing with 2 issues here now. The orig pyudev thing and a new systemd-journald doesn't start issue.
I filed https://bugzilla.redhat.com/show_bug.cgi?id=2035812 on the systemd-journald / dbus starting issue. Lets keep this for the pyudev issue.

So, on the udev thing... 

it seems like ldconfig isn't working right: 

[anaconda root@fedora ~]# python3
Python 3.10.1 (main, Dec  7 2021, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> from ctypes.util import find_library
>>> find_library("udev")
>>>
[anaconda root@fedora ~]# ldconfig -p
[anaconda root@fedora ~]# 

There's a cache file, but no output from ldconfig -p.... none of gcc, objdump or ld is installed.

Moving to glibc for comment on the ldconfig -p on 32bit arm not working...

I'm trying a new compose with glibc-2.34.9000-33.fc36 untagged to see if thats indeed the issue.

I see you're now stuck in a game of Compose Dependency Whack-A-Mole?

So, untagging zstd-1.5.1-1.fc36 gets us past that bug and systemd boots things fine again. 

Now we are left with the orig bug here which I think is due to ldconfig -p on armv7 not working, causing anaconda to not be able to import the libudev library via pyudev.

I cannot easily untag glibc-2.34.9000-33.fc36 because a number of things have been built with it and they seem to have >= requirements for it. (Althought perhaps thats only firefox?)

Does ldconfig -p show some output if your run ldconfig yourself?

(In reply to Florian Weimer from comment #16)
> Does ldconfig -p show some output if your run ldconfig yourself?

Nope. If I spin a scratch koji image build, go to the builder, virsh console into the install, go to the second tty:

[anaconda root@fedora ~]# ldconfig -p
[anaconda root@fedora ~]# 
[anaconda root@fedora ~]# ldconfig -V
ldconfig (GNU libc) 2.34.9000
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Andreas Jaeger.

What about ldconfig -v? Does it find some objects? And does ldconfig -p show anything afterwards? Thanks.

glibc-2.34.9000-32.fc36.armv7hl works fine, glibc-2.34.9000-33.fc36.armv7hl breaks.

Tested in fedora-rawhide-armhfp mock on armv7-test02.fedorainfracloud.org )you can do that as well).

<mock-chroot> sh-5.1$ ldconfig -p
<mock-chroot> sh-5.1$ ldconfig -v
ldconfig: Path `/usr/lib' given more than once
(from <builtin>:0 and <builtin>:0)
/lib: (from <builtin>:0)
	libxxhash.so.0 -> libxxhash.so.0.8.1
	librpmbuild.so.9 -> librpmbuild.so.9.2.0
	libopcodes-2.37-22.fc36.so -> libopcodes-2.37-22.fc36.so
	libctf.so.0 -> libctf.so.0.0.0
	libctf-nobfd.so.0 -> libctf-nobfd.so.0.0.0
	libbfd-2.37-22.fc36.so -> libbfd-2.37-22.fc36.so
	libdw.so.1 -> libdw-0.186.so
	libasm.so.1 -> libasm-0.186.so
	libdebuginfod.so.1 -> libdebuginfod-0.186.so
	libdevmapper.so.1.02 -> libdevmapper.so.1.02
	libcryptsetup.so.12 -> libcryptsetup.so.12.7.0
	libauthselect.so.3 -> libauthselect.so.3.1.2
	libpamc.so.0 -> libpamc.so.0.82.1
	libpam_misc.so.0 -> libpam_misc.so.0.82.1
	libpam.so.0 -> libpam.so.0.85.1
	libpwquality.so.1 -> libpwquality.so.1.0.2
	librpmio.so.9 -> librpmio.so.9.2.0
	libnghttp2.so.14 -> libnghttp2.so.14.21.1
	libkmod.so.2 -> libkmod.so.2.3.7
	libgcrypt.so.20 -> libgcrypt.so.20.3.4
	libverto.so.1 -> libverto.so.1.0.0
	libgpg-error.so.0 -> libgpg-error.so.0.32.1
	librpm.so.9 -> librpm.so.9.2.0
	libnss_resolve.so.2 -> libnss_resolve.so.2
	libgomp.so.1 -> libgomp.so.1.0.0
	libp11-kit.so.0 -> libp11-kit.so.0.3.0
	libgssrpc.so.4 -> libgssrpc.so.4.2
	libffi.so.6 -> libffi.so.6.0.2
	libeconf.so.0 -> libeconf.so.0.4.0
	liblber-2.4.so.2 -> liblber-2.4.so.2.11.7
	libnss_myhostname.so.2 -> libnss_myhostname.so.2
	libdb-5.3.so -> libdb-5.3.so
	libkdb5.so.10 -> libkdb5.so.10.0
	libbrotlienc.so.1 -> libbrotlienc.so.1.0.9
	libbrotlidec.so.1 -> libbrotlidec.so.1.0.9
	libbrotlicommon.so.1 -> libbrotlicommon.so.1.0.9
	libcrypto.so.3 -> libcrypto.so.3.0.0
	libpcreposix.so.0 -> libpcreposix.so.0.0.7
	libpkgconf.so.3 -> libpkgconf.so.3.0.0
	libargon2.so.1 -> libargon2.so.1
	libsasl2.so.3 -> libsasl2.so.3.0.0
	libkeyutils.so.1 -> libkeyutils.so.1.9
	libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2
	libjson-c.so.5 -> libjson-c.so.5.1.0
	libtirpc.so.3 -> libtirpc.so.3.0.0
	libgdbm_compat.so.4 -> libgdbm_compat.so.4.0.0
	libgdbm.so.6 -> libgdbm.so.6.0.0
	libkrb5support.so.0 -> libkrb5support.so.0.1
	libudev.so.1 -> libudev.so.1.7.3
	p11-kit-trust.so -> libnssckbi.so
	libseccomp.so.2 -> libseccomp.so.2.5.3
	libexpat.so.1 -> libexpat.so.1.8.1
	libfdisk.so.1 -> libfdisk.so.1.1.0
	libarchive.so.13 -> libarchive.so.13.5.2
	libssh.so.4 -> libssh_threads.so.4.8.7
	libkrad.so.0 -> libkrad.so.0.0
	libssl.so.1.1 -> libssl.so.1.1.1l
	libcrypto.so.1.1 -> libcrypto.so.1.1.1l
	libldap_r-2.4.so.2 -> libldap_r-2.4.so.2.11.7
	libssl.so.3 -> libssl.so.3.0.0
	libpcre.so.1 -> libpcre.so.1.2.13
	libsigsegv.so.2 -> libsigsegv.so.2.0.6
	libxml2.so.2 -> libxml2.so.2.9.12
	libmount.so.1 -> libmount.so.1.1.0
	libsystemd.so.0 -> libsystemd.so.0.33.0
	libnsl.so.3 -> libnsl.so.3.0.0
	libk5crypto.so.3 -> libk5crypto.so.3.1
	libutempter.so.0 -> libutempter.so.1.2.1
	libcurl.so.4 -> libcurl.so.4.7.0
	libkrb5.so.3 -> libkrb5.so.3.3
	libblkid.so.1 -> libblkid.so.1.1.0
	libsemanage.so.2 -> libsemanage.so.2
	libslapi-2.4.so.2 -> libslapi-2.4.so.2.11.7
	libselinux.so.1 -> libselinux.so.1
	libcrack.so.2 -> libcrack.so.2.9.0
	libnss_systemd.so.2 -> libnss_systemd.so.2
	libtasn1.so.6 -> libtasn1.so.6.6.2
	libidn2.so.0 -> libidn2.so.0.3.7
	libmpfr.so.6 -> libmpfr.so.6.1.0
	libauparse.so.0 -> libauparse.so.0.0.0
	libdrop_ambient.so.0 -> libdrop_ambient.so.0.0.0
	libcap-ng.so.0 -> libcap-ng.so.0.0.0
	liblz4.so.1 -> liblz4.so.1.9.3
	libcom_err.so.2 -> libcom_err.so.2.1
	libaudit.so.1 -> libaudit.so.1.0.0
	libreadline.so.8 -> libreadline.so.8.1
	libhistory.so.8 -> libhistory.so.8.1
	libpsx.so.2 -> libpsx.so.2.48
	libunistring.so.2 -> libunistring.so.2.1.0
	liblua-5.4.so -> liblua-5.4.so
	liblzma.so.5 -> liblzma.so.5.2.5
	libsmartcols.so.1 -> libsmartcols.so.1.1.0
	libtinfo.so.6 -> libtinfo.so.6.2
	libtic.so.6 -> libtic.so.6.2
	libpanelw.so.6 -> libpanelw.so.6.2
	libpanel.so.6 -> libpanel.so.6.2
	libncursesw.so.6 -> libncursesw.so.6.2
	libpcprofile.so -> libpcprofile.so
	libmemusage.so -> libmemusage.so
	libutil.so.1 -> libutil.so.1
	libthread_db.so.1 -> libthread_db.so.1
	librt.so.1 -> librt.so.1
	libresolv.so.2 -> libresolv.so.2
	libpthread.so.0 -> libpthread.so.0
	libnss_files.so.2 -> libnss_files.so.2
	libnss_dns.so.2 -> libnss_dns.so.2
	libnss_compat.so.2 -> libnss_compat.so.2
	libm.so.6 -> libm.so.6
	libdl.so.2 -> libdl.so.2
	libc_malloc_debug.so.0 -> libc_malloc_debug.so.0
	libc.so.6 -> libc.so.6
	libanl.so.1 -> libanl.so.1
	libSegFault.so -> libSegFault.so
	libBrokenLocale.so.1 -> libBrokenLocale.so.1
	ld-linux-armhf.so.3 -> ld-linux-armhf.so.3
	libpsl.so.5 -> libpsl.so.5.3.3
	libacl.so.1 -> libacl.so.1.1.2301
	libmagic.so.1 -> libmagic.so.1.0.0
	libattr.so.1 -> libattr.so.1.1.2501
	libsepol.so.2 -> libsepol.so.2
	libgmp.so.10 -> libgmp.so.10.4.1
	libuuid.so.1 -> libuuid.so.1.3.0
	libcap.so.2 -> libcap.so.2.48
	libpcre2-posix.so.3 -> libpcre2-posix.so.3.0.1
	libzstd.so.1 -> libzstd.so.1.5.0
	libelf.so.1 -> libelf-0.186.so
	libbz2.so.1 -> libbz2.so.1.0.8
	libpcre2-8.so.0 -> libpcre2-8.so.0.10.4
	libz.so.1 -> libz.so.1.2.11
	libncurses.so.6 -> libncurses.so.6.2
	libmenuw.so.6 -> libmenuw.so.6.2
	libmenu.so.6 -> libmenu.so.6.2
	libformw.so.6 -> libformw.so.6.2
	libform.so.6 -> libform.so.6.2
	libgcc_s.so.1 -> libgcc_s-11-20211203.so.1
	libstdc++.so.6 -> libstdc++.so.6.0.29
	liblua-5.3.so -> liblua-5.3.so
	libsqlite3.so.0 -> libsqlite3.so.0.8.6
	libpopt.so.0 -> libpopt.so.0.0.1
	libcrypt.so.2 -> libcrypt.so.2.0.0
ldconfig: Can't create temporary cache file /etc/ld.so.cache~: Permission denied
<mock-chroot> sh-5.1$ ldconfig -p
<mock-chroot> sh-5.1$ 



With root:

<mock-chroot> sh-5.1# ldconfig -p
<mock-chroot> sh-5.1# ldconfig -v
ldconfig: Path `/usr/lib' given more than once
(from <builtin>:0 and <builtin>:0)
/lib: (from <builtin>:0)
	libxxhash.so.0 -> libxxhash.so.0.8.1
	librpmbuild.so.9 -> librpmbuild.so.9.2.0
	libopcodes-2.37-22.fc36.so -> libopcodes-2.37-22.fc36.so
	libctf.so.0 -> libctf.so.0.0.0
	libctf-nobfd.so.0 -> libctf-nobfd.so.0.0.0
	libbfd-2.37-22.fc36.so -> libbfd-2.37-22.fc36.so
	libdw.so.1 -> libdw-0.186.so
	libasm.so.1 -> libasm-0.186.so
	libdebuginfod.so.1 -> libdebuginfod-0.186.so
	libdevmapper.so.1.02 -> libdevmapper.so.1.02
	libcryptsetup.so.12 -> libcryptsetup.so.12.7.0
	libauthselect.so.3 -> libauthselect.so.3.1.2
	libpamc.so.0 -> libpamc.so.0.82.1
	libpam_misc.so.0 -> libpam_misc.so.0.82.1
	libpam.so.0 -> libpam.so.0.85.1
	libpwquality.so.1 -> libpwquality.so.1.0.2
	librpmio.so.9 -> librpmio.so.9.2.0
	libnghttp2.so.14 -> libnghttp2.so.14.21.1
	libkmod.so.2 -> libkmod.so.2.3.7
	libgcrypt.so.20 -> libgcrypt.so.20.3.4
	libverto.so.1 -> libverto.so.1.0.0
	libgpg-error.so.0 -> libgpg-error.so.0.32.1
	librpm.so.9 -> librpm.so.9.2.0
	libnss_resolve.so.2 -> libnss_resolve.so.2
	libgomp.so.1 -> libgomp.so.1.0.0
	libp11-kit.so.0 -> libp11-kit.so.0.3.0
	libgssrpc.so.4 -> libgssrpc.so.4.2
	libffi.so.6 -> libffi.so.6.0.2
	libeconf.so.0 -> libeconf.so.0.4.0
	liblber-2.4.so.2 -> liblber-2.4.so.2.11.7
	libnss_myhostname.so.2 -> libnss_myhostname.so.2
	libdb-5.3.so -> libdb-5.3.so
	libkdb5.so.10 -> libkdb5.so.10.0
	libbrotlienc.so.1 -> libbrotlienc.so.1.0.9
	libbrotlidec.so.1 -> libbrotlidec.so.1.0.9
	libbrotlicommon.so.1 -> libbrotlicommon.so.1.0.9
	libcrypto.so.3 -> libcrypto.so.3.0.0
	libpcreposix.so.0 -> libpcreposix.so.0.0.7
	libpkgconf.so.3 -> libpkgconf.so.3.0.0
	libargon2.so.1 -> libargon2.so.1
	libsasl2.so.3 -> libsasl2.so.3.0.0
	libkeyutils.so.1 -> libkeyutils.so.1.9
	libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2
	libjson-c.so.5 -> libjson-c.so.5.1.0
	libtirpc.so.3 -> libtirpc.so.3.0.0
	libgdbm_compat.so.4 -> libgdbm_compat.so.4.0.0
	libgdbm.so.6 -> libgdbm.so.6.0.0
	libkrb5support.so.0 -> libkrb5support.so.0.1
	libudev.so.1 -> libudev.so.1.7.3
	p11-kit-trust.so -> libnssckbi.so
	libseccomp.so.2 -> libseccomp.so.2.5.3
	libexpat.so.1 -> libexpat.so.1.8.1
	libfdisk.so.1 -> libfdisk.so.1.1.0
	libarchive.so.13 -> libarchive.so.13.5.2
	libssh.so.4 -> libssh_threads.so.4.8.7
	libkrad.so.0 -> libkrad.so.0.0
	libssl.so.1.1 -> libssl.so.1.1.1l
	libcrypto.so.1.1 -> libcrypto.so.1.1.1l
	libldap_r-2.4.so.2 -> libldap_r-2.4.so.2.11.7
	libssl.so.3 -> libssl.so.3.0.0
	libpcre.so.1 -> libpcre.so.1.2.13
	libsigsegv.so.2 -> libsigsegv.so.2.0.6
	libxml2.so.2 -> libxml2.so.2.9.12
	libmount.so.1 -> libmount.so.1.1.0
	libsystemd.so.0 -> libsystemd.so.0.33.0
	libnsl.so.3 -> libnsl.so.3.0.0
	libk5crypto.so.3 -> libk5crypto.so.3.1
	libutempter.so.0 -> libutempter.so.1.2.1
	libcurl.so.4 -> libcurl.so.4.7.0
	libkrb5.so.3 -> libkrb5.so.3.3
	libblkid.so.1 -> libblkid.so.1.1.0
	libsemanage.so.2 -> libsemanage.so.2
	libslapi-2.4.so.2 -> libslapi-2.4.so.2.11.7
	libselinux.so.1 -> libselinux.so.1
	libcrack.so.2 -> libcrack.so.2.9.0
	libnss_systemd.so.2 -> libnss_systemd.so.2
	libtasn1.so.6 -> libtasn1.so.6.6.2
	libidn2.so.0 -> libidn2.so.0.3.7
	libmpfr.so.6 -> libmpfr.so.6.1.0
	libauparse.so.0 -> libauparse.so.0.0.0
	libdrop_ambient.so.0 -> libdrop_ambient.so.0.0.0
	libcap-ng.so.0 -> libcap-ng.so.0.0.0
	liblz4.so.1 -> liblz4.so.1.9.3
	libcom_err.so.2 -> libcom_err.so.2.1
	libaudit.so.1 -> libaudit.so.1.0.0
	libreadline.so.8 -> libreadline.so.8.1
	libhistory.so.8 -> libhistory.so.8.1
	libpsx.so.2 -> libpsx.so.2.48
	libunistring.so.2 -> libunistring.so.2.1.0
	liblua-5.4.so -> liblua-5.4.so
	liblzma.so.5 -> liblzma.so.5.2.5
	libsmartcols.so.1 -> libsmartcols.so.1.1.0
	libtinfo.so.6 -> libtinfo.so.6.2
	libtic.so.6 -> libtic.so.6.2
	libpanelw.so.6 -> libpanelw.so.6.2
	libpanel.so.6 -> libpanel.so.6.2
	libncursesw.so.6 -> libncursesw.so.6.2
	libpcprofile.so -> libpcprofile.so
	libmemusage.so -> libmemusage.so
	libutil.so.1 -> libutil.so.1
	libthread_db.so.1 -> libthread_db.so.1
	librt.so.1 -> librt.so.1
	libresolv.so.2 -> libresolv.so.2
	libpthread.so.0 -> libpthread.so.0
	libnss_files.so.2 -> libnss_files.so.2
	libnss_dns.so.2 -> libnss_dns.so.2
	libnss_compat.so.2 -> libnss_compat.so.2
	libm.so.6 -> libm.so.6
	libdl.so.2 -> libdl.so.2
	libc_malloc_debug.so.0 -> libc_malloc_debug.so.0
	libc.so.6 -> libc.so.6
	libanl.so.1 -> libanl.so.1
	libSegFault.so -> libSegFault.so
	libBrokenLocale.so.1 -> libBrokenLocale.so.1
	ld-linux-armhf.so.3 -> ld-linux-armhf.so.3
	libpsl.so.5 -> libpsl.so.5.3.3
	libacl.so.1 -> libacl.so.1.1.2301
	libmagic.so.1 -> libmagic.so.1.0.0
	libattr.so.1 -> libattr.so.1.1.2501
	libsepol.so.2 -> libsepol.so.2
	libgmp.so.10 -> libgmp.so.10.4.1
	libuuid.so.1 -> libuuid.so.1.3.0
	libcap.so.2 -> libcap.so.2.48
	libpcre2-posix.so.3 -> libpcre2-posix.so.3.0.1
	libzstd.so.1 -> libzstd.so.1.5.0
	libelf.so.1 -> libelf-0.186.so
	libbz2.so.1 -> libbz2.so.1.0.8
	libpcre2-8.so.0 -> libpcre2-8.so.0.10.4
	libz.so.1 -> libz.so.1.2.11
	libncurses.so.6 -> libncurses.so.6.2
	libmenuw.so.6 -> libmenuw.so.6.2
	libmenu.so.6 -> libmenu.so.6.2
	libformw.so.6 -> libformw.so.6.2
	libform.so.6 -> libform.so.6.2
	libgcc_s.so.1 -> libgcc_s-11-20211203.so.1
	libstdc++.so.6 -> libstdc++.so.6.0.29
	liblua-5.3.so -> liblua-5.3.so
	libsqlite3.so.0 -> libsqlite3.so.0.8.6
	libpopt.so.0 -> libpopt.so.0.0.1
	libcrypt.so.2 -> libcrypt.so.2.0.0
<mock-chroot> sh-5.1# ldconfig -p
<mock-chroot> sh-5.1#

(my laptop)$ strace ldconfig -p
...
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=246095, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 246095, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f638283b000
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
newfstatat(1, "", {st_mode=S_IFIFO|0600, st_size=0, ...}, AT_EMPTY_PATH) = 0
write(1, "4788 libs found in cache `/etc/l"..., 40964788 libs found in cache `/etc/ld.so.cache'
        p11-kit-trust.so (libc6,x86-64) => /lib64/p11-kit-trust.so
        lib3mf.so.2 (libc6,x86-64) => /lib64/lib3mf.so.2
        libz3.so.4.8 (libc6,x86-64) => /lib64/libz3.so.4.8
        libzzipwrap-0.so.13 (libc6,x86-64) => /lib64/libzzipwrap-0.so.13
        libzzipmmapped-0.so.13 (libc6,x86-64) => /lib64/libzzipmmapped-0.so.13
        libzzipfseeko-0.so.13 (libc6,x86-64) => /lib64/libzzipfseeko-0.so.13

(mock-fedora-rawhide-armh7vl)$ strace ldconfig -p
...
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=14189, ...}) = 0
close(3)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

$ ls -l /etc/ld.so.cache 
-rw-r--r--. 1 root root 14189 Dec 28 19:14 /etc/ld.so.cache

On i386, I see this for ldconfig -p:

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE) = 3
statx(3, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=6669, ...}) = 0
mmap2(NULL, 0, PROT_READ, MAP_PRIVATE, 3, 0) = -1 EINVAL (Invalid argument)

(glibc-2.34.9000-33.fc36.i686)

It looks like something goes wrong inside glibc between the statx system call and the struct stat64 result from fstat64.

I'll revert this commit:

commit a6d2f948b71adcb5ea395cb04833bc645eab45e6
Author: Adhemerval Zanella <adhemerval.zanella>
Date:   Thu May 20 14:20:18 2021 -0300

    Use LFS and 64 bit time for installed programs (BZ #15333)

Raised here upstream: https://sourceware.org/pipermail/libc-alpha/2021-December/134716.html

Many thanks for the quick revert!

I suppose this was fixed.

On the subject of ldconfig -p output being empty on armv7, I found this bug.

It was broken with glibc-2.34.9000-33.fc36.armv7hl but it appears to be fixed
with glibc-2.34.9000-34.fc36.armv7hl

Sounds like we're OK here now. Re-open if anything remains.