Bug 2019329 - nosync.so is linked against GLIBC_2.34 symbols, it breaks mock
Summary: nosync.so is linked against GLIBC_2.34 symbols, it breaks mock
Keywords:
Status: ASSIGNED
Alias: None
Product: Fedora
Classification: Fedora
Component: nosync
Version: 40
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
Assignee: Mikolaj Izdebski
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2254328 (view as bug list)
Depends On:
Blocks: 2036722
TreeView+ depends on / blocked
 
Reported: 2021-11-02 08:51 UTC by Pavel Raiskup
Modified: 2024-02-16 07:23 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2036722 (view as bug list)
Environment:
Last Closed:
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Pavel Raiskup 2021-11-02 08:51:13 UTC
I'm not sure what changed, but the new dependency is breaking
Mock builds for older distributions (e.g. even Fedora 34 doesn't
provide 2.34 symbols).

$ cat ~/.config/mock.cfg | grep nosync
config_opts['nosync'] = True
$ mock -r fedora-34-x86_64 --shell
...
  Running scriptlet: ca-certificates-2021.2.50-1.0.fc34.noarch                                                                                                                                                                       83/134 
/bin/sh: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/tmp/tmp.mock.khvcffv5/lib64/nosync.so)
error: %prein(ca-certificates-2021.2.50-1.0.fc34.noarch) scriptlet failed, exit status 1

Error in PREIN scriptlet in rpm package ca-certificates
  Installing       : krb5-libs-1.19.2-2.fc34.x86_64                                                                                                                                                                                  84/134 
error: ca-certificates-2021.2.50-1.0.fc34.noarch: install failed
...
Failed:
  ca-certificates-2021.2.50-1.0.fc34.noarch                                                                                   tpm2-tss-3.1.0-1.fc34.x86_64                                                                                  

Error: Transaction failed

Comment 1 Tom Hughes 2021-11-02 08:54:43 UTC
There are two GLIBC_2.34 symbols that nosync.so is requiring:

0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  pthread_testcancel
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.34  dlsym

Comment 2 Mikolaj Izdebski 2021-11-02 09:24:16 UTC
I can reproduce this issue in mock with F34 chroot on F35 x86_64 host.

Package versions (on the F35 host):
mock-2.12-2.fc35.noarch
nosync-1.1-11.fc35.x86_64

Reproducer command:
mock -r rep.cfg --shell

Mock config:
config_opts['basedir'] = '/tmp/nosync-rep'
config_opts['cache_topdir'] = '/tmp/nosync-rep'
config_opts["rpmbuild_networking"] = False
config_opts["use_host_resolv"] = False
config_opts["isolation"] = "simple"
config_opts["use_bootstrap"] = False
config_opts["plugin_conf"]["yum_cache_enable"] = False
config_opts["plugin_conf"]["root_cache_enable"] = False
config_opts["plugin_conf"]["package_state_enable"] = False
config_opts["nosync"] = True
config_opts["nosync_force"] = True
config_opts["root"] = "nosync-rep"
config_opts["target_arch"] = "x86_64"
config_opts["chroot_setup_cmd"] = "install shadow-utils"

config_opts["yum.conf"] = """
[main]
keepcache=0
debuglevel=2
reposdir=/dev/null
gpgcheck=0
assumeyes=1
install_weak_deps=0
metadata_expire=-1

[f34]
name=f34
baseurl=https://ftp.icm.edu.pl/pub/Linux/dist/fedora/linux/releases/34/Everything/x86_64/os/
enabled=1
"""

Actual result
- several scriplet failures during mock chroot setup
- mock fails with the following error:
ERROR: Command failed: 
 # /usr/sbin/groupadd -g 135 mock

Disabling nosync in mock fixes the issue.

Comment 3 Pavel Raiskup 2021-11-02 11:40:16 UTC
> Disabling nosync in mock fixes the issue.

Yes.  Well: s/fixes the issue/makes the build succeed/  because we don't want to disable nosync in mock :-)

It seems that nosync.so+mock always worked this way (nosync from host used inside the target-chroot).
It's weird that this problem is happening for the first time right now.

Comment 4 Tom Hughes 2021-11-02 12:02:11 UTC
Well mock.so uses very few symbols from libc so the risk is low but glibc 2.34 has merged libdl into libc which means all dl* symbols have new versions as they are now in libc and dlsym is the one function that mock.so does have to use...

There's also a pthread symbol being referenced somehow which suffers from the same problem because libpthread has been merged into libc.

Comment 5 Pavel Raiskup 2021-12-06 06:55:30 UTC
Is there some plan for this issue?  What would happen if we built
the nosync module statically?

Comment 6 Tom Hughes 2021-12-06 07:03:04 UTC
Is this bug even in the right place? I mean nosync hasn't really done anything wrong - would it be possible for mock to add nosync to the buildroot (so that it comes from the right distribution) instead of injecting it from outside?

Comment 7 Pavel Raiskup 2021-12-06 08:54:09 UTC
> Is this bug even in the right place?

I believe it is.  That's how it was designed.

> would it be possible for mock to add nosync to the buildroot

Mock has been using nocache.so for _all_ chroots (when turned on).  But it
would be hard (or even impossible) to make the plugin available inside all
the supported chroots (consider also third-party configs people are using
daily on their own).

Technically it is possible.  Not a 5-minute work, but possible.

Comment 8 Mikolaj Izdebski 2021-12-06 15:47:58 UTC
The bug is correctly reported against nosync.
I need to find time to work on the issue. Thanks for bumping the bug.

My current plan is to try to link nosync against oldest versioned glibc symbols. I think that on older systems unversioned symbols from libdl/libpthread should be able to satisfy versioned requirements, but I'll need to confirm that. In general, I'll need to explore how exactly symbol versioning in glibc works.

Comment 9 Tom Hughes 2021-12-06 16:46:21 UTC
I don't think that is possible sadly - as far as I know the only way to do that is to have an older version of glibc-devel that has the old versions as the primary names.

Comment 10 Kevin Fenzi 2021-12-09 01:06:34 UTC
Just upgrading builders to f35 and hit this. ;( 

I guess I will disable nosync for now until we sort it out...

Comment 11 Pavel Raiskup 2021-12-15 10:19:54 UTC
This patch could help to link against 2.2.5 symbols:
https://copr-dist-git.fedorainfracloud.org/cgit/praiskup/nosync-libc-2.2.5/nosync.git/commit/?id=0bb4978918b7de10e2d8ea7c49ccc03bf94b3856

Though this fails in i386, and this library needs to be built multilib:
https://copr.fedorainfracloud.org/coprs/praiskup/nosync-libc-2.2.5/build/3048930/

....
cc -shared -fPIC -ldl -lpthread -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m32 -march=i686 -mtune=generic -msse2 -mfpmath=sse -mstackrealign -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -o nosync.so fsync.o open.o
/usr/bin/ld: nosync.so: no symbol version section for versioned symbol `dlsym.5'
collect2: error: ld returned 1 exit status

Even if the glibc versioning on i386 was fixed (or whatever is causing
this), this doesn't seem to be a long-term solution?

It would be nice if nosync was a glibc subpackage with statically
linked dlsym() and pthread_testcancel().  But this is probably unrealistic.

Comment 12 Miroslav Suchý 2022-02-18 20:46:05 UTC
Ping. Any progress here?

Comment 13 Nicolas Chauvet (kwizart) 2022-04-13 08:38:36 UTC
@pavel,
Have you reproduced this build error only on copr or also in koji scratch build ?
(is it related to how koji provide the glibc32 package ?)

Can we have this fix conditionally applied on x86_64 at least ?
EPEL doesn't build i686 packages, so we might not hit the issue there.

Comment 14 Nicolas Chauvet (kwizart) 2022-04-13 08:54:07 UTC
Side note:

readelf -a /usr/lib/libc.so.6 |grep dlsym
  3259: 00087340   192 FUNC    GLOBAL DEFAULT   15 dlsym
  3261: 00087340   192 FUNC    GLOBAL DEFAULT   15 dlsym@@GLIBC_2.34
   706: 00087310    33 FUNC    LOCAL  DEFAULT   15 dlsym_doit
  2421: 0015b9e0    75 FUNC    LOCAL  DEFAULT   15 do_dlsym
  2880: 0015bb30   169 FUNC    LOCAL  DEFAULT   15 __libc_dlsym
  4241: 00087340   192 FUNC    LOCAL  DEFAULT   15 ___dlsym
  7007: 00087340   192 FUNC    GLOBAL DEFAULT   15 dlsym
  8429: 00087340   192 FUNC    GLOBAL DEFAULT   15 dlsym@@GLIBC_2.34

So for x86, the symbol version is 2.0, not 2.2.5
(it might be different for others architectures).

Comment 15 Ben Cotton 2022-11-29 17:12:38 UTC
This message is a reminder that Fedora Linux 35 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 35 on 2022-12-13.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '35'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 35 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 16 Nicolas Chauvet (kwizart) 2022-11-30 08:15:42 UTC
I wonder if it would be possible to solve this problem by building nosync using an el8 (or even el7) target and later tag it for newer fedora releases ?
(that would be easier than messing with hardcoded symbols version that could be different for each architecture.)


Of course this break assumption for all fedora build, I'm not sure FESCO could allow such.
It will open it's can of worms for sure.

Alternative would be to provide it into the infra-tag koji repo for infra (and outside infra) consumption...

Comment 17 Miroslav Suchý 2022-12-12 09:43:49 UTC
bump bump. Please.

BTW one way how to fix this is to build it statically https://docs.fedoraproject.org/en-US/packaging-guidelines/#packaging-static-libraries

Comment 18 Nicolas Chauvet (kwizart) 2022-12-12 10:14:08 UTC
@praiskup 
Can you fix the patch to pick the 2.0 dlsym symbol for i386 ? that seems the easier way forward...
See also https://bugzilla.redhat.com/show_bug.cgi?id=2019329#c14

> BTW one way how to fix this is to build it statically https://docs.fedoraproject.org/en-US/packaging-guidelines/#packaging-static-libraries
@msuchy as #c14 it doesn't sym that easy as the needed symbols aren't part of the static libc, as I understand.

Comment 19 Miroslav Suchý 2022-12-12 14:35:20 UTC
> Can you fix the patch 

We cannot. I and Pavel - we are reporters. Mikolaj is the one who can fix this.

Comment 20 Pavel Raiskup 2022-12-15 01:55:54 UTC
(In reply to Nicolas Chauvet (kwizart) from comment #18)
> @praiskup 
> Can you fix the patch to pick the 2.0 dlsym symbol for i386 ? that seems the
> easier way forward...

I tried, but I realized that the nosync.so doesn't work correctly with EL7
chroots (while with nosync.so built on F34 everything works fine):

  Installing : gawk-4.0.2-4.el7_3.1.x86_64   46/172 
/sbin/install-info: relocation error: /var/tmp/tmp.mock.2qns6qkr/lib64/nosync.so: symbol dlsym, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference
  Installing : diffutils-3.3-5.el7.x86_64    47/172 
/sbin/install-info: relocation error: /var/tmp/tmp.mock.2qns6qkr/lib64/nosync.so: symbol dlsym, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference
  Installing : 1:findutils-4.5.11-6.el7.x86_64  48/172 
/sbin/install-info: relocation error: /var/tmp/tmp.mock.2qns6qkr/lib64/nosync.so: symbol dlsym, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference
  Installing : cpio-2.11-28.el7.x86_64 49/172 
/sbin/install-info: relocation error: /var/tmp/tmp.mock.2qns6qkr/lib64/nosync.so: symbol dlsym, version GLIBC_2.2.5 not defined in file libc.so.6 with link time reference

Feel free to experiment with nosync-1.1-13.1.copr5146043
packages (x86_64 and i686) from `dnf copr enable praiskup/nosync-libc-2.2.5`.

The updated patch is here:
https://copr-dist-git.fedorainfracloud.org/cgit/praiskup/nosync-libc-2.2.5/nosync.git/tree/link-against-old-symbols.patch?id=ffdd562af15f182f3c744d4963861d6e7c9e255d

Comment 21 Jan Kratochvil 2023-05-24 09:39:11 UTC
Still a problem for: nosync-1.1-14.fc38.x86_64

Comment 22 Aoife Moloney 2023-11-23 00:06:52 UTC
This message is a reminder that Fedora Linux 37 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora Linux 37 on 2023-12-05.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
'version' of '37'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, change the 'version' 
to a later Fedora Linux version. Note that the version field may be hidden.
Click the "Show advanced fields" button if you do not see it.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora Linux 37 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora Linux, you are encouraged to change the 'version' to a later version
prior to this bug being closed.

Comment 23 Aoife Moloney 2024-02-15 22:54:18 UTC
This bug appears to have been reported against 'rawhide' during the Fedora Linux 40 development cycle.
Changing version to 40.

Comment 24 Pavel Raiskup 2024-02-16 06:43:50 UTC
*** Bug 2254328 has been marked as a duplicate of this bug. ***

Comment 25 Nicolas Chauvet (kwizart) 2024-02-16 07:23:30 UTC
The current workaround for this bug is to build nosync with epel7 (aka minimal build target) and use it on newer fedora builders.


Note You need to log in before you can comment on or make changes to this bug.