Bug 1946518

Summary: binutils-2.30-98 are causing go binaries to crash due to segmentation fault on aarch64
Product: Red Hat Enterprise Linux 8 Reporter: Ondřej Budai <obudai>
Component: binutilsAssignee: Nick Clifton <nickc>
binutils sub component: system-version QA Contact: Miloš Prchlík <mprchlik>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: bstinson, carl, drjones, fweimer, jcajka, jwboyer, Maurice.Smulders, mprchlik, obudai, ohudlick, rjones, skolosov
Version: CentOS StreamKeywords: Bugfix, Reopened, Triaged
Target Milestone: beta   
Target Release: ---   
Hardware: aarch64   
OS: Unspecified   
Whiteboard:
Fixed In Version: binutils-2.30-100.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 19:31:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ondřej Budai 2021-04-06 10:24:34 UTC
Description of problem:
When I run go test on the latest CentOS Stream, the compiled test binaries fails with "signal: segmentation fault (core dumped)".


Version-Release number of selected component (if applicable):
binutils-2.30-98

How reproducible:
Always


Steps to Reproduce (aarch64 only!):
dnf install -y git-core golang
git clone https://github.com/osbuild/osbuild-composer.git
cd osbuild-composer
go test ./...

Actual results:
The tests fail with multiple "signal: segmentation fault (core dumped)" errors.


Expected results:
The tests pass.


Additional info:
I installed binutils-2.30-99 from the latest RHEL 8.5 nightly and the issue disappeared, therefore I'm fairly certain that the issue is actually in binutils.

Comment 1 Nick Clifton 2021-04-06 11:49:19 UTC
This is a known bug in the 2.30-98.el8 binutils.  Fortunately it has been fixed in the 2.30-99.el8 binutils, which is now in the buildroot for RHEL 8.

Comment 2 Ondřej Budai 2021-04-06 11:55:47 UTC
I agree but I opened this bug against CentOS Stream where this bug still exists.

I wonder what's the correct way to track bugs in CentOS Stream - my gut feeling is that CLOSED status isn't appropriate when the bug is still present.

Comment 3 Nick Clifton 2021-04-06 12:16:23 UTC
Hi Ondřej,

  Sorry about closing this BZ - I looked at the Product field and skipped the Version field in the header...

  I am unfamiliar with how the RHEL-8 version of CentOS Stream gets its sources.  Does it inherit from the RHEL-8.5.0 branches ?  If so, then presumably the fact that the binutils there has been updated means that the centos version will be updated soon.

  I did try to clone the centos binutils to see if I could trigger a rebuild but unforunatly it appears that I do not have permission to access these sources.  (Which is strange given that I can clone other centos packages).

  I will leave this BZ in ASSIGNED for now.   Please can you let me know if binutils in CentOS Stream does get updated to 2.30-99.el8 ?

Cheers
  Nick

Comment 4 Carl George 🤠 2021-04-06 16:32:22 UTC
Yes, currently CS8 is getting RHEL-8.5.0 content.  binutils-2.30-99.el8 has already been built in the CentOS koji [0], and is included in the next compose, which I'm hoping to push out today.

[0] https://koji.mbox.centos.org/koji/buildinfo?buildID=17192

Comment 5 MauriceS 2021-04-07 03:46:52 UTC
(In reply to Carl George 🤠 from comment #4)
> Yes, currently CS8 is getting RHEL-8.5.0 content.  binutils-2.30-99.el8 has
> already been built in the CentOS koji [0], and is included in the next
> compose, which I'm hoping to push out today.
> 
> [0] https://koji.mbox.centos.org/koji/buildinfo?buildID=17192

Binutil is fixed now - thank you. But binutil-devel is still the old version, and I do not know whether it's safe to mix them - and binutil-devel is needed to build / change kernel drivers...
[root@node2222 pi]# yum update
Last metadata expiration check: 0:00:05 ago on Wed 07 Apr 2021 03:35:16 AM UTC.
Error: 
 Problem: package binutils-devel-2.30-98.el8.aarch64 requires binutils = 2.30-98.el8, but none of the providers can be installed
  - cannot install both binutils-2.30-99.el8.aarch64 and binutils-2.30-98.el8.aarch64
  - cannot install the best update candidate for package binutils-devel-2.30-98.el8.aarch64
  - cannot install the best update candidate for package binutils-2.30-98.el8.aarch64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Comment 6 Nick Clifton 2021-04-07 08:55:19 UTC
(In reply to MauriceS from comment #5)

Hi Maurice,

> But binutil-devel is still the old
> version, and I do not know whether it's safe to mix them - and binutil-devel
> is needed to build / change kernel drivers...

If you can update to 2.30-99 then I would definitely recommend it.  The 2.30-98
version is broken when it comes to AArch64 code generation.

That said, it would not be wise to update binutils without updating binutils-devel at the same time.  The two packages do need to be in sync.

Cheers
  Nick

Comment 8 Ondřej Budai 2021-04-09 06:57:51 UTC
CentOS Stream now ships both binutils-2.30-99 and binutils-devel-2.30-99, thanks for the quick fix, Carl!

Comment 9 Richard W.M. Jones 2021-04-09 09:21:53 UTC
I don't think -99 is good either.  The binaries don't crash immediately,
but anything using threads will crash eventually.  See
https://bugzilla.redhat.com/show_bug.cgi?id=1946977

Comment 10 Nick Clifton 2021-04-09 09:57:12 UTC
(In reply to Richard W.M. Jones from comment #9)
> I don't think -99 is good either.  The binaries don't crash immediately,
> but anything using threads will crash eventually.

Sadly this is true. :-(  

[Resetting this BZ to ASSIGNED]

I have a fix which should resolve the issue, but I have to wait for an ITM to be set on BZ 1946977 before I can push the patch.

Comment 13 Nick Clifton 2021-04-14 12:53:09 UTC
Now fixed in binutils-2.30-100.el8

Comment 14 MauriceS 2021-04-15 15:42:25 UTC
Any eta when this will be available in Centos Stream?

Comment 15 Nick Clifton 2021-04-16 10:42:16 UTC
(In reply to MauriceS from comment #14)
> Any eta when this will be available in Centos Stream?

Not sure.  (I assume that you are referring to CentOS 8 and not CentOS 9).  The -101 build has just completed gating and been tagged into the rhel-8 buildroot, so I would assume that if the automatic synchronization is working, then an update should be happening at any moment.

Cheers
  Nick

PS.  The -101 build is the same as the -100 build, just with an extra CVE fix added.

Comment 18 Carl George 🤠 2021-04-29 16:55:21 UTC
binutils-2.30-101.el8 was built for CentOS Stream 8 on 2021-04-17 [0], composed on 2021-04-21 [1], and published to the mirrors shortly after that.

[0] https://koji.mbox.centos.org/koji/buildinfo?buildID=17267
[1] https://composes.centos.org/CentOS-Stream-8-20210421.n.0/compose/BaseOS/x86_64/os/Packages/binutils-2.30-101.el8.x86_64.rpm

Comment 19 Miloš Prchlík 2021-05-03 09:24:06 UTC
Verified with both CentOS 8 binutils-2.30-101.el8.aarch64 and RHEL 8.5.0 binutils-2.30-101.el8.aarch64.

Comment 22 errata-xmlrpc 2021-11-09 19:31:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: binutils security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4364