Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2224555

Summary: annocheck reports missing build notes in nbdkit even though it is built corectly
Product: Red Hat Enterprise Linux 9 Reporter: Richard W.M. Jones <rjones>
Component: annobinAssignee: Nick Clifton <nickc>
annobin sub component: system-version QA Contact: qe-baseos-tools-bugs
Status: CLOSED NOTABUG Docs Contact:
Severity: unspecified    
Priority: unspecified CC: chhu, fweimer, hongzliu, juzhou, lersek, mpolacek, mxie, tyan, tzheng, vwu, xiaodwan
Version: 9.3Keywords: Bugfix, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-24 17:08:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Richard W.M. Jones 2023-07-21 12:51:34 UTC 
Description of problem:

(From Florian ...)

I think it can't find the debuginfo for some reason.  Before debuginfo
stripping, the build is clean:

# annocheck /builddir/build/BUILD/nbdkit-1.34.1/plugins/null/.libs/nbdkit-null-plugin.so
annocheck: Version 12.12.
Hardened: nbdkit-null-plugin.so: PASS.

But the stripped build in BUILDROOT is not:

# annocheck /builddir/build/BUILDROOT/nbdkit-1.34.1-1.el9.x86_64/usr/lib64/nbdkit/plugins/nbdkit-null-plugin.so
annocheck: Version 12.12.
Hardened: nbdkit-null-plugin.so: MAYB: test: notes, reason: notes not found and no DWARF info found (could there be a separate debuginfo file ?)
Hardened: nbdkit-null-plugin.so: MAYB: test: optimization, reason: could not determine how the code was created
Hardened: nbdkit-null-plugin.so: MAYB: test: pic, reason: no valid notes found regarding this test
Hardened: nbdkit-null-plugin.so: MAYB: test: stack-clash, reason: could not determine how the code was created
Hardened: nbdkit-null-plugin.so: MAYB: test: stack-prot, reason: could not determine how the code was created
Hardened: Rerun annocheck with --verbose to see more information on the tests.
Hardened: nbdkit-null-plugin.so: Overall: FAIL (due to MAYB results).

This is sort of expected because the debuginfo isn't installed
system-wide.  The error goes away if I supply the correct path:

# annocheck --debug-file=/builddir/build/BUILDROOT/nbdkit-1.34.1-1.el9.x86_64/usr/lib/debug/usr/lib64/nbdkit/plugins/nbdkit-null-plugin.so-1.34.1-1.el9.x86_64.debug /builddir/build/BUILDROOT/nbdkit-1.34.1-1.el9.x86_64/usr/lib64/nbdkit/plugins/nbdkit-null-plugin.so
annocheck: Version 12.12.
Hardened: nbdkit-null-plugin.so: PASS.

But if I install the RPMs:

nbdkit-server-1.34.1-1.el9.x86_64
nbdkit-nbd-plugin-1.34.1-1.el9.x86_64
nbdkit-debuginfo-1.34.1-1.el9.x86_64
nbdkit-nbd-plugin-debuginfo-1.34.1-1.el9.x86_64

it still fails:

# annocheck /usr/lib64/nbdkit/plugins/nbdkit-null-plugin.so
annocheck: Version 12.12.
Hardened: nbdkit-null-plugin.so: MAYB: test: notes, reason: notes not found and no DWARF info found (could there be a separate debuginfo file ?)
Hardened: nbdkit-null-plugin.so: MAYB: test: optimization, reason: could not determine how the code was created
Hardened: nbdkit-null-plugin.so: MAYB: test: pic, reason: no valid notes found regarding this test
Hardened: nbdkit-null-plugin.so: MAYB: test: stack-clash, reason: could not determine how the code was created
Hardened: nbdkit-null-plugin.so: MAYB: test: stack-prot, reason: could not determine how the code was created
Hardened: Rerun annocheck with --verbose to see more information on the tests.
Hardened: nbdkit-null-plugin.so: Overall: FAIL (due to MAYB results).

Looking at the verbose output:

Hardened: nbdkit-null-plugin.so: build_id_len: 20, name: d7d16e8ead09960e637e87e48eeea95d258106.
Hardened: nbdkit-null-plugin.so:  try: /usr/lib/debug/.build-id/05/d7d16e8ead09960e637e87e48eeea95d258106.debug.
Hardened: nbdkit-null-plugin.so: Could not find separate debuginfo file based on build-id.

That looks like a BFD or annocheck bug, given that the file is there:

# file -L  /usr/lib/.build-id/05/d7d16e8ead09960e637e87e48eeea95d258106 
/usr/lib/.build-id/05/d7d16e8ead09960e637e87e48eeea95d258106: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=05d7d16e8ead09960e637e87e48eeea95d258106, stripped


Version-Release number of selected component (if applicable):

annobin-annocheck-12.12-1.el9.x86_64

How reproducible:

100%

Steps to Reproduce:

See above, but another way is:

$ brew download-build nbdkit-1.34.1-1.el9 --arch=x86_64 --arch=debug          
$ annocheck -v --skip-cf-protection --skip-glibcxx-assertions --skip-glibcxx-assertions --skip-stack-realign --section-size=.gnu.build.attributes --ignore-gaps nbdkit-server-1.34.1-1.el9.x86_64.rpm --debug-rpm=nbdkit-debuginfo-1.34.1-1.el9.x86_64.rpm                            

Additional info:

https://lists.corp.redhat.com/archives/rhel-devel/2023-July/005569.html
Comment 1 Marek Polacek 2023-07-24 15:41:24 UTC 
Fixing Assignee.
Comment 2 Nick Clifton 2023-07-24 16:47:32 UTC 
(In reply to Richard W.M. Jones from comment #0)
Hi Richard,

> But if I install the RPMs:
> 
> nbdkit-nbd-plugin-1.34.1-1.el9.x86_64
> nbdkit-nbd-plugin-debuginfo-1.34.1-1.el9.x86_64

> Looking at the verbose output:
> 
> Hardened: nbdkit-null-plugin.so: build_id_len: 20, name:
> d7d16e8ead09960e637e87e48eeea95d258106.
> Hardened: nbdkit-null-plugin.so:  try:
> /usr/lib/debug/.build-id/05/d7d16e8ead09960e637e87e48eeea95d258106.debug.
> Hardened: nbdkit-null-plugin.so: Could not find separate debuginfo file
> based on build-id.

Are you sure that the installed rpms are nbdkit-nbd-plugin-1.34.1-1.el9.x86_64 and nbdkit-nbd-plugin-debuginfo-1.34.1-1.el9.x86_64 ?

I ask because when I download the debuginfo rpm from brew and look at its contents:

  $ rpm -q -l -l nbdkit-nbd-plugin-debuginfo-1.34.1-1.el9.x86_64.rpm 
  /usr/lib/debug
  /usr/lib/debug/.build-id
  /usr/lib/debug/.build-id/58
  /usr/lib/debug/.build-id/58/52a3cd64671beb1d470e6c23f1d8820b83256b
  /usr/lib/debug/.build-id/58/52a3cd64671beb1d470e6c23f1d8820b83256b.debug
  /usr/lib/debug/usr
  /usr/lib/debug/usr/lib64
  /usr/lib/debug/usr/lib64/nbdkit
  /usr/lib/debug/usr/lib64/nbdkit/plugins
  /usr/lib/debug/usr/lib64/nbdkit/plugins/nbdkit-nbd-plugin.so-1.34.1-1.el9.x86_64.debug

Note how the build-id is completely different.
Comment 3 Richard W.M. Jones 2023-07-24 17:08:34 UTC 
Without the -nbd- in the middle.

However I have now realised what the problem is, and it's not annocheck at all.
The problem is we didn't have the right debuginfo packages installed.

nbdkit is a meta-package that pulls in nbdkit-server, nbdkit-basic-plugins
and nbdkit-basic-filters.  Therefore nbdkit-debuginfo is essentially empty:

$ rpm -ql nbdkit-debuginfo 
/usr/lib/debug
/usr/lib/debug/.dwz
/usr/lib/debug/.dwz/nbdkit-1.34.1-1.el9.x86_64

The real package containing symbols is nbdkit-server-debuginfo:

$ rpm -ql nbdkit-server-debuginfo
/usr/lib/debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/1c
/usr/lib/debug/.build-id/1c/2263f9378081bbc4db3ebd289087995584d2eb
/usr/lib/debug/.build-id/1c/2263f9378081bbc4db3ebd289087995584d2eb.debug
/usr/lib/debug/.build-id/5f
/usr/lib/debug/.build-id/5f/10af993a94d9c07796a8845360e76b745a5418
/usr/lib/debug/.build-id/5f/10af993a94d9c07796a8845360e76b745a5418.debug
/usr/lib/debug/usr
/usr/lib/debug/usr/lib64
/usr/lib/debug/usr/lib64/nbdkit
/usr/lib/debug/usr/lib64/nbdkit/plugins
/usr/lib/debug/usr/lib64/nbdkit/plugins/nbdkit-null-plugin.so-1.34.1-1.el9.x86_64.debug
/usr/lib/debug/usr/sbin
/usr/lib/debug/usr/sbin/nbdkit-1.34.1-1.el9.x86_64.debug

And with that installed, annocheck passes fine.

=> NOTABUG!
Comment 4 Nick Clifton 2023-07-25 08:06:57 UTC 
(In reply to Richard W.M. Jones from comment #3)

> nbdkit is a meta-package that pulls in nbdkit-server, nbdkit-basic-plugins
> and nbdkit-basic-filters.  Therefore nbdkit-debuginfo is essentially empty:

Just a thought - would it be possible to do a similar trick with the ndbkit-debuginfo
rpm ?  Ie have it pull in the ndbkit-server-debuginfo, ndbkit-basic-plugins-debuginfo
and ndbkit-basic-filters-debuginfo rpms ?


> And with that installed, annocheck passes fine.
> => NOTABUG!

Now that is my kind of bug report ! :-)
Comment 5 Richard W.M. Jones 2023-07-25 08:14:23 UTC 
(In reply to Nick Clifton from comment #4)
> (In reply to Richard W.M. Jones from comment #3)
> 
> > nbdkit is a meta-package that pulls in nbdkit-server, nbdkit-basic-plugins
> > and nbdkit-basic-filters.  Therefore nbdkit-debuginfo is essentially empty:
> 
> Just a thought - would it be possible to do a similar trick with the
> ndbkit-debuginfo
> rpm ?  Ie have it pull in the ndbkit-server-debuginfo,
> ndbkit-basic-plugins-debuginfo
> and ndbkit-basic-filters-debuginfo rpms ?

debuginfo packages are generated by deeply complex RPM macros so I guess not.

I think what really matters is that dnf debuginfo-install /usr/sbin/nbdkit
pulls in the correct debuginfo RPMs, which I just checked now and it does.
It doesn't do it based off the binary's build ID, but by taking the package
name (nbdkit-server) and adding -debuginfo to the end.
Comment 6 Florian Weimer 2023-07-25 12:34:05 UTC 
Oh, right, I missed that /usr/lib64/nbdkit/plugins/nbdkit-null-plugin.so is not actually /usr/lib64/nbdkit/plugins/nbdkit-nbd-plugin.so (different path, different RPM packages, so different debuginfo packages as well).