RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2018937 - RPM assumes file signatures in the same package always have the same length
Summary: RPM assumes file signatures in the same package always have the same length
Keywords:
Status: CLOSED ERRATA
Alias: None
Deadline: 2021-12-14
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: rpm
Version: 9.0
Hardware: Unspecified
OS: Unspecified
urgent
unspecified
Target Milestone: rc
: ---
Assignee: Michal Domonkos
QA Contact: Eva Mrakova
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-01 10:11 UTC by Patrick Uiterwijk
Modified: 2022-05-17 16:35 UTC (History)
19 users (show)

Fixed In Version: rpm-4.16.1.3-11.el9
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-17 16:01:23 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Rough patch (3.07 KB, application/mbox)
2021-11-01 10:11 UTC, Patrick Uiterwijk
no flags Details
reproducer-rpm (5.87 KB, application/octet-stream)
2021-11-26 08:24 UTC, Patrick Uiterwijk
no flags Details
Scratch SRPM with the fix (4.20 MB, application/x-rpm)
2021-12-06 09:20 UTC, Michal Domonkos
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github rpm-software-management rpm issues 1833 0 None open segfault installing packages (in very strange edge cases) 2021-11-25 09:43:22 UTC
Red Hat Issue Tracker RHELPLAN-101217 0 None None None 2021-11-01 12:20:56 UTC
Red Hat Product Errata RHBA-2022:4021 0 None None None 2022-05-17 16:01:36 UTC

Internal Links: 2177088

Description Patrick Uiterwijk 2021-11-01 10:11:33 UTC
Created attachment 1838825 [details]
Rough patch

RPM stores a single FILESIGNATURELENGTH header value (in sighdr), to indicate how big every individual file signature (FILESIGNATURES in sighdr) is, and then uses that for every individual entry during hex decoding.
However, this is based on an invalid assumption: individual signatures can be one byte (2 hex characters) longer, depending on the specific number that is encoded[1], leading to signatures of different lengths in the same RPM package.

However, because the signatures are encoded as an array of hex strings, the length is not needed in order to decode the signature.
RPM should ignore the FILESIGNATURLENGTH header (and probably not put them in anymore), and use the string length to decode the signature.

The attachment is a draft patch that I tested and worked for me (but is not the most efficient probably).


[1]: https://github.com/openssl/openssl/blob/OpenSSL_1_0_1c/crypto/ecdsa/ecs_lib.c#L244

Comment 1 Panu Matilainen 2021-11-09 13:46:03 UTC
rpmfilesFSignature() returns a const pointer which callers must *not* free so this would mean every call to rpmfilesFSignature() leaks memory and we can't very well change the API in rpm 4.16. 
 

The different length case is just a peculiarity of ECDSA, right? How about we just declare that unsupported for file signatures in RHEL-9... 

BTW there's another leak in the implementation that fi->signatures is copied as independently allocated strings which are never freed. You don't need to do that, you could call headerGet() with HEADERGET_ALLOC to get a single allocated blob for the whole thing, and as this is a common operation in rpmfi, there's an ugly shortcut macro. Replacing the call to copy_array() with this should do it all, error checks included:

  _hgfi(h, RPMTAG_FILESIGNATURES, &td, defFlags, fi->signatures);

Comment 2 Peter Robinson 2021-11-23 15:11:54 UTC
> The different length case is just a peculiarity of ECDSA, right? How about
> we just declare that unsupported for file signatures in RHEL-9... 

Product Management asked us to use ECDSA as the size is about 200 bytes vs 2048 - 4096 for RSA depending on the keysize, it also has a speed advantage. We can't invariably just go and mark it unsupported now as the ship has already sailed. 
 
> BTW there's another leak in the implementation that fi->signatures is copied
> as independently allocated strings which are never freed. You don't need to
> do that, you could call headerGet() with HEADERGET_ALLOC to get a single
> allocated blob for the whole thing, and as this is a common operation in
> rpmfi, there's an ugly shortcut macro. Replacing the call to copy_array()
> with this should do it all, error checks included:
> 
>   _hgfi(h, RPMTAG_FILESIGNATURES, &td, defFlags, fi->signatures);

It was marked as a draft patch for a reason, it was intended as a proof of concept not intended as the final solution as we aren't specialists in the rpm codebase.

Comment 3 Panu Matilainen 2021-11-24 08:55:18 UTC
So the ship sailed with a previously untested configuration. Nice.

Comment 4 Peter Robinson 2021-11-24 08:58:01 UTC
(In reply to Panu Matilainen from comment #3)
> So the ship sailed with a previously untested configuration. Nice.

If we're going to do sarcastic replies I would ask where was the testing when the feature landed and where's the tests that would have would have picked this up from the outset.

Comment 5 Panu Matilainen 2021-11-24 09:26:29 UTC
I wasn't around at the time so cannot comment. However I have been making quite a lot of noise against IMA first in Fedora and then in RHEL because it's not production ready IMO but nobody listened. Yet here we are, and yes I am a little grumpy.

Comment 6 Peter Robinson 2021-11-24 09:34:49 UTC
(In reply to Panu Matilainen from comment #5)
> I wasn't around at the time so cannot comment. However I have been making
> quite a lot of noise against IMA first in Fedora and then in RHEL because
> it's not production ready IMO but nobody listened. Yet here we are, and yes
> I am a little grumpy.

The fact of the matter is we have product requirements and security requirements and unfortunately these often drive things from not ready to ready. If some people had their way it would never be ready. We need to be able to secure products, and there is numerous ways of doing this, security needs multiple layers like an onion. Unfortunately sometimes if we sit around and wait until things are ready we miss that market. So the situation is what is it and salty comments doesn't help engineers like myself work to get us a working product. So I get that you're grumpy but please don't take it out on other engineers that are just trying to do their job.

Comment 7 Panu Matilainen 2021-11-24 10:02:46 UTC
Yes, stuff only gets ready by people pushing and working on them, but that gruntwork is supposed to happen in Fedora. We never received a single patch regarding IMA despite all the complaints people raised at the time, and apparently some different algorithm was used. I don't know if IMA even initially supported ECDSA - I certainly didn't know it did even now. Yet here we have the entire distro now signed with what is clearly a previously untested algorithm. That's not how these things are supposed to be done, which is what I'm grumpy about.

Comment 8 Peter Robinson 2021-11-24 10:30:04 UTC
Actually the ECDSA key was used in Fedora when it was tested so it was the same algorithm. Bugs happen, software is complex. I some how doubt it's going to be the last issue we find.

We found one bug and provided a patch, the issue that was found in el8 and was fixed in 8.4.

Ultimately it wasn't enabled in Fedora due to various reasons and we had to have this in place for el9 GA because of the requirement of signing all rpms as part of the el9 shipping requirements. This is functionality we couldn't wait until el10 for. It's not the first time a feature landed in an enterprise release due to Fedora community push back, I doubt it will be the last.

Ultimately the timing was out of my control and you are of course allowed to be grumpy but that:
1) doesn't move us forward to get this bug fixed
2) is not useful for the Fedora/RHEL ecosystem
3) improve RHEL/Fedora's ability to provide users the ability to better secure their platforms

We do plan on resubmitting the feature to Fedora but timing/dealines/resources has restricted that and having to deal with politics/pitchforks hasn't helped that. We already have plans to resubmit it for F-36.

Comment 10 Panu Matilainen 2021-11-25 09:43:22 UTC
As for the actual issue, this is missing a reproducer entirely. I understand the description of the issue, but having a some real-world specimen is necessary to develop + validate a proper fix. Also there's zero indication as to what happens because of this, ie severity. A later, independent report at upstream seems to suggest this causes segfaults which is of course quite severe.

So a reproducer please, and then we can deal with this with an appropriate priority.

Comment 12 Patrick Uiterwijk 2021-11-26 08:24:18 UTC
Created attachment 1843675 [details]
reproducer-rpm

The attached RPM will show this: this has two files (/usr/share/example1 and /usr/share/example2) that are both IMA signed with ECDSA, one has an 80 bytes long signature, the other 79 bytes.

When you look at the package contents with for example Koji's sighdr parsing:
import koji
sighdr = koji.rip_rpm_sighdr("test-1.0-1.fc34.noarch.rpm")
sighdr = koji.RawHeader(sighdr)
sighdr.dump()

This contains the following contents for FILESIGNATURES:
Tag: 274 [?], Type: 8, Offset: 232, Count: 2
String(162): b'030204a598255400483046022100e5117bdafa73baaeb1f1dc46ecaa46981a62d417745a33532572b63dc6d95d16022100c789107ac5b91e2d915e1df3c7b78414f6b3f50899d44c1de381d0e938dfc82b'
String(160): b'030204a598255400473045022100c10943795bff5d9c0db53dd4f8e4b845615fd08a2be295c30a80f5bdb4e6a41302203038840cc6abaab92acb56cb3e3ce520b17f22ff7444a8d5d0f703a44d5307a3'

Note that the second one is slightly shorter than the first.

And the following the FILESIGNATURELENGTH:
Tag: 275 [?], Type: 4, Offset: 378, Count: 1
[0, 0, 0, 80]


After installing the package with rpm-plugin-ima installed, this results in the following extended attributes:
  ▸ getfattr -e hex -n security.ima /usr/share/example2
getfattr: Removing leading '/' from absolute path names
# file: usr/share/example2
security.ima=0x030204a598255400473045022100c10943795bff5d9c0db53dd4f8e4b845615fd08a2be295c30a80f5bdb4e6a41302203038840cc6abaab92acb56cb3e3ce520b17f22ff7444a8d5d0f703a44d5307a3

  ▸ getfattr -e hex -n security.ima /usr/share/example1
getfattr: Removing leading '/' from absolute path names
# file: usr/share/example1
security.ima=0x030204a598255400483046022100e5117bdafa73baaeb1f1dc46ecaa46981a62d417745a33532572b63dc6d95d16022100c789107ac5b91e2d915e1df3c7b78414f6b3f50899d44c1de381d0e938dfc8


Note that /usr/share/example1 is missing the final byte of its signature: it ends in c8, instead of c82b, which would lead in an invalid signature.

Comment 14 Panu Matilainen 2021-11-26 09:08:27 UTC
(In reply to Patrick Uiterwijk from comment #12)

> The attached RPM will show this: this has two files (/usr/share/example1 and
> /usr/share/example2) that are both IMA signed with ECDSA, one has an 80
> bytes long signature, the other 79 bytes.
> 

Thanks, this should do. But if you have pointers to real-world packages with more files exhibiting this, that wouldn't hurt.

FWIW, this is easily observable with rpm cli queries, no need to program in Python:

$ rpm -qp --qf "[%{filesignatures}\n]" test-1.0-1.fc34.noarch.rpm 
warning: test-1.0-1.fc34.noarch.rpm: Header V4 RSA/SHA512 Signature, key ID 09daef21: NOKEY
030204a598255400483046022100e5117bdafa73baaeb1f1dc46ecaa46981a62d417745a33532572b63dc6d95d16022100c789107ac5b91e2d915e1df3c7b78414f6b3f50899d44c1de381d0e938dfc82b
030204a598255400473045022100c10943795bff5d9c0db53dd4f8e4b845615fd08a2be295c30a80f5bdb4e6a41302203038840cc6abaab92acb56cb3e3ce520b17f22ff7444a8d5d0f703a44d5307a3
$

Also when looking at header details, --xml query format is handy (if ugly) when you're not sure what you're looking for:

$ rpm -qpl --xml test-1.0-1.fc34.noarch.rpm 
[...]
  <rpmTag name="Filesignatures">
        <string>030204a598255400483046022100e5117bdafa73baaeb1f1dc46ecaa46981a62d417745a33532572b63dc6d95d16022100c789107ac5b91e2d915e1df3c7b78414f6b3f50899d44c1de381d0e938dfc82b</string>
        <string>030204a598255400473045022100c10943795bff5d9c0db53dd4f8e4b845615fd08a2be295c30a80f5bdb4e6a41302203038840cc6abaab92acb56cb3e3ce520b17f22ff7444a8d5d0f703a44d5307a3</string>
  </rpmTag>
  <rpmTag name="Filesignaturelength">
        <integer>80</integer>
  </rpmTag>
[...]

Comment 15 Panu Matilainen 2021-11-26 10:43:16 UTC
Proposed fix at https://github.com/rpm-software-management/rpm/pull/1844

Please test, I have only verified to the point that the installed signatures now match what's in the header and it doesn't blow up in the process.

Comment 16 Michal Domonkos 2021-12-01 12:20:00 UTC
Assigning to myself for the backporting work.

Comment 17 Neal Gompa 2021-12-03 00:34:46 UTC
I was pointed to this BZ after reaching my wit's end trying to figure out why most of my package builds on the openSUSE Build Service have been failing for the past month: https://build.opensuse.org/project/show/home:Pharaoh_Atem:CS9EPELDev

If someone has a SRPM of rpm I can build with a test patch for CS9, I can build it and test to see if it fixes the issues I'm seeing in my build environments.

Comment 18 Michal Domonkos 2021-12-06 09:20:14 UTC
Created attachment 1844867 [details]
Scratch SRPM with the fix

Comment 19 Michal Domonkos 2021-12-06 09:25:38 UTC
Neal, I've attached an SRPM that you can try out.  I've tested it against the RPM provided by Patrick above and it worked.

Comment 20 Michal Domonkos 2021-12-06 09:27:41 UTC
Just to add - the SRPM is based off of the latest c9s branch at:
https://gitlab.com/redhat/centos-stream/rpms/rpm

Comment 21 Neal Gompa 2021-12-06 13:37:19 UTC
I'm building the package here: https://build.opensuse.org/package/show/home:Pharaoh_Atem:CS9EPELDev:baseos-dev/rpm

I've configured it to be a dependency of https://build.opensuse.org/project/show/home:Pharaoh_Atem:CS9EPELDev

As rpm gets built in each of them, it'll trigger a mass build of all the packages in there.

If x86_64 gets to all green, then we did it. :)

Comment 37 Michal Domonkos 2022-02-14 10:47:15 UTC
Upstream PR:

https://github.com/rpm-software-management/rpm/pull/1914

Comment 44 errata-xmlrpc 2022-05-17 16:01:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: rpm), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4021


Note You need to log in before you can comment on or make changes to this bug.