Hide Forgot
Sorry, attachment hit sent the BZ sooner than I wished. Steps to Reproduce: 1. Download corrupt.rpm onto a fresh/updated RHEL 7.6 VM. 2. `yum install corrupt.rpm` -- it will bring in dependencies as it is PKI-base with a diff of one byte. 3. see segfault. Expected results: I'd expect a signature invalid error as gpgcheck=1 is in /etc/yum.conf, or at least not a segfault since `rpm -Kvv` parses the file correctly.
This happens with a lot of bytes in the signature header. Anything which corrupts the gpg signature in the first 300-380 bytes of the RPM header causes these segfaults. The exceptions seem to be something like this: Loaded plugins: product-id, search-disabled-repos, subscription-manager error: /root/tmp2/corrupt/corrupt-387.rpm: rpmReadSignature failed: region trailer: BAD, tag 15872 type 2047 offset 28672 count 4096 Cannot open: ./corrupt-387.rpm. Skipping. (offset byte 387 has been decremented by one in that example from the original RPM) The majority just cause yum to segfault like this initial bug report: Creating RPM: 389+0 records in 389+0 records out 389 bytes (389 B) copied, 0.00131576 s, 296 kB/s 414062+0 records in 414062+0 records out 414062 bytes (414 kB) copied, 1.04906 s, 395 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.000233682 s, 4.3 kB/s 0xc0 Cleaning up failed transaction: BDB2053 Freeing read locks for locker 0xc8e: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc90: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc91: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc92: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc93: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc94: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc95: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc96: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc97: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc98: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc99: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc9a: 19170/140688457258816 BDB2053 Freeing read locks for locker 0xc9b: 19170/140688457258816 Loaded plugins: product-id, subscription-manager Cleaning up unfinished transaction journals Cleaning up 2018-12-17.20:16.27 rpm -Kvv ./corrupt-389.rpm D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key D: loading keyring from rpmdb D: opening db environment /var/lib/rpm cdb:0x401 D: opening db index /var/lib/rpm/Packages 0x400 mode=0x0 D: locked db index /var/lib/rpm/Packages D: opening db index /var/lib/rpm/Name 0x400 mode=0x0 D: read h# 558 Header SHA1 digest: OK (22824a5fcbad49190bcc2dd45d06b4b53aad2447) D: added key gpg-pubkey-fd431d51-4ae0493b to keyring D: read h# 559 Header SHA1 digest: OK (2355c07f9978587c537d385c0bdd9353214f7032) D: added key gpg-pubkey-2fa658e0-45700c69 to keyring D: Using legacy gpg-pubkey(s) from rpmdb D: Expected size: 414452 = lead(96)+sigs(1284)+pad(4)+data(413068) D: Actual size: 414452 ./corrupt-389.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD Header SHA1 digest: OK (bd82675d3064d12c83e0b79c5afa628979496ca4) V3 RSA/SHA256 Signature, key ID fd431d51: OK MD5 digest: OK (c425fe585468c6c9afcf51429a6f6210) D: closed db index /var/lib/rpm/Name D: closed db index /var/lib/rpm/Packages D: closed db environment /var/lib/rpm yum install ./corrupt-389.rpm -y Loaded plugins: product-id, search-disabled-repos, subscription-manager Examining ./corrupt-389.rpm: pki-base-10.5.9-6.el7.noarch Marking ./corrupt-389.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package pki-base.noarch 0:10.5.9-6.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =================================================================================================================================================================================================================== Package Arch Version Repository Size =================================================================================================================================================================================================================== Installing: pki-base noarch 10.5.9-6.el7 /corrupt-389 2.0 M Transaction Summary =================================================================================================================================================================================================================== Install 1 Package Total size: 2.0 M Installed size: 2.0 M Downloading packages: Running transaction check Running transaction test Transaction test succeeded Running transaction Segmentation fault (core dumped) (corrupt-389.rpm is an archive where byte at offset 389 has been decremented by one).
> Expected results: > > I'd expect a signature invalid error as gpgcheck=1 is in /etc/yum.conf, or at least not a segfault since `rpm -Kvv` > parses the file correctly. Obviously. What's not so obvious is that when you install a local package, gpgcheck=1 does not apply. If you try it with "yum -y --setopt localpkg_gpgcheck=1 install corrupt.rpm" it'll actually error out as it should: --- Downloading packages: error: /home/pmatilai/corrupt.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD Problem opening package corrupt.rpm --- I suppose the rationale behind not checking signatures by default on local packages is that rpm doesn't check them either, but what yum does is actually worse than what rpm does: rpm always checks signatures if present and not explicitly disabled, it just doesn't require a signature to be present. So rpm does catch this, iff the key is imported (so yes it's all pretty hysterical): --- [root@zoo pmatilai]# rpm -Uvh corrupt.rpm error: corrupt.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD error: corrupt.rpm cannot be installed --- The interesting thing here is that even if you bypass the signature check, installing with rpm does *not* segfault on the same package, it installs quite merrily. With yum, it crashes inside file fingerprinting which shouldn't have anything to do with this all. It could be that the additional test-transaction that yum performs is causing silent memory corruption or something like that, or ... it could be something else. Need to investigate deeper, starting with rpm.
Oookay, so this is one funny bug, and old as ages. This happens with both yum and dnf (so we'll need to clone for rhel-8 and fedora too, but lets figure out the details here first). The problem is that both yum and dnf fail to notice the error code from their test-transaction, both only stop when problem objects were created by rpm but that doesn't happen in this case. When this failed test-transaction is resubmitted to rpm, it crashes inside fingerprinting because normally valid assumptions don't hold. Obviously optimally it wouldn't do so. All of this has relatively little to do with the invalid signature, it just happens to be the trigger that causes the unexpected failure to open the corrupted package once inside the transaction. Ultimately all three (rpm, yum and dnf) need fixes for this, but making rpm do the right thing here is where it should start, and which will minimally fix it for both yum and dnf. Both yum and dnf need to start checking for the error codes correctly though, there could be other similar corner-cases lurking inside rpm.
FWIW, the problematic place in yum in cli.py: testcb = RPMTransaction(self, test=True) tserrors = self.ts.test(testcb) del testcb if len(tserrors) > 0: errstring = _('Transaction check error:\n') ...versus the full error code check rpmtrans.py explains: # ts.run() exit codes are, hmm, "creative": None means all ok, empty # list means some errors happened in the transaction and non-empty # list that there were errors preventing the ts from starting...
Oh and FWIW, the ultimate reason for this failure is that for packages whose signature checking is disabled, yum/dnf honor this during initially reading packages and populating the transaction. However both use rpm defaults (which is to check signatures if present) to actually run the transactions (test + real), which causes packages failing to open in the transaction callback. This mismatch is what causes the failures in unexpected places, and also means that yum/dnf --nogpgcheck cannot be used to install packages with bad signatures, only ones with no signature at all or ones with whose key we don't have imported (which is not necessarily a bad thing, just different from what rpm's --nosignature does)
Very interesting, thanks for the explanation and investigation. :) > # ts.run() exit codes are, hmm, "creative": None means all ok, empty > # list means some errors happened in the transaction and non-empty > # list that there were errors preventing the ts from starting... I definitely stumbled upon the comment about ts.run() error codes while playing around (and stepping through gdb), but missed the significance at the other location. Nice catch! Makes it sound like a C<->Python issue ultimately. > so we'll need to clone for rhel-8 and fedora too F27 has this problem, but interestingly F28 and F29 weren't working on one of the test packages. (I didn't try all byte offsets and this was a different package with fewer dependencies). > What's not so obvious is that when you install a local package, gpgcheck=1 does not apply. Yeah, my guidance to the lab was going to include createrepo. (I noticed that _OK was returned in the validate package signature section while in gdb...) I'd also argue that installing a package with an invalid signature to a known key should also be a big no-no (without the --nogpgcheck or --very-unsafe-I-know-what-is-happening... even if it is a local file), but that's me. :) Anyhow, thanks! :)
Proposed upstream fix: https://github.com/rpm-software-management/rpm/pull/651
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1114
FEDORA-EPEL-2020-3ef1e07e82 has been pushed to the Fedora EPEL 7 stable repository. If problem still persists, please make note of it in this bug report.